Abstract: Convolutional neural networks are an important model of deep learning and are widely used in image processing and other fields. The commonly used neural network model is complex and has many parameters, which is not suitable for running on the mobile end. Based on the idea of modularization and hardware reuse, this paper presents a hardware implementation of handwritten digital character recognition network based on FPGA. Based on the principle of MobileNet, the structure is improved, and the algorithm hardware acceleration is realized, and the number of parameters of the network and the overall calculation amount are effectively reduced. The experimental results based on the MNIST dataset show that compared with the traditional neural network, the parameter size of the improved structure is reduced by 23.26%, and the calculation amount is reduced by 31.32%. The entire network is implemented with less resources and lower power consumption while maintaining the same speed.