Abstract: Sign language recognition plays a vital role in the communication between deaf-mute and normal people. In order to solve the problem of insufficient gesture feature extraction, caused by the multi-scale characteristics of hand features, and loss of detail information in feature fusion of traditional sign language recognition algorithm, an improved algorithm based on EfficientDet-D0 is proposed. To be specific, this algorithm adds spatial attention mechanism in the backbone of EfficientDet-D0, making it be capable of locating the hand features effectively. Then, in the feature fusion network, the idea of Laplacian Pyramid and the cross-level connection is used, which enable it to fuse detailed feature maps and make full use of features of resolutions. So that, the information of high-level feature map is richer, and the high-frequency detail information lost by downsampling is fully described. Moreover, the transfer learning technology and Adam optimizer are used to train the whole network. Experiments show that this new algorithm can quickly and accurately identify different sign language actions in various backgrounds. The final accuracy rate reaches 94.1%. In a word, it has higher accuracy rate and stronger robustness than traditional algorithms. And, a two-way translation application is designed based on the algorithm, which is practical.