Abstract: Aiming at the problem of large difference in similar action classes, small difference between classes in action video and low recognition accuracy, a action recognition method based on two-stream convolution network and double-center loss is proposed. The method first constructs a two-stream convolutional network structure, and uses the C3Dnet model as the basic model of the two-stream structure to extract the apparent short-term motion information in the multi-scale RGB video frame and the long-term motion information in the stacked optical flow map respectively; Then, the depth information extracted by the two-stream structure is parsed by a long and short time memory (LSTM) network to perform feature fusion; Finally, the 2C-softmax objective function based on dual-center loss is used to maximize the distance between classes and minimize the distance within the class, so as to classify and identify similar actions. The experimental results on the data set KTH show that the method can accurately identify similar actions, and the recognition accuracy can reach 98.2%, which has a good recognition effect.