Abstract: In order to solve the background interference and robustness problems of object tracking algorithm in moving targets, an asymmetric residual attention network algorithm based on Siamese RPN++ was proposed.First, aAsymmetric residual attention structure is added to the network corresponding to the template branch, so as to extract the common features of the sampled images, form a relatively stable target contour, and solve the problem that the target moving background changes.The adaptive weight updating method is adopted to fuse the output features of candidate network modules in different regions to obtain more robust expression of scale change features and solve the problem of target deformation.The experimental results show that the proposed improved algorithm has good tracking accuracy and robustness in challenging tracking test videos, and can deal with the changes of moving background and scale.