Abstract: For the traditional Mean shift algorithm fixed bandwidth and direction of the ellipse kernel, so the location of object tracking is inaccurate or lost.For this shortage, this thesis puts forward a kind of scale-orientation adaptive object tracking algorithm. First of all, tracked the object with the SIFT features. And then analysis of the SIFT feature's scale and orientation, found that the scale change of the SIFT feature point is proportional to that of the object, the Angle change of the SIFT feature point is the same as that of the object, puts forward the calculation method of object scale and orientation. Furthermore introduce the similarity transformation, use variable-bandwidth and orientation ellipse to improve the traditional Mean shift algorithm, let it can track the object that scale and orientation changes. Finally, the algorithm utilizes linear weighted method to fuse the tracking results of SIFT and improved Mean shift, obtaining the final tracking results. Experiment shows that the proposed algorithm can effectively track the object that the scale and orientation change, the scale and orientation parameter of object is higher, and the localization is more accurate.