Abstract: In real time tracking system, the size of tracking window should be updated when the scale of the moving object is being varied, which is corresponding to the scale change of mean shift. In order to solve the problem, this paper proposed a method to catch the adaptive scale change. Firstly, the corner points are detected in the tracking window. Secondly, the detected points are matched with the preceding frame, and the affine model is constructed based on the matching results. Thirdly, least square method is taken to induce the scale coefficients. Lastly, the obtained scale is applied to the mean shift tracking framework. Experimental results show the improved algorithm could track objects precisely when scale of the object changes progressively, as well as is robust and operated real time.