Abstract: To solve the problem of drift in the tracking, in this paper, a re-detection object tracking algorithm based on fast multi-scale estimation is proposed. In the tracking process, the tracker locates the target position through the maximum value of the response and detects the reliability of the current position using a new self-adaptive detection criterion. In contrast to other detection criteria, our new detection criterion reduces the dependence on the maximum response value. If the current location is determined to be unreliable, our method can generate target candidate boxes by using the edge boxes algorithm and select the best target location by applying the non-maximum suppression (NMS) and the Euclidean metric methods. Furthermore, we proposed an adaptive updating method to reduce the errors caused by tracking failure. Experimental results show that our approach has a good performance in terms of precision and success rates.