Abstract: In view of the dissatisfactory accuracy and stability caused by outliers in the process of estimating fundamental matrix, a novel fundamental matrix estimation algorithm was presented in this paper. First a new packet sampling strategy was adopted to the matched points, which emphasized the randomness and uniformity. Under rules of the distance between the point matches and the corresponding epipolar lines, a new inlier set of matched points and original fundamental matrix were obtained via the twice-mid-value method. Finally, the thought of dynamic punishment weighing was introduced to nonlinearly optimize the new point set, which gained the accurate epipolar geometry. Results of a mass of experiments on simulated data and real images indicate that the proposed approach is a practical way and yields the precise and robust fundamental matrix in the case of mismatch and Gaussian noise.