Abstract: This paper proposes hand gesture recognition and fingertip tracking system based on multiple levels of processors.The system contains a 64×64 Processing Elements (PE) array,a coprocessor group consists of 64 coprocessors and a RISC CPU. The PE array can carry out noise cancelation, smooth filtering, and image morphology.The coprocessor group is responsible for feature statistics and feature extraction.The RISC CPU controls the whole system and is capable of conducting complex arithmetic operations.It can also perform data exchange with other systems.The proposed system is implemented in FPGA.Due to its multiple level processors architecture the processing rate is significantly improved.Experiments demonstrated that our proposed system has good robustness and fully satisfy the real-time hand gesture recognition and finger-tip tracking requirement.