Abstract: A Multi-GPU parallel framework of DNN is given out, and the implementation method and its performance optimization are presented, relying on the powerful synergy parallel computing ability of Multi-GPU, combining with the characteristics of data parallel, the fast and efficient training of Deep Neural Network is realized. The application of speech recognition, the model convergence rate and model performance have achieved effective promotion. Compared with single GPU, it has 4.6 times speedup ratio, billions of training days convergence, and word error rate is reduced about 10%.