Abstract: Scenes and objects in videos include abundant emotional clues. Traditional video emotion recognition mainly focuses on human behavior in video, and context clues such as scenes and objects are often ignored. This paper proposed a behavior context based video emotion recognition method. Firstly, multiple modal features of video scene, object and behavior are extracted based on convolution neural network(CNN). Secondly, emotional scores of video frames are determined according to its feature information. Thirdly, multiple modal feature fusion model is built based on Pyramid framework to recognize video emotion. The method is implemented based on Caffe. The experiment results show that our method is superior to the existing methods in performance.