Abstract: In recent years, with the rapid development of computer technology and the extensive application of human computer interaction technology, the video based recognition of facial expression is becoming one of the hot fields, and has obtained practical applications. This paper proposes a video based fusion algorithm on the spatial and temporal emotion characteristics. Firstly, we get spatial and temporal characteristics and the corresponding cube (cuboids) of emotional video. And then, the feature vectors of cuboids, extracted by the operator by Piotr Dollar proposed and the CBP_TOP operator, are fused as the final feature vector of the spatial and temporal characteristics. Finally, the final characteristics of the emotional video are extracted by means of words bag model, which will be used for subsequent expression classification. Simulation results show that the proposed algorithm can improve the recognition speed greatly and ensure the recognition accuracy.