Abstract: Speech emotion recognition has been widely used in various fields such as vehicle driving systems, service industries, education, and medical care. In order to make the computer recognize the speaker's emotion more accurately, this paper proposes an emotional speech recognition method based on the combination of multi-task 3D convolutional neural network and bidirectional long-short term memory network with attention mechanism. Based on the multi-spectral feature fusion group map, the deep speech emotion features are extracted by three-dimensional convolutional neural network, and the multi-task learning mechanism of gender classification is combined to improve the accuracy of speech emotion recognition. Finally, experimental results show that the proposed model can attain higher accuracy on CASIA Chinese emotional corpus.