Abstract: Facial action unit recognition task is one of the most important aspects of understanding facial expressions. However, the extremely unbalanced categories and the multi-label classification bring great difficulties for the design of recognition algorithm. By means of deep learning technique, this paper proposes a face action unit recognition algorithm. Firstly, based on the transfer learning theory and driven by the face recognition task, some large-scale datasets are used to pre-train the convolutional network so as to make this model has the ability to extract abstract features of face. Secondly, an objective function is designed to dynamically weigh the sample loss according to the classification confidence for making the model focus more on optimizing a few samples. Finally, two related tasks including the multi-label co-occurrence relationship fitting and the key-point regression of face are combined to jointly train and test the model. It is shown from the experimental results that the proposed method can effectively improve the classification accuracy and F1-score on the relevant datasets CK+ and MMI.