Abstract: The sign language recognition technology based on computer vision brings great convenience to bilingual teaching in deaf schools. One of the difficulties of sign language recognition technology is the extraction of video key frames. According to the characteristics of sign language video keyframes and sign language habits of sign language users, this paper proposes a video key frame extraction and optimization algorithm for sign language recognition. Firstly, the convolutional auto-encoder is used to extract the deep features of video frames, and then, K-means clustering is performed. In each kind of video frames, the clearest video frames are selected as the keyframes for the first time. Then, the point density method is used to optimize the first extracted key frames, and the final extracted key frames are obtained for sign language recognition. Finally, it is shown via experimental results that the proposed algorithm can reduce substantial redundant frames and improve the accuracy and efficiency of sign language recognition.