Qun Ren, Journal of Information Processing Systems Vol. 17, No. 3, pp. 556-570, Jun. 2021  

https://doi.org/10.3745/JIPS.02.0156
Keywords: facial expression recognition, Multi-Mode Deep Learning, Multiplicative Fusion, Optical Flow Method, Spatial Convolutional Neural Network, Time Convolutional Neural Network
Fulltext: PDF Full Text PubReader

Abstract

The existing video expression recognition methods mainly focus on the spatial feature extraction of video expression images, but tend to ignore the dynamic features of video sequences. To solve this problem, a multimode convolution neural network method is proposed to effectively improve the performance of facial expression recognition in video. Firstly, OpenFace 2.0 is used to detect face images in video, and two deep convolution neural networks are used to extract spatiotemporal expression features. Furthermore, spatial convolution neural network is used to extract the spatial information features of each static expression image, and the dynamic information feature is extracted from the optical flow information of multiple expression images based on temporal convolution neural network. Then, the spatiotemporal features learned by the two deep convolution neural networks are fused by multiplication. Finally, the fused features are input into support vector machine to realize the facial expression classification. Experimental results show that the recognition accuracy of the proposed method can reach 64.57% and 60.89%, respectively on RML and Baum-ls datasets. It is better than that of other contrast methods.

Statistics
Cite this article