Multimodal sarcasm recognition based on RCBA model

At present, most of the sarcasm recognition models are based on text data, and the image data contained in the tweets are not used effectively, which leads to the low accuracy of the sarcasm recognition task. Aiming at this problem, a joint neural network model combined with the attention mechanisms model is proposed, use for recognizing the multimodal data which mixed image and text whether contains sarcasm. The RCBA model first uses the 101-layer deep residual network which is combined with spatial attention mechanism and channel attention mechanism to adaptively extract images feature. At the same time, using the image attribute classifier to extract image attribute features. Secondly, the image attribute features are regarded as the initial state of the bidirectional long-short-term memory neural network to complete the extraction of text features. Then, the image features, image attribute features, and text features are fused by a two-layer neural network. Finally, construct a two-layer backpropagation network as a classifier to complete the sarcasm recognition task. RCBA is tested on the public dataset of image-text mixed Twitter sarcasm. Compared with the baseline model of the image-text sarcasm detection task, the accuracy and F1 value are respectively improved by 6.19% and 5.29%. The experimental results show that the RCBA model can effectively extract multi-modal data features and has better performance in the sarcasm recognition task.