Text categorization on D-BGRU with self-attention mechanism

Aiming at the problems that the traditional recurrent neural network (RNN) modeling is too stressful and it is easy to ignore the local details and the convolutional neural network (CNN) cannot capture the long-distance dependencies, a text classification model method based on disconnected information flow is proposed. This method introduces the disconnected information flow into the bidirectional gated recurrent unit (BGRU), which can extract the long-distance dependence of the context and has the feature position invariance similar to the convolution kernel, thus taking into account the temporal and spatial characteristics of the text. On this basis, the self-attention mechanism is integrated to further learn the dependencies between features, assign larger weights to important features to reduce noise redundancy, strengthen the model's ability to extract key information, and realize the optimization of text features. Experiments on five real data sets including AGnews, DBPedia, Yelp P., etc., the accuracy of this method is higher than that of multiple baseline algorithms, reaching 95.8%, 99.7%, 98.1%, 70.4%, 77.5% respectively. It is verified that the model can realize text categorization more effectively and has good application prospects.