Abstract: In order to solve the problem that the dynamic prototype unit model based on memory unit does not make full use of multi-level features and does not consider the structural differences between abnormal and normal events when detecting video anomalies, an anomaly detection model combining multi-scale memory module and multi-scale structural similarity is proposed. The new model constructs a multi-scale memory Module, which uses memory units of different scale space to encode the features of the encoding layer, and concatenates the encoding results with the features of the decoding layer, which can not only preserve the shallow details of the network, but also promote the diversity of normal patterns. In order to constrain the learning of structural information in normal events, the multi-scale structure similarity index error and L_1 error are combined as objective functions to make the event structure in the predicted video closer to the normal event, and improve the prediction error of abnormal events in the video. The experimental results on the standard datasets UCSD Ped1, UCSD Ped2 and Avenue show that the frame level AUC of the proposed model is improved by 0.8%, 3.4% and 1.0% compared with the original model, respectively. And the frame rate reaches 142.9 FPS.