Abstract: In order to solve the problems of less research, low segmentation accuracy in crop segmentation of multi-temporal remote sensing images at present, this paper designs a model based on 3D Feature Pyramid Net (FPN) structure, and introduces the Atrous Spatial Pyramid Pooling (ASPP) module, making the network capture the multi-scale features of the image and fuse more contextual semantic information of the image. In addition, in order to solve the problem of imbalance in the number of categories and difficulty of distinguishing in the dataset, the Focal loss function is introduced. By setting the weight factor, the network pays more attention to the categories with a small number or indistinguishable. Using the multi-temporal remote sensing images of Qiqihar City in 2019 as the data source, the segmentation of corn, soybean, rice and other four categories was carried out. The experimental results show that overall accuracy reachs 93.6%, average recall reachs 93.2%, average precision reachs 94.0%, average F1 score reachs 93.5%, and average Intersection over Union(IoU) reachs 88.6%, which is better than other crop segmentation methods such as 3D FCN, 3D Unet, 2D Unet+CLSTM, proving the effectiveness of the proposed model.