Abstract: To address the problem that independent implementation of tongue segmentation and single-label classification tasks have difficulty in providing the pathological feature information required by the tongue clinic, a multi-task network framework for joint tongue segmentation and multi-label classification is proposed through a shared layer extraction feature strategy. Firstly, shared layer adopts a lightweight module, combined with pyramid spilt attention to fuse the deep and shallow features of the tongue image, and improve the feature extraction ability of the shared layer. Secondly, there is no obvious correlation between different labels of tongue images, and it is difficult to model the correlation of different labels, so a two-stream branch network is designed to achieve multi-label classification: One of the branches designs a filter background module based on an adaptive segmentation mask to improve performance of tongue crack recognition, and the other branch uses spatial pyramid pooling on the basis of coding blocks to achieve tongue coating classification. Finally, in the early training process, the segmentation loss is much smaller than the classification loss, and the equal loss weighting strategy will result in the segmentation task not learning the optimal parameters, so the performance of multiple tasks is improved simultaneously by an optimized uncertainty weighting strategy. Experiments have proved that the multi-task learning can effectively jointly optimize each task and improve performance while extracting shared features and reducing network parameters. Compared with multi-task learning networks such as Y-Net and MT-UNet, it has better tongue segmentation and muti-label classification performance have been improved.