Abstract: Fully Convolutional Neural Networks (FCN) have been applied in various domains of deep learning in recent years. It can be used not only for simple image classification tasks but also for such as object detection, semantic/image segmentation and generative tasks based on Generative Adversarial Networks (GAN). Typical FCN includes not only traditional convolutional layers but also deconvolutional layers, which are computationally intensive. Most researchers are now focusing on the design optimization of the convolutional layer, but deconvolutional layer receives less attention. This paper proposes a Fully Convolutional Neural Networks accelerator using bi-direction systolic data flow, which can efficiently process the general convolution layer and deconvolution layer simultaneously. Several representative FCN models, such as DCGAN, Cascaded-FCN, etc., have been selected in the experiment. Compared with the traditional unoptimized acceleration scheme, the proposed approach can achieve 2.8X speed up on average, and the energy consumption is reduced by 46.3%.