Abstract: In order to fully explore the parallelism of convolutional neural network (CNN) computing, hardware accelerators are more attractive for their characteristics of high speed, low cost and high fault tolerance. A novel algorithm that can optimize the CNN network layer by layeris proposed, and the corresponding instruction set is designedinthis paper. The proposed algorithm can be used to find an optimal acceleration scheme for differ-ent networks with specific computing and storage resources. In the optimization process, different types of data can be quantized to half-precision to reduce memory access. Based on the 40 nm CMOS process and the proposed algorithm, aprogrammable accelerator for CNN is designed, which can achieve peak performance of 416 GOP/s under 200 MHz working frequency. VGG is implemented on our accelerator as a case study, and the latency of the total network is 116 ms.