A Parallel Acceleration Strategy for Distributed DBN in Spark

DDBN(Distributed Deep Belief Network, DDBN) has many problems in Spark, such as data skew, lack of fine-grained data replacement, and unable to cache data with high re-usability automatically, resulting in high complexity and low timeliness of DDBN computing. In order to improve the timeliness of DDBN, a parallel acceleration strategy is proposed for DDBN in Spark, which includes LSRP(Label Set based on Range Partition, LSRP) algorithm and CRWS(Cache Replacement based on Weight Statistics, CRWS) algorithm. The problem of data skew is solved by LSRP algorithm, and CRW algorithm is used to solve the problem of RDD reuse and cached data caused by insufficient memory space. The results show that compared with the traditional DBN, the training speed of DDBN is increased by about 2.3 times, and the distributed parallelism of DDBN is greatly improved through LSRP and CRW.