杨鹏, 林俊晖. 一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案[J]. 微电子学与计算机, 2018, 35(4): 68-72, 78.
引用本文: 杨鹏, 林俊晖. 一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案[J]. 微电子学与计算机, 2018, 35(4): 68-72, 78.
YANG Peng, LIN Jun-hui. A Scheme for Massive Unstructured Iot Data Processing Based on MongoDB and Hadoop[J]. Microelectronics & Computer, 2018, 35(4): 68-72, 78.
Citation: YANG Peng, LIN Jun-hui. A Scheme for Massive Unstructured Iot Data Processing Based on MongoDB and Hadoop[J]. Microelectronics & Computer, 2018, 35(4): 68-72, 78.

一种基于MongoDB和Hadoop的海量非结构化物联网数据处理方案

A Scheme for Massive Unstructured Iot Data Processing Based on MongoDB and Hadoop

  • 摘要: 随着物联网数据种类的增多和数据规模的增大, 对物联网数据的存储和计算提出了新的挑战, 为了应对海量非结构化物联网数据的存储和计算要求, 提出了一种NoSQL数据库技术与MapReduce编程框架相结合的方案.使用典型的NoSQL数据库MongoDB作为主数据库来存储海量非结构化的物联网数据, 使用Hadoop MapReduce作为对物联网数据分析处理的计算框架.通过对MongoDB集群和Hadoop集群的重叠部署, 降低了计算时数据传输的开销, 构建了一套高可用、高性能的物联网大数据处理平台.通过使用该方案对海量非结构化物联网数据的处理分析实验验证了该方案的高可用性及高效性.

     

    Abstract: As the IoT data type and data volume increase, a new challenge is proposed to the storage and calculation of IoT data.To meet the storage and calculation demands of the massive unstructured IoT data, a scheme that NoSQL database technology combined with MapReduce programming framework has been presented in this paper. Using the MongoDB, a typical database of NoSQL, as the main database to store massive unstructured IoT data. At the same time, the Hadoop MapReduce was adopted as the calculation framework for IoT data analysis and processing. Through the overlapping deployment of MongoDB cluster and Hadoop cluster, the cost of data transmission is reduced, and a set of high availability and high performance large data processing platform is established. The analysis of the experiment that using this scheme to deal with massive IoT data has verified the high efficiency and high efficiency of this method.

     

/

返回文章
返回