Research on Efficient Data Filtering Based on FPGA

With the explosive growth of data volume, how to process the massive data within a reasonable time is becoming more and more important. Recently, Hadoop has got a great attention in academia and industry with high reliability, high expansibility and MapReduce programing model. While Hadoop was designed to handle large scale gathering mission, and this kind of mission need to scan the whole data-sets.In order to increase the efficiency of data processing, FPGA was injected into Hadoop cluster. FPGA will filter the data stored in HDFS(Hadoop Distributed File System). In this way MapRedeuce just need to process the filtered data. A series experiment shows that FPGA could filter useless data and increase the efficiency of data processing.