Data Mining Based on Improved Map/Reduce and Pattern Space Division

In order to realize data mining through map/reduce based on key/value pairs,the way of processing the many-to-many corresponding relationship between the data set and the pattern set is adopted.For some of the more complex types of pattern,the pattern set is so large because of combinatorial explosion that the corresponding relationship cannot always be processed directly by the map/reduce.Therefore a way of pattern space is proposed to convert the problem of processing the many-to-many corresponding relationship between the date set and the pattern set to the problem of processing the many-to-many corresponding relationship between the data set and the set of the sub-pattern sets.At the same time,the scheduling mechanisms of the map/reduce cluster and the way of organizing the key/value pairs is improved to enhance the ability of map/reduce to execute pattern mining tasks.The results show that higher parallelism is achieved on map/reduce clusters by using this idea than the map/reduce of the traditional algorithm in mining some of the more complex types of pattern.