A Scheduler for Hadoop Based on Dynamic Resource Collection

In order to overcome the problem of wasting of resources in hadoop cluser based on the slot resource model,a scheduler based on the dynamic resource collection has been proposed.According to collect theCPU,Memory and IO resource utilization of part of tasks in the hadoop job,We estimate the resource demand of the rest tasks in the job,and then schedule these tasks according to their resource demand and the load of every Tasktracker.The contrast experiment shows the scheduler we proposed can increase the resource utilization of the hadoop cluser and reduce the completion time of the Hadoop job.