刘涛, 武淑红, 强彦. 用于多个MapReduce作业的任务调度算法[J]. 微电子学与计算机, 2013, 30(12): 156-159.
引用本文: 刘涛, 武淑红, 强彦. 用于多个MapReduce作业的任务调度算法[J]. 微电子学与计算机, 2013, 30(12): 156-159.
LIU Tao, WU Shu-hong, QIANG Yan. Task Scheduling Algorithm Used for Multiple MapReduce Tasks[J]. Microelectronics & Computer, 2013, 30(12): 156-159.
Citation: LIU Tao, WU Shu-hong, QIANG Yan. Task Scheduling Algorithm Used for Multiple MapReduce Tasks[J]. Microelectronics & Computer, 2013, 30(12): 156-159.

用于多个MapReduce作业的任务调度算法

Task Scheduling Algorithm Used for Multiple MapReduce Tasks

  • 摘要: 提出了一个可用于工作流服务的,包含多个M apReduce作业的调度机制。工作流应用程序具有优先级依赖约束(可以用有向无环图DAG表示)。同时,对于带宽有限的低数据传输成本,数据依赖限制应该在调度系统中。该调度机制能够提供:调度带有优先级约束的M apReduce任务;在网络上保存数据时,通过考虑了数据依赖限制的数据预放置策略来减少传输成本。

     

    Abstract: MapReduce has been widely used in data-intensive applications;it has several advantages,such as good programming interface,high availability and fault tolerance.It is suitable for distributed processing on large-scale data,for instance,network analysis,bioinformatics and high-performance computing.In shared cluster,there are many researches on scheduling issue.However,in multi-processers nodes,the scheduling method based on priority constrain which conclude multi MapReduce tasks is uncommon.This paper provides a scheduling method based on priority constrain which conclude multi MapReduce tasks. Workload applications have a priority dependency constraint (which can be represented by DAG).Meanwhile,for the limited bandwidth of the low cost of data transmission,data dependence limit should be concluded in the scheduling system.The proposed algorithm can provide:MapReduce task scheduling with precedence constraints;by considering the data dependencies restrict the data pre-placement strategy to reduce transmission costs.

     

/

返回文章
返回