高级检索

  • ISSN 1006-3080
  • CN 31-1691/TQ

大数据流计算环境下的低延时高可靠性的资源调度方法

孙怀英 虞慧群 范贵生 陈丽琼

孙怀英, 虞慧群, 范贵生, 陈丽琼. 大数据流计算环境下的低延时高可靠性的资源调度方法[J]. 华东理工大学学报(自然科学版), 2017, (6): 855-862. doi: 10.14135/j.cnki.1006-3080.2017.06.016
引用本文: 孙怀英, 虞慧群, 范贵生, 陈丽琼. 大数据流计算环境下的低延时高可靠性的资源调度方法[J]. 华东理工大学学报(自然科学版), 2017, (6): 855-862. doi: 10.14135/j.cnki.1006-3080.2017.06.016
SUN Huai-ying, YU Hui-qun, FAN Gui-sheng, CHEN Li-qiong. Low Latency and High-Reliability Resource Scheduling Method in Big Data Streaming Computing Environment[J]. Journal of East China University of Science and Technology, 2017, (6): 855-862. doi: 10.14135/j.cnki.1006-3080.2017.06.016
Citation: SUN Huai-ying, YU Hui-qun, FAN Gui-sheng, CHEN Li-qiong. Low Latency and High-Reliability Resource Scheduling Method in Big Data Streaming Computing Environment[J]. Journal of East China University of Science and Technology, 2017, (6): 855-862. doi: 10.14135/j.cnki.1006-3080.2017.06.016

大数据流计算环境下的低延时高可靠性的资源调度方法

doi: 10.14135/j.cnki.1006-3080.2017.06.016
基金项目: 

国家自然科学基金(61173048,61300041);高等学校博士学科点专向科研基金(20130074110015);中央高校基本科研业务费专项基金(WH1314038)

Low Latency and High-Reliability Resource Scheduling Method in Big Data Streaming Computing Environment

  • 摘要: 在大数据处理过程中,如何保证流数据处理的可靠性及实时性变得日益重要。本文使用数据流图(DSG)对大数据流应用过程进行描述,并将DSG表示为扩展的Petri网以便对数据流过程进行建模。提出了基于CPU利用率平均变化率的资源熵算法计算资源组可靠性,并根据资源熵算法提出了基于时间和可靠性的资源调度算法(TRS-SCHE)以获得高可靠性、低延时的资源调度方案。通过仿真实验,模拟实现soda交通大数据分析应用并进行资源的调度,验证了TRS-SCHE相比于Storm隔离调度算法在响应时间、请求失败率和算法时间复杂度方面的优势。

     

  • [1] SUN D,ZHANG G,YANG S,et al.Re-Stream:Real-time and energy-efficient resource scheduling in big data stream computing environments[J].Information Sciences,2015,319:92-112.
    [2] KANOUN K,VAN d S M.Big-data streaming applications scheduling with online learning and concept drift detection[C]//Design,Automation & Test in Europe Conference & Exhibition.USA:IEEE,2015:1547-1550.
    [3] XHAFA F,NARANJO V,CABALLE S.Processing and analytics of big data streams with Yahoo!S4[C]//IEEE International Conference on Advanced Information Networking & Applications.USA:IEEE,2015:263-270.
    [4] MURRAY D G,MCSHERRY F,ISAACS R,et al.Naiad:Atimely dataflow system[C]//Twenty-Fourth ACM Symposium on Operating Systems Principles.USA:ACM,2013:439-455.
    [5] SHIM S J,LEE D S,LEE W S.CP-tree:An adaptive synopsis structure for compressing frequent itemsets over online data streams[J].Information Sciences,2014,278:559-576.
    [6] ZAHARIA M,DAS T,LI H,et al.Discretized streams:Fault-tolerant streaming computation at scale[C]//Twenty-Fourth ACM Symposium on Operating Systems Principles.USA:ACM,2013:423-438.
    [7] XU J,CHEN Z,TANG J,et al.T-Storm:Traffic-Aware online scheduling in storm[C]//IEEE International Conference on Distributed Computing Systems.USA:IEEE,2014:535-544.
    [8] OUSTERHOUT K,WENDELL P,ZAHARIA M,et al.Sparrow:Distributed,low latency scheduling[C]//Twenty-Fourth ACM Symposium on Operating Systems Principles.USA:ACM,2013:69-84.
    [9] XIANG Y,WANG L,FU T.A preliminary study of power system reliability considering cloud service reliability[C]//International Conference on Power System Technology.USA:IEEE,2014:2031-2036.
    [10] QIAN Zhengping,HE Yong,SU Chunzhi,et al.TimeStream:Reliable stream computation in the cloud[C]//ACM European Conference on Computer Systems.New York,USA:ACM,2013:1-14.
    [11] CHEN H,WANG F Z.Spark on entropy:A reliable & efficient scheduler for low-latency parallel jobs in heterogeneous cloud[C]//IEEE International Workshop on Cloud-based Networks and Applications.USA:IEEE,2015:708-713.
    [12] CHRISTODOULOU S,ELLINAS G,ASLANI P.Entropy-based scheduling of resource-constrained construction projects[J].Automation in Construction,2009,18(7):919-928.
    [13] XIE Z,HAN L,BALDOCK R.Augmented Petrinet cost model for optimisation of large bioinformatics workflows using cloud[C]//European Modelling Symposium.USA:ACM,2013:201-205.
    [14] ZHU Lianzhang,TAN Shouchao,ZHANG Weishan,et al.Validation of pervasive cloud task migration with colored Petri net[J].Tsinghua Science and Technology,2016,21(1):89-101.
    [15] RIBAS M,FURTADO C G,BARROSO G,et al.Modeling the use of spot instances for cost reduction in cloud computing adoption using a Petri net framework[C]//IEEE International Symposium on Integrated Network Management.USA:IEEE,2015:312-9.
  • 加载中
图(1)
计量
  • 文章访问数:  1411
  • HTML全文浏览量:  215
  • PDF下载量:  340
  • 被引次数: 0
出版历程
  • 收稿日期:  2016-12-27
  • 刊出日期:  2017-12-28

目录

    /

    返回文章
    返回