CMS workflow execution using intelligent job scheduling and data access strategies

Hasham, K. print, Delgado Peris, A. print, Anjum, A. print, Evans, D. print, Gowdy, S. print, Hernandez, J. M. print, Huedo, E. print, Hufnagel, D. print, van Lingen, F. print and McClatchey, R. print (2011) CMS workflow execution using intelligent job scheduling and data access strategies. IEEE Transactions on Nuclear Science, 58 (3). pp. 1221-1232. ISSN 0018-9499

[img]
Preview
 PDF - Accepted Version
330Kb

Publisher's URL: http://dx.doi.org/10.1109/TNS.2011.2146276

Abstract

Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies, such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficient and robust analysis environment. In this paper, we discuss an example of a pilot job based infrastructure (as used in the CMS Tier0 analysis workflow at CERN) and intelligent data reuse and jobs execution strategies to minimize its scheduling, queuing, execution and data access latencies, which have helped to achieve significant gains in the overall turnaround time of the workflow.

Item Type:Article
Additional Information:© 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.
Uncontrolled Keywords:data cahce, grid, latency, pilot jobs, workflow
Faculty/Department:Faculty of Environment and Technology > Department of Computer Science and Creative Technologies
~Pre-2010 Faculty Structure > Environment and Technology > Bristol Institute of Technology > Centre for Complex Co-operative Systems
~Pre-2012 Faculty Structure > Faculty of Environment and Technology > Centre for Complex Co-operative Systems
~Pre-2012 Faculty Structure > Faculty of Environment and Technology > Department of Computer Science and Creative Technologies
ID Code:11617
Deposited By: Professor R. print Mcclatchey
Deposited On:15 Nov 2010 14:42
Last Modified:01 Dec 2012 18:36

Repository Staff Only: item control page

Total Document Downloads in Past 12 Months

Document Downloads

Total Document Downloads

More statistics for this item...
Copyright 2013 © UWE better together