CMS workflow execution using intelligent job scheduling and data access strategies
Hasham, K. print, Delgado Peris, A. print, Anjum, A. print, Evans, D. print, Gowdy, S. print, Hernandez, J. M. print, Huedo, E. print, Hufnagel, D. print, van Lingen, F. print and McClatchey, R. print (2011) CMS workflow execution using intelligent job scheduling and data access strategies. IEEE Transactions on Nuclear Science, 58 (3). pp. 1221-1232. ISSN 0018-9499
Publisher's URL: http://dx.doi.org/10.1109/TNS.2011.2146276
Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies, such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficient and robust analysis environment. In this paper, we discuss an example of a pilot job based infrastructure (as used in the CMS Tier0 analysis workflow at CERN) and intelligent data reuse and jobs execution strategies to minimize its scheduling, queuing, execution and data access latencies, which have helped to achieve significant gains in the overall turnaround time of the workflow.
Repository Staff Only: item control page
Total Document Downloads in Past 12 Months
Total Document DownloadsMore statistics for this item...