Advancing next-generation sequencing data analytics with scalable distributed infrastructure

With the emergence of popular next-generation sequencing (NGS)-based genome-wide protocols such as chromatin immunoprecipitation followed by sequencing (ChIP-Seq) and RNA-Seq, there is a growing need for research and infrastructure to support the requirement of effectively analyzing NGS data. Such research and infrastructure do not replace but complement algorithmic advances developments in analyzing NGS data. We present a runtime environment, Distributed Application Runtime Environment, that supports the scalable, flexible, and extensible composition of capabilities that cover the primary requirements of NGS-based analytics. In this work, we use BFAST as a representative stand-alone tool used for NGS data analysis and a ChIP-Seq pipeline as a representative pipeline-based approach to analyze the computational requirements. We analyze the performance characteristics of BFAST and understand its dependency on different input parameters. The computational complexity of genome-wide mapping using BFAST, amongst other factors, depends upon the size of a reference genome and the data size of short reads. Characterizing the performance suggests that the mapping benefits from both scaling-up (increased fine-grained parallelism) and scaling-out (task-level parallelism - local and distributed). For certain problem instances, scaling-out can be a more efficient approach than scaling-up. On the basis of investigations using the pipeline for ChIP-Seq, we also discuss the importance of dynamical execution of tasks. Copyright © 2013 John Wiley & Sons, Ltd.