A lightweight method for designing and massive processing of genomics pipelines

The analysis of genomics data involves multistep complex tasks that, in addition to consume a substantial amount of computational resources, require intervention by the operator for execution. A pipeline, comprising preconfigured steps, is designed to maximize efficiency and reproducibility. In this context, pipelines facilitate the automation and orchestration of complex processes, minimizing the need for manual intervention. In this paper, we propose a FLexible ENgine for massive processing of Pipelines in genomics (FLENP). It orchestrates the processing according to the steps defined in an ad-hoc template, by maximizing the CPU usage during the computation. Our method exhibited high robustness and an inherent flexibility. Furthermore, it has proven not to be expensive in terms of memory, and it does not introduce a significant latency during the execution. We also evaluated its performance by designing a template on a real use case for the Transcript-level Quantification of RNA-seq experiments.