Reducing Costs by Up to 87% Using AWS Batch with Seqera

In the life sciences, researchers are hard at work predicting potential pandemics, developing lifesaving therapeutics, and studying complex biological systems. In 2013, a team of scientists in Barcelona led by Paolo Di Tommaso, chief technology officer and cofounder at Seqera, created Nextflow—an open-source project to accelerate the pace of discovery through bioinformatics. Although the team used to operate its own HPC data centers, it soon found that because bioinformatics operations have sudden bursts of use followed by periods of less activity, these workloads are well suited for the cloud. The organization started using AWS services in the mid-2010s, and Nextflow fully embraced AWS in 2017 after the release of AWS Batch—a service that empowers developers, scientists, and engineers to efficiently run hundreds of thousands of batch and machine learning computing jobs while optimizing compute resources.

In 2018, the Nextflow team formed Seqera to help users streamline their journeys into data collaboration at scale. In 2020, Seqera launched Seqera Platform (formerly Nextflow Tower), a centralized command post for Nextflow pipelines. “As our project and user base grew over time, we developed strong connections with the AWS team,” says Floden. “AWS support has been critical to the emergence of our open-source community.” By 2023, Seqera had more than 7,000 global members in its nf-core community, an independent effort to collect a curated set of high-quality pipelines built using Nextflow. Moreover, Seqera is now an AWS Partner serving people in more than 47 countries.

Seqera has seen an overwhelmingly positive response to the launch of Seqera Platform, which comes in two versions: one self-hosted, the other on the cloud. The company reports that the cloud solution, which was built on AWS, has grown by 160 percent to roughly 8,000 users in 18 months, during which time the corporate customer base has more than doubled. “We have increased revenue by six times over the past 24 months,” says Floden.

With its solutions on AWS, Seqera is meeting its goal of delivering cost-optimized HPC solutions to the scientific community. An estimated 20,000 scientists, engineers, and developers use Nextflow daily, and downloads have grown by 190 percent in 18 months. “On AWS, we offer customers convenience, cost-effectiveness, and high throughput,” says Floden.

Seqera reduced costs by implementing a resource optimization feature into Seqera Platform. “With the new Seqera resource optimization feature, we’ve demonstrated that production-scale pipelines on AWS can run up to 40 percent faster,” says Floden. “We’ve also dramatically reduced the cost of pipeline runs for our customers—up to 87 percent in some cases.”

Seqera customers have several options for deploying Seqera Platform. They can use Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload, to provide access to scalable infrastructure on demand. If they wish, they can also use Amazon EC2 Spot Instances, a service to run fault-tolerant workloads for up to 90 percent lower costs compared to Amazon EC2 On-Demand pricing. Beyond the Amazon EC2 family, Seqera customers can deploy to Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service to run Kubernetes on AWS, and to AWS Fargate, a serverless, pay-as-you-go compute engine. By making Seqera Platform compatible with so many services, Seqera empowers customers to choose the infrastructure that makes sense for them. (See figure 1. Seqera Cloud on the AWS Cloud.)

The company has continually expanded its offerings by using AWS services. For example, most of its customers use Amazon Simple Storage Service (Amazon S3), an object storage service offering exceptional scalability, data availability, security, and performance. However, because some of its customers require a high-performance file system, Seqera also uses Amazon FSx for Lustre, which provides fully managed shared storage with the scalability and performance of Lustre. “Using Amazon FSx makes it incredibly simple to deploy Lustre in an on-premises environment, which is normally very complicated,” says Di Tommaso. “That makes it attractive to our customers.”

Seqera teams are excited to speed up the pace of innovation on AWS. “Our customers used to deploy HPC clusters in on-premises data centers and had to wait weeks or months for the clusters to be ready for use,” says Di Tommaso. “Now, we automatically deploy compute environments using AWS Batch.”