DEV Community: 毅硕科技

Sentieon | Application Tutorial: Accelerating Custom Algorithms Using the Sentieon Python API Engine

毅硕科技 — Tue, 17 Dec 2024 02:55:08 +0000

Background

All modules in the Sentieon suite are several times to dozens of times faster than their corresponding open-source software. While using these modules, users sometimes hope that the Sentieon team can help accelerate their custom-developed software. To help these users enjoy the speed of Sentieon modules in their own software, we have developed a Python API system to meet the needs of secondary development and self-acceleration.

API Introduction

*The Sentieon Python API is essentially a communication system that connects users' data analysis scripts with Sentieon's high-speed engine, accelerating the process while also improving the readability and maintainability of the scripts.
*
Sentieon's data processing engine is the core of multiple Sentieon modules, capable of high-speed analysis of BAM/CRAM and FASTA format data files. The engine supports both single-pass and multithreaded execution data flow methods. The multithreaded data flow is faster but relatively more complex. It divides the genome into fragments with a default length of 1Gb. The Sentieon engine processes each fragment independently in parallel across multiple threads. Each fragment is further divided into smaller segments (Steps) with a default length of 1Kb, which the engine processes linearly. During this process, the data processing logic of the user's software will be executed at high speed.

Implementation Case Study

We will demonstrate the acceleration effect of Sentieon through a collaborative case with the CREST software team from St. Jude Children's Research Hospital in the United States. CREST (Clipping REveals STructure) is a well-known software in the industry for detecting structural variations in cancer genomes, primarily using breakpoints as clues to identify structural variations in the genome. Specifically, the CREST software workflow includes steps such as soft-clip detection, assembly, post-assembly alignment, breakpoint confirmation, and structural variation confirmation. The assembly and alignment steps mainly rely on third-party tools. While CREST's advantage lies in its high accuracy, its speed limitations are equally apparent. For a standard 30x tumor whole-genome paired sample, the processing time on a 20-thread workstation can take up to 24 hours, which is often insufficient to meet user demands.

After learning about the capabilities of the Sentieon Python API, the CREST team reimplemented CREST's functionality using this system. In test data, the Sentieon-accelerated version of CREST achieved a 10-fold speed increase, with results identical to the original CREST. On a 20-thread workstation, it reduced the processing time for the vast majority of samples to under 1 hour.

Next, we'll introduce two more application acceleration case studies. Quality control is a crucial step in NGS data processing workflows. Although the logic is relatively simple, it involves extensive reading of BAM/CRAM files. These tools often struggle to balance speed, multi-threaded parallelism, and code maintainability.

The Sentieon Python API can separate the algorithmic logic of quality control tools from data reading, simultaneously improving speed and code readability. As implementation examples, we used the Python API to accelerate Picard's CollectInsertSizeMetrics tool for rapid insert size statistics, and GATK's CalculateTargetCoverage tool for quick depth statistics in target regions. Users can also refer to these cases to accelerate their own custom quality control tools.

Technical Support

The Sentieon Python API allows users' scripts to communicate with the Sentieon engine, enabling high-speed parallel reading of BAM/CRAM/FASTA files, resulting in over 10-fold speed improvements.** Users can utilize this platform for secondary development to accelerate their custom software**. We are more than willing to provide comprehensive technical support.

Introduction to Sentieon Software

Sentieon offers a comprehensive, purely software-based solution for secondary analysis in genetic variant detection. Its analysis pipeline remains entirely faithful to the mathematical models of gold standards such as BWA, GATK, MuTect2, STAR, Minimap2, Fgbio, Picard, and others. While matching the analysis results of open-source workflows, Sentieon significantly improves the analysis efficiency and detection accuracy for sequencing data from WGS, WES, Panel, UMI, ctDNA, RNA, and other sources. It is compatible with all current second and third-generation sequencing platforms.

The Sentieon software team possesses rich experience in software development and algorithm optimization engineering. They are dedicated to solving speed and accuracy bottlenecks in biological data analysis, providing efficient and precise software solutions for partners from various fields such as molecular diagnostics, drug development, clinical medicine, population cohorts, and animal and plant research, jointly promoting the development of genetic technology.

As of the end of 2023, Sentieon has provided services to over 1300 users worldwide and has been widely cited in top-tier impact factor journals such as NEJM, Cell, and Nature, with nearly a thousand citations. Furthermore, Sentieon has consistently won accolades in authoritative evaluations such as Precision FDA and Dream Challenges for several consecutive years, gaining widespread recognition in the industry.

Introduction to Sentieon

毅硕科技 — Thu, 07 Nov 2024 09:41:04 +0000

Sentieon Vision

01.Sentieon Vision

Sentieon is dedicated to solving the bottleneck of speed and accuracy in bioinformatics data analysis, through the deep optimization of algorithms and enterprise-level software engineering, to significantly improve the efficiency, accuracy and reliability of NGS data processing.

Sentieon provides software solutions to partners and research institutions in the fields of molecular diagnostics, drug discovery and development, clinical medicine, animal and plant genomics, etc. Sentieon is committed to advancing the development of genetic technology and realizing the vision of “Achieving Precision Data, Serving Precision Medicine”.

02.Bioinformatics Industry Pain Points

Currently, for secondary analysis of second-generation sequencing data (SNP/Indel variant detection), BWA + GATK, as a best-practice process, has been widely recognized by the industry after more than a decade of repeated validation with a large number of samples, and is widely acknowledged as the gold standard.

However, the BWA + GATK process has several significant problems:

Long runtime, low resource utilization, and high computational cost;
GATK's random downsampling in high sequencing depth regions significantly affects the sensitivity and accuracy of variant detection, especially in the case of low abundance (VAF) variants in tumors, and GATK causes inconsistency in the results when repeated runs are performed.

03.Sentieon Solutions

Sentieon introduces a complete software-only secondary analysis solution that covers the entire process from gene matching to mutation detection for both germline mutation detection and somatic mutation detection.

Sentieon software is faithful to the mathematical models of gold standards such as BWA, GATK, MuTect, MuTect2, Minimap2, etc., and improves the analysis efficiency by more than 10 times under the premise of ensuring that the results exactly match the gold standards.

04.Awards

Since Sentieon officially launched its software product in 2015, it has won first place or tied for first place in several international raw letter competitions:

Bio-IT Innovative Practices Award.
PrecisionFDA Truth Challenge.
PrecisionFDA Consistency Challenge
ICGC-TCGA Dream Mutation Calling Challenge.
PrecisionFDA Hidden Treasures Warm-up.
PrecisionFDA NCI-CPTAC Crowdsourced Multi-omics Sample Mislabeling Big Data Challenge.

software functionality

01.Highlights

Pure soft solution: can be efficiently and flexibly deployed on different computing environments. No special hardware is required, reducing operation and maintenance costs. Can support X86\ARM architecture processor at the same time.

Highlights

1.Accelerated analysis: 10x faster than original BWA/GATK process

2.No downsampling: no differences between batches, better for high depth and low abundance detection.

3.Accurate results: 99.7% consistency compared to BWA/GATK/Mutect2/Minimap2 process.

4.Cost Savings: Cost savings of up to 90%, dramatically improving the accuracy of shallow sequencing results.

5.Joint Calling: One-time joint call operation for 100,000 WGS samples, no need for large memory and no intermediate steps.

6.Enterprise-level service: Provide comprehensive customer support from trial to use and establish long-term partnership with customers.

02.functional module

Sentieon software is very flexible, you can use the complete process directly, or according to the specific needs of the relevant modules to replace the original module. Users can directly use the complete process, but also according to the specific needs of the relevant modules to replace the original process modules, Sentieon software input and output files in strict accordance with industry norms for the operation, the user can easily get started to reduce the learning time. Currently Sentieon software can provide customers with the use of the module and the specific functions please refer to the following table:

03.application scenario

As an alternative acceleration solution to the industry gold-standard BWA/GATK/Mutect2/STAR processes, Sentieon software is also suitable for basic research, translational medicine, clinical and other fields. Some of the applicable scenarios are listed below:

Industry Recognition

Sentieon software is widely certified and used by nearly 1,000 commercial companies and research organizations, including:

Several large multinational pharmaceutical companies sequencing service providers and molecular diagnostic companies worldwide.
Hospitals and medical research institutions: including the O'Brien Project led by 15 large hospitals and pharmaceutical companies in the U.S., Fudan University Pediatrics Hospital, Shanghai Jiaotong University Children's Hospital, Xiangya Hospital, and CUHK Cancer Medical Center.
Governmental organizations: including the U.S. Food and Drug Administration (FDA), the China Food and Drug Administration (CFDA), the U.S. National Institutes of Health (NIH), and the U.S. National Cancer Institute (NCI).
Universities and research institutes: including Harvard, Yale, Stanford, Cornell, Chinese Academy of Sciences, Union Medical College, Fudan University, Shanghai Jiaotong University, Sun Yat-sen University, Yunnan University, Chinese Academy of Agricultural Sciences, State Key Laboratory of Crop Genetic Improvement and so on.

So far, Sentieon software has run and processed more than 1430 PB of data, with millions of samples and more than 360 million kernel hours, and has been cited many times in journals with world-class impact factors, including NEJM, Cell, and Nature.