The industry’s top contenders went head-to-head in this year’s Battle of the Bioinformatics Pipelines at AMP Europe 2018. Variants were identified and interpreted and classifications were given, but in the end, QIAGEN outperformed competitors and took the checkered flag.
The contest was called to address a current industry challenge: standardizing variant interpretation and reporting. The increasing demand for next-generation sequencing (NGS)-based tests has led to a high degree of variability in how members of the global molecular genetics and pathology community classify variants and prepare final reports. To better understand the limitations of standardization and move forward in the same direction, Agilent Technologies, Thermo Fisher Scientific, and QIAGEN engaged in a friendly competition to determine which company had the most accurate and consistent NGS analysis and interpretation platform: the Alissa Interpret platform, Ion Torrent platform, and QIAGEN’s Biomedical Genomics Workbench and QIAGEN Clinical Insight (QCI™) Interpret, respectively.
Erasmus University Medical Center (Erasmus MC) Rotterdam generated sequence data from five clinical samples using Thermo Fisher’s Ion Torrent platform. The three contestants were given this sequence data and instructed to identify clonal and subclonal mutations in the tumors down to an allele frequency level of at least five percent. Then, the contestants were asked to identify and annotate the variants, calculate allele frequency, and interpret the variants according to a five-tier classification system ranging from “benign” to “clinically significant.”
This was no easy task.
Initially, seven companies expressed interest in meeting the challenge. In the end, only three crossed the finish line. According to Professor Winand Dinjens, the organizer of this year’s competition, and head of molecular diagnostics in the Department of Pathology at Erasmus MC Rotterdam, most contenders dropped out after struggling to analyze the sequencing data, which was produced with the Ion S5 XL System, a version of the Ion Torrent platform. As the race unfolded, unfamiliarity with the provided sequencing data caused more than half of the competitors to exit the track.
QIAGEN Clinical Insight (QCI™) Interpret
So, was there a home advantage?
The sequencing data in question was produced by the Ion Torrent platform, which means the Thermo Fisher team analyzed the data using tools custom-built to work with this type of sequence data. For example, during the variant calling process, the Thermo Fisher team was able to apply “flow space” information for each base as it related to Ion Torrent chemistry.
Even so, QIAGEN didn’t back down.
While Thermo Fisher may have had greater familiarity with this particular “track,” QIAGEN’s bioinformatics solutions are open platforms and have experienced over 750,000 “races” on all types of “tracks” across the world. The QIAGEN Knowledge Base is the industry’s largest, most up-to-date clinical database with the direct experience of analyzing over 750,000 human samples. This cumulative experience, in addition to more than 16 million knowledge base findings across 23,000 genes, gives QCI Interpret greater scope and depth.
For each of the five cancer samples, Erasmus supplied FASTQ files plus aligned BAM files, from which the task was to perform variant calling and interpretation and to generate a list of clinically reportable findings. Tim Bonnert, QIAGEN’s Associate Director of Bioinformatics Field Application Scientists, EMEA first used QIAGEN’s Biomedical Genomics Workbench to process the FASTQ files and perform the variant calling. Then he assessed the variants with QCI Interpret, which has curated content that triggered built-in ACMG/AMP and AMP/ASCO/CAP guidelines.
When the results from all contenders were in, none were identical.
QIAGEN’s software solutions performed significantly better than the competition. The organizers stated that there were a total of 12 variants across the five samples. QIAGEN reported 11 of the 12 variants; the variant had been identified in a homopolymer region, and the QIAGEN team identified this variant as a potential false positive. However, this classification does not change the overall clinical actionability recommendation for the patient. By contrast, Agilent and Thermo Fisher missed multiple calls. In some cases, neither platform detected variants in a sample that did in fact have clinically actionable findings; for example, the Torrent Variant Caller pipeline generated no calls for three clinically meaningful variants.
Even though QIAGEN outperformed competitors and achieved almost 100% concordance, these kinds of “battles” illustrate the importance of having pre-defined and clear standards for bioinformatics workflows. Classification and reporting of variants to healthcare providers is critical for patient care. This process requires: accurate reporting of the tumor response to targeted therapy; establishment of professional guidelines for patient care; and collaborative institutional clinical trials, thereby supporting the need for standardization among laboratories performing these tests.
According to Bonnert, “While Erasmus MC no doubt had these standards nailed down in their routine testing pipelines, for this competition there was, for example, uncertainty about required depth of coverage, acceptable distance into the intron, and other confidence metrics and standards that we believe are essential to any routine clinical bioinformatics approach. The need for standards in routine testing also extends to employing clearly defined rules and assessment criteria supported by curated clinical evidence.”
Standards are important for ensuring consistency of secondary and tertiary analysis workflows and for generating actionable data. The faithful detection of variants from Ion Torrent data by the Biomedical Genomics Workbench and the standardized clinical decision support provided by QCI Interpret proved the winning combination.
We are honored to have participated in the Battle of the Bioinformatics Pipelines. Here at QIAGEN, we are committed to building the most reliable, robust and technologically-advanced bioinformatics tools that are sequencer-agnostic because we want to empower more labs, more clinicians, and more patients to do and know more. Just as our pipeline proved victorious with Ion Torrent data in this battle, our research and clinical solutions work successfully with QIAGEN’s own GeneReader platform, as well as with data from Illumina.
Learn how the Biomedical Genomics Workbench and QCI Interpret can help you Standardize, Streamline, and Scale-up!
Get in touch with one of our experts today! CONTACT US