A high-throughput population screening laboratory sees significant scale-up with implementation of QIAGEN Clinical Insight (QCI®)
INTRODUCTION
Genetic disease is the leading cause of infant death in the United States, accounting for approximately 20% of annual infant mortality.1 Screening for genetic disease has been a long-established part of preconception and prenatal care, with a community wide screening program for Tay-Sachs disease (TSD) dating back to the 1970s; however, traditional methods of carrier screening have been offered gene-by-gene, disorder-by-disorder.
Recent developments in laboratory technologies have led to the commercial availability of expanded carrier screening (ECS) panels capable of assessing hundreds of mutations associated with genetic diseases. ECS panels have the ability to identify mutations that would otherwise not be detected. While many of the disorders on these panels are individually rare, the overall risk of having an affected offspring is 1 in 280, which is higher than the risk of having a child with a neural tube defect, for which screening is universal.2
In 2012, one of the first DNA testing and genetic counselling companies to offer ECS in the United States launched a flagship ECS panel that used next-generation sequencing (NGS) technology to assess thousands of mutations associated with more than 175 of the most relevant recessive diseases. For cancer-focused screens, the lab developed a 36 gene panel for hereditary cancer risk assessment.2
In the first three years of offering ECS, the lab screened over 400,000 individuals.3 By 2016, the lab served a network of more than 10,000 health professionals, and demand for preconception screening was soaring, owing to the increasing public awareness of the ill effects related to the transfer of genetic disease.4 Unique to the lab’s ECS offering was the company’s “real-time manual curation” to support the classification of each genetic variant they encountered. Extremely thorough and highly accurate, the lab’s manual literature curation enabled the company to elevate the actionable information provided to the ordering physicians and the patients they served. However, this process was labor-intensive and costly, which was ironic given the dwindling cost of DNA sequencing and the supporting technology. The question became how to scale-up without cutting corners.
AUTOMATING GENOMIC VARIANT CURATION
Clinical decision support solutions have long been touted as the way of the future for clinical genetic testing laboratories. Combining big data analytics with advanced tools and knowledge bases, clinical decision support solutions are designed to organize, filter, and present useful information at the appropriate point in time to the person who can use it to make a decision. In 2017, the lab evaluated the use of a clinical decision support solution to help scale their genomic interpretation processes: QIAGEN Clinical Insight (QCI).*
QCI is QIAGEN’s clinical decision support solution for genetic testing laboratories. Software that reproducibly converts highly complex NGS data into clinician-ready reports, QCI is the tool through which actionable information is extracted from the sequencing results. Unlike any other clinical decision support solution on the market, QCI is largely powered by manual curation.
The knowledge base inside QCI is maintained by hundreds of Ph.D. scientists certified in clinical case curation who are committed to reading and recording all publications for a given mutation. This information is then mapped to over 2.8 million ontology classes contained within the QIAGEN Knowledge Base, providing further context by establishing relationships between variants, genes, tissue types, and pathways. When a genetic testing lab runs NGS data through QCI, the software computes the ACMG classification based on evidence curated from full-text articles, public, and private data sources. The knowledge extracted from full-text articles include observed genes, variants, function, phenotype, drug, dose, clinical cases, etc. With all this information stored in a structured knowledge base, the QIAGEN KB can quickly retrieve the relevant evidence that triggers all 28 ACMG criteria to more accurately compute an ACMG classification. Further this evidence is presented at the clinician’s fingertips for quick reference. Additionally, using natural language processing, the QIAGEN KB can auto-generate a one-sentence “finding” that is representative of the relevant evidence found in the published article.
This critical feature—automated curation of manually sourced content—saves genetic testing labs considerable time and effort when searching for variant-specific articles to satisfy the levels of evidence needed to definitively determine a classification. Especially for ECS, which is a testing practice that frequently encounters novel rare variants, the value of automation is fast becoming a necessity. To accurately and robustly appraise a novel rare variant’s pathogenicity, lab personnel must manually curate multiple lines of evidence to assess clinical significance. Therefore, if the majority of this information was autogenerated, the genomic interpretation process could be economically shortened.
The lab recognized the opportunity of integrating QCI into their curation workflow and designed a study to evaluate the concordance between the clinical evidence that QCI automatically retrieves for each observed variant classification and the clinical evidence that the lab’s curation team locates and ultimately uses in the physician reports. If the results were comparable, QCI could introduce significant time and cost savings.
EVALUATING SOFTWARE PERFORMANCE AND ACCURACY
The lab’s manual curation workflow is outlined in Figure 1. A semi-automated process, the workflow utilizes proprietary software to initially classify variants into three categories: those with high population frequency; those that have never been reported; and those needing more information before pathogenicity can be assessed. For those remaining variants, the curation team manually searches online databases, in-house article libraries, and other available resources to find variant-specific references.
Figure 1. The lab’s curation workflow
The curation workflow used to determine clinical significance of variants is summarized graphically. (a) The curation process is shown in the context of the overall laboratory workflow, in which inbound samples are eventually transformed into patient reports. (b) The curation workflow contributes lines of primary evidence that are reviewed manually, which are then combined with multiple lines of autogenerated supporting evidence to assess clinical significance.
Once evidence is collected for a variant —if any is to be found—the information is then used to assess the variant’s potential pathogenicity. As recommended by the American College of Medical Genetics (ACMG) and the Association for Molecular Pathology (AMP) published guidelines for the assessment of variants in genes associated with Mendelian diseases, the lab classifies variants following a two-step process:
First, the collected evidence is categorized into one of 28 defined criteria set forth by the ACMG-AMP guidelines and assigned a code that addresses the strength of evidence, such as population data, case-control analyses, functional data, computational predictions, allelic data, segregation studies, and de novo observations. Each code is assigned a weight (stand-alone, very strong, strong, moderate, or supporting) and direction (benign or pathogenic).
Next, the lab combines these evidence codes to arrive at one of five classifications: pathogenic (P), likely pathogenic (LP), variant of uncertain significance (VUS), likely benign (LB), or benign (B). Important in this step is the lab’s ability to modify the strength of individual criteria based on expert discretion—a safeguard that goes away with computerized systems.
To determine whether QCI could provide value to the lab’s curation team, the software was tasked with pulling a bibliography for 2,324 variants that had been recently detected by the lab’s ECS and hereditary cancer risk assessment panels. For each of these variants, the curation team had been able to match at least one published article with a specific disease-gene reference. QCI’s variant bibliography was expected to present the same quantity and quality of clinical evidence.
CONCORDANCE RESULTS
The study found that QCI’s variant bibliography was highly concordant with lab’s manual curation efforts. Of the 2,324 unique article-variant pairs identified by the lab, QCI pulled 2,075 of the references (89.3%) and an additional 13,938 article-variant pairs not captured by the lab’s curation team.
Figure 2. Overlap of bibliographic content
Figure 2 shows the overlap in content quantity between the two sources. As depicted, QCI (QIAGEN) presents significantly more data for the evaluated variants. This outcome reflects the comprehensive nature of QIAGEN’s article-centric approach, which aims to collect all publications for a given variant. While exhaustive and not always necessary, QCI’s ability to glean information from numerous sources affords the software greater accuracy in predicting variant classifications, which is seen in the second phase of the lab’s evaluation.
QUALITY OVER QUANTITY
More important than the number of bibliographic sources, accuracy of cited content ultimately dictates clinical significance. Counsyl measured the quality of QCI’s variant bibliography by looking at how the software would classify variants based on the information it pulled. What they found was a concordance of 98.8% of the pathogenic cases (Figure 2).
During the study period, a total of 682 variants were classified as pathogenic by lab’s genetic scientists. Of these, only eight would be downgraded to VUS utilizing only QCI bibliographies. Therefore, the false negative rate for using QCI’s bibliographies was ~1.2% and is expected to decrease to <1%. Further, for a sample of 50 VUS variants examined, none would change classification with additional unique references in QCI, primarily because QCI includes secondary reports and studies for other disease contexts that may be listed as ‘reviewed but not curated’ in their curations.
As a result of these positive findings, QCI bibliographies have been integrated into the lab’s manual curation workflow, eliminating the need for manual searches in the majority of cases. (Left: variant-specific page in QCI). After several months, a comparison of the time taken for reference searches before and after the adoption of QCI was performed (Figure 3).
Figure 3. Before and after adopting QCI
CONCLUSION
The goal of this evaluation was to assess whether utilization of QIAGEN’s variant-specific bibliographies could match the level of accuracy and quality of the lab’s more time-intensive manual article selection approach. Investigators concluded that there are clear benefits for adopting QCI for reference identification: an exceptionally high variant-specific article coverage, and significant time savings in a search process that can take up to ~45 minutes.
The results also serve to validate the efficacy of the lab’s previous article search and selection method, with the vast majority of variant classifications being unaltered by use of QIAGEN’s bibliographies. The lab now employs QCI bibliographies for every curated variant. Consequently, manual search methods are still employed at the lab, but can now be reserved for variants nearer VUS/pathogenic evidence thresholds.
QCI has already proven a valuable resource for increasing the efficiency of the lab’s in-house curation. Work is underway to additionally incorporate QIAGEN’s continually-updated bibliographies into the automated components of our variant classification workflows: the initial software-based auto-curation step for newly-identified variants, and the identification of those requiring re-curation in response to new publications becoming available. Accordingly, we expect QCI to further contribute to the lab’s continuing efforts to improve turnaround time by increasing curation efficiency while maintaining classification accuracy in patient reports.
*Data taken from a joint study conducted by Counsyl and QIAGEN: Cox et al. ClinGen 2017. Counsyl has since been acquired.
Learn more about QIAGEN Clinical Insight for here.
References
- Chokoshvili D, Vears D, Borry P. Expanded carrier screening for monogenic disorders: where are we now? Prenat Diagn. 2018;38:59–66.
- Haque IS, Lazarin GA, Kang HP, Evans EA, Goldberg JD, Wapner RJ. Modeled fetal risk of genetic diseases identified by expanded carrier screening. JAMA. 2016;316:734–742.
- “Global Genetic Testing Market Outlook 2022”
- https://www.healthcaredive.com/press-release/