INTRODUCTION
The COVID-19 pandemic has challenged public health systems worldwide, not only for patient care or surveillance and control but also to guarantee the quality of SARS-CoV-2–related diagnostic tools. For instance, multiple in vitro RT-qPCR diagnostic kits are available in the market for the detection of SARS-CoV-2. Some of them have received Emergency Use Authorization (EUA) from the U.S. Food and Drug Administration (FDA), whereas many others just report limited validations made by the manufacturers. The CDC-designed FDA EUA 2019-nCoV CDC kit (IDT) is based on N1 and N2 probes to detect SARS-CoV-2 and RNase P as an RNA extraction quality control, being considered gold standard worldwide for SARS-CoV-2 RT-PCR diagnosis.1–5
Among the commercial kits available in South America for SARS-CoV-2 diagnosis, some of them have FDA authorization. On the other hand, SARS-CoV-2 RT-qPCR kits without clinical use authorization from their country of origin are exported and sold in South America for clinical use. For instance, the 2019-nCoV kit (Da An Gene) is a commercially available multiplex SARS-CoV-2 RT-qPCR assay for N and ORF1ab genes that is authorized for clinical use by the Chinese CDC. However, the GenomeCoV19 kit (ABM) is also a commercially available multiplex SARS-CoV-2 RT-qPCR kit for N and RdRp genes, lacking clinical use authorization in Canada.6 We herein present a comparison of the analytical and clinical performance of 2019-nCoV kit (Da An Gene) and GenomeCoV19 kit (ABM) for SARS-CoV-2 RT-qPCR diagnosis from nasopharyngeal samples.
MATERIALS AND METHODS
Study design.
In total, 107 clinical specimens (nasopharyngeal swabs collected on 0.5 mL Tris-EDTA (TE) pH 8 buffer) were included on this study. Also, negative controls (TE pH 8 buffer) were included as control for carryover contamination, one for each set of RNA extractions.
RNA extraction and RT-qPCR for SARS-CoV-2 diagnosis using 2019-nCoV CDC kit.
All the samples included in the study were tested following an adapted version of the CDC protocol1: using the AccuPrep Viral RNA extraction kit (Bioneer, Daejeon, South Korea) as an alternate RNA extraction method,2 and using CFX96 BioRad instrument.7–12
RT-qPCR for SARS-CoV-2 diagnosis using 2019-nCoV kit (Da An Gene).
The same RNA extractions from all the samples included in the study were tested using the 2019-nCoV kit (Da An Gene), following the manufacturer’s instruction manual.13 To avoid RNA degradation, samples were processed for the 2019-nCoV CDC kit and 2019-nCoV kit (Da An Gene) within the same day or stored at −80° C for next day processing.
RT-qPCR for SARS-CoV-2 diagnosis using GenomeCoV19 kit (ABM).
The same RNA extractions from all the samples included in the study were tested using the GenomeCoV19 kit (ABM), following the manufacturer’s instruction manual.14 To avoid RNA degradation, samples were processed by using the 2019-nCoV CDC kit and GenomeCoV19 kit (ABM) within the same day or stored at −80° C for next day processing.
Analytical sensitivity.
The limit of detection (LoD) was performed using the commercially available 2019-nCoV N positive control (IDT); provided at 200.000 genome equivalents/mL, it was used for calibration curves to obtain the viral loads of the samples. Viral loads can be expressed as copies/µL of RNA extraction or copies/mL of sample; the conversion factor is 200 as 0.2 mL of sample is used for RNA extraction and 40 µL is used as final elution volume of RNA extraction.
Statistics.
95% confident intervals were calculated using Jamovi Statistical Software.
Ethics statement.
All samples have been submitted for routine patient care and diagnostics. Ethics approval was not sought because the study involves laboratory validation of test methods and the secondary use of anonymous pathological specimens that falls under the category “exempted” by Comité de Etica para Investigación en Seres Humanos from “Universidad de Las Américas.”
RESULTS
Clinical performance of GenomeCoV19 kit (ABM) compared with the CDC gold standard protocol.
In total, 107 samples were tested for SARS-CoV-2 following both GenomeCoV19 kit (ABM) and 2019-nCoV CDC kit protocols, as described in the Methods section. For the 2019-nCoV CDC EUA kit, 68 samples tested positive and 39 samples tested negative (Table 1 and Supplemental Table 1). All 39 samples tested negative by the 2019-nCoV CDC kit were also SARS-CoV-2 negative by the GenomeCoV19 kit (ABM), so the specificity obtained in our study was 100%.
Clinical performance of GenomeCoV19 kit (ABM) and 2019-nCoV kit (Da An Gene) using the CDC protocol as a gold standard (% values: sensitivity)
| RT-PCR kit | Positive samples (including “inconclusive” samples) | False-negative samples | Total SARS-CoV-2–positive samples |
|---|---|---|---|
| 2019-nCoV kit (Da An Gene) | 68 (100%) | 0 | 68 |
| GenomeCoV19 kit (ABM) | 51 (75%) | 17 | 68 |
Only SARS-CoV-2–positive samples included in the study are detailed.
Considering only true-positive samples by the 2019-nCoV kit (amplification of both N and ORF1ab genes according to the manufacturer’s protocol), 26 samples tested positive and 81 samples were negative; but if we consider as positives not only true positives but also “inconclusive” samples (only amplification of N gene), 51 samples tested positive and 56 were negative. In summary, sensitivity of the GenomeCoV19 kit (ABM) compared with the 2019-nCoV CDC EUA was 38.2% (35.7–40.7, CI 95%) if we considered only true positives, but up to 75% (71.3–78.7, CI 95%) if we considered also “inconclusive” samples as positives for SARS-CoV-2 (Table 1 and Supplemental Table 1).
The viral loads of SARS-CoV-2–positive samples that tested negative by the GenomeCoV19 kit were as high as 649 copies/µL (129.800 copies/mL). Actually, four SARS-CoV-2–positive samples with viral loads more than 40 copies/µL (8,000 copies/mL) tested negative by the GenomeCoV19 kit. So, even if we considered the LoD for the GenomeCoV19 kit (ABM) at 40 copies/µL, the sensitivity obtained will be 89.2%, as 33 of 37 SARS-CoV-2–positive samples were detected.
Clinical performance of 2019-nCoV kit (Da An Gene) compared with the CDC gold standard protocol.
In total, 107 samples were tested for SARS-CoV-2 following both 2019-nCoV kit (Da An Gene) and 2019-nCoV CDC kit protocols, as described in the Methods section. As it was detailed earlier, for the 2019-nCoV CDC EUA kit, 68 samples tested positive and 39 samples tested negative (Table 1 and Supplemental Table 1). All 39 samples tested negative according to the 2019-nCoV CDC kit were also SARS-CoV-2 negative by the 2019-nCoV kit (Da An Gene), so the specificity obtained in our study was 100%.
Considering only true-positive samples by the 2019-nCoV kit (amplification of both N and Orf1ab genes according to the manufacturer’s protocol), 66 samples tested positive and 41 samples were negative; but if we consider as positive not only true positives but also “inconclusive” samples (only amplification of N gene or Orf1ab gene), all the 68 SARS-CoV-2–positive samples tested positive. In summary, sensitivity of the 2019-nCoV kit (Da An Gene) compared with the 2019-nCoV CDC EUA was 97% (94.1–99.9, CI 95%) if we considered only true positives, but up to 100% if we considered also “inconclusive” samples as positives for SARS-CoV-2 (Table 1 and Supplemental Table 1).
The viral loads of SARS-CoV-2–positive samples that tested “inconclusive” by the 2019-nCoV kit (Da An Gene) were as low as 3.51 and 2.80 copies/µL, both even below the LoD of 1,000 copies/mL for the CDC protocol (see samples 65 and 68 at Supplemental Table 1).
Analytical sensitivity: Calculation of the LoD of 2019-nCoV kit (Da An Gene).
The viral loads detailed in Supplemental Table 1 were calculated using a calibration curve with 2019-nCoV N positive control (IDT). In previous studies, the LoD for the CDC protocol was set at 1,000 viral RNA copies per mL of sample (or five RNA copies/µL of RNA extraction solution).1,5,7–11 For the 2019-nCoV kit (Da An Gene), the positive control included on the kit is reported as a “pseudovirus” of unknown concentration, making its use impossible for LoD determination. We used a SARS-CoV-2–positive RNA extraction of known viral load to set a calibration curve for values ranging from 1 to 10 copies/µL, equivalent to 200–2,000 copies/mL (Table 2). As the LoD is defined as the lowest viral load in which all replicates are detected (100% sensitivity), our data indicate that the LoD for N and ORF1ab gene was 2,000 viral RNA copies/mL of sample (10 copies/µL of RNA extraction solution), as it is detailed in Table 2.
DISCUSSION
Although the main limitation of our study is the sample size (107 specimens), our results support that the 2019-nCoV kit (Da An Gene) had a great performance in terms on sensitivity and specificity compared with 2019-nCoV CDC EUA, with sensitivity up to 100%. As we have described in the Results section, we could calculate the LoD for the 2019-nCoV kit (Da An Gene) on 2,000 viral RNA copies/mL of sample; although the manufacturer’s manual reported an LoD of 500 copies/mL,13 the LoD and sensitivity are quite acceptable for a reliable SARS-CoV-2 diagnosis considering the viral load frequency population distribution already reported.14,15
On the other hand, our results support that the GenomeCoV19 kit (ABM) had lower performance in terms on sensitivity than 2019-nCoV CDC EUA, with values up to 75% if “inconclusive” (N gene amplification only) samples were considered as SARS-CoV-2 positive, and strikingly low as 38.2% when true positives were considered. So far, the performance of primers and probes for the RdRp target at the GenomeCoV19 kit (ABM) is questionable for a good-quality SARS-CoV-2 diagnosis. As we have described in the Results section, we could estimate the LoD of the GenomeCoV19 kit (ABM) to be at least more than 8,000 viral RNA copies/mL of sample, as four SARS-CoV-2–positive samples above this threshold failed to amplify neither N nor RdRp gene targets. It is worrisome that the manufacturer’s manual reported an LoD of 5 copies/reaction,16 considering that the sample reaction volumen sample for the Genome CoV19 kit (ABM) is 5 µL,16 which would mean an LoD of 1 copy/µL of RNA extraction. This value is equivalent to 200 copies/mL under our experimental conditions, which is clearly much lower than the LoD experimentally obtained.
Table 3 summarizes the analytical parameters and other features of the GenomeCoV19 kit (ABM) and 2019-nCoV kit (Da An Gene) kits. As we have detailed in the Introduction section, although the 2019-nCoV kit (Da An Gene) has clinical use authorization by the People’s Republic of China CDC, the Canadian public health authorities do not allow the clinical use of the GenomeCoV19 kit (ABM) in Canada.6,16 Considering the remarkable clinical performance of the 2019-nCoV kit (Da An Gene) and the poor clinical performance of the GenomeCoV19 kit (ABM), our study suggests that a good public policy for developing countries like Ecuador would be to allow the importation of SARS-CoV-2 RT-PCR diagnosis kits with clinical use authorization from their country of origin. We have previously published other clinical evaluation studies that endorse this same idea. For instance, whereas the SARS-CoV-2 RT-PCR and RT-LAMP from the South Korean companies MiCo BioMed, Bioneer, and M Monitor have poor clinical performance and lack clinical use authorization by the Korean CDC,5,12,17 the Viasure SARS-CoV-2 RT PCR kit from the Spanish company Certest shows a good clinical performance and has clinical use authorization in Spain.11 So far, we expressed our deep ethical concern toward companies that export SARS-CoV-2 diagnosis kits to developing countries, although they are not allowed to sell them for diagnostic purposes of their fellow citizens.
Comparison of 2019-nCoV CDC EUA (IDT), GenomeCoV19 kit (ABM) and 2019-nCoV kit (Da An Gene) kits
| SARS-CoV-2 RT-PCR kit (company/country) | Viral gene targets | Limit of detection observed (promised by manufacturer) | EUA at country of origin |
|---|---|---|---|
| 2019-nCoV CDC EUA (IDT) | N1 and N2 | 1,000 viral copies/mL | Yes (FDA). |
| GenomeCoV19 kit (ABM) | N and RdRp | > 8,000 viral copies/mL (66 viral copies/mL) | NO |
| 2019-nCoV kit (Da An Gene) | N and ORF1ab | 2,000 viral copies/mL (500 viral copies/mL) | Yes (C-CDC) |
C-CDC = Chinese CDC; EUA = Emergency Use Authorization; FDA = Federal Drug Administration. For the limit of detection indicated, the sensitivity obtained was 100% for the 2019-nCoV kit and 89.2% for the GenomeCoV19 kit.
Considering the high worldwide demand for reagents for SARS-CoV RT-qPCR diagnosis, supply shortage is a fact, affecting harder the developing countries like Ecuador. In this scenario, independent clinical performance and analytical sensitivity evaluations are mandatory for companies exporting their products to developing countries, usually those lacking reliable regulatory agencies. This is crucial to guarantee the quality of the supplies in the market for every country in the world, and it is a matter of global justice and human rights.
REFERENCES
- 1.↑
Lu X et al. 2020. US CDC real-time reverse transcription PCR panel for detection of severe acute respiratory syndrome coronavirus 2. Emerging Infect Dis 26: 8.
- 2.↑
CDC, 2020. Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens from Persons for Coronavirus Disease 2019 (COVID-19). Atlanta, GA: Center for Diseases Control and Prevention. Available at: https://www.cdc.gov/coronavirus/2019-ncov/lab/guidelines-clinical-specimens.html. Accessed May 9, 2020.
- 3.
Rhoads DD, Cherian SS, Roman K, Stempak LM, Schmotzer CL, Sadri N, 2020. Comparison of Abbott ID Now, diasorin simplexa, and CDC FDA EUA methods for the detection of SARS-CoV-2 from nasopharyngeal and nasal swabs from individuals diagnosed with COVID-19. J Clin Microbiol 58: e00760-20.
- 4.
Nallaa AK, Castob AM, Huanga MLW, Perchettia GA, Sampoleoa R, Shresthaa L, Weia Y, Zhua H, Jeromea KR, Greningera AL, 2020. Comparative performance of SARS-CoV-2 detection assays using seven different primer/probe sets and one assay kit. J Clin Microbiol 58: e00557-20.
- 5.↑
Freire-Paspuel B, Vega-Mariño P, Velez A, Castillo P, Cruz M, Garcia-Bereguiain MA, 2020. Evaluation of nCoV-QS (MiCo BioMed) for RT-qPCR detection of SARS-CoV-2 from nasopharyngeal samples using CDC FDA EUA qPCR kit as a gold standard: an example of the need of validation studies. J Clin Virol 128: 104454.
- 7.↑
Freire-Paspuel B, Vega-Mariño P, Velez A, Castillo P, Gomez-Santos EE, Cruz M, Garcia-Bereguiain MA, 2020. Cotton-tipped plastic swabs for SARS-CoV-2 RT-qPCR diagnosis to prevent supply shortages. Front Cell Infect Microbiol 10: 356.
- 8.
Freire-Paspuel B, Vega-Mariño P, Velez A, Castillo P, Cruz M, Garcia-Bereguiain MA, 2020. Sample pooling of RNA extracts to speed up SARS-CoV-2 diagnosis using CDC FDA EUA RT-qPCR kit. Virus Res 290: 198173.
- 9.
Freire-Paspuel B, Vega-Mariño P, Velez A, Castillo P, Masaquiza C, Cedeño-Vega R, Lozada T, Cruz M, Garcia-Bereguiain MA, 2020. “One health” inspired SARS-CoV-2 surveillance: the Galapagos Islands experience. One Health 11: 100185.
- 10.
Freire-Paspuel B, Garcia-Bereguiain MA, 2020. Analytical sensitivity and clinical performance of a triplex RT-qPCR assay using CDC N1, N2 and RP targets for SARS-CoV-2 diagnosis. Int J Infect Dis 102: 14–16.
- 11.↑
Freire-Paspuel B, Vega-Mariño P, Velez A, Cruz M, Perez F, Garcia-Bereguiain MA, 2020. Analytical and clinical comparison of viasure (CerTest Biotec) and 2019-nCoV CDC (IDT) RT-qPCR kits for SARS-CoV2 diagnosis. Virology 553: 154–156.
- 12.↑
Freire-Paspuel B, Garcia-Bereguiain MA, 2020. Poor sensitivity of “AccuPower SARS-CoV-2 real time RT-PCR kit (Bioneer, South Korea)”. Virol J 17: 178.
- 14.↑
Lavezzo E et al. 2020. Suppression of a SARS-CoV-2 outbreak in the Italian municipality of Vo’. Nature 584: 425–429.
- 15.↑
Kleiboeker S, Cowden S, Grantham J, Nutt J, Tyler A, Berg A, Altrich A, 2020. SARS-CoV-2 viral load assessment in respiratory samples. J Clin Virol 129: 104439.
- 17.↑
Freire-Paspuel B, Garcia-Bereguiain MG, 2021. Low clinical performance of “Isopollo COVID19 detection kit” (Monitor, South Korea) for RT-LAMP SARS-CoV-2 diagnosis: a call for action against low quality products for developing countries. Int J Infect Dis 104: 303–305.