INTRODUCTION
Japanese encephalitis (JE) virus is the leading cause of viral neurologic disease and disability in Asia.1,2 Disease manifestations primarily occur in children, leading to either death or long-term neurologic disability in ≈70% of those with clinical illness.3–6 In many countries known to be at risk for JE transmission, the burden of disease is not known. The clinical presentation of JE cannot be accurately differentiated from other etiologies of meningoencephalitis; therefore, laboratory diagnostic confirmation is required for definitive diagnosis.
Because affordable JE diagnostics are generally not available, cases of encephalitis and other severe neurologic illnesses that are truly due to JE virus may not be recognized or reported to public health officials. Limited awareness of disease burden means less chance to control this vaccine-preventable disease. To address this situation, the World Health Organization (WHO) has planned initiatives to support and build laboratory capacity to strengthen JE surveillance.7 Identification of one or more reliable, simple, and available diagnostic tests would facilitate this effort.
Although neutralizing antibody assays of paired serum samples are considered the gold standard for JE diagnostics, the capacity to perform such assays is rarely present in many affected nations. Instead, the simpler immunoglobulin M (IgM) antibody capture enzyme-linked immunosorbent assay (MAC ELISA) for cerebrospinal fluid (CSF) and serum has become the practical standard for the diagnosis of JE.2,8–10 “In-house” JE ELISA diagnostics are available at many research facilities; however, the performance of in-house tests in less sophisticated field laboratories has generally proven unsatisfactory (Nisalak A, unpublished data).
The ELISA format for confirmatory diagnosis of JE infection in Asia is attractive for several reasons. In the context of the goal of measles elimination, WHO has established a Measles Laboratory Network to strengthen laboratory capacity for measles testing.11 Because ELISAs are also widely used for measles diagnosis, the equipment and trained personnel to perform ELISAs already exist in many places and JE testing can be easily integrated. The time required to complete testing is relatively short, particularly with these newly developed commercial kits (which is a particular advantage over some in-house kits).
Although not commercially available at the time of this study, three standardized JE MAC ELISA diagnostic kits had been developed with the intent for commercial sale: the Japanese Encephalitis–Dengue IgM Combo ELISA Test manufactured by Panbio Limited, the Japanese Encephalitis IgM ELISA manufactured by InBios International, Inc., and the JEV CheX kit manufactured by XCyton Diagnostics Ltd.12–15 All three tests use a cell culture-derived recombinant particulate JE antigen; the Panbio test also uses recombinant dengue 1–4 antigens.
The Department of Virology, United States Army Medical Component–Armed Forces Research Institute of Medical Sciences (USAMC-AFRIMS), a regional reference laboratory for Asia designated as a WHO Collaborating Center, has an in-house JE MAC ELISA. This assay has been widely used across the region for many years and was shown to be very accurate when compared with JE hemagglutination inhibition assay results during the test’s development.16 The goal of this study was to evaluate the sensitivity, specificity, and usability of the three diagnostic kits against this regionally recognized JE MAC ELISA using specimens containing either JE IgM, dengue IgM, or neither. The inclusion of dengue IgM positive specimens was considered necessary because dengue virus co-circulates in many regions of Asia where JE virus is found, and the potential for cross-reactivity between flaviviruses is well recognized.3,4
MATERIALS AND METHODS
Samples.
Serum samples without personal identifiers were selected from the AFRIMS archived samples. The panel of 360 acute-phase samples consisted of 121 JE IgM-positive/ dengue IgM-negative samples, 111 JE IgM-negative/dengue IgM-positive samples, and 128 JE IgM-negative/dengue IgM-negative samples. The samples were originally collected in Nepal in 2004 (JE IgM-positive samples) and Thailand in 2001 (dengue and negative control samples). All samples had been initially characterized using AFRIMS JE and dengue ELISAs and then stored at −20°C after the date of collection. The average age of Thai patients contributing serum samples was 7.3 years (range 0.4–34 years) from 75 males and 103 females (61 unknown). Associated age and sex data were not available for the Nepal samples. Two quality-control technicians coded all samples numerically and maintained the code table matching identical samples. The technicians thawed and divided each archived sample into five aliquots: one for testing with each of the three diagnostic kits, one for the AFRIMS ELISAs, and one to be used in case of human error or deviation from the test instructions. The technicians froze all aliquots.
Test methods.
The AFRIMS in-house JE IgM ELISA was used as the reference standard for this evaluation. The AFRIMS JE IgM ELISA was run according to the standard operating procedures (SOPs) in duplicate.9,16,17 If a pair of AFRIMS test results was not concordant, then the initial results were discarded and the sample testing was repeated in duplicate. The same procedure was undertaken using the AFRIMS dengue ELISA. A second trained AFRIMS technician checked for proper test completion and sample use.
A single AFRIMS technician with > 17 years of experience performed all evaluation testing. This technician was blinded both to the manufacturer of each kit as well as to the classification of each sample. The technician thawed each sample only once prior to testing. AFRIMS stored all kits in the manner specified by the three commercial manufacturers in the package inserts. The AFRIMS technician ran the tests sequentially using each manufacturer’s written SOPs/ dilutions. Samples with equivocal results were retested with the reagents provided when indicated by the SOP. Duplicate testing was not undertaken for the test kits as this would not typically be undertaken when testing is conducted in the field. The technician classified the AFRIMS results as “JE infection,” “negative,” “primary dengue,” or “secondary dengue” and classified the kit results according to the cut-off values provided by the manufacturer as “positive,” “negative,” or “equivocal” for JE (if indicated). The Panbio kit used a two-step testing algorithm that requires both JE and dengue IgM results to diagnose JE.
The AFRIMS laboratory technician also completed a questionnaire on the same day of testing regarding operational aspects of each kit and its ease of use. The protocol was approved by the institutional review board of the Walter Reed Army Institute of Research.
Statistical methods.
For the purpose of this evaluation, the AFRIMS test results were considered in two categories, JE positive and negative. Samples with AFRIMS results of “JE infection” were labeled “positive” for JE infection and the remainder were “negative” for JE infection (i.e., those with results of “negative,” “primary dengue,” and “secondary dengue”). Equivocal results from test kits were coded negative for analysis of agreement, sensitivity, and specificity. Sensitivity was defined as the proportion of the samples with an AFRIMS result of “JE infection positive” that had a positive kit test result. Specificity was defined as the proportion of the samples with an AFRIMS result of “JE infection negative” that had a negative kit test result. Agreement was assessed using Cohen’s Kappa. Data were analyzed using SPSS version 11.5 (SPSS Inc., Chicago, IL), and quadratic 95% confidence intervals were obtained using Epi Info 6.04d (Centers for Disease Control and Prevention, Atlanta, GA).
RESULTS
Kit testing was completed between April and July 2005. The paired results of the duplicate AFRIMS tests were concordant for all 360 samples; no retesting was required.
Quantitative analysis.
Comparing the results of the diagnostic kits from each of three manufacturers to the AFRIMS IgM ELISA results showed the Panbio kit had sensitivity and specificity of 89.3% and 99.2%, respectively (Figure 1 and Table 1); the InBios kit had sensitivity and specificity of 99.2% and 56.1%, respectively; and the XCyton kit had sensitivity and specificity of 96.7% and 65.3%, respectively. The agreement with the AFRIMS IgM ELISA was 90.4%, 45.6%, and 53.4% for the Panbio, InBios, and XCyton kits, respectively.
Test specificity was also calculated using only the JE IgM-negative/dengue IgM-negative samples where cross-reactivity was unlikely. In this group, the Panbio, InBios, and XCyton kits had specificities of 98.4%, 96.1%, and 96.1%, respectively.
Table 2 provides a breakdown of the subclassification of results for each kit compared with the AFRIMS test results for each category of sample tested (i.e., JE infection, negative, primary dengue, and secondary dengue). The Panbio kit correctly identified all specimens that were JE IgM-negative/ dengue IgM-positive as negative for JE infection. The Panbio kit had 10.7% false negatives among the JE IgM-positive/ dengue IgM-negative specimens; of the 13 false negatives, using Panbio criteria, 6 were negative and 7 were equivocal. The InBios kit had 3.9% false positives among the JE IgM-negative/dengue IgM-negative samples; however, an additional 22.7% of these samples were classified as equivocal (and considered negative in this analysis). The InBios kit did not correctly identify any of the 111 JE IgM-negative/dengue IgM-positive samples as JE negative. The XCyton kit correctly identified 96.1% of JE IgM-negative/dengue IgM-negative samples as JE negative. However, the XCyton kit incorrectly identified 70.3% of the JE IgM-negative/dengue IgM-positive samples as JE positive.
Qualitative analysis.
Running ≈100–200 samples required 3–3.5 hours of “hands-on” time and 5–8 hours in total to complete testing (Table 3). The technician estimated that for 20 samples it would require 1–1.5 hours and 4–6 hours of “hands-on” and total time, respectively. The AFRIMS IgM ELISA needs to be incubated overnight, therefore requiring ≈ 20 hours from start to finish.
The AFRIMS technician highlighted advantages and disadvantages of each kit in relation to several factors, including the number of steps involved, the time to complete the procedure, and the temperature requirements for storage of kit components (see a summary of qualitative results in Table 3). Ultimately, however, the technician did not prefer any one kit to the others. The technician also described issues that could affect ease of use in a peripheral laboratory. Each kit required components or procedures that, although easy to obtain or perform in a reference laboratory setting, might be challenging to obtain or conduct in a peripheral laboratory.
DISCUSSION
In January 2006, new WHO JE surveillance standards recommended wide use of JE diagnostics to confirm the etiology of acute encephalitis syndrome (AES) cases.18 This has been hindered by lack of standardized reagents. This evaluation provided an opportunity to assess three new ELISA JE IgM diagnostic kits. Wider commercial availability of these tests may allow greater use in settings that need it most.
Use of diagnostics should be evaluated according to the desired endpoint of testing. If data are to be used for JE surveillance, then sensitivity and specificity need to be balanced with the needs of the system. Different diseases have differing diagnostic needs and health-system responses. Whereas sensitivity is important for viral infections when specific antiviral treatment options exist, such as herpes simplex, specificity may be more important where specific public health responses (e.g., mass vaccination) are needed to control disease, such as yellow fever. If the test is being used in a research setting then sensitivity and specificity can be pursued with additional testing in more sophisticated and expensive laboratory settings. In JE surveillance that is used for public health planning, improved sensitivity and specificity can be obtained by controlling the patient population that is tested (i.e., testing only those with AES), balanced with quality assurance and quality control mechanisms designed to address the needs of the system.
In this evaluation, a panel of well-characterized sera was used to assess the performance of each diagnostic kit. Serum samples, rather than CSF, were chosen for two reasons. First, laboratory confirmation for a case of JE in the WHO JE surveillance standards includes “presence of JE virus-specific IgM antibody in a single sample of CSF or serum, as detected by an IgM-capture ELISA specifically for JE virus.” Second, although CSF is the preferred specimen for diagnosis, in some JE-endemic countries, often only a serum specimen is collected for diagnosis. For these reasons, assessment of kit performance with serum was considered important, as issues of cross-reactivity are likely to be more of a problem than in CSF. Currently, labeling on the Panbio and InBios kits allows only for use with sera; however, correspondence with both companies indicated that this was because CSF samples were not readily available for validating and optimizing performance.
The panel chosen for this evaluation included samples from patients with acute dengue virus infection. The problem of cross-reactivity when testing for antibodies to flaviviruses is a well-recognized phenomenon.3,4 In most JE endemic areas, co-circulation of dengue and other flaviviruses frequently occurs. The Panbio kit has an advantage when used in settings where dengue co-circulates. This was the only one of the three JE kits with at least 90% agreement to the AFRIMS standard when both JE and dengue IgM-positive samples were considered in the analysis. In this situation, sensitivity was good for the InBios and XCyton kits but lower for the Panbio kit. However, specificity was low for both the InBios and XCyton kits as a result of cross-reactivity with dengue antibodies.
The relative importance of dengue cross-reactivity is directly related to the application of the test in real clinical settings. If the test is used exclusively in patients presenting with an acute encephalitis syndrome, then issues with dengue cross-reactivity are less relevant. Although dengue infections occasionally cause encephalopathy, such infections are not considered to cause encephalitis. Therefore, when testing samples from encephalitic patients, dengue is not likely to be the pathogen, so the issue of dengue cross-reactivity is of less importance. There is a chance that if dengue virus was circulating in a JE-endemic area, serum samples only were collected, and the InBios or XCyton kits were used for diagnosis of an encephalitic patient, then a non-JE patient could be misclassified as JE. For example, if a child were to present with non-JE encephalitis but with a concurrent (possibly asymptomatic) or recent dengue infection, this child could be misclassified as JE with these two kits. However, this is not likely to be a frequent occurrence. It is likely that the positive predictive value (PPV) with any of the three kits is high when used in diagnosing persons with an acute encephalitis syndrome. All three kits showed specificity of 96% or above when dengue cross-reactivity was not a factor. This issue will be further evaluated in field trials of the kits.
For both the InBios and XCyton kits, altering the cut-off values would, of course, improve specificity, with an associated reduction in sensitivity. However, it was apparent that the problem of low specificity could not be overcome sufficiently in sample pools that include JE IgM-negative/dengue IgM-positive samples. A two-step testing algorithm in which dengue testing is performed, as the Panbio kit does, may be the only way to achieve both adequate sensitivity and specificity in a population where dengue-like illnesses (i.e., febrile illnesses) are tested for JE and dengue cross-reactivity is an issue.
More than one encephalitic flavivirus may be present at the same time as JE virus—for example, West Nile virus or Murray Valley encephalitis virus. These flaviviruses are closely related to JE virus and would likely cause false positives with any of the kits identified. This was not tested in this evaluation but may be investigated in the future. Differentiating between flaviviral infections is important, and until comprehensive and simple diagnostics are available for all flaviviruses, use of epidemiologic data and referral laboratories for quality assurance and quality control will be needed.
In considering use of diagnostic kits, sensitivity, specificity, ease of use, affordability, and availability are all important factors. Price could not be considered in this assessment as the kits were not available in the marketplace at the time of the assessment. Fortunately, equipment already exists for conducting ELISAs, and ELISA methodology is relatively simple, which will improve affordability and feasibility. However, cost will be an important factor to be considered as companies establish their public-sector pricing.
This study confirms the appropriateness of at least one commercial kit for use in any JE-endemic setting. This is an important step in meeting the demand for diagnostics from public health surveillance programs and in determining the real burden of JE in Asia. Forthcoming field evaluations will assess PPV in a patient population with encephalitis in an area known to have both JE and dengue fever, as well as assess usability in peripheral hospital laboratories. These evaluations will determine if more than one kit is appropriate for widespread use and will provide additional information on the accuracy and usability of these kits in a field setting.
Positive and negative results for three Japanese encephalitis (JE) diagnostic kits compared to AFRIMS IgM ELISA
| AFRIMS JE IgM ELISA | |||
|---|---|---|---|
| Test | + | - | |
| Panbio Ltd. (8 equivocals coded negative) | + | 108 | 2 |
| − | 13 | 237 | |
| InBios International, Inc. (41 equivocals coded negative) | + | 120 | 105 |
| − | 1 | 134 | |
| XCyton Diagnostics Ltd. | + | 117 | 83 |
| − | 4 | 156 | |
Complete subclassification of results for three Japanese Encephalitis diagnostic kits compared to AFRIMS test results
| Test | JE virus infection | Negative | Primary Dengue | Secondary Dengue | Total | |
|---|---|---|---|---|---|---|
| Panbio Ltd. | + | 108 | 2 | 0 | 0 | 110 |
| − | 6 | 125 | 30 | 81 | 242 | |
| +/− | 7 | 1 | 0 | 0 | 8 | |
| InBios International, Inc. | + | 120 | 5 | 22 | 78 | 225 |
| − | 0 | 94 | 0 | 0 | 94 | |
| +/− | 1 | 29 | 8 | 3 | 41 | |
| XCyton Diagnostics Ltd. | + | 117 | 5 | 10 | 68 | 200 |
| − | 4 | 123 | 20 | 13 | 160 |
Selected qualitative findings by AFRIMS technician regarding operational features and ease of use of three Japanese encephalitis diagnostic kits
| Panbio Ltd. | InBios International, Inc. | XCyton Diagnostics Ltd. | |
|---|---|---|---|
| Volume of sample required | 10 μL sera | 4 μL sera | 5 μL sera or 10 μL CSF |
| Maximum number of samples able to be run on a 96-well plate (not run in duplicate) | 43 | 44 | 88 |
| Actual time to complete assay for study (≈ 100–200 samples) | 8 hours (3 hours hands-on) | 8 hours (3.5 hours hands-on) | 5 hours (3 hours hands-on) |
| Estimated time to complete an assay with 20 samples | 6 hours (1 hour hands-on) | 6 hours (1.5 hours hands-on) | 4 hours (1.3 hours hands-on) |
| Temperature requirements for storage | 2–8°C | 2–8°C for most kit components, −70°C for antigens | 2–8°C |
| Advantages of kit reported by technician | Small number of steps involved. | Ready-to-use reagents. | Short time period required to run assay. Steps of procedure easy. |
| Challenges of kit reported by technician | Reagent preparation complicated. Care needed to add JE and dengue antigens to correct wells. | Procedure for addition of samples to wells somewhat complicated. Care needed to add the JE recombinant and normal cell antigens to the correct wells. | The brevity of the incubation periods during the conjugate step means it is difficult to perform testing with more than one plate at a time. Technician adapted to a weak optical density reading for the weak positive control in the prerun by keeping the plate in a prewarmed box during the conjugate step and preparing the substrate 15 minutes in advance. |
| Suggestions to improve field use of kit | Provide reagents ready to use. Change incubation temperature so it will be easy to use in a peripheral laboratory (currently requires incubation at 37°C). | Avoid need to store kit components at −70°C. Change incubation temperature so it will be easy to use in a peripheral laboratory (currently requires incubation at 37°C). | Change incubation temperature so it will be easy to use in a peripheral laboratory (currently requires incubation at 37°C). |
Summary of percentages of agreement, sensitivity, and specificity for three Japanese encephalitis diagnostic kits compared with AFRIMS IgM ELISA.
Citation: The American Journal of Tropical Medicine and Hygiene 77, 1; 10.4269/ajtmh.2007.77.164
Summary of percentages of agreement, sensitivity, and specificity for three Japanese encephalitis diagnostic kits compared with AFRIMS IgM ELISA.
Citation: The American Journal of Tropical Medicine and Hygiene 77, 1; 10.4269/ajtmh.2007.77.164
Summary of percentages of agreement, sensitivity, and specificity for three Japanese encephalitis diagnostic kits compared with AFRIMS IgM ELISA.
Citation: The American Journal of Tropical Medicine and Hygiene 77, 1; 10.4269/ajtmh.2007.77.164
Address correspondence to Julie A. Jacobson, Japanese Encephalitis Project, PATH, 1455 NW Leary Way, Seattle, Washington 98107. E-mail: jjacobs@path.org
Authors’ addresses: Julie A. Jacobson, Susan L. Hills, and Jennifer L. Winkler, Japanese Encephalitis Project, PATH, 1455 NW Leary Way, Seattle, WA 98107, Telephone: +1 (206) 285-3500, Fax: +1 (206) 285-6619, E-mails: jjacobs@path.org, shills@path.org, and jwinkler@path.org. Mammen Mammen, Butsaya Thaisomboonsuk, and Robert V. Gibbons, Department of Virology, United States Army Medical Component–Armed Forces Research Institute of Medical Sciences, 315/6 Rajvithi Road, Bangkok 10400, Thailand, Telephone: +662 644 5644, Fax: +662-644-4760, E-mails: mammen.mammen@us.army.mil, ButsayaT@afrims.org, and Robert.Gibbons@afrims.org. Anthony A. Marfin, Division of Global Migration and Quarantine, Centers for Disease Control and Prevention, 2201 Sixth Avenue, Seattle, WA 98121, Telephone: +1 (206) 615-3600, Fax: +1 (206) 615-2481, E-mail: aam0@cdc.gov,
Acknowledgments: The authors thank Khun Nawarat Charoensri and Khun Panor Srisongkram, USAMC-AFRIMS, for their performance of the tests; Dr. Ananda Nisalak of USAMC-AFRIMS for technical advice; and Dr. Martha Roper for her contributions to the analyses of the results.
Financial support: This work was funded by the Bill & Melinda Gates Foundation.
REFERENCES
- 1↑
Vaughn DW, Hoke CH, 1992. The epidemiology of Japanese encephalitis: prospects for prevention. Epidemiol Rev 14 :197–221.
- 2↑
Halstead SB, Tsai TF, 2004. Japanese encephalitis vaccines. Plotkin SA, Orenstein WA, eds. Vaccines. Fourth edition. Philadelphia: WB Saunders, 919–958.
- 3↑
Tsai TF, 2000. New initiatives for the control of Japanese encephalitis by vaccination. Minutes of a WHO/CVI meeting, October 13–15, 1998, Bangkok, Thailand. Vaccine 18 (Suppl 2):1–25.
- 4↑
Solomon T, Dung NM, Kneen R, Gainsborough M, Vaughn DW, Khanh VT, 2000. Japanese encephalitis. J Neurol Neurosurg Psychiatry 68 :405–415.
- 5
Kalayanarooj S, 1995. Japanese encephalitis: Clinical manifestations, outcome and management. Southeast Asian J Trop Med Public Health 26 (Suppl 3):9.
- 6↑
Misra UK, Kalita J, 2002. Prognosis of Japanese encephalitis patients with dystonia compared to those with parkinsonian features only. Postgrad Med J 78 :238–241.
- 7↑
World Health Organization, Regional Office for South-East Asia, 2006. Vaccine Preventable Disease Surveillance Bulletin. New Delhi, India: WHO, 10(40).
- 9↑
Burke DS, Nisalak A, Ussery MA, Laorakpongse T, Chantavibul S, 1985. Kinetics of IgM and IgG responses to Japanese encephalitis virus in human serum and cerebrospinal fluid. J Infect Dis 151 :1093–1099.
- 10↑
Cuzzubbo AJ, Endy TP, Vaughn DW, Solomon T, Nisalak A, Kalayanarooj S, Dung NM, Warrilow D, Aaskov J, Devine PL, 1999. Evaluation of a new commercially available immunoglobulin M capture enzyme-linked immunosorbent assay for diagnosis of Japanese encephalitis infections. J Clin Microbiol 37 :3738–3741.
- 11↑
World Health Organization, 2005. Global measles and rubella laboratory network update. Wkly Epidemiol Rec 80 :384–388.
- 12↑
E-JED01C–Japanese encephalitis–Dengue IgM Combo website. Available at: http://www.panbio.com/modules.php?name=catalog&file=product_info&cPath=21&products_id=627. Accessed December 18, 2006.
- 13
JE DETECT IgM ELISA website. Available at: http://www.inbios.com/products.php?code=elisaJEM. Accessed December 18, 2006.
- 14
Hunt AR, Cropp CB, Chang GJ, 2001. A recombinant particulate antigen of Japanese encephalitis virus produced in stably-transformed cells is an effective noninfectious antigen and subunit immunogen. J Virol Methods 97 :133–149.
- 15↑
Ravi V, Desai A, Balaji M, Apte MP, Lakshman L, Subbakrishna DK, Sridharan G, Dhole TN, Ravikumar BV, 2006. Development and evaluation of a rapid IgM capture ELISA (JEV-Chex) for the diagnosis of Japanese encephalitis. J Clin Virol 35 :429–434.
- 16↑
Burke DS, Nisalak A, 1982. Detection of Japanese encephalitis virus immunoglobulin M antibodies in serum by antibody capture radioimmunoassay. J Clin Microbiol 15 :353–361.
- 17↑
Burke DS, Nisalak A, Ussery MA, 1982. Antibody capture immunoassay detection of Japanese encephalitis virus immunoglobulin M and G antibodies in cerebrospinal fluid. J Clin Microbiol 16 :1034–1042.
- 18↑
World Health Organization, 2006. WHO-Recommended Standards for Surveillance of Selected Vaccine-Preventable Diseases. WHO/V&B/03.01. Geneva, Switzerland: World Health Organization.