• 1.

    Wouters OJ, Shadlen KC, Salcher-Konrad M, Pollard AJ, Larson HJ, Teerawattananon Y, Jit M, 2021. Challenges in ensuring global access to COVID-19 vaccines: production, affordability, allocation, and deployment. Lancet 397: 10231034.

    • Search Google Scholar
    • Export Citation
  • 2.

    Brat GA et al., 2020. International electronic health record-derived COVID-19 clinical course profiles: the 4CE Consortium. npj Digit Med 3: 109.

    • Search Google Scholar
    • Export Citation
  • 3.

    Madhi SA et al., 2021. Efficacy of the ChAdOx1 nCoV-19 COVID-19 vaccine against the B. 1.351 variant. N Engl J Med 384: 18851898.

  • 4.

    Shrotri M, Swinnen T, Kampmann B, Parker EP, 2021. An interactive website tracking COVID-19 vaccine development. Lancet Glob Health 9: e590e592.

    • Search Google Scholar
    • Export Citation
  • 5.

    Voysey M et al., 2021. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet 397: 99111.

    • Search Google Scholar
    • Export Citation
  • 6.

    Dagan N, Barda N, Kepten E, Miron O, Perchik S, Katz MA, Hernán MA, Lipsitch M, Reis B, Balicer RD, 2021. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 384: 14121423.

    • Search Google Scholar
    • Export Citation
  • 7.

    Hernán MA, Robins JM, 2016. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol 183: 758764.

    • Search Google Scholar
    • Export Citation
  • 8.

    Westreich D, Hudgens MG, 2016. Invited commentary: beware the test-negative design. Am J Epidemiol 184: 354356.

  • 9.

    Griffith GJ, Morris TT, Tudball MJ, Herbert A, Mancano G, Pike L, Sharp GC, Sterne J, Palmer TM, Smith GD, 2020. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat Commun 11: 112.

    • Search Google Scholar
    • Export Citation
  • 10.

    Fell DB, Dimitris MC, Hutcheon JA, Ortiz JR, Platt RW, Regan AK, Savitz DA, 2021. Guidance for design and analysis of observational studies of fetal and newborn outcomes following COVID-19 vaccination during pregnancy. Vaccine 39: 18821886.

    • Search Google Scholar
    • Export Citation
  • 11.

    Fewell Z, Hernán MA, Wolfe F, Tilling K, Choi H, Sterne JA, 2004. Controlling for time-dependent confounding using marginal structural models. Stata J 4: 402420.

    • Search Google Scholar
    • Export Citation
  • 12.

    Danaei G, Rodriguez LA, Cantero OF, Logan R, Hernan MA, 2013. Observational data for comparative effectiveness research: an emulation of randomised trials of statins and primary prevention of coronary heart disease. Stat Methods Med Res 22: 7096.

    • Search Google Scholar
    • Export Citation
  • 13.

    Mahase E, 2021. COVID-19: Israel sees new infections plummet following vaccinations BMJ 372: n338. doi: .

  • 14.

    Akanbi MO, Ocheke AN, Agaba PA, Daniyam CA, Agaba EI, Okeke EN, Ukoli CO, 2012. Use of electronic health records in sub-Saharan Africa: progress and challenges. J Med Trop 14: 16.

    • Search Google Scholar
    • Export Citation
  • 15.

    Salyer SJ et al., 2021. The first and second waves of the COVID-19 pandemic in Africa: a cross-sectional study. Lancet 397: 12651275.

  • 16.

    Makoni M, 2020. COVID-19 vaccine trials in Africa. Lancet Respir Med 8: e79e80.

 
 
 

 

 
 
 

 

 

 

 

 

 

The Need for a Practical Approach to Evaluate the Effectiveness of COVID-19 Vaccines for Low- and Middle-Income Countries

View More View Less
  • 1 Rwanda Biomedical Centre, Kigali, Rwanda;
  • | 2 Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada;
  • | 3 The International COVID-19 Data Alliance, Health Data Research UK, London, United Kingdom;
  • | 4 Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada;
  • | 5 Department of Experimental Medicine, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada

ABSTRACT.

The global demand for coronavirus disease 2019 (COVID-19) vaccines currently far outweighs the available global supply and manufacturing capacity. As a result, securing doses of vaccines for low- and middle-income countries has been challenging, particularly for African countries. Clinical trial investigation for COVID-19 vaccines has been rare in Africa, with the only randomized clinical trials (RCTs) for COVID-19 vaccines having been conducted in South Africa. In addition to addressing the current inequities in the vaccine roll-out for low- and middle-income countries, there is a need to monitor the real-world effectiveness of COVID-19 vaccines in these regions. Although RCTs are the superior method for evaluating vaccine efficacy, the feasibility of conducting RCTs to monitor COVID-19 vaccine effectiveness during mass vaccine campaigns will likely be low. There is still a need to evaluate the effectiveness of mass COVID-19 vaccine distribution in a practical manner. We discuss how target trial emulation, the application of trial design principles from RCTs to the analysis of observational data, can be used as a practical, cost-effective way to evaluate real-world effectiveness for COVID-19 vaccines. There are several study design considerations that need to be made in the analyses of observational data, such as uncontrolled confounders and selection biases. Target trial emulation accounts for these considerations to improve the analyses of observational data. The framework of target trial emulation provides a practical way to monitor the effectiveness of mass vaccine campaigns for COVID-19 using observational data.

Equitable access to coronavirus disease 2019 (COVID-19) vaccines will be vital for controlling the COVID-19 pandemic. Currently, with the global demand for COVID-19 vaccines far outweighing the available global supply and manufacturing capacity, securing doses of vaccines for African countries and other low- and middle-income countries (LMICs) has been challenging.1 The COVID-19 Vaccines Global Access (COVAX) facility as a multilateral initiative has been leading the procurement and equitable distribution of COVID-19 vaccines for lower income countries, but the COVAX supply to date has been far below the need in LMICs. As of June 10, 2021, COVAX has delivered more than 81 million vaccine doses to 129 eligible countries globally, including 26 eligible countries in Africa.2

In Africa, where 16% of the global population resides, clinical trial investigation for COVID-19 vaccines has been limited. In fact, COVID-19 vaccine trials have only been conducted in South Africa.3,4 This is potentially concerning because geographic factors such as different prevalent variants may limit COVID-19 vaccine effectiveness. For instance, although a pooled interim analysis of the ChAdOx1 nCoV-19 COVID-19 vaccine (the Oxford-AstraZeneca vaccine) based on 11,636 participants from the United Kingdom and Brazil showed an overall efficacy of 70.4% (95% confidence interval, 54.8–80.6%) for the outcome of polymerase chain reaction-confirmed symptomatic COVID-19 more than 14 days after the full dose,5 the same vaccine did not show protective effects in South Africa, presumably as a result of the predominant circulation of the B.1.351 variant.3

With continued mass vaccine campaigns, monitoring the real-world effectiveness of COVID-19 vaccines in regions where the original trials were not conducted will be critical. Although randomized clinical trials (RCTs) are the superior method for evaluating vaccine efficacy, the feasibility of conducting RCTs to monitor COVID-19 vaccine effectiveness during mass vaccine campaigns will likely be low. There is still a need to monitor the effectiveness of mass COVID-19 vaccine distribution in a practical manner.

When RCTs are not feasible, it is possible to emulate a randomized trial to estimate the causal effect of the COVID-19 vaccination on patient health outcomes using observational data.6,7 Target trial emulation is the application of trial design principles from RCTs to the analysis of observational data. This framework involves coming up with a protocol for a hypothetical RCT (target trial) that provides the information of interest, including specification of eligibility criteria, treatment strategies, assignment procedure, study follow-up period, outcomes, estimands (e.g., intention-to-treat or per-protocol effects), and statistical analysis plan to estimate the causal contrasts of interest.7 Then, based on the target trial protocol, observational data are analyzed accordingly using appropriate statistical methods in an attempt to emulate the target trial.7 Similar to protocols of RCTs, target trial protocols can be made available online and published to ensure transparency.

Using observational data to estimate vaccine effectiveness has its challenges. Because most countries have adopted risk tiers based vaccination distribution plans, confounding is an important issue. To adjust for confounding, one can identify all important prognostic variables such as age, gender, geographic location, and comorbidities associated with vaccination.6 Randomization can then be emulated by applying a suitable statistical method to adjust for confounders. Eligibility criteria of a target trial may need to be restricted further to reduce the risk of information bias and confounding.6 For example, a test-negative design could be used, in which only individuals presenting COVID-19 symptoms to primary care are eligible, to reduce confounding by health-care-seeking behavior and treatment misclassification, with the appropriate sensitivity analyses to address selection bias and lack of generalizability.8 Although risk of bias is always an issue for any non-randomized study, the target trial emulation framework provides comprehensive methodological evaluation of observational data.

In its most basic form, target trial emulation provides a rigorous and theoretical foundation for observational analyses aimed at identifying causal effects of health-care interventions. The specification of the target trial protocol for practical application to an observational database requires consultation with domain experts. In the case of target emulation for mass vaccine campaign evaluation, this would likely be LMIC-based clinicians and epidemiologists familiar with the local health-care setting. These experts can identify important confounders to identify comparable patient cohorts that offer generalizability to the broader population and relevance to public health. The explicit definition of a causal diagram during the study design phase helps identify confounders (non-colliders) that should be controlled and colliders that should be left uncontrolled. This, in turn, helps prevent collider bias (a distortion that modifies an association between an exposure and an outcome arising from attempts to control for a common effect of the exposure and outcome), which is often a concern with test-negative designs.9

Because most licensed COVID-19 vaccines typically require two doses, there may be a need to compare risks in partially vaccinated individuals. For this, it may be important to consider the time-varying effect of vaccination in LMICs.10 A large body of literature on non-random loss to follow-up and time-varying confounding has been developed in the context of target trial emulation.11 Some of these techniques for trial emulation can improve precision of estimates by considering multiple instances of eligibility per individual.6,12

To our best knowledge, target trial emulation for COVID-19 vaccine effectiveness has only been applied to the mass distribution of the BNT162b2 messenger RNA vaccine (the Pfizer-BioNTech vaccine) in Israel.6 Since launching their mass vaccination campaign December 20, 2020, Israel has seen its cases and fatalities decrease in association with the mass distribution of the Pfizer-BioNTech vaccine.13 Although the lessons from Israel are highly impactful, their applicability elsewhere is an ongoing source of debate. The target trial emulation framework can be applicable to wide range of settings, but it is important to note how different LMICs have varied enormously with regard to their vaccination coverage, pace of vaccination, vaccine types, baseline COVID-19 incidence, and potential variants of concern. Particularly in Africa, there are many unknowns about the epidemiology and risk factors for COVID-19, and how they differ from those in high-income regions, where the majority of COVID-19 vaccine clinical trial research has taken place so far. Diseases such as HIV infection, tuberculosis, and malaria are more prevalent in Africa than high-income countries. However, the effects of these infections and diseases on the immune response to COVID-19 vaccines are not well understood. In addition to these factors, differing vaccination schedules and vaccine-handling logistics, and health systems across Africa could influence the real-world effectiveness of COVID-19 vaccines.

It is also important to point to the requirements for high-quality data in target trial emulation. A key to the success of the Israeli study was access to well-organized digital health records. Organized data in this format is essential for the purposes of transformation and matching methodologies to ensure appropriate confounding is accounted for. Digital health records have variable coverage between African countries, although in certain settings coverage is high.14 Many of these countries use a standardized health record platform (OpenMRS), which may improve cross-collaborative analyses and data flows. Where data are non-digitized or missing, alternative data capture can be deployed quickly, as evidenced by existing rapid cross-sectional analyses coordinated via the African Centers for Disease Control and Prevention, capturing more than 2.5 million COVID-19 cases and more than 65,000 deaths.15

When RCTs are feasible, they are preferable over target trial emulation. However, because clinical trial evaluation of the vaccine campaign is likely infeasible in LMICs, target trial emulation can provide an effective and practical way to enable causal analyses of real-world data sets. Normally in the analyses of observational data sets, several measures need to be made to minimize selection biases and control for confounders. Target trial emulation outlines these measures in a target trial protocol to improve the analyses of observational data. Because there has been a systematic under-representation of randomized, controlled COVID-19 vaccine trials across the African subcontinent,16 there is a critical need to support these countries in performing evaluation of the real-world effectiveness for COVID-19 vaccines in these countries. Given that RCTs for the evaluation of mass vaccine campaigns for COVID-19 are likely not feasible, target trial emulation can provide an effective and practical way to enable causal analyses of real-world data.

ACKNOWLEDGMENTS

The American Society of Tropical Medicine and Hygiene has waived the Open Access fee for this article due to the ongoing COVID-19 pandemic.

References

  • 1.

    Wouters OJ, Shadlen KC, Salcher-Konrad M, Pollard AJ, Larson HJ, Teerawattananon Y, Jit M, 2021. Challenges in ensuring global access to COVID-19 vaccines: production, affordability, allocation, and deployment. Lancet 397: 10231034.

    • Search Google Scholar
    • Export Citation
  • 2.

    Brat GA et al., 2020. International electronic health record-derived COVID-19 clinical course profiles: the 4CE Consortium. npj Digit Med 3: 109.

    • Search Google Scholar
    • Export Citation
  • 3.

    Madhi SA et al., 2021. Efficacy of the ChAdOx1 nCoV-19 COVID-19 vaccine against the B. 1.351 variant. N Engl J Med 384: 18851898.

  • 4.

    Shrotri M, Swinnen T, Kampmann B, Parker EP, 2021. An interactive website tracking COVID-19 vaccine development. Lancet Glob Health 9: e590e592.

    • Search Google Scholar
    • Export Citation
  • 5.

    Voysey M et al., 2021. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet 397: 99111.

    • Search Google Scholar
    • Export Citation
  • 6.

    Dagan N, Barda N, Kepten E, Miron O, Perchik S, Katz MA, Hernán MA, Lipsitch M, Reis B, Balicer RD, 2021. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 384: 14121423.

    • Search Google Scholar
    • Export Citation
  • 7.

    Hernán MA, Robins JM, 2016. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol 183: 758764.

    • Search Google Scholar
    • Export Citation
  • 8.

    Westreich D, Hudgens MG, 2016. Invited commentary: beware the test-negative design. Am J Epidemiol 184: 354356.

  • 9.

    Griffith GJ, Morris TT, Tudball MJ, Herbert A, Mancano G, Pike L, Sharp GC, Sterne J, Palmer TM, Smith GD, 2020. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat Commun 11: 112.

    • Search Google Scholar
    • Export Citation
  • 10.

    Fell DB, Dimitris MC, Hutcheon JA, Ortiz JR, Platt RW, Regan AK, Savitz DA, 2021. Guidance for design and analysis of observational studies of fetal and newborn outcomes following COVID-19 vaccination during pregnancy. Vaccine 39: 18821886.

    • Search Google Scholar
    • Export Citation
  • 11.

    Fewell Z, Hernán MA, Wolfe F, Tilling K, Choi H, Sterne JA, 2004. Controlling for time-dependent confounding using marginal structural models. Stata J 4: 402420.

    • Search Google Scholar
    • Export Citation
  • 12.

    Danaei G, Rodriguez LA, Cantero OF, Logan R, Hernan MA, 2013. Observational data for comparative effectiveness research: an emulation of randomised trials of statins and primary prevention of coronary heart disease. Stat Methods Med Res 22: 7096.

    • Search Google Scholar
    • Export Citation
  • 13.

    Mahase E, 2021. COVID-19: Israel sees new infections plummet following vaccinations BMJ 372: n338. doi: .

  • 14.

    Akanbi MO, Ocheke AN, Agaba PA, Daniyam CA, Agaba EI, Okeke EN, Ukoli CO, 2012. Use of electronic health records in sub-Saharan Africa: progress and challenges. J Med Trop 14: 16.

    • Search Google Scholar
    • Export Citation
  • 15.

    Salyer SJ et al., 2021. The first and second waves of the COVID-19 pandemic in Africa: a cross-sectional study. Lancet 397: 12651275.

  • 16.

    Makoni M, 2020. COVID-19 vaccine trials in Africa. Lancet Respir Med 8: e79e80.

Author Notes

Address correspondence to Jay J. H. Park, VGH Research Pavilion 575-828, West 10th Ave., Vancouver, British Columbia V5Z 1L8, Canada. E-mail: jayhpark1@alumni.ubc.ca

Authors’ addresses: Sabin Nsanzimana and Jean Paul Uwizihiwe, Rwanda Biomedical Centre, Kigali, Rwanda, E-mails: nsabinco@gmail.com and uwizihiwejp@gmail.com. Alind Gupta and Paul Arora, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada, E-mails: alind.gupta@mail.utoronto.ca and paul.arora@utoronto.ca. Jonas Haggstrom, The International COVID-19 Data Alliance, Health Data Research UK, London, UK, E-mail: haggstrom.jonas@gmail.com. Louis Dron, Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada, E-mail: louisdron@gmail.com. Jay J. H. Park, Department of Experimental Medicine, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada, E-mail: jayhpark1@gmail.com.

Save