The Need for a Practical Approach to Evaluate the Effectiveness of COVID-19 Vaccines for Low- and Middle-Income Countries

Equitable access to coronavirus disease 2019 (COVID-19) vaccines will be vital for controlling the COVID-19 pandemic. Currently, with the global demand for COVID-19 vaccines far outweighing the available global supply and manufacturing capacity, securing doses of vaccines for African countries and other low- and middle-income countries (LMICs) has been challenging.¹ The COVID-19 Vaccines Global Access (COVAX) facility as a multilateral initiative has been leading the procurement and equitable distribution of COVID-19 vaccines for lower income countries, but the COVAX supply to date has been far below the need in LMICs. As of June 10, 2021, COVAX has delivered more than 81 million vaccine doses to 129 eligible countries globally, including 26 eligible countries in Africa.²

In Africa, where 16% of the global population resides, clinical trial investigation for COVID-19 vaccines has been limited. In fact, COVID-19 vaccine trials have only been conducted in South Africa.^3,4 This is potentially concerning because geographic factors such as different prevalent variants may limit COVID-19 vaccine effectiveness. For instance, although a pooled interim analysis of the ChAdOx1 nCoV-19 COVID-19 vaccine (the Oxford-AstraZeneca vaccine) based on 11,636 participants from the United Kingdom and Brazil showed an overall efficacy of 70.4% (95% confidence interval, 54.8–80.6%) for the outcome of polymerase chain reaction-confirmed symptomatic COVID-19 more than 14 days after the full dose,⁵ the same vaccine did not show protective effects in South Africa, presumably as a result of the predominant circulation of the B.1.351 variant.³

With continued mass vaccine campaigns, monitoring the real-world effectiveness of COVID-19 vaccines in regions where the original trials were not conducted will be critical. Although randomized clinical trials (RCTs) are the superior method for evaluating vaccine efficacy, the feasibility of conducting RCTs to monitor COVID-19 vaccine effectiveness during mass vaccine campaigns will likely be low. There is still a need to monitor the effectiveness of mass COVID-19 vaccine distribution in a practical manner.

When RCTs are not feasible, it is possible to emulate a randomized trial to estimate the causal effect of the COVID-19 vaccination on patient health outcomes using observational data.^6,7 Target trial emulation is the application of trial design principles from RCTs to the analysis of observational data. This framework involves coming up with a protocol for a hypothetical RCT (target trial) that provides the information of interest, including specification of eligibility criteria, treatment strategies, assignment procedure, study follow-up period, outcomes, estimands (e.g., intention-to-treat or per-protocol effects), and statistical analysis plan to estimate the causal contrasts of interest.⁷ Then, based on the target trial protocol, observational data are analyzed accordingly using appropriate statistical methods in an attempt to emulate the target trial.⁷ Similar to protocols of RCTs, target trial protocols can be made available online and published to ensure transparency.

Using observational data to estimate vaccine effectiveness has its challenges. Because most countries have adopted risk tiers based vaccination distribution plans, confounding is an important issue. To adjust for confounding, one can identify all important prognostic variables such as age, gender, geographic location, and comorbidities associated with vaccination.⁶ Randomization can then be emulated by applying a suitable statistical method to adjust for confounders. Eligibility criteria of a target trial may need to be restricted further to reduce the risk of information bias and confounding.⁶ For example, a test-negative design could be used, in which only individuals presenting COVID-19 symptoms to primary care are eligible, to reduce confounding by health-care-seeking behavior and treatment misclassification, with the appropriate sensitivity analyses to address selection bias and lack of generalizability.⁸ Although risk of bias is always an issue for any non-randomized study, the target trial emulation framework provides comprehensive methodological evaluation of observational data.

In its most basic form, target trial emulation provides a rigorous and theoretical foundation for observational analyses aimed at identifying causal effects of health-care interventions. The specification of the target trial protocol for practical application to an observational database requires consultation with domain experts. In the case of target emulation for mass vaccine campaign evaluation, this would likely be LMIC-based clinicians and epidemiologists familiar with the local health-care setting. These experts can identify important confounders to identify comparable patient cohorts that offer generalizability to the broader population and relevance to public health. The explicit definition of a causal diagram during the study design phase helps identify confounders (non-colliders) that should be controlled and colliders that should be left uncontrolled. This, in turn, helps prevent collider bias (a distortion that modifies an association between an exposure and an outcome arising from attempts to control for a common effect of the exposure and outcome), which is often a concern with test-negative designs.⁹

Because most licensed COVID-19 vaccines typically require two doses, there may be a need to compare risks in partially vaccinated individuals. For this, it may be important to consider the time-varying effect of vaccination in LMICs.¹⁰ A large body of literature on non-random loss to follow-up and time-varying confounding has been developed in the context of target trial emulation.¹¹ Some of these techniques for trial emulation can improve precision of estimates by considering multiple instances of eligibility per individual.^6,12

To our best knowledge, target trial emulation for COVID-19 vaccine effectiveness has only been applied to the mass distribution of the BNT162b2 messenger RNA vaccine (the Pfizer-BioNTech vaccine) in Israel.⁶ Since launching their mass vaccination campaign December 20, 2020, Israel has seen its cases and fatalities decrease in association with the mass distribution of the Pfizer-BioNTech vaccine.¹³ Although the lessons from Israel are highly impactful, their applicability elsewhere is an ongoing source of debate. The target trial emulation framework can be applicable to wide range of settings, but it is important to note how different LMICs have varied enormously with regard to their vaccination coverage, pace of vaccination, vaccine types, baseline COVID-19 incidence, and potential variants of concern. Particularly in Africa, there are many unknowns about the epidemiology and risk factors for COVID-19, and how they differ from those in high-income regions, where the majority of COVID-19 vaccine clinical trial research has taken place so far. Diseases such as HIV infection, tuberculosis, and malaria are more prevalent in Africa than high-income countries. However, the effects of these infections and diseases on the immune response to COVID-19 vaccines are not well understood. In addition to these factors, differing vaccination schedules and vaccine-handling logistics, and health systems across Africa could influence the real-world effectiveness of COVID-19 vaccines.

It is also important to point to the requirements for high-quality data in target trial emulation. A key to the success of the Israeli study was access to well-organized digital health records. Organized data in this format is essential for the purposes of transformation and matching methodologies to ensure appropriate confounding is accounted for. Digital health records have variable coverage between African countries, although in certain settings coverage is high.¹⁴ Many of these countries use a standardized health record platform (OpenMRS), which may improve cross-collaborative analyses and data flows. Where data are non-digitized or missing, alternative data capture can be deployed quickly, as evidenced by existing rapid cross-sectional analyses coordinated via the African Centers for Disease Control and Prevention, capturing more than 2.5 million COVID-19 cases and more than 65,000 deaths.¹⁵

When RCTs are feasible, they are preferable over target trial emulation. However, because clinical trial evaluation of the vaccine campaign is likely infeasible in LMICs, target trial emulation can provide an effective and practical way to enable causal analyses of real-world data sets. Normally in the analyses of observational data sets, several measures need to be made to minimize selection biases and control for confounders. Target trial emulation outlines these measures in a target trial protocol to improve the analyses of observational data. Because there has been a systematic under-representation of randomized, controlled COVID-19 vaccine trials across the African subcontinent,¹⁶ there is a critical need to support these countries in performing evaluation of the real-world effectiveness for COVID-19 vaccines in these countries. Given that RCTs for the evaluation of mass vaccine campaigns for COVID-19 are likely not feasible, target trial emulation can provide an effective and practical way to enable causal analyses of real-world data.