Can Outreach Training and Supportive Supervision Improve Competency in Malaria Service Delivery? An Evaluation in Cameroon, Ghana, Niger, and Zambia

ABSTRACT. Outreach Training and Supportive Supervision (OTSS) of malaria services at health facilities has been adopted by numerous malaria-endemic countries. The OTSS model is characterized by a hands-on method to enhance national guidelines and supervision tools, train supervisors, and perform supervision visits. An independent evaluation was conducted to evaluate the effectiveness of OTSS on health worker competence in the clinical management of malaria, parasitological diagnosis, and prevention of malaria in pregnancy. From 2018 to 2021, health facilities in Cameroon, Ghana, Niger, and Zambia received OTSS visits during which health workers were observed directly during patient consultations, and supervisors completed standardized checklists to assess their performance. Mixed-effects logistic regression models were developed to assess the impact of increasing OTSS visit number on a set of eight program-generated outcome indicators, including overall competency and requesting a confirmatory malaria test appropriately. Seven of eight outcome indicators showed evidence of beneficial effects of increased OTSS visits. Odds of health workers reaching competency thresholds for the malaria-in-pregnancy checklist increased by more than four times for each additional OTSS visit (odds ratio [OR], 4.62; 95% CI, 3.62–5.88). Each additional OTSS visit was associated with almost four times the odds of the health worker foregoing antimalarial prescriptions for patients who tested negative for malaria (OR, 3.80; 95% CI, 2.35–6.16). This evaluation provides evidence that successive OTSS visits result in meaningful improvements in indicators linked to quality case management of patients attending facilities for malaria diagnosis and treatment, as well as quality malaria prevention services received by women attending antenatal services.


Inclusion of health facilities which met competency thresholds on their first OTSS visit under Impact Malaria in pooled dose-response models -a sensitivity analysis
Method A sensitivity analysis attempted to assess the impact of decision to remove baseline high performing facilities from dose response models.Baseline high performing facilities are those which scored ≥90% on the specific domain checklist on their first OTSS visit under the Impact Malaria project.
Outcome indicators and mixed effect logistic regression models were otherwise the same as those presented in the main manuscript, whereby facilities receiving only one OTSS visit were excluded, country was retained in models as a fixed effect, and a random effect included for the individual facility to account for non-independence of observations from the same facilities.S1: Summary of eight mixed effect models generated to assess association between OTSS visit iteration (continuous) received by a facility and key malaria case management outcome indicators.This sensitivity analysis permitted inclusion of facilities with baseline high performance.

Method
Pooled models were tested with interaction terms between country and OTSS visit iteration for each of the defined outcomes.A likelihood ratio test was used to test the null hypothesis that there was no interaction between country and OTSS visit iteration in each case management outcome model.
Where there was evidence for an interaction, country-specific models were generated for further description of the effect of OTSS on malaria case management in each country.

Results
Evidence for interaction between country and OTSS visit iteration is indicated for six of the outcomes assessed.Microscopy competence could not be assessed since the model with interaction term did not successfully converge.
In stratum-specific models, an effect of increasing OTSS rounds on health worker competence in malaria clinical management and malaria in pregnancy management were seen in all four countries, although with varying effect size.Effects on successful patient classification, and on withholding antimalarials from patients with negative tests were only found in Cameroon.Effect of increasing OTSS rounds on RDT competency was found in Ghana, Niger, and Zambia but not in Cameroon.
Improvements in IPTp provision with increasing OTSS rounds were found in Ghana and Niger.
The primary limitation of the country-specific models is the small amount of data for some outcomes, limiting the ability to fully assess the impact of OTSS on outcomes within these countryspecific models.S1: Forest plot displaying estimated effect of increasing OTSS visit iteration on the six outcomes where there was evidence for interaction between country and OTSS iteration.For each outcome, the effect estimate from country-specific models (supplementary table 5) and corresponding effect estimate from pooled model (Table 3) are presented.

Method
A sensitivity analysis attempted to further investigate associations between increasing OTSS visits (as a continuous exposure) and summary OTSS checklist scores combining visits under MalariaCare and Impact Malaria.This analysis was intended to capture the longer history of OTSS in Ghana and Zambia, where facilities may have received more OTSS rounds.
MalariaCare data was available summarized by facility-round (not facility and health worker observation), necessitating aggregation of Impact Malaria data to the same level.Combining the Impact Malaria and MalariaCare datasets assumed that health facility names remained consistent between the two projects, although it should be noted that this likely underestimates the extent of repeat visits across the two project eras, since some facilities may have existed under different, unlinkable names in both datasets.
Health facilities which had received only one OTSS visit during the Impact Malaria project were excluded from the logistic regression models dataset.In addition, health facilities which were high performing on their first visit were excluded from the dataset: high performing was defined as scoring ≥90 on the relevant checklist (e.g. for outcome indicators on the clinical checklist this was defined an overall clinical checklist score ≥90), as these facilities could not be measurably improved by further OTSS intervention.Logistic regression models included country as a fixed effect and had a random effect for the individual facility to account for non-independence of observations from the same facilities.
Models were prepared for the summary 'competence' indicators for the overall clinical, RDT and microscopy checklists, where competence was defined as scoring ≥90% on the checklist.Malaria in pregnancy scores were not available for the MalariaCare project.
Supplemental Table S4

Table S2 :
Likelihood ratio test results comparing models with and without interaction term between country and OTSS visit iteration

Table S3 :
Country-specific model results for the six outcomes where interaction between country and OTSS visit iteration was indicated

:
Summary of OTSS data available from Cameroon, Ghana, Niger, and Zambia for sensitivity analysis combining OTSS visits from MalariaCare and Impact Malaria projects