Home Visits to Assess the Reliability of Caregiver-Reported Use of Insecticide-Treated Bednets by Children in Machinga District, Malawi

Insecticide-treated bednets (ITNs) can dramatically reduce malaria transmission1 and are a cornerstone of malaria control programs throughout sub-Saharan Africa.2 In community-based surveys, however, use of ITNs often falls below the World Health Organization's universal coverage target.2 Evaluations of ITN use frequently rely on self- or caregiver report because direct observation at night is challenging due to logistical considerations, security issues, and community acceptance. The few studies that have validated reported ITN use through nighttime checks have found that survey-reported use generally overestimates actual ITN use by 6–32% points.3–5 Over-reporting of ITN use, usually attributed to recall and/or social desirability bias,6,7 can result in significant underestimates of ITN effectiveness in reducing malaria risk.8,9 As part of a prospective study examining the effectiveness of ITNs to prevent malaria in a fixed cohort of young children (aged 6–59 months) in six rural villages in Machinga District, Malawi, we compared caregiver verbal reports of ITN use at monthly interviews at community centers with visual confirmation of ITNs at home visits.

The cohort study was conducted from April 2012 to December 2013 and included baseline distribution of new PermaNet® 2.0 long-lasting ITNs (Vestergaard, Lausanne, Switzerland) in May 2012 to all 2,178 registered households. The distribution provided one ITN per every two household residents (plus an additional ITN for odd numbers). Caregivers of enrolled children were interviewed once per month at a community center away from their homes using a standardized questionnaire. Caregivers' responses regarding whether their child slept under an ITN on the night prior to the monthly interview served as the caregiver verbal report of ITN use.

The study protocol originally included nighttime checks on samples of enrolled children to validate caregiver-reported ITN use. After consultation with the study community, however, nighttime checks were deemed unacceptable and were replaced with unannounced daytime home visits. A study staff person, blinded to the caregivers' reports of ITN use, attempted to visit the homes of selected children within three days after the monthly interview. Using a standardized questionnaire, the staff person requested visual confirmation of the ITN the child used after the caregiver was asked whether the child slept under an ITN the night before the home visit; a child was considered to have used an ITN at the home visit if the caregiver responded positively and the child's unpackaged ITN was visually observed by the staff person. The caregivers present at the monthly interview and at the home visit were not necessarily the same individual.

Children who attended their scheduled monthly interviews were selected by simple random sampling for follow-up home visits during the wet season (survey 1: December 2012–January 2013; N = 211 [18% of enrolled children]) and the dry season (survey 2: September–October 2013; N = 325 [26% of enrolled children]). Sample sizes were calculated to provide 80% power to detect a difference of 5% (assuming 98% caregiver-reported ITN use and 93% visual confirmation of ITN use) at α = 0.05.

As neither caregiver verbal report nor visual confirmation of an ITN at the home visit can be considered a gold standard measure of ITN use by a child, we could not calculate directly the sensitivity and specificity of either measure. For concordance in a diagnostic test, proportionate agreements in positive and negative classifications are measures analogous to sensitivity and specificity. These two indices may be more desirable to assess agreement than measures such as Cohen's kappa when marginal totals are significantly imbalanced.10 To measure the level of agreement between caregiver verbal reports of ITN use and visual confirmation of ITNs at the home visit, we calculated the proportion of positive agreement as

View Expanded

where N is the total number of children included in each survey, a is the number of children with ITN use both reported by the caregiver and confirmed during home visit, and d is the number of children who did not use ITNs as reported by the caregiver and confirmed during home visit.11 The proportion of agreement in negative responses was similarly calculated as

View Expanded

Bootstrap 95% confidence intervals (CIs) for both measures were constructed using 1,000 bootstrap samples. The χ² test was used to compare proportions. Statistical analyses were conducted using R version 3.0.2 (R Core Team, Vienna, Austria).

Ethical approval for this study was obtained from the Centers for Disease Control and Prevention (Atlanta, GA; protocol no. 6227) and the University of Malawi's College of Medicine (Blantyre, Malawi; protocol no. 1151). All caregivers of the study participants provided written informed consent.

During the first survey, a median of two days passed between monthly interviews and home visits (range 0–8 days), with some outliers due to heavy rains. Caregiver-reported ITN use was 98.1%, whereas ITN use at the home visit was 98.6% (Table 1). Five responses were discordant between caregiver reports and home visits. The proportion of positive agreement was 98.8% (95% CI 97.6–99.8%) and the proportion of negative agreement was 28.6% (95% CI 0–75.0%).

During the second survey, a median of one day passed between monthly interviews and home visits (range 1–3 days). Caregiver-reported ITN use was 96.0% (Table 1), which was not statistically different from the first survey (P = 0.213). Home visit ITN use was 88.6%, which was lower than in the first survey (P < 0.001). Forty responses were discordant between caregiver reports and home visits. The proportion of positive agreement in the second survey was 93.3% (95% CI 91.2–95.3%) and the proportion of negative agreement was 20.0% (95% CI 0.1–35.0%).

Reasons for the lower ITN use as confirmed by home visit during the second survey may include increased physical damage to ITNs,12 higher temperatures during the dry season,3 and/or increased time since ITN distribution.13 This analysis included only two time points, making it difficult to tease apart the effects of seasonality and time on ITN use.

The high level of agreement in positive classification of ITN use between caregiver reports and home visits was similar between surveys, suggesting that caregiver report is a reliable proxy for home visits when ITN use is reported positively. However, the proportionate agreement for negative responses was very low in both surveys suggesting that negative responses by caregivers in either setting may not be reliable. The major limitation of our study was the inability to validate caregiver-reported ITN use against a robust gold standard, such as direct nighttime observation. In addition, because only 27 of the 536 home visits (5.0%) occurred on the same day as the monthly interviews, most caregivers were asked about their child's ITN use on different nights. Although conducting home visits on the same day as the monthly interview would have been ideal, this was not possible in most cases due to resource constraints. We also note the possibility that different respondents could have been asked about ITN use at the monthly interview compared with the home visit.

ITN use by children in Machinga District, Malawi, was generally high (≥ 88.6%) in this study. There was high (> 90%) agreement for positive responses to ITN use, suggesting that caregiver report can be used in place of more intensive home visits as a measure of ITN use in epidemiologic surveys. The reasons for low agreement on negative responses are unclear but suggest that reported non-use of ITNs may not be a reliable measure.