INTRODUCTION
Neglected tropical diseases (NTDs) affect more than one billion people worldwide who live in conditions of poverty. Among these illnesses, those that present with skin lesions, including cutaneous leishmaniasis (CL), are classified as skin NTDs.1–3 After Brazil, Colombia reports the second-highest number of cases of CL in the Americas (6,093 cases in 2021, incidence 22.7 per 100,000 inhabitants).4 Around 82% of patients with CL live in remote rural areas with limited access to a health system because of geographic, economic, and cultural barriers.5 Departments (first-level administrative divisions) with the highest incidence of CL in Colombia are Guaviare, Caldas, Santander, Tolima, and Risaralda, reporting annual incidences from 67 to 242 per 100,000 inhabitants.5
Standard of care for CL recommended by national guidelines includes weekly monitoring of treatment and evaluation of clinical response at 3, 6, and 12 months after treatment conclusion.6 Consequently, administration of treatment and follow-up of CL patients in remote rural areas is particularly challenging owing to the aforementioned barriers to access to health services.7 Currently, data on therapeutic responses to anti-leishmanial treatments in the Americas are largely limited to randomized clinical trials of efficacy; hence, the effectiveness of the diverse treatments administered is unknown.4
The use of mobile technologies has transformed some health services. Mobile health is defined as health practice supported by mobile devices such as mobile phones, patient monitoring devices, and personnel digital technologies.8 Mobile health (mHealth) has improved access to service, outcomes, and monitoring of visceral leishmaniasis in the community health system.9 Health interventions coupled with mHealth and community health worker participation have helped to bridge the gap between health providers and patients in dispersed rural areas.3,9,10
Mobile applications have also been developed to facilitate the diagnosis, clinical management, and epidemiological surveillance of CL. These apps are intended mainly for use by health professionals.11–13 The “Guaral” mHealth app for presumptive diagnosis of CL developed by our research team for use by community health volunteers was previously shown to reduce the time to diagnosis from 8 to 4 weeks.14,15 Recently, a complementary mobile app named “Guaral: seguimiento a tratamiento (Guaral+ST)” was designed and developed to facilitate monitoring of treatment and assessment of therapeutic response in CL patients. The high accuracy of the remote evaluation of therapeutic response using the photographs taken with the app was previously demonstrated.16
In the current study, an effectiveness-implementation hybrid design17,18 was used to assess the effectiveness of Guaral+ST for monitoring adherence to treatment, adverse reactions, and therapeutic response of patients with CL. Key implementation outcomes (fidelity, usability, and acceptability) are reported for patients in rural areas of two municipalities of Colombia having a high incidence of CL.
MATERIALS AND METHODS
Ethical approval and consent to participate.
This research was approved and monitored by the Institutional Ethical Review Board of the Centro Internacional de Entrenamiento e Investigaciones Médicas (CIDEIM) in accordance with national and international regulations (reference number 1295). Written informed consent was obtained from all participants, community health leaders (CHLs), health workers, and patients, as well as guardians of participants under 18 years of age. Assent was obtained from children over 7 years of age. Approval of the indigenous governor of the Emberá Chamí community in the Pueblo Rico study site was obtained prior to the initiation of the study, in addition to individual consent by community members. Waiver of informed consent was obtained from the institutional ethics committee for use of records of historical standard-of-care controls. This trial was registered with ClinicalTrials.gov (registration number NCT05533736).
Study design.
An effectiveness-implementation hybrid type 2 design18–20 with a sequential explanatory mixed-methods approach was carried out between 2020 and 2021 at two study sites: Pueblo Rico, Risaralda and Rovira, Tolima. A quasi-experimental study with historical control (standard of care) was designed to evaluate the effectiveness of the mHealth intervention to improve monitoring of treatment of patients with CL by CHLs using the Guaral+ST app. The effectiveness indicators, including 1) number of follow-up contacts, 2) adherence to treatment, 3) adverse drug reactions, and 4) therapeutic response, were compared between the intervention and control groups. Acceptability and usability were evaluated qualitatively at three levels: CHLs, health workers, and patients. Fidelity was evaluated through quantitative methods (Figure 1). The study is reported in accordance with the Standards for Reporting Implementation Studies (StaRi) guidelines.21
Study design. CHL = community health leader; CL = cutaneous leishmaniasis; Guaral+ST = Guaral: seguimiento a tratamiento.
Citation: The American Journal of Tropical Medicine and Hygiene 109, 4; 10.4269/ajtmh.22-0805
Study sites and context.
The study was conducted in two municipalities: Pueblo Rico in the Department of Risaralda and Rovira in the Department of Tolima. During 2020, Risaralda and Rovira reported CL incidences of 67.5 and 114 per 100,000 inhabitants, respectively; both were above the national average of 52.4 per 100,000 inhabitants.5
The town of Pueblo Rico is located on the eastern side of the western range of the Andes, (latitude 5.22156, longitude −76.0292), with a mean altitude of 1,560 meters above sea level (m.a.s.l.).22 This municipality extends 632 km2, and 99.9% of its territory is rural. Pueblo Rico is composed of 84 villages. Inhabitants belong to three main ethnic groups: indigenous (Emberá Chamí) (40%), Afro-Colombian (12%), and mestizo (48%).23 This diversity presents barriers for health interventions related to cultural, lingual, and literacy factors. Emberá is the indigenous language spoken; most of the indigenous population lives in remote mountain locations, and their access to and level of formal education are low.7 In contrast, the other two ethnic groups speak Spanish and customarily live closer to road infrastructure. The main health facility in Pueblo Rico, San Rafael Hospital, is located in the urban center. Participant recruitment for the study was conducted at the Santa Cecilia health post, which is the nearest health facility for most CL patients in the municipality. Personnel of the Santa Cecilia health post performed microscopic analysis of the direct smear of lesions to diagnose CL and medical evaluations prior to prescription of treatment.
The town of Rovira is located in the Department of Tolima, on the eastern slope of the central Andean range (latitude 4.23933, longitude −75.23968), having a mean elevation of 1,109 m.a.s.l.22 This municipality extends 736 km2 with a rural area comprising 99.9%.24 Rovira is composed of 82 villages; approximately 96.4% of the population is mestizo.25 San Vicente Hospital, the main health facility in the urban center, provides emergency and outpatient medical services as well as laboratory diagnosis, including microscopy of direct smear of lesions and pretreatment laboratory tests.
Study participants.
Patients with active CL and historical records of patients with confirmed parasitological diagnosis were included in the quantitative evaluation. Further, a subgroup of CHLs, health workers, and patients participated in the qualitative evaluation. Eligibility criteria for study participants are summarized in Table 1.
Inclusion criteria for participants in quantitative and qualitative evaluations
| Inclusion criteria for participants in quantitative evaluation (effectiveness) | |
| Patients with active CL | Diagnosis confirmed by microscopy of tissue smear or culture |
| Any age and gender | |
| Medical prescription for anti-leishmanial treatment | |
| Approved and signed informed consent | |
| Availability of CHL to monitor treatment of the patients in the community | |
| Historical controls (standard of care) | Clinical records of patients with confirmed CL at most 2 years prior to the start of this study |
| Any age and gender | |
| Received anti-leishmanial treatment based on national guidelines6 | |
| Inclusion criteria for participants in qualitative evaluations (acceptability and usability) | |
| CHLs | Residents of the rural areas of study sites |
| Voluntary participation in the monitoring of patients using the mobile app | |
| Completion of the project training program on clinical aspects of CL, ethics, and use of the mobile app | |
| Had followed at least two patients using the app | |
| Signed and approved informed consent | |
| Health workers | Professionals involved in the clinical management and assessment of the CL patients using the mobile app |
| Approved and signed informed consent | |
| Patients | Patients with active CL who were monitored by CHL using the app |
| Approved and signed informed consent | |
CHL = community health leader; CL = cutaneous leishmaniasis.
Sample size.
Sample size for the effectiveness evaluation was calculated using the Fleiss equation for differences between two proportions.26,27 The parameters were 95% confidence, 95% statistical power, and the expected proportions of patients with at least one follow-up estimated at 7% in the historical control group and 50% in the intervention (app) group.28 A ratio of two historical controls to one active CL case (2:1) was assumed, as were losses to follow-up of 20%. This yielded 48 historical controls and 24 patients with active CL. Active CL patients to be monitored using the app were enrolled consecutively, and historical records were selected by simple random sampling. For qualitative evaluation, the sample size of the active CL group was increased (n = 57) to include at least five CHLs at each study site who completed follow-up monitoring of at least two CL patients each. In addition, seven health workers who monitored treatment of patients with the app and six patients were included in the qualitative analysis.
Intervention and implementation strategy.
Based on efficacy results of a previous study using the Guaral app for presumptive diagnosis and the Guaral+ST app by CHLs,14–16 expert consensus,29,30 and recommendations of the national guidelines,6 a community-based strategy utilizing the Guaral+ST app was undertaken to monitor the treatment and follow the therapeutic response of CL patients.
A context analysis was performed before beginning the study to appropriately build the relationship with health authorities, adapt the implementation strategy to the context, and identify and train CHLs. In addition, adjustments were made to the Guaral+ST mobile app to improve its operation in areas without Internet connectivity and to upload data and photos to the iCloud when and where access to connectivity by the users in the study sites was available.
After confirmation of CL diagnosis by microscopy of tissue smear or culture, a prescription for treatment was issued by local hospital physicians in accordance with national guidelines.6 Active CL patients who met the inclusion criteria and provided informed consent were enrolled in the study. The strategy included the following steps. First, active CL patients were registered in the (Guaral) app for presumptive diagnosis by CHLs.15 Sociodemographic data as well as the number and localization of lesions were recorded, and photographs of lesions were taken before treatment was initiated. Second, a CHL was assigned by the field study coordinator to each patient based on place of residence through the System for Neurological Diseases (SND) desktop app.15 Third, during treatment, adherence and adverse drug reactions were recorded weekly by CHLs using the app. Photographs of lesions were taken at the end of treatment and on days 90 and 180 after initiation of treatment. Data and photographs were transmitted via the Internet to a cloud-based system. Finally, based on these photographs, the study physician evaluated the clinical response for each patient using the SND app (Figure 2).
Schematic of implementation strategy. (A) Sociodemographic and clinical data and pretreatment lesion photographs of CL patients were recorded and transmitted to the cloud by the CHL using the Guaral app. (B) A CHL was assigned to each patient through the desktop Guaral: seguimiento a tratamiento (Guaral+ST) app by the field study coordinator using the System for Neurological Diseases (SND) app. (C) Weekly evaluation of adherence and adverse drug reactions was performed by the CHL in the Guaral+ST app. (D) Clinical response was evaluated by the study physician based on photographs taken by the CHL at the end of treatment and on days 90 and 180 ±2 weeks after initiating treatment using the Guaral+ST app. CHL = community health leader; CL = cutaneous leishmaniasis.
Citation: The American Journal of Tropical Medicine and Hygiene 109, 4; 10.4269/ajtmh.22-0805
To support the implementation of the strategy, CHLs and health workers were trained in ethical research conduct, clinical aspects of CL, and the study protocol and procedures and were evaluated on their knowledge and use of the Guaral+ST through accompaniment by the study coordinator when monitoring treatment with their first patients.
Effectiveness of the community-based strategy with the Guaral+ST app for follow-up of CL patients.
Effectiveness was estimated by comparing the outcomes between active CL patients in the intervention group and those who previously received standard of care. The following proportions were compared between the two groups: 1) follow-up at end of treatment, day 90, day 180, and days 90 and/or 180 in accordance with the national guidelines for CL6; 2) number of follow-up visits after treatment; 3) record of adherence to treatment; 4) patients with adverse drug reactions recorded; and 5) record of therapeutic response at days 90 and/or 180. We considered windows of ±1 week for end of treatment and ±2 weeks for the last two visits to be acceptable.
Patient treatment was prescribed by the attending physician in the health facilities of the study sites in accordance with national guidelines.6 Systemic treatments administered during the study were meglumine antimoniate (MA), pentamidine, and miltefosine.
Initial and final cures were defined as the absence of new leishmaniasis lesions and complete re-epithelialization of the ulcers (based on photographs) at days 90 and 180 after initiating treatment, respectively. Therapeutic failure was defined as new lesions or less than 100% re-epithelization of lesions 90 or 180 days after initiating treatment.29 Patients who failed treatment were referred by the CHL to the local health facility for rescue treatment. Decisions on rescue treatment were made by the physicians at local health facilities on the basis of Colombian guidelines.6
Implementation effectiveness.
Implementation outcomes were defined according to the Proctor model.31
Fidelity.
Defined as compliance with the implementation strategy as described in the protocol,31 fidelity was measured at two levels: CHL and physician. Fidelity was measured in terms of the following: 1) record of treatment adherence; 2) record of adverse drug reactions; 3) existence of photographs of lesions at day 90 or 180 ± 2 weeks; and 4) evaluation of therapeutic response. The CHLs were responsible for aspects 1 to 3, whereas aspect 4 was the responsibility of study physicians. Each area of fidelity was measured as a binary variable (either acceptable or unacceptable).
Treatment adherence and adverse drug reactions were evaluated weekly in the field while the patient received medication. The following numbers of evaluations were considered acceptable: three for MA, four for miltefosine, and two for pentamidine. Whether CHLs had obtained and recorded photographs was evaluated before treatment, at the end of treatment, and on or around days 90 and 180 after initiation of treatment. Compliance with evaluation of therapeutic response by physicians was measured at these same time intervals.
Acceptability.
Referring to implementers’ perception that the app is appealing, and it was considered either suitable.19,31 In this study, acceptability was measured qualitatively via semi-structured individual and group interviews at three levels: CHLs, health workers, and patients. Because of the Covid-19 pandemic, these activities were performed through videoconference or in person. Acceptability was evaluated based on three analytical categories: 1) relevance: whether the app is necessary, important, or useful for health needs, patients, and users; 2) suitability: whether the app is appropriate in the socio-health context; and 3) added value: whether the app has advantages or favorable differences with respect to routine or standard care.31–34
Usability.
This term refers to the degree to which the app can be used by CHLs and health workers to achieve the quantified objectives of effectiveness, efficiency, and satisfaction.35 In the current study, usability was evaluated using the same qualitative approaches as acceptability. Analytical categories that oriented the interviews were 1) attractiveness: the appealing presentation of the app to the users; 2) ease of use: ability of users to accomplish follow-up monitoring with the app; 3) speed of recording information: the time needed for users to record the data; 4) memorability: the ability of a user to use the app effectively on subsequent occasions; and 5) frequency of errors: how well the user can complete the respective task according to the user’s perception.35–41
DATA ANALYSES
The management of data from these mHealth apps was described previously.42 Access was controlled by creating password-protected user accounts with encrypted passwords. Data transfers were also encrypted. Data collected by CHLs using the app and from interviews (particularly recordings and transcripts) were identified with a code to guarantee confidentiality. Clinical monitoring was performed three times during the study.
Quantitative analysis was performed using Stata 12 and R version 4.1.0 software. For categorical variables, groups were compared using either the χ2 or Fisher’s exact test, depending on the magnitude of the expected values. For dichotomous variables, odds ratios were calculated, with confidence intervals using Fay’s method,43 implemented in the “exact2 × 2” R package. Post hoc analysis was performed to evaluate possible confounding and effect modification by age, ethnicity, and type of medication. Crude and adjusted odds ratios were estimated between the effectiveness outcome (whether patient follow-up was done at 90 and/or 180 days) and study group (Guaral+ST app by CHL or standard of care). Confounding was determined to exist if the percentage change between the crude and adjusted ORs was 10% or greater.44 Stratum-specific odds ratios were calculated, and the Breslow-Day test was performed to evaluate homogeneity between strata.
For qualitative analysis, N-vivo software was used for coding. Following grounded theory procedures, the data were analyzed through a process of open, axial, and selective coding, identifying recurrent emerging themes, which were subsequently grouped into categories of analysis.45 The constant comparative analysis method was used to establish the differences and similarities between level of evaluations (CHLs, health workers, and patients)46 and study sites to ensure credibility. The coding process was carried out by a sociologist research assistant and was reviewed by the researcher who led the fieldwork to ensure confirmability.47
RESULTS
Enrollment and participant characteristics.
Characteristics of patients recruited in the quantitative analysis.
Participants were enrolled for quantitative analysis (effectiveness evaluation) at the two study sites; 57 patients with parasitologically confirmed active CL were prospectively followed up with the Guaral+ST app by CHLs, and 48 historical control patients were managed under standard of care–based follow-up. In the intervention group, 17.5% of patients (10/57) with CL were lost to follow-up before completing treatment (n = 1) or had incomplete follow-up (n = 9), whereas 96% (46/48) in the standard-of-care group had no follow-up after treatment (Figure 3).
Schematic summary of the enrollment of study participants in quantitative and qualitative evaluations. (A) Evaluation of effectiveness: CL patients followed either by community health leader (CHL) using the Guaral: seguimiento a tratamiento (Guaral+ST) app or according to standard of care. (B) Evaluation of acceptability and usability at three levels: CHLs, health workers (HWs), and patients. CL = cutaneous leishmaniasis.
Citation: The American Journal of Tropical Medicine and Hygiene 109, 4; 10.4269/ajtmh.22-0805
Regarding the patients’ demographic and clinical characteristics, the distribution by gender was similar, with males proportionately higher in both groups, whereas differences in age, ethnicity, and treatment received were identified (Table 2). These differences reflect and underscore the difference in access of indigenous populations and young children in the study sites to health facilities. Pentamidine, which is administered in four doses, was prescribed to patients in Pueblo Rico with limited capacity to return to health facilities (pediatric and indigenous patients) for daily treatment during the 3 to 4 weeks required for treatment with MA or miltefosine, respectively.
Demographic and clinical characteristics of patients with cutaneous leishmaniasis
| Characteristics | Community-based strategy using Guaral+ST app (n = 57) | Standard of care (n = 48) | P value |
|---|---|---|---|
| Demographic characteristics | |||
| Gender, n (%) | |||
| Male | 31 (54.4) | 29 (60.4) | 0.56* |
| Female | 26 (45.6) | 19 (39.6) | |
| Age (years), n (%) | |||
| 0–12 | 30 (52.6) | 16 (33.3) | 0.05* |
| > 12 | 27 (47.4) | 32 (66.7) | |
| Ethnicity, n (%) | |||
| Mestizo | 31 (54.4) | 36 (75.0) | 0.04† |
| Indigenous | 25 (43.9) | 12 (25.0) | |
| Afro-Colombian | 1 (1.7) | 0 (0) | |
| Clinical characteristics | |||
| Number of lesions, n (%) | |||
| 1–3 | 56 (98.2) | ND | – |
| > 3 | 1 (1.8) | ND | |
| Duration of oldest lesion (months), n (%) | |||
| 0–3 | 28 (49.1) | 13 (27.1) | 0.07* |
| 3.1–5.9 | 7 (12.3) | 9 (18.8) | |
| > 6 | 22 (38.6) | 26 (54.1) | |
| Location of lesions, n (%) | |||
| Head | 22 (24.4) | ND | – |
| Trunk | 7 (7.8) | ND | |
| Arms | 30 (33.3) | ND | |
| Legs | 31 (34.4) | ND | |
| Total lesions | 90 | – | – |
| Treatment, n(%) | |||
| Meglumine antimoniate | 25 (43.9) | 45 (93.8) | < 0.001† |
| Miltefosine | 2 (3.5) | 2 (4.2) | |
| Pentamidine | 30 (52.6) | 1 (2.0) | |
ND = No data available. P values in boldface mean that are statistically significant.
χ2 test.
Fisher’s exact test.
In terms of study site characteristics, the gender and age distributions in Pueblo Rico were similar for the community-based strategy using the app and the historical group attended at the local hospital. Differences were found between ethnicity and the treatment received. The predominant ethnicity in the intervention group was indigenous (81%; 25/31) followed by mestizo (16.1%; 5/31), whereas the historical control group was composed of 54.5% (12/22) and 45.5% (10/22) (P = 0.03), respectively. Regarding treatment, all patients in the community-based strategy group in Pueblo Rico received pentamidine, whereas 95.5% received MA in the historical patient control group (P ≤ 0.001). In contrast, no significant differences in demographic and clinical characteristics were found between the intervention and control study groups in Rovira. In Pueblo Rico, most participants in the community-based strategy group were children (74%; 0 to 12 years of age), whereas the opposite occurred in Rovira, where 73% of CL participants were > 12 years of age.
Characteristics of participants in the qualitative analysis.
Twenty-four participants were included in the qualitative analysis (acceptability and usability evaluation) (Figure 3). In Pueblo Rico, of the seven CHLs who participated, their ethnicity was mainly indigenous (5/7); only one was affiliated with the local hospital and another with the Departmental Secretary of Health. All health workers were mestizo and had completed technical education. Patients had received less formal education compared with health workers. In Rovira, all participants, six health workers, and three patients, were mestizo. The CHLs were affiliated with the local hospital and had a complete technical education as auxiliary nurses. Demographic characteristics of the CHLs, health workers, and patients who participated in the qualitative evaluation are detailed in Table 3.
Demographic characteristics of participants in the evaluation of acceptability and usability
| Study site | Participants | Ethnicity | Gender | Level of education |
|---|---|---|---|---|
| Pueblo Rico |
|
|
|
|
|
Mestizo (n = 5) |
|
Completed technical education (n = 5) | |
|
|
|
| |
| Rovira |
|
Mestizo (n = 9) |
|
Completed technical education (n = 4) |
|
|
| ||
|
|
|
CHL = community health leader.
Effectiveness of community-based strategy using the Guaral+ST app.
The proportion of patient who fulfilled follow-up at day 90 and/or 180 increased from 4.2% among historical controls (standard of care) to 82.5% among participants in the community-based strategy with the Guaral+ST app. The recording of treatment adherence, adverse drug reactions, and therapeutic response also increased significantly (P < 0.001). Adherence to treatment by > 90% was higher in the app group at 78.9% (45/57) compared with standard of care at 43.8% (21/48). Therapeutic response was not found in case histories of most of the patients (95.8%) whose follow-up was conducted under the standard of care in the historical control group. Twelve patients (12/57; 25.5%) in the app group failed treatment. Sixty-six percent (8/12) were children 0 to 12 years of age (Table 4).
Effectiveness of community-based strategy using the Guaral+ST app
| Effectiveness evaluation | Community-based strategy Guaral+ST app (n = 57) | Standard of care (n = 48) | Odds Ratio (95% CI)* | P value |
|---|---|---|---|---|
| Register of follow-up | ||||
| End of treatment | 42 (73.7) | 0 (0.0) | 0 (0.000–0.030) | < 0.001† |
| Day 90 after initiating treatment | 40 (70.2) | 1 (2.1) | 0.009 (0.000–0.065) | < 0.001† |
| Day 180 after initiating treatment | 43 (75.4) | 1 (2.1) | 0.007 (0.000–0.051) | < 0.001† |
| Day 90 or 180 after initiating treatment | 47 (82.5) | 2 (4.2) | 0.009 (0.001–0.047) | < 0.001† |
| Day 90 and 180 after initiating treatment | 36 (63.2) | 0 (0.0) | 0 (0.000–0.048) | < 0.001† |
| Patients with adverse drug reactions registered | ||||
| Yes | 22 (38.6) | 4 (8.3) | 0.144 (0.033–0.488) | < 0.001† |
| No | 35 (61.4) | 44 (91.7) | ||
| Number of follow-up visits AST | ||||
| 0 | 0 (0.0) | 46 (95.8) | – | < 0.001‡ |
| 1 | 7 (12.3) | 2 (4.2) | – | |
| 2 | 12 (21.0) | 0 (0.0) | – | |
| > 3 | 38 (66.7) | 0 (0.0) | ||
| Registration of treatment adherence | ||||
| Yes | 56 (98.2) | 26 (54.2) | 0.021 (0.001–0.149) | < 0.001‡ |
| No | 1 (1.8) | 22 (45.8) | ||
| Treatment adherence | ||||
| < 90% | 11 (19.3) | 5 (10.4) | – | < 0.001† |
| > 90% | 45 (78.9) | 21 (43.8) | – | |
| Unknown | 1 (1.8) | 22 (45.8) | – | |
| Registration of therapeutic response at day 90 or 180 AST | ||||
| Yes | 47 (82.5) | 2 (4.2) | 0.009 (0.001–0.048) | < 0.001‡ |
| No | 10 (17.5) | 46 (95.8) | ||
| Therapeutic response at day 90 or 180 AST (n = 49) | ||||
| Definitive cure | 35 (74.5) | 0 (0.0) | NC | 0.077‡ |
| Failure | 12 (25.5) | 2 (100) | – | |
AST = after start of treatment; NC = not calculable; OR = odds ratio; Guaral+ST app = Guaral: seguimiento a tratamiento. P values in boldface mean that are statistically significant.
For dichotomous variables, standard of care relative to app.
χ2 test.
Fisher’s exact test.
Treatment failures were concentrated in pediatric patients. Therapeutic failure with pentamidine was 29.6% (8/27). Of the eight failures, six were children 12 years of age or younger. Failure for MA was 16.7% (3/18). The only patient who failed treatment among the four receiving miltefosine was 3 years of age (Supplemental Table 1).
Data on adverse events (AEs) were available for more participants in the mobile application group than in the historical group: 22/57 (101 events) compared with 4/48 (50 events). Most AEs were mild. Among patients followed with the app, 17 received MA and 5 received pentamidine. Among these 22, the most frequent AEs for MA (n = 94) were injection site reactions, arthralgia, headache, and myalgia. Fifty-nine percent of these patients had 1 to 5 events, with the remaining 41% having 6 to 11. Among the seven AEs registered, in patients receiving pentamidine were dizziness and fever. In the historical group, the three patients who received MA reported 39 AEs (12 to 14 each). The remaining patient with AE data in this group received miltefosine and reported 11 cevents (including vomiting, nausea, dizziness, and headache).
Post hoc analysis showed confounding by age, ethnicity, or treatment medication between groups, with the percentage changes between crude and adjusted odds ratios being higher than 10% (Supplemental Table 2). After adjustment, however, associations remained between patient follow-up at day 90 or 180 and the group (app or historical).
Implementation outcomes.
Fidelity.
Overall, fidelity to the strategy using the app was between 70% and 100% for the recording of 1) adherence to treatment (87.7%); 2) adverse reactions (87.7%); 3) photographs of lesions taken by CHL before treatment, at end of treatment, and at day 90 or 180 (98.2%, 73.7%, and 82.7%, respectively); and 4) evaluation of therapeutic response at day 90 or 180 by study physicians via photos taken by CHLs (100%) (Figure 4).
Fidelity of CHL and physician to the implementation strategy. CHL = community health leader.
Citation: The American Journal of Tropical Medicine and Hygiene 109, 4; 10.4269/ajtmh.22-0805
Both recordings of adherence by CHLs using the app based on medication doses received and adverse drug reactions were similarly fulfilled at 87.7%, as these variables were captured together within the same app follow-up session. According to the protocol, patients should have been followed up weekly by CHLs until the end of treatment. Some patients who lived in particularly remote rural areas were, in fact, followed up at 2-week intervals or at an unprogrammed time according to their availability. Adherence of CHLs in obtaining lesion photos and evaluation of clinical response by the study physician at days 90 and 180 ± 2 weeks after the start of treatment was 82.5% and 100%, respectively (Figure 4).
Evaluation of the acceptability of the app by community health workers, health workers, and patients.
The app was considered acceptable by all participants. The principal themes that emerged were CL management, health care in rural areas, efficiency of health institutions, CL monitoring, technological innovation, and opportunities to use the app for other diseases. Participants emphasized the importance of facilitating patient follow-up, case detection, expediting health workers’ tasks, and data collection (Supplemental Table 3).
“As a health worker […] one goes back to the patient’s home and the follow-up is not only for them but also their family and the people around them or within their community who may also have leishmaniasis, so the app also helps detect cases.” (Group interview. CHL/Health worker. December 4, 2020).
“[…] all the information is saved there [in the app] and there’s no risk that, on the way to visit the patients, paper documents get damaged and we lose information” (Focus group 01. CHL. April 26, 2021).
“They feel support since it’s more supervised because before they were evaluated to see whether it (the lesion) had healed or not after a while, whereas now it is almost every day, there is more commitment to the patient, and for them they feel that support […] ” (Focus group 01. CHL. April 26, 2021).
“With the app we realize if the person is healing and how long it’s taking to heal, and if it is failing, the new lesions, […] those people for whom the treatment fails and who otherwise would not be detected” (Group interview 05. CHL/Health worker. December 4, 2020).
“…with the app the follow-up is very good, especially because of the photos, and because the information is kept there, it is very continuous, then above all, the evidence of the photographic record is very important, if one did it manually, that is, a lot is lost…” (Interview 05. CHL/Health worker. August 19, 2020).
In terms of other uses of the app, the CHLs in both municipalities noted that it would be pertinent to apply this mHealth strategy to management of other diseases. Health workers in Pueblo Rico expressed interest in establishing a departmental network in which the photographic record would be sent from rural communities and assessed by physicians at the first-level (primary care) administrative division (Department). In addition, they pointed out the app’s potential use by health workers affiliated with health institutions.
Evaluation of usability of the app by CHLs and health workers.
Overall, CHLs and health workers expressed a positive perception of the app’s usability. Participants considered that the follow-up app was easy to use and to remember and that the process of recording information was rapid. They also mentioned limitations, especially in Pueblo Rico, due to Internet connection coverage and the CHL’s ability to use a cell phone and the app (Supplemental Table 4).
In relation to the attractiveness analytical category, CHLs and health workers in both municipalities mentioned that they liked the design of the app and stated that it did not need changes. Community health leaders, who were the main users of the app, noted that the app has attractive colors, a legible font, and good graphic representations, which facilitate its use.
“The easiest part is recording the administration of treatment, it [the app] says the day and how many vials, it’s really easy. Also, registering the daily symptoms that the patient may have, like fever, headache.” (Focus Group 02. Community health leader. November 27, 2020).
… “The steps are there. We know that if we ask the series of questions, we have to wait to start taking the photos, then you know more or less what you have to do. With the questions, every question leads you”. (Interview 01. Community leader. August 19, 2020).
With regard to the speed of recording information, for CHLs in Pueblo Rico, there were aspects that some considered fast and others slow, suggesting that they may have been isolated incidents or associated with each leader’s proficiency in the use of cell phones and the app. However, there was a clear trend in the perception of the rapidity of data registration, especially the questions about patient follow-up and adverse effects.
“The only hard thing is the synchronization… to synchronize, myself, I find it complicated, because you always have to have data [in your cellphone plan] or be connected to the Wi-Fi network, and synchronizing consumes a lot of data, and connecting to the network is slow… or where I am, it takes quite a while to synchronize” (Interview 06. Community leader. August 19, 2020).
In terms of frequency of errors, CHLs in both municipalities mentioned that the errors were initially due to user mistakes but were later overcome. As for health workers, they noted that if they paid attention when using the app, there were no errors and that in addition, the app provides the option to correct the user errors before saving data.
DISCUSSION
Our findings indicate that this community-based strategy using the Guaral+ST app improved the follow-up of CL patients in rural settings in Colombia. The proportion of patients with follow-up visits increased from 4.2% under standard of care to 82.5% using the app (P < 0.001). The effect of the community-based strategy using the app also expanded information on treatment adherence, adverse drug reactions, and therapeutic response (P < 0.05 for all comparisons). Regarding implementation outcomes, in this mixed-methods study, we determined from the CHL, health worker, and patient perspectives that the app was accepted because it met a public health need and was suitable for use in rural areas. It was perceived as easy to use, particularly regarding data input, and photographic recording of lesions was considered easy to perform. Overall, the app and its use by community leaders and health workers facilitated case management, patient follow-up, and access to clinical results, including treatment effectiveness.
One of the knowledge gaps in real-life CL management and care is the lack of data on treatment effectiveness. In the Americas, 38.5% of cases lack information on treatment outcomes, and in Colombia and several other countries of the region, this information is unavailable.4 The Guaral+ST app demonstrated potential in reducing this gap, particularly considering that Colombia continues to improve connectivity, particularly in urban centers, and has committed to bringing Internet connectivity to the entire country by 2025.48 The findings substantiate the feasibility and contribution of remote evaluation of treatment to overcoming the geographic and economic barriers to medical care that CL patients often face.7,49 Guaral+ST and other mobile applications for NTDs improve access to medical assistance and monitoring of clinical results at the community level.9 These results align with the WHO NTD roadmap.2
The Guaral+ST app, which was perceived as relevant, suitable, and adding value to monitoring CL patients, had high acceptability. These perceptions are facilitators for wider adoption and potential scale-up of the app.50 The use of human-centered design principles in the development of this tool likely facilitated its ease of use, as well as the application’s fitness within the local population and context.15 Other facilitators described for different mHealth apps used in diverse interventions include user-centered design, motivational aspects for the user, effective marketing, humanizing technology, accessibility, and patient–provider interaction oriented to personnel values.51–53 Another factor that will facilitate the adoption of this intervention is its alignment with the Colombian treatment guidelines for CL.6
Application usability was positively valued: The step-by-step workflow and graphic instructions and illustrations facilitated its use. However, usability was negatively affected by limited connectivity, which impacted data synchronization. Limited access to the Internet in rural areas is a potential barrier to mHealth. Variance in Internet service between rural and urban areas is wide in Colombia; at the end of 2020, 56.5% of the population had Internet access in large cities compared with only 23.8% in rural and remote rural areas.54 Nevertheless, this limitation was managed through the offline capture of data by the app and data synchronization when the user was in a location with Internet service. Notably, the Guaral+ST app was evaluated and perceived as easy to use by CHLs with relatively limited formal education (mostly nonprofessionals), supporting its potential for wider use in community-based and primary levels of care.
We observed that participants in Pueblo Rico had more difficulties in using the technology than those from Rovira and needed more training. Overall, barriers to usability included institutional capacity of the local health system, lower network coverage, differences in education levels, technological literacy, and Spanish language proficiency (especially among indigenous populations), as described elsewhere.52,55,56 Implementation and scaling of skin NTD mHealth apps should take into account local needs and context, perceived utility and simplicity, users’ training needs and empowerment, technological requirements to ensure a long-term mHealth platform, cost assessment, and a monitoring and evaluation process.3,57
Fidelity to the strategy of the different users (CHLs, health workers) was between 70% and 100% for recording of lesion photographs, adherence to treatment, adverse drug reactions, and evaluation of clinical response by physicians based on photos. These results support the viability of implementation of this strategy for monitoring patients with CL in remote rural areas. Although the strategy called for weekly monitoring of adherence to treatment and adverse drug reactions during treatment, in some cases, the follow-up was performed at a different interval according to circumstances. The community-based approach involving community members as CHLs offers flexibility in the programming of follow-up.
A particular strength of this study was the use of mixed methods to estimate the intervention’s effectiveness and to subsequently understand participants’ attitudes toward use of the app and difficulties with the technology. This information is critical to the scale-up of the intervention. The design also facilitated the interdisciplinary integration of health and social sciences.
The main weakness in the evaluation of effectiveness was the nonrandom allocation for the intervention group using the app. Historical patients were included as a control group considering the difficulties, time, and cost of implementing the randomization process in the field, as well as ethical concerns. Participants in the historical control group were selected by a simple randomization process to minimize selection bias. Differences were found in baseline characteristics between the two groups (community-based strategy with the app and standard of care) in terms of age, ethnicity, and treatment. Post hoc analysis showed positive confounding in relation with the first two variables, age and ethnicity, and negative confounding in the treatment. Differences between the two groups also suggest that the population that consults and reaches the health system through the passively administered standard of care is different from the patient population accessed by active community-based monitoring. Most of the patients in Pueblo Rico were children and of indigenous ethnicity, which facilitated the provision of health services to these vulnerable populations.
Qualitative evaluation and data collection were performed virtually. Connectivity issues, communication barriers, and cultural characteristics of Pueblo Rico represented challenges for virtual data collection and may have impacted the study participants’ responses. The use of virtual tools for qualitative data collection has been widely implemented and described by social scientists in their research.58,59 During the Covid-19 pandemic, recommendations for collecting information virtually, ways to enhance platform security, and resources for the virtual collection of data were developed to adapt to social distancing and mobility restrictions.60
Interactions with local authorities and stakeholders during and after this study encouraged implementation of mHealth strategies to improve health system access to dispersed communities and vice versa. The cost of the intervention, its sustainability, and the interoperability between the app and the information systems of local hospitals were not assessed within the scope of the current study, yet receptivity has been engendered. These issues, as well as organizational and programmatic procedures in place, are important considerations in the scale-up and sustainability of mHealth according to the WHO model.61 They will be the focus of the next phase of the implementation of mHealth interventions.
Mobile technology offers opportunities for improving services provided by community health workers to manage diverse illnesses (maternal and child health, HIV/AIDS, and sexual and reproductive health). The most common mobile apps are focused on ensuring community health workers’ compliance with standards for health services, emergency referrals, receiving alerts and reminders by community health workers and caregivers, collecting patient data through mobile phones, and community education and training.10,62,63
Several apps targeting health care professionals and patients have been developed for clinical management and epidemiological surveillance of NTDs in low- and middle-income countries settings.3 Among these is the skin NTD app developed by the WHO to facilitate the diagnosis and clinical management of patients with diverse transmissible dermatological conditions. This app is aimed principally at health care workers.12 Similarly, the LeishCare app was developed to allow health care professionals to record the evolution of CL skin lesions using photos, calculate a score that is defined to identify the risk of death from visceral leishmaniasis, and provide guidance in the management of these diseases in settings where both occur.11 SkinApp supports health workers in peripheral locations in recognizing the early signs and symptoms of skin diseases, including skin NTDs, and to opportunely initiate treatment or refer patients for more advanced diagnostic evaluation.13 The Guaral+ST app evaluated in this report aims to assist and empower CHLs and health care professionals in monitoring treatment of patients in remote rural areas. Future efforts will focus on scale-up and evaluation of penetration and sustainability. The future development of an app for direct interaction of patients with health providers, including transmission of photographs of lesions to health workers for clinical evaluation by medical personnel, could also build upon the findings of this investigation.
The intervention with the app allows clinical monitoring of CL patients. The effectiveness of MA was similar to that reported in other publications.64–66 Among the eight patients who received pentamidine and failed treatment, it is noteworthy that 6/8 patients were children. Clinical trials to evaluate pentamidine’s efficacy in the pediatric population are needed.
This study was a collaborative effort between the research team, affected communities, and health system authorities sponsored by a pharmaceutical company. Future global health innovation for NTDs will require intersectoral collaboration among investigators and other stakeholders to respond to patient needs. Implementation research and innovative, integrated approaches to diagnose and treat leishmaniasis and other NTDs using mHealth technologies developed, evaluated, and implemented in partnership with communities, health authorities, and multidisciplinary research teams can bridge the gap in access to diagnosis and treatment.
The Guaral+ST app is an effective, easy-to-use mHealth tool that is acceptable for community leaders and health volunteers for monitoring and determining the effectiveness of treatment of CL. The community-based approach used in this study alleviates geographic barriers to the monitoring of interventions in areas where populations are dispersed and medical personnel are scarce. Use of the mHealth app by CHLs provided a link between health personnel in health facilities and patients in their rural communities for the exchange of actionable information.
Supplemental Materials
ACKNOWLEDGMENTS
The authors are grateful for the support of the Secretary of Health of the Departments of Risaralda and Tolima, the San Rafael and Santa Cecilia hospitals, and the Local Health Directorate from Pueblo Rico, Hospital San Vicente de Paul, and the Secretary of Health of the municipality of Rovira. We especially thank the CHLs, communities, indigenous authorities, and patients who trusted us to conduct this work within their communities. We also acknowledge with gratitude the support of the Epidemiology and Biostatistics Unit of CIDEIM in developing the databases and assisting in the analysis of data.
REFERENCES
- 1.↑
Reed SL , Mckerrow JH , 2018. Why funding for neglected tropical diseases should be a global priority. Clin Infect Dis 67: 323–326.
- 2.↑
World Health Organization , 2021. Ending the Neglect to Attain the Sustainable Development Goals: A Road Map for Neglected Tropical Diseases 2021–2030. Available at: https://www.who.int/publications/i/item/9789240010352. Accessed June 11, 2022.
- 3.↑
Carrion C , Robles N , Sola-Morales O , Aymerich M , Ruiz-Postigo JA , 2020. Mobile health strategies to tackle skin neglected tropical diseases with recommendations from innovative experiences: systematic review. JMIR Mhealth Uhealth 8: 1–14.
- 4.↑
Pan American Health Organization , WHO , 2022. Leishmaniasis: Epidemiological Report for the Americas. Available at: https://iris.paho.org/handle/10665.2/56831. Accessed June 11, 2022.
- 5.↑
Instituto Nacional de Salud , Ministerio de Salud , 2020 . Informe del Evento Leishmaniasis Cutanea, Mucosa y Visceral. Colombia. Available at: https://www.ins.gov.co/buscador-eventos/Informesdeevento/LEISHMANIASIS_2020.pdf. Accessed June 11, 2022.
- 6.↑
Ministerio de Salud y Protección Social , 2023. Lineamientos de Atencion Clınica Integral para Leishmaniasis en Colombia. Versión 4. Available at: https://www.minsalud.gov.co/sites/rid/Lists/BibliotecaDigital/RIDE/VS/PP/PAI/Lineamientos-leishmaniasis.pdf. Accessed Aug 8, 2023.
- 7.↑
Gomez-Bautista MM , Doerfler J , Castro MdM , 2022. Barriers to cutaneous leishmaniasis care faced by indigenous communities of rural areas in Colombia: a qualitative study. BMC Infect Dis 22: 302.
- 8.↑
World Health Organization , Global Observatory for eHealth , 2011. mHealth: New Horizons for Health through Mobile Technologies. Available at: https://apps.who.int/iris/handle/10665/44607. Accessed June 20, 2022.
- 9.↑
Tilahun B , Gashu KD , Mekonnen ZA , Endehabtu BF , Angaw DA , 2021. Mapping the role of digital health technologies in prevention and control of COVID-19 pandemic: review of the literature. Yearb Med Inform 30: 26–37.
- 10.↑
Mahmood H , Mckinstry B , Saturnino L , Fairhurst K , Nasim S , Hazir T , Collaboration R , 2020. Community health worker-based mobile health (mHealth) approaches for improving management and caregiver knowledge of common childhood infections: a systematic review. J Glob Health 10: 1–21.
- 11.↑
da Silva PEF , dos Santos Fonseca G Jr. , Ambrozio RB , Costa MGST , Machado GB , de Carvalho SFG , de Oliveira EJ , Jorge DC , de Almeida Silva Teixeira L , 2020. LeishCare®: a software designed for the management of individuals with leishmaniases. Am J Trop Med Hyg 103: 909–916.
- 12.↑
WHO , 2020. Neglected Tropical Diseases of the Skin: WHO Launches Mobile Application to Facilitate Diagnosis. Available at: https://www.who.int/news/item/16-07-2020-neglected-tropical-diseases-of-the-skin-who-launches-mobile-application-to-facilitate-diagnosis. Accessed June 20, 2022.
- 13.↑
Mieras LF , Taal AT , Post EB , Ndeve AGZ , van Hees CLM , 2018. The development of a mobile application to support peripheral health workers to diagnose and treat people with skin diseases in resource-poor settings. Trop Med Infect Dis 3: 1–7.
- 14.↑
Navarro A , Rubiano L , Arango JD , Rojas CA , Alexander N , Saravia NG , Aronoff-Spencer E , 2018. Developing mobile health applications for neglected tropical disease research. PLoS Negl Trop Dis 12: 1–6.
- 15.↑
Rubiano L et al., 2021. Adaptation and performance of a mobile application for early detection of cutaneous leishmaniasis. PLoS Negl Trop Dis 15: 1–15.
- 16.↑
Del Castillo AM , del Mar Castro M , Cossio A , García Luna JA , Rincón D , Castillo RM , Prieto MD , Rebellón-Sánchez DE , Navarro A , Alexander N , 2022. Sensitivity and specificity of the remote evaluation of therapeutic response in cutaneous leishmaniasis using photographs from a mobile application. Am J Trop Med Hyg 107: 662–668.
- 17.↑
Landes SJ , McBain SA , Curran GM , 2019. An introduction to effectiveness-implementation hybrid designs. Psychiatry Res 280: 112513.
- 18.↑
Curran GM , Bauer M , Mittman B , Pyne JM , Stetler C , 2012. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 50: 217–226.
- 19.↑
Peters DH , Adam T , Alonge O , Agyepong IA , Tran N , 2013. Implementation research: what it is and how to do it. BMJ (Online) 347: 1–7.
- 20.↑
Landes SJ , McBain SA , Curran GM , 2020. Reprint of: An introduction to effectiveness-implementation hybrid designs. Psychiatry Res 283: 1–16.
- 21.↑
Pinnock H et al., 2017. Standards for reporting implementation studies (StaRI) statement. BMJ (Online) 356: 1–9.
- 22.↑
Ocampo CB , Guzmán-Rodríguez L , Moreno M , Castro MdM , Valderrama-Ardila C , Alexander N , 2021. Integration of phlebotomine ecological niche modelling, and mapping of cutaneous leishmaniasis surveillance data, to identify areas at risk of under-estimation. Acta Trop 224: 106122.
- 23.↑
Alcaldía Municipal de Pueblo Rico - Risaralda , Plan de Desarrollo 2020–2023. Available at: https://puebloricorisaralda.micolombiadigital.gov.co/sites/puebloricorisaralda/content/files/000420/20964_pdt-pueblo-rico-unidos-hacemos-mas-20202023_2.pdf. Accessed June 22, 2022.
- 24.↑
Gobernación del Tolima , Secretaría de Planeacion y TIC. Estadísticas 2011–2014. Rovira. Available at: https://www.tolima.gov.co/images/tolima/cifras-y-estadisticas/Rovira.pdf. Accessed June 22, 2022.
- 25.↑
Municipio de Rovira , Plan de Desarrollo Municipal. Con la Gente para la Gente 2020–2023. Available at: https://roviratolima.micolombiadigital.gov.co/sites/roviratolima/content/files/000391/19548_plan-de-desarrollo-rovira-final.pdf. Accessed June 22, 2022.
- 26.↑
Whitley E , Ball J , 2002. Statistics review 4: sample size calculations. Crit Care 6: 335–341.
- 27.↑
Fleiss J , 1981. Statistical Methods for Rates and Proportions. 3rd ed. New York, NY: J. Wiley and Sons.
- 28.↑
Castro M del M , Martinez-Valencia AJ , Cossio A , Jojoa SJ , 2014. Seguimiento de pacientes con leishmaniasis tegumentaria: experiencia en un centro de referencia. Infectio 18: 71.
- 29.↑
Olliaro P et al., 2018. Harmonized clinical trial methodologies for localized cutaneous leishmaniasis and potential for extensive network with capacities for clinical evaluation. PLoS Negl Trop Dis 12: e0006141.
- 30.↑
Olliaro P , Vaillant M , Arana B , Grogl M , Modabber F , Magill A , Lapujade O , Buffet P , Alvar J , 2013. Methodology of clinical trials aimed at assessing interventions for cutaneous leishmaniasis. PLoS Negl Trop Dis 7: e2130.
- 31.↑
Proctor E , Silmere H , Raghavan R , Hovmand P , Aarons G , Bunger A , Griffey R , Hensley M , 2011. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 38: 65–76.
- 32.↑
WHO , 2016. Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment. Available at: https://apps.who.int/iris/handle/10665/252183. Licencia: CC BY-NC-SA 3.0 IGO. Accessed June 22, 2022.
- 33.↑
Chan AHY , Honey MLL , 2022. User perceptions of mobile digital apps for mental health: Acceptability and usability - An integrative review. J Psychiatr Ment Health Nurs 29: 147–168.
- 34.↑
Peters DH , Adam T , Alonge O , Agyepong IA , Tran N , 2014. Implementation research: what it is and how to do it. Br J Sports Med 48: 731–736.
- 35.↑
Patel NN , Dalal P , 2013. Usability evaluation of mobile applications. Int J Eng Res Tech (IJERT), 2: 299–302.
- 36.↑
Harrison R , Flood D , Duce D , 2013. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 1: 1.
- 37.↑
Zahra F , Hussain A , Mohd H , 2017. Usability evaluation of mobile applications; where do we stand? AIP Conf Proc 1891: 020056.
- 38.↑
Zhang D , Adipat B , 2005. Challenges, methodologies, and issues in the usability testing of mobile applications. Int J Hum Comput Interact 18: 293–308.
- 39.↑
Zaid B , Jamaludin R , Wafaa B , 2012. A comparative study of usability methods for mobile applications. Int J Sci Eng Res 3: 1–4.
- 40.↑
Kortum P , Sorber M , 2015. Measuring the usability of mobile applications for phones and tablets. Int J Hum Comput Interact 31: 518–529.
- 41.↑
Moumane K , Idri A , Abran A , 2016. Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards. Springerplus 5: 548.
- 42.↑
Oviedo Sarmiento OJ , del Mar Castro M , Lerma YO , Bernal LV , Navarro A , Alexander NDE , 2021. Data management plan for a community-level study of the hidden burden of cutaneous leishmaniasis in Colombia. BMC Res Notes 14: 1–6.
- 43.↑
Fay MP , 2010. Confidence intervals that match Fisher’s exact or Blaker’s exact tests. Biostatistics 11: 373–374.
- 44.↑
Szklo M , Nieto J , 2018. Identifying noncausal associations: confounding. Epidemiology Beyond the Basics. 4th ed. Burlington, MA: Jones & Bartlett Learning, 175–203.
- 45.↑
Strauss A , Corbin J , 1990. Grounded theory research: procedures, canons, and evaluative criteria. Qual Sociol 13: 13–21.
- 46.↑
Kolb SM , 2012. Grounded theory and the constant comparative method. Valid Res Strategies Educ 3: 83–86.
- 47.↑
Forero R , Nahidi S , De Costa J , Mohsin M , Fitzgerald G , Gibson N , McCarthy S , Aboagye-Sarfo P , 2018. Application of four-dimension criteria to assess rigour of qualitative research in emergency medicine. BMC Health Serv Res 18: 1–11.
- 48.↑
Defelipe S , 2020. Greater Coverage in Rural Connectivity Will Only Be Real in 2025. Available at: https://impactotic.co/en/Greater-coverage-in-rural-connectivity-will-only-be-real-in-2025/. Accessed June 22, 2022.
- 49.↑
Eid D , San Sebastian M , Hurtig AK , Goicolea I , 2019. Leishmaniasis patients’ pilgrimage to access health care in rural Bolivia: a qualitative study using human rights to health approach. BMC Int Health Hum Rights 19: 1–9.
- 50.↑
Odendaal WA , Watkins JA , Leon N , Goudge J , Griffiths F , Tomlinson M , Daniels K , 2020. Health workers’ perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database Syst Rev 3: CD011942.
- 51.↑
Shabir H , D’Costa M , Mohiaddin Z , Moti Z , Rashid H , Sadowska D , Alam B , Cox B , 2022. The Barriers and Facilitators to the Use of Lifestyle Apps: A Systematic Review of Qualitative Studies. Eur J Investig Health Psychol Educ 12: 144–165.
- 52.↑
Feroz AS , Ali NA , Khoja A , Asad A , Saleem S , 2021. Using mobile phones to improve young people sexual and reproductive health in low and middle - income countries: a systematic review to identify barriers, facilitators, and range of mHealth solutions. Reprod Health 18: 9.
- 53.↑
Austin J , Shukla N , Halley M , Nava V , Budaraju J , Zhang L , Linos E , 2021. Barriers and facilitators to mobile health and active surveillance use among older adults with the skin disease. Health Expect 24: 1582–1592.
- 54.↑
Bnamericas , 2021. Spotlight: Colombia’s Vast Digital Divide between Cities and Rural Areas. Available at: https://www.bnamericas.com/en/news/spotlight-colombias-vast-digital-divide-between-cities-and-rural-areas. Accessed June 22, 2022.
- 55.↑
Scott Kruse C , 2019. Barriers to the use of mobile health in improving health outcomes in developing countries: systematic review. J Med Internet Res 21: e13263.
- 56.↑
Scott Kruse C , Karem P , Shifflett K , Vegi L , Ravi K , Brooks M , 2018. Evaluating barriers to adopting telemedicine worldwide: a systematic review. J Telemed Telecare 24: 4–12.
- 57.↑
WHO , 2010. Nine Steps for Developing a Scaling-Up Strategy. Available at: https://apps.who.int/iris/handle/10665/44432. Accessed June 22, 2022.
- 58.↑
Hope J , Snee H , Hine C , Yvette M & Roberts S Digital Methods for Social Science. London, United Kingdom: Palgrave Macmillan.
- 59.↑
Chen P , Hinton S , 1999. Realtime interviewing using the World Wide Web. Sociol Res Online 4: 63–81.
- 60.↑
Lobe B , Morgan D , Hoffman KA , 2020. Qualitative data collection in an era of social distancing. Int J Qual Methods 19: 1–8.
- 61.↑
World Health Organization, United Nations Foundation, UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction & Johns Hopkins University , 2015. The MAPS Toolkit: mHealth Assessment and Planning for Scale. Available at: https://apps.who.int/iris/handle/10665/185238. Accessed August 10, 2023.
- 62.↑
Braun R , Catalani C , Wimbush J , Israelski D , 2013. Community health workers and mobile technology: a systematic review of the literature. PLoS One 8: 4–9.
- 63.↑
Agarwal S , Perry HB , Long LA , Labrique AB , 2015. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health 20: 1003–1014.
- 64.↑
Oliveira LF , Schubach AO , Martins MM , Passos SL , Oliveira RV , Marzochi MC , Andrade CA , 2011. Systematic review of the adverse effects of cutaneous leishmaniasis treatment in the New World. Acta Trop 118: 87–96.
- 65.↑
Pinart M , Rueda J-R , Romero GAS , Pinzón-Flórez CE , Osorio-Arango K , Silveira Maia-Elkhoury AN , Reveiz L , Elias VM , Tweed JA , 2020. Interventions for American cutaneous and mucocutaneous leishmaniasis. Cochrane Database Syst Rev 2020:CD004834.
- 66.↑
Castro M et al., 2023. Cutaneous leishmaniasis treatment and therapeutic outcomes in special populations: a collaborative retrospective study. PLoS Negl Trop Dis 17: e0011029.