Measuring Implementation Strength for Integrated Community Case Management in Malawi: Results from a National Cell Phone Census

Program managers, investors, and evaluators need real-time information on how program strategies are being scaled up and implemented. Integrated Community Case Management (iCCM) of childhood illnesses is a strategy for increasing access to diagnosis and treatment of malaria, pneumonia, and diarrhea through community-based health workers. We collected real-time data on iCCM implementation strength through cell phone interviews with community-based health workers in Malawi and calculated indicators of implementation strength and utilization at district level using consensus definitions from the Ministry of Health (MOH) and iCCM partners. All of the iCCM implementation strength indicators varied widely within and across districts. Results show that Malawi has made substantial progress in the scale-up of iCCM since the 2008 program launch. However, there are wide differences in iCCM implementation strength by district. Districts that performed well according to the survey measures demonstrate that MOH implementation strength targets are achievable with the right combination of supportive structures. Using the survey results, specific districts can now be targeted with additional support.


INTRODUCTION
One challenge to program evaluation at scale is determining the strength of implementation, defined here as the quantity of a program delivered to a population. Too often, large-scale evaluations are not able to address questions of program effectiveness, because implementation is incomplete or of poor quality. 1,2 Program managers face an equally important challenge in trying to assess and improve their programs without real-time information on the status of implementation, especially when expanding pilot or demonstration programs to scale. The quality of data produced by Health Management Information Systems-even when these systems include relevant data points-is often unknown and records are universally suspected to be incomplete. 3 New and complementary methods are needed to produce up-to-the-moment snapshots of implementation strength for use in improving health service provision, calibrating routine monitoring systems, and interpreting changes (or lack of change) in measures of program outcome and impact.
Integrated Community Case Management (iCCM) is a community-based strategy that uses trained and supervised community health workers (CHWs) to assess, classify, and treat diarrhea, malaria, and pneumonia among children under 5 years of age. 4 iCCM holds promise as a strategy to improve access to correct case management for the major infectious causes of child deaths. 5 There is a growing body of implementation research addressing the challenges of scaling-up iCCM in lowincome countries, and the first full evaluations of the strategy are starting to appear. The program assumption is that deploying iCCM-trained CHWs who are appropriately trained, supplied, and supervised, and at sufficient density in populations without access to fixed health facilities will improve access to appropriate treatment, will be used by the population, and will reduce child mortality from childhood pneumonia, diarrhea, and malaria.
Malawi was the first country in sub-Saharan Africa to implement iCCM at national scale. 6 Beginning in 2008, the Ministry of Health (MOH), with support from the World Health Organization (WHO) and the United Nations Children's Fund (UNICEF), trained existing and newly recruited CHWs, who are called Health Surveillance Assistants (HSAs) in Malawi, and deployed them across 10 districts to areas that District Health Management Teams (DHMTs) defined as hard-to-reach, with limited access to fixed heath facilities. By 2010, the MOH was implementing the iCCM strategy in all 28 districts of Malawi, with support from various other donors and implementation partners (Box 1).
Starting in 2008, an independent evaluation team co-led by the National Statistical Office (NSO) and the Johns Hopkins Institute for International Programs worked with the MOH and WHO to document iCCM and other maternal newborn and child health programs in 10 districts. In 2011, an iCCM stakeholder technical working group (TWG) led by the MOH developed a list of consensus indicators for monitoring the strength of routine iCCM implementation at the level of the HSA and agreed to report on them regularly across all districts. In 2012, despite these efforts, the information on iCCM implementation remained incomplete. Therefore, the evaluation team collaborated with the MOH to develop, test, and implement a method to generate rapid, cross-sectional data on iCCM implementation through a cell phone survey.
We report here on the first full application of this phonebased implementation "snapshot" approach at national scale, conducted among a census of iCCM-trained HSAs in Malawi. A report on the validation study and cost of the method was published previously. 3

MATERIALS AND METHODS
We conducted telephone interviews with all HSAs identified by the MOH as trained in iCCM and deployed to provide iCCM services. Implementation strength indicators. We selected the indicators to be measured from among those defined by the MOH and other iCCM stakeholders in 2011 (see Table 1 for a list of indicators and their definitions). We defined currently working in iCCM as an HSA who had seen at least one sick child in the 3 months before the survey. We included the small number of HSAs who completed on-the-job training rather than the official 6-day iCCM training course provided they met the currently working criterion. Each DHMT defines the hard-to-reach areas (HTRA) in their district using criteria established by the MOH. At time of the survey, these included areas more than 8 km from a health facility. However, there was no official listing available to match HSA catchment areas with these DHMT-defined HTRAs as is required to calculate TWG indicator 2a. Therefore, we asked HSAs to self-report whether they worked in HTRA and reported on this modified indicator. For indicator 3a, HSAs only reported the duration of the most recent stockout in the previous 3-6 months.
For indicators 4 and 5, supervision specifically refers a trained supervisor going out to the HSA's village clinic. iCCM-trained HSAs can also be mentored by qualified clinical staff when they come into the health facility. Reinforcement of clinical practice is defined as the supervisor either directly observing case management practices in the field or clinic or presenting case scenarios to the HSA. More general supervision activities such as protocol reviews or supply replenishing are not considered

BOX 1 iCCM in Malawi
Malawi has primarily used its established cadre of MOH-used CHWs called HSAs to implement the iCCM program. The program guidelines require that HSAs reside in the community they serve, although this has been difficult to implement. In addition to their foundational training as HSAs, HSAs in iCCM target areas receive a 6-day intensive training in iCCM protocols. The MOH provides each HSA with an initial box of iCCM drugs, which is restocked by field supervisors based out of a local health facility. Once trained and supplied, the HSAs hold regular (i.e., 2 days per week) sick child clinics in their assigned catchment area, as well as providing sick childcare on demand. Depending on their location, some HSAs are also engaged in other health promotion activities. Various NGOs provide technical, logistical, and financial support to the MOH for iCCM implementation; most operate at the district or subdistrict level.
The MOH and a coalition of partners have worked together throughout the iCCM scale-up to document program implementation. UNICEF and STC supported a monitoring and evaluation officer based in the MOH from 2009 to 2011, and again beginning in 2013, who was charged with tracking districtlevel iCCM training, deployment, and supervision using routine reports. A web-based commodities management system called cStock was developed, piloted, and has been taken to scale since 2013 with the support of John Snow Inc. The system tracks iCCM supply flows from central stocks down to the level of the HSAs using mobile applications that provide near-real-time data to program managers at national, district, and facility levels (http://sc4ccm.jsi.com/countries/malawi/). reinforcement of clinical practice. To distinguish supervision episodes that included reinforcement of clinical practice, HSAs were asked about the specific activities that occurred during their most recent supervision and mentorship experiences in the previous 3 months.
Enumeration of HSAs. We used a multistep process to identify all currently working HSAs who received iCCM training. District-level MOH officials provided lists of iCCM-trained HSAs organized by health facility. The study team called the in-charge of each health facility to review the accuracy and completeness of these lists. We used several approaches to further check the completeness of the enumeration process. In 22 of the 29 districts, we compared the lists to training and reporting records provided by nongovernmental organizations (NGOs) working in the area. In the seven districts without NGO data, we randomly selected 1-2 health facilities that district officials identified as not having iCCM activities and called the in-charge to verify whether this was actually true.
Telephone interviews. Teams from NSO and Johns Hopkins piloted the telephone interview guide in 2011, and in 2012, we conducted a validation study among 200 HSAs in Ntcheu and Dowa districts in Malawi. 3 The final interview guide for the national census was translated into Chichewa from the original English, and the translation was field tested and back translated. Five teams of four interviewers and one supervisor each conducted all telephone interviews from a private room at the NSO headquarters in Zomba, Malawi. National MOH officials notified the DHMTs before the start of any interviews in their districts.
We attempted to contact each enumerated iCCM-trained HSA by cell phone. The interviewers followed a structured protocol that required up to eight documented call attempts over multiple days before an HSA was determined to be unreachable. If the interviewer was not successful in reaching the HSA within the first four call attempts over a 2-day period, he or she referred the case to the team supervisor who contacted the HSA supervisor or health facility in-charge to develop a strategy for reaching that HSA (e.g., arranging a time to call the HSA via the supervisor's or another HSA's phone). Verbal consent was obtained from each HSA before beginning the interview.
Supervisors were on site with the interviewers during data collection. They directly observed 5% of telephone interviews using a checklist to assess interviewer performance and provide individualized feedback. For quality control purposes, during the first 2 weeks of data collection, two questionnaires per week were randomly selected for each interviewer and the HSAs were called back and reinterviewed by the team supervisor.
Other data collection. The district-level MOH provided the total number of HSAs working in each district. The total district population under 5 years of age was taken from 2013 projections released by the NSO based on the 2008 National Census. 7 We report on a total of 29 districts, because in 2009, the MOH divided Mzimba District into two management units, Mzimba North and Mzimba South.
Data processing and analysis. Responses captured on the paper questionnaires were double entered in CSPro (United States Census Bureau) and exported into STATA 11.0 (StataCorp LP, College Station, TX) for analysis. 8,9 We calculated the proportion of all HSAs trained in iCCM by dividing the number of surveyed HSAs in each district who reported receiving the MOH's 6-day training by the total number of HSAs in each district as reported by district-level MOH offi-cials. To facilitate interpretation across the 29 districts, we ranked and divided the districts into tertiles for select implementation strength and utilization indicators. Figure 1 outlines the process of identifying HSAs currently working in iCCM. There were total 9,555 HSAs working in Malawi at the time of the survey. District-level MOH and  facility in-charges identified 3,725 HSAs as trained in iCCM. The survey team reached all but eight of these HSAs by phone (99.7%). One HSA did not have a personal cell phone and was contacted using a supervisor's phone. Among those reached, 3,392 (91.4%) reported having seen a sick child in the previous 3 months and were classified as currently working in iCCM for the analysis. Table 2 presents results for TWG indicators 1 and 2a. The mean density of deployed HSAs per 1,000 under 5 children was 3.3. The highest density was in Likoma, a sparsely populated island in Lake Malawi, and the lowest density was in predominantly urban Lilongwe District. Nationally, 38.4% of HSAs were trained in iCCM. District-level results ranged from 10.3% trained in Mangochi and 11.7% in Lilongwe compared with over 90% in Likoma, Mwanza, and Neno. Table 3 provides an overview of demographic characteristics and work history of the interviewed HSAs. The majority were male (72.3%) and in their mid-30s. Almost all had at least some secondary education (97.2%), which is consistent with requirements for the Malawi Civil Service. The survey participants had been working as HSAs for an average of about 9.5 years including 2.5 years of iCCM experience. Table 4 presents the findings for the remaining TWG indicators. Consistent with the national strategy for iCCM targeting, the majority (87.1%) of surveyed HSAs self-reported working in a HTRA including three districts-Likoma, Blantyre, and Chiradzulu-where all surveyed HSAs reported working in HTRA. Regarding deployment indicator 2c, 77% percent of HSAs currently working in iCCM reported treating a sick child in the previous 7 days. These HSAs saw an average of 15 children in the previous week ( Figure 2). The actual number of sick children treated by each HSA varied widely both within and between the districts (Supplemental Table 1).

RESULTS
According to MOH strategy, HSAs are supposed to live in the catchment areas where they provide services (Box 1). TWG indicator 6 measures compliance with this policy. Nationally, 69.5% of HSAs working in iCCM reported living in their catchment area. The island of Likoma was the outlier, with only one of its six HSAs currently working in iCCM living in his or her catchment area.
Nationally, nearly 60% of HSAs reported no stockouts longer than 7 days for the four primary iCCM drugs. Five districts-Likoma, Mzimba North, Nkhata Bay, Rumphi, and Nsanje-met the 80% target set by the MOH for indicator 3a. The lowest rates were in Mchinji, Mwanza, and Mangochi. Districts varied in which individual drugs were more likely to be stocked out. HSAs were much less likely to be stocked out of any single iCCM drug compared with paracetamol and eye ointment, which are included in HSA drug kits, but not used for iCCM protocols (results not shown).
Lumefantrine-artemether (LA), co-trimoxizole, and oral rehydration solution are essential for timely life-saving treatment of malaria, pneumonia, and diarrhea, respectively. Therefore, a stockout of any of these drugs is considered unacceptable under TWG indicator 3b. Nationally, half of HSAs reported no stockout of life-saving drugs in the previous 3 months. Only Indicators 4 and 5 relate to supervision of HSAs carrying out iCCM activities. Nationally, 43.7% of iCCM HSAs received field-based supervision and 42.0% received clinic-based mentorship in the previous 3 months. Over half of HSAs (57.6%) reported receiving field-based supervision and/or mentorship with reinforcement of clinical practice in the previous 3 months. Supervision and/or mentorship rates with reinforcement of clinical practice varied by district, with a high of 96.0% in Kasungu and a low of 21.7% in Chitipa. In some districts (e.g., Salima and Mulanje), rates of facility-based mentorship were higher than field-based supervision, while in others (e.g., Chitipa and Kasungu), field-based supervision was more common than facility-based mentorship (results not shown).

DISCUSSION
This is the first report of a cross-sectional implementation strength snapshot for iCCM. The results show that Malawi has made substantial progress in the scale-up of iCCM service provision since the launch of the program in 2008. There are trained and active HSAs providing services across all of Malawi's 29 districts. Overall utilization of HSA services in Malawi is higher than at the start of the program in 2009, 10 and higher than in similar programs in other African countries. 11 Although such progress is noteworthy, nationally, Malawi is not reaching the 80% benchmarks for supply and supervision indicators set by the MOH.
The survey shows that implementation strength and utilization of iCCM services varies widely at the district level (Table 5). We reviewed the results with national MOH program leadership and other partners supporting the implementation of iCCM to identify factors that might explain the district-level variability. Findings of particular importance to those implementing iCCM programs are discussed here.
Districts in which iCCM program activities were targeted to district-defined HTRAs had, on average, HSAs that were more active and seeing greater numbers of sick children than districts in which iCCM program activities were not targeted only to HTRAs. Five districts in the Southern Zone supported by the NGO partner Population Services International (Machinga, Neno, Mwanza, Thyolo, and Zomba) adopted a universal cov-erage strategy that aimed to train and deploy all HSAs for iCCM, regardless of whether they worked in an HTRA. Survey findings for training, deployment, and utilization indicators were consistent with differences in targeting strategy, with the five universal coverage districts among those showing the lowest rates of utilization per HSA. This finding suggests that, in non-HTRAs, children were most likely taken for care to health facilities rather than to the HSAs.
The survey findings suggest an association between HSAs living in their catchment area and increased utilization by the population. Among districts in the highest tertile for indicator 6 (HSA residence in the catchment area), only one was in the lowest tertile for the utilization indicator whereas among those in the lowest tertile for indicator 6, only one was in the highest tertile for the utilization indicator.
Given that both mentorship and restocking of HSAs supplies occur with supervisor assistance at the health facility, we predicted that the two indicators would perform similarly at district level. Unexpectedly, only two of the nine districts in the highest tertile for the drug and supplies indicator were also in the highest tertile for the supervision indicator (Kasungu, Blantyre), and three of these higher performing districts for drugs and supplies were in the lowest supervision tertile (Nkhata Bay, Rumphi, and Likoma).
Supervision appears to occur more frequently in districts with NGO partner support (  Successful supply chain systems did not appear to depend on the support of large NGOs, as several districts without major NGO implementing partner support were among the top performers in this area (Nkhata Bay, Rumphi, and Likoma). Since 2011, the MOH has been rolling out a district-level supply chain system management tool called cStock (see Box 1). Among the nine top-performing districts for indicator 3a, five were supported by the new system (Mzimba North, Nkhata Bay, Kasungu, Nkhotakota, and Nsanje); all of these except Mzimba North had been using cStock since 2011 (i.e., early adopters) ( Table 6). Five of the nine worst performing districts for indicator 3a were also implementing cStock although all had adopted the system more recently. Unexpectedly, Rumphi had neither a major NGO implementation partner nor cStock, but still reached the 80% MOH target for indicator 3a.
Districts with consistently higher scores demonstrate that MOH implementation strength targets are achievable with the right combination of supportive structures. Overall, Kasungu was the most consistently high-scoring district for the core implementation strength elements including proportion of iCCM-trained HSAs providing services as well as the supply and supervision related indicators. Kasungu also ranks in the highest tertile for proportion of HSAs reporting working in HTRA and proportion of HSAs residing in their catchment area. Utilization was also high in Kasungu. It is notable that Kasungu is perhaps a unique best-case scenario in terms of government and partner support. The district was one of the original rapid scale-up districts that received WHO and UNICEF support early in implementation, and a pilot district for cStock. Kasungu had stable district-level management and consistent support from other NGO partners since 2008. It was also one of two districts that participated in a U.S. Agency for International Development-funded pilot project focused on improving capture, reporting, and utilization of iCCM monitoring data by HSAs. The project included site visits to Kasungu and Dowa districts between February and May 2013. 12 However, Nsanje, Phalombe, and Balaka had a similar set of factors (other than the pilot data project), but did not have same consistently high results across the implementation strength indicators. One possible explanatory factor is that compared with Kasungu, these three districts are located further from the capital city Lilongwe and potentially received less frequent oversight from the more senior central MOH and NGO staff.
The cell phone methodology worked well in Malawi. Malawi's cellular signal coverage rates are quite high, with more than 93% of areas covered. 13 We were able to reach 99.7% of HSAs identified by the MOH. More than half of these HSAs were reached within a single call attempt, and 93% were reached within four call attempts. There are limitations that might influence the interpretation and application of our findings. As designed, the study provides a snapshot of iCCM program implementation at a specific point in time. The indicators related to iCCM training and supply chains are particularly responsive to time-sensitive events, including training cycles (e.g., additional training could have taken place just weeks before or after the survey), and high-level supply chain issues (e.g., national-level stockouts). The cell phone interview approach was validated against supervisor reports and written supervision records, which themselves are subject to reporting bias. In the validation study, the sensitivity and specificity of the supply indicators were lower than others but all indicators were above the 80% threshold we set for adequacy. 3 According to the national MOH, there were no system-wide interruptions in iCCM drug supplies during the time of the survey. The survey was subject to MOH-level limitations around identifying the official HTRA HSA catchment areas, and therefore not able to report on the official indicator 2b. The survey does not include household-level reports of service delivery, and it does not capture whether iCCM utilization by households is adequate compared with the expected burden of disease and access to health facilities. A large household survey was conducted by the NSO shortly after the census of HSAs, and includes some data that might help answer these questions. Despite these limitations, the cell phone methodology produced muchneeded and consistently collected information on iCCM implementation strength across all districts, which was put to use immediately by the MOH and partners to improve program implementation. Work is under way now to compare the results of the implementation strength snapshot to those routine monitoring reports collected from implementing partners, to determine whether there are systematic biases in the routine systems that might be able to be addressed. The scale-up of the cStock system, which was in pilot phase during this study, to all districts in 2014 should improve the completeness of routinely reported supply indicators in Malawi. We recommend the telephone method for use in other contexts that lack strong routine or real-time monitoring systems, or as a tool to help periodically assess the quality and completeness of routinely collected data, although there may be limits in settings where cell phone coverage is less complete than in Malawi.