Quality Assurance and Quality Control in the Global Trachoma Mapping Project

Abstract. In collaboration with the health ministries that we serve and other partners, we set out to complete the multiple-country Global Trachoma Mapping Project. To maximize the accuracy and reliability of its outputs, we needed in-built, practical mechanisms for quality assurance and quality control. This article describes how those mechanisms were created and deployed. Using expert opinion, computer simulation, working groups, field trials, progressively accumulated in-project experience, and external evaluations, we developed 1) criteria for where and where not to undertake population-based prevalence surveys for trachoma; 2) three iterations of a standardized training and certification system for field teams; 3) a customized Android phone–based data collection app; 4) comprehensive support systems; and 5) a secure end-to-end pipeline for data upload, storage, cleaning by objective data managers, analysis, health ministry review and approval, and online display. We are now supporting peer-reviewed publication. Our experience shows that it is possible to quality control and quality assure prevalence surveys in such a way as to maximize comparability of prevalence estimates between countries and permit high-speed, high-fidelity data processing and storage, while protecting the interests of health ministries.

1 asked countries to prepare a first draft of the survey protocol local adoption of a methodology without local understanding of why each of its elements was important failure to build local capacity.
The GTMP did this in each of its constituent projects.
imposition of locally inappropriate survey elements if local partners feel inhibited about challenging a template difficulties in survey implementation.
2 ensured that the draft survey protocol was consistent with WHO recommendations, working with the health ministry and local partners to refine the draft as needed international inconsistency prevalence estimates that could not be compared between settings.
The GTMP did this in each of its constituent projects.
3 assisted countries to frame EUs of appropriate sizes (generally at the level of the local administrative unit for health care management and ideally containing framing of inappropriately large EUs potential to miss significant pockets of disease. In Yobe State, Nigeria, a previous population-based trachoma prevalence survey covered a population of > 2 million people in 2 populations of 100,000-250,000 persons 35 ) a single EU. 43 framing of inappropriately small EUs excessive use of resources for mapping, or extrapolation of results from a small EU to provide prevalence estimates for a larger population. 44 In the Solomon Islands, the regions of Rennell-Bellona (estimated population 3041) and Temotu (estimated population 21 362) had sufficiently similar socio-economic and environmental characteristics to be combined to form a single EU. 31 4 assisted countries to design epidemiologically valid cluster selection methods within EUs selection of too many clusters excessive use of resources for mapping.
Due to a misunderstanding, twice the required number of clusters were selected and visited in one GTMP-Mozambique EU that had been formed by combining 2 adjacent districts. We subsequently included explicit discussion of the implications of EU formation on cluster selection in our conversations with health ministries.
selection of too few clusters potential to miss significant pockets of disease.
In some trachoma prevalence surveys conducted prior to the GTMP, 7-14 clusters were selected for inclusion. [45][46][47] selection of clusters using a biased generation of inaccurate All GTMP-supported surveys applied epidemiologically- Within the GTMP, 3 EUs in Chad and 3 EUs in Egypt had to be resurveyed because field teams only examined 1-9-year-olds; in Cambodia, only households in which 1-9-year-olds lived were enrolled 50 ; in Viet Nam, only 1-9year-olds and ≥50-year-olds were enrolled. 49 failure to examine children 47 inability to estimate a meaningful prevalence of trachomatous inflammation-follicular.
The GTMP did this in each of its constituent projects.
creation of an incentive for household residents very keen to be examined, or very keen not to be examined, to misrepresent their age bias in prevalence estimates.
In recent trials of a trachomatous trichiasis-only survey methodology, when only those aged ≥40 years were examined, unexpectedly large numbers of individuals claiming to be aged 40-45 years were enrolled. 51 10 supported health ministries to obtain local ethical clearance before surveys started neglect of locally important ethical considerations in survey design failure to "take into consideration the laws and regulations of the country or countries in which the research is to be performed as well as applicable international norms The GTMP did this in each of its constituent projects. 6 and standards", as required by the Declaration of Helsinki. 52 SUPPLEMENTAL 1 closely reviewed budgets against the agreed methodology and a standard budget template and ensured that the methodology was consistently reflected in the budgeting assumptions 53 essential activities omitted, or excessive resources requested to undertake mapping mapping activities not aligned with agreed methodology, inefficient use of resources, or shortfall in funding with consequent failure of one or more of the GTMP's constituent project.
Resources to support field team supervisors added to budgets in several projects. In one country, excessive requests were trimmed, resulting in a budget reduction of 28%.
2 provided Android smartphones for survey teams, with (if possible) survey forms already pre-loaded possible local purchase of phones with outdated versions of the Android operating system and/or difficulties in ensuring correct software installation delays in commencing surveys, or reversion to the use of paperbased data collection.
The GTMP tried to do this in each of its constituent projects. Where it did not (because, for example, it was difficult to import phones, or lead times were too short), phone cost was often higher, and internet bandwidth occasionally made it challenging to download survey software. 7 3 provided binocular, 2.5× magnifying loupes for graders to use lack of provision of loupes by programmes; or provision of loupes that were uncomfortable to wear, prone to breakage, or of the wrong magnification failure to use loupes, or use of loupes with the wrong magnification, leading to reduction in diagnostic accuracy.
The GTMP did this in each of its constituent projects. The GTMP supported grader trainees from Lao People's Democratic Republic 59 and Cambodia 50 to be trained in part of the training process then asked to identify [55][56][57][58] Ethiopia, at the invitation of the Oromia Regional Health Bureau.
8 only deployed graders who had demonstrated their trachoma grading competency through formal inter-grader agreement exercises on real subjects, using the assessments of a GTMPcertified grader trainer 10 as the gold standard deployment of graders whose competency had been assessed only through grading of slides or photographs of trachoma [55][56][57][58]60 uncertainty about grader competence.
The GTMP did this in each constituent project, other than Viet Nam, 49 where travel of grader trainees to a more highly endemic country could not be undertaken.
Though not allowing previouslyexperienced graders who did not pass the test to continue was controversial at the beginning, it was subsequently seen as an important demonstration of how important quality was to the GTMP. In some contexts, however, managing disappointed grader trainees became an issue in its own right. 1 deployed in-service supervisors, each of whom were required to first pass the formal inter-grader agreement exercises on real subjects, using as the gold standard the assessments of a GTMPcertified grader trainer lack of appropriate supervision b drift in accuracy of grading over time; errors in application of fieldwork protocol; or unreported social, economic, health or supply issues that could adversely affect field team performance.
The GTMP did this in each of its constituent projects.
2 rapidly reviewed raw data over-estimation of the mean number of residents per household in available census data failure to examine a number of individuals in each EU that would permit calculation of prevalence estimates with acceptable precision. c In Southern Nations, Nationalities and Peoples' Region of Ethiopia, 66 we requested that 4 clusters be added to an EU because there were too few 1-9-year-olds examined. b We intend to further improve the standard and consistency of supervision in trachoma impact and surveillance surveys, through the use of a dedicated training package for supervisors c We did not keep a sufficiently close eye on this issue early in project implementation. (In Oromia, for example, there were 2 EUs in which 651 and 653 1-9-year-olds were initially examined, but because of delays in data upload, teams had moved to other zones before this came to light.) Pre-GTMP surveys which stipulated a given number of subjects to be examined per cluster did not run this risk, but instead risked biased selection.
13 under-estimation of the mean number of residents per household in available census data examination of more individuals in each EU than necessary to permit calculation of prevalence estimates with acceptable precision, leading to inefficient use of resources. d In Guinea, where 23 clusters were included per EU, the range of 1-9year-olds examined per EU was 1113-3137.
3 telephoned field supervisors as soon as a record of trachomatous trichiasis in a child was identified erroneous recording of the presence of trachomatous trichiasis in a child potential mobilisation of a paediatric ophthalmologist or oculoplastic surgeon to provide service; or if undetected, tacit encouragement of a lack of concentration in the field.
In raw data from 55 projects, 519 cases of trichiasis were reported amongst 1 146 644 1-9-year-olds; 249 of those cases were confirmed when checked with field teams. 4 discussed and resolved fieldwork problems as they arose uncertainty, confusion, inconsistency between teams reductions in the accuracy and/or repeatability of prevalence estimates.
The GTMP did this in each of its constituent projects. d We did not do this well enough. Pre-GTMP surveys which stipulated a given number of subjects to be examined per cluster did not run this risk, but instead risked biased selection.
14 SUPPLEMENTAL failure to ensure high-fidelity transfer of data from paper to electronic format for the purposes of data analysis, and/or long delays while paper-based surveys are managed (photocopied, double entered, etc.) data used to generate prevalence estimates not reflecting the findings observed in the field, and/or long intervals between survey completion and programmatic decision-making.
The GTMP did this in each of its constituent projects.
2 ensured that the LINKS-GTMP app did not permit fields to be skipped e failure to collect or record available data missing data, and uncertainty in analyses.
The GTMP did this in each of its constituent projects.
3 included "don't know" and/or "other" options in all multiple choice questions data recorders being forced to stop, enter junk data or use a parallel reporting system junk data, loss of system integrity or inability to proceed.
The GTMP did this in each of its constituent projects. 4 used check screens requesting recorders to verify data just entered lack of flagging of entry of rare outcomes, such as trachomatous missed opportunities to correct erroneous keystrokes at source.
The GTMP did this in each of its constituent projects. e The dangers inherent in allowing skip fields are illustrated by the GPS data collected (for each household) in the GTMP. Because GPS signals are sometimes difficult or impossible to access, LINKS-GTMP allows recorders to proceed to the next question without collecting GPS coordinates, which teams were instructed to press only after the Android had tried to triangulate its location for at least 60s without success. In one project, a group of recorders rapidly developed a habit of always skipping the GPS field. In our next-generation app developed for Tropical Data, skipping GPS data collection is only possible after system-driven timeout. trichiasis 5 restricted responses to sensible rangesage, for example, could only be recorded as 1-100 years; for those reporting an age at last birthday of > 100 years, 100 years was recorded errors potentially uninterpretable data. The GTMP did this in each of its constituent projects.
SUPPLEMENTAL The GTMP did this in each of its constituent projects.
2 applied 128-bit encryption at the transport layer, 10 and carried out subsequent data review, cleaning and approval only through a secure website with transport layer security, IP-restricted firewall, and site authentication and authorization, to which access could be gained only through password-protected login.
paper-based or electronic data containing personally identifiable information stored without adequate security potential inadvertent disclosure of personally identifiable information on survey subjects, violating standard ethical principles for investigators.
The GTMP did this in each of its constituent projects; for Tropical Data, 13, 14 256-bit encryption is being used, including encryption of data at rest.
3 went to extreme lengths to recover data stored on micro-SD cards that were, in the rare case, damaged in the field loss of data and potential loss of faith in the reliability of data storage on Android smartphones possible need to repeat surveys, and possible need to design a new system.
The GTMP lost data from 1 cluster (due to a lost Android), of a total of 12,631 clusters visited. Where Androids were damaged or corrupted, all stored data were 20 SUPPLEMENTAL decisions made using data not powered to provide the estimates generated.
The GTMP did this in each of its constituent projects.
2 calculated 95% confidence intervals for prevalence estimates by bootstrapping adjusted cluster-level proportions, with replacement, over 10,000 replications confidence intervals around prevalence estimates not calculated, not calculated using an appropriate methodology, or not reported assessment of precision of prevalence estimates made difficult.
The GTMP did this in each of its constituent projects.
3 provided epidemiologist support to local authors, as needed, to draft projectspecific manuscripts once each project was complete; and brokered agreement to publish the resulting papers, if accepted under the journal's normal criteria, in a series of supplements to a peer-reviewed journal, with financial support from the project to make those data not published in peerreviewed journals lower visibility of data and of local efforts.
There has been considerable interest in the published output of the GTMP, with a number of special issues of Ophthalmic