Evaluation of geospatial methods to generate subnational HIV prevalence estimates for local level planning

Supplemental Digital Content is available in the text


Introduction
Historically, country epidemics have been considered as fairly homogenous and been broadly classified as 'generalized' or 'concentrated' [1]. However, such an approach fails to capture the often substantial local level variation in the patterns of risk and transmission, key drivers of the epidemic, and availability of services observed [2]. Indeed, 13 of 33 countries in sub-Saharan Africa report at least five-fold differences in adult prevalence between provinces [3].
This spatial heterogeneity has profound implications for all aspects of monitoring the epidemic and planning the response. There have been a large number of examples of ways in which spatial data can be used to improve HIV planning, including identification of places or populations at highest risk [4], allocation of resources across locations [5], in understanding local level changes and monitoring the epidemic [6], in interpreting gaps in service provision [7], in understanding reasons for different biases in available surveillance data [8], and in tailoring services and local level targeting of intervention [9].
Although it is critical that planners have an understanding of local level variations in the intensity of the HIV epidemic, practical and financial constraints that restrict the size of surveys and surveillance systems commonly used to monitor generalized epidemics may inhibit this. Robust estimates of subnational HIV prevalence are typically available at the first administrative level (often termed 'provinces'), but not at more local levels which may be needed for programme planning (such as the district or county levels).
There are several candidate methods that could be used to generate local subnational estimates of HIV prevalence (Table 1). These include the method that the Joint United Nations Programme on HIV/AIDS (UNAIDS) has used to generate maps of high-burden countries [23] and a method which has been used to estimate malaria prevalence patterns [14,15]. However, the performance of these methods has not been evaluated or compared in the context of HIV epidemiology. Here, we conduct a formal evaluation and comparison of subnational HIV prevalence estimates generated by these candidate methods.

Methods
Six candidate methods were included in this study (labelled models 1-6 in Table 1). Key characteristics are described in Table 1.
Some mapping strategies use sampled HIV prevalence data from household surveys only and do not use ancillary spatial information (e.g. road network or vegetation coverage). PrevR (model 1) is such a method [10], and it has been recommended by UNAIDS on the basis of it being straightforward to implement and available for immediate use.
This approach is compared with methods that do leverage information on other features ('covariates') which can enhance predictions of HIV prevalence (data sources are described in Table S1, http://links.lww.com/QAD/ A890). Such data are particularly useful if they are sampled at a greater geographical resolution than the HIV prevalence data. Models 2-6 fall into this category. Among these, there are important differences in the theoretical framework behind them, which influence their ability to deal with uncertainty and the computational load (Table 1).
Finally, some of these methods estimate a continuous 'surface' of HIV prevalence (models 1-3), whereas others instead aim to provide predictions for the aggregate level, namely the subnational units under consideration directly, for example, a district level prevalence estimate (models 4-6).
Two different validation procedures were used to assess the performance of the methods, depending on whether the method produces a 'surface' of prevalence or can give estimates at aggregate subnational level. Data from three countries (Tanzania, Kenya, Malawi) with generalized epidemics were used. These countries were chosen as they encompass variation in epidemic patterns and data availability.

Internal validation
The performance of those models that produce continuous HIV prevalence surfaces (models 1-3) was assessed using internal validation. A proportion of observed data is held back as a 'test'dataset, before the method is used. Then, the test data are used to challenge the model prediction at sample locations. This test dataset was either a single point, namely a demographic and health survey (DHS) cluster [leave out one cross validation (LOOCV), (models 1 and 3)] or a larger proportion of the data [partitioned data hold back (PDHD), (models 1 and 2)]. Which strategy was applied (i.e. LOOCV or PDHD) for a given spatial method was dependent on the computational intensity of the mapping approach.
The resulting root mean squared error (RMSE) (Eq. 1) between prediction and data was calculated. The RMSE estimates are in the same units as the surface (i.e. prevalence) and the lower the RMSE, the closer the model prediction is to the observed data.
where E is the predicted values, O the observed values, N the total number of locations, and i each omitted location.

External validation (cross-year comparison of estimates)
External validation was also conducted through comparing mapped predictions with data from an earlier survey year. This approach takes advantage of the location of clusters often being different in different survey years in a country, and assumes that the true spatial pattern of HIV prevalence is conserved over time. All three countries considered have surveys for more than one DHS round The approach was repeated for all methods at the level of the first administrative unit for Malawi (district level), as not all methods produce continuous prevalence surfaces. For those methods which produced predicted surfaces (models 1-3), the average of the values of the surface within each district boundary was calculated, to allow for comparison with those methods which produce district level estimates directly (models 4-6). The accuracy of district level predictions in comparison to the data from the earlier survey year was summarized by the RMSE. Figure 1a presents the continuous prevalence surfaces for each country using models 1-3. Very substantial withincountry variations in HIV prevalence are revealed by all methods. In particular, methods indicate a prevalence gradient from east to west in Kenya, south to north in Malawi, and a focus of high prevalence in southwest Tanzania.

Results
There is substantial variation in the degree of local level variation suggested by the models. Methods that bring in additional data tend to produce estimates with a more complex spatial structure, which is related to road networks, among other factors, although uncertainty in the estimates about this is great (not shown). In contrast, the PrevR approach gives a smooth surface.
The internal validation procedure suggests that all methods can produce estimates of HIV prevalence at unsampled locations with a similar, and reasonable, level of accuracy (RMSE values: Fig. 1a). Among the methods, the Bayesian geostatistical approach (model 2), gives marginally the best RMSE values consistently across all the countries (Fig. 1: RMSE values displayed in the key for each panel).
The external validation for the continuous surfaces shows that the methods typically are successful in predicting prevalence in unsampled locations (Fig. 1b). The greatest difference in prediction error is between countries rather than between methods, and all methods have similar spatial pattern in the errors. Among models 1-3 (those that could do this test), model 2 gives the lowest RMSE across all countries. All methods give higher RMSE values than in the internal validation exercise described above as they are being used to predict the spatial distribution from a different year and the epidemic will have changed.
The external validation for the district level in Malawi (Fig. 1c), shows that whereas all the methods give the same broad spatial trends, differences in some districts are quite pronounced and overall errors are much greater. As prevalence may vary widely within an administrative region, particularly between urban and rural areas, an averaged value for each administrative region gives wider differences than the point by point comparison. Again, model 2 gives the lowest RMSE, and gives the most accurate prediction.

Discussion
Generating subnational estimates of HIV prevalence will be crucial to informing a locally tailored response to the HIVepidemic. This analysis has provided a number of key insights as to how countries can best utilise available spatial data.
First, the magnitude of the error and accuracy of predictions appears to depend most on the prevalence level in the country of interest and the characteristics of the survey sample, rather than the estimation method used. Because of this, we see a greater difference in the accuracy of predictions between countries rather than between methods. For this reason, some confidence in predictions from these spatial methodologies comes from their relative consistency in performance across all validation procedures described. The method used already by UNAIDS (model 1: PrevR) performs similarly to most other methods and so greater confidence can be afforded in the results. As a result of these analyses, we recommend that Bayesian geostatistical approach (model 2) be developed further, as the performance of this method was consistently the strongest. This method has been applied extensively to other infections [13], and has many desirable characteristics, in particular a formal accounting of uncertainty and the explicit leveraging of other geographic data.
Second, the methods appear to work reasonably well, and can capture the broad spatial trends in prevalence observed across countries. Arguably, in these high prevalence settings, these methods would usefully distinguish areas  of very high prevalence from those with very low prevalence.
The mapping methods described can be further developed, particularly through integration of different data sources alongside the DHS, in particular, antiretroviral therapy and prevention of mother-to-child transmission programme data, antenatal clinic surveillance, and, in the future, case-reporting data. Doing so would require building upon earlier work [8] to assess how different data sources may feed into prevalence mapping in a manner that reflects the different biases, underlying populations and spatial coverage of these data. Furthermore, although the tools outlined can fill critical gaps at this time, they do not mitigate the need for future additional local data collection and reporting to strengthen more localized responses to the HIVepidemic.
National epidemics cannot continue to be assessed as a whole when there is clear evidence of substantial subnational heterogeneity. Subnational indicators should be integrated into all national planning, monitoring, and evaluating processes performed routinely. Existing tools, such as Spectrum/Estimation and Projection Package modelling software, are already being adapted to explicitly examine the epidemic in subnational areas [24]. The increasing availability of georeferenced data and mapping tools provides us with the opportunity to be responsive to the subnational features of HIVepidemics to improve intervention planning.