Setting health research priorities using the CHNRI method: I. Involving funders

In 2007 and 2008, the World Health Organization's Department for Child and Adolescent Health and Development (later renamed as WHO MNCAH – Maternal, Newborn, Child and Adolescent Health) commissioned five large exercises to define research priorities related to the five major causes of child deaths for the period up to the year 2015. The exercises were based on the CHNRI (Child Health and Nutrition Research Initiative) method, which was just being introduced at the time [1,2]. The selected causes were childhood pneumonia, diarrhoea, birth asphyxia, neonatal infections and preterm birth/low birth weight [3–7]. The context for those exercises was clearly defined: to identify research that could help reduce mortality in children under 5 years of age in low and middle income countries by the year 2015. The criteria used in all five exercises were the “standard” CHNRI criteria: (i) answerability of the research question; (ii) likelihood of the effectiveness of the resulting intervention; (iii) deliverability (with affordability and sustainability); (iv) potential to reduce disease burden; and (v) effect on equity [3–7]. 
 
The five criteria used by the scorers were intuitive as they followed the path from generating new knowledge to having an impact on the cause of death. They were chosen with a view to identifying research questions that were most likely to contribute to finding effective solutions to the problems. However, after the five exercises – all of which were published in respected international journals [3–7] – the WHO officers were left with an additional question: how “fundable” were the identified priorities, ie, how attractive were they to research funders? More specifically, should another criterion be added to the CHNRI exercises, which would evaluate the likelihood of obtaining funding support for specific research questions? 
 
To answer these questions, coordinators of the CHNRI exercises at the WHO agreed that it would be useful to invite a number of representatives from large funding organizations interested in child health research to take part in a consultation process at the WHO. The process aimed to explore funders perspectives in prioritization of health research. The funders would be presented with the leading research priorities identified through the CHNRI exercises and asked to discuss any potential variation in their likelihood of being funded. If all the leading priorities were equally attractive to funders and likely to attract funding support, this would indicate that the “standard” CHNRI criteria were sufficient for the process of prioritization. However, if there were large differences in attractiveness of the identified research priorities to funders, then adding another criterion to the exercise – “likelihood of obtaining funding support”, or simply “fundability” – would be a useful addition to the standard CHNRI framework.

were published in respected international journals [3][4][5][6][7] -the WHO officers were left with an additional question: how "fundable" were the identified priorities, ie, how attractive were they to research funders? More specifically, should another criterion be added to the CHNRI exercises, which would evaluate the likelihood of obtaining funding support for specific research questions?
To answer these questions, coordinators of the CHNRI exercises at the WHO agreed that it would be useful to invite a number of representatives from large funding organizations interested in child health research to take part in a consultation process at the WHO. The process aimed to explore funders' perspective in prioritization of health research. The funders would be presented with the leading research priorities identified through the CHNRI exercises and asked to discuss any potential variation in their likelihood of being funding. If all the leading priorities were equally attractive to funders and likely to attract funding support, this would indicate that the "standard" CHNRI criteria were sufficient for the process of prioritization. However, if there were large differences in attractiveness of the identified research priorities to funders, then adding another criterion to the exercise -"likelihood of obtaining funding support", or simply "fundability" -would be a useful addition to the standard CHNRI framework. Having explained the aims of the consultation meeting to the representatives of funding agencies, the 16 participants were presented with a list of the top 10 research priorities for each of the five major causes of child deaths: pneumonia, diarrhea, birth asphyxia, neonatal infections and preterm birth/low birth weight [3][4][5][6][7]. This set of 50 research priorities represented roughly the top 5% of all the research ideas submitted for scoring during the CHNRI exercises.
The WHO coordinators (RB and JM) explained each of the 50 leading research priorities to the 16 donor representatives. Then, the 16 donor representatives were provided with the list of research priorities and asked to individually identify those that were most likely to receive funding support from their respective organizations.
Funding attractiveness was measured in two ways. First, funder representatives were asked to rank the identified research priorities according to their likelihood to receive funding support under an organization' s current investment policies and practices. Second, funding attractiveness was measured by asking funder representatives to distribute a theoretical US$ 100 among the research priorities that seem most fundable. Results were used to facilitate discussion on what makes a research question attractive (or unattractive) for funding support. The scoring sheet that was given to meeting participants is shown in Figure 1. While they did not need to provide their name or organization, they were asked to assign ranks 1-10 to the ten research priorities identified for each of the five causes of death (column 1), and also to distribute a hypothetical US$ 100 to different research priorities in concordance to the likely funding support that they may obtain.
Sixteen participants scored the identified research priorities according to the instructions (Figure 1). The average ranks across the 16 participants (1 = most likely to be funded; 10 = least likely to be funded) assigned to the 50 research priorities ranged from 3.7 to 7.2. The average US$ amount assigned to research priorities ranged from US$ 20.1 to US$ 2.5. There was general consistency between ranks and the US$ assigned to research priorities.
In 2007 and 2008, the World Health Organization's Department for Child and Adolescent Health and Development commissioned five large research priority setting exercises using the CHNRI (Child Health and Nutrition Research Initiative) method. The aim was to define research priorities related to the five major causes of child deaths for the period up to the year 2015. The selected causes were childhood pneumonia, diarrhoea, birth asphyxia, neonatal infections and preterm birth/low birth weight. The criteria used for prioritization in all five exercises were the "standard" CHNRI criteria: answerability, effectiveness, deliverability, potential for mortality burden reduction and the effect on equity. Having completed the exercises, the WHO officers were left with another question: how "fundable" were the identified priorities, i.e. how attractive were they to research funders? Importantly, the analysis of the collective input based on the 2nd column (ie, assigned US$), presented in Figure 2, clearly shows that there was a rather substantial departure of the assigned funds from that expected at random: if all research priorities were equally likely to obtain support from the funders, then all the bars would be extending only to the line that represents an investment of US$ 10.0. Furthermore, 4 research priorities (8%) clearly stood out from the rest [8]. It was agreed that they might provide a starting point from which MNCAH Department could concentrate its efforts. These 4 research priorities are shown in Table 1. Table 1. The 4 research priorities (8%) that were identified as positive outliers in terms of their likelihood to obtain funding support Evaluate the quality of community workers to adequately assess, recognize danger signs, refer and treat acute respiratory infections (ARI) in different contexts and settings. What are the barriers against appropriate use of oral rehydration therapy? What are the feasibility, effectiveness and cost of different approaches to promote the following home care practices (breastfeeding, cord/skin, care seeking, handwashing)? What are the feasibility, effectiveness and cost of a scheme of routine home visits for initiation of supportive practices, detection of illness and newborn survival?  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50

AN ANALYSIS OF THE EXERCISE WITH FUNDER REPRESENTATIVES
The results were analysed after the first day of the meeting and presented to donor representatives at the beginning of the second day of the meeting. An open discussion was held with participants to understand and interpret the results of their collective input. Participants agreed that the most important criteria for research prioritisation differed between researchers and funders. Researchers tended to value answerability, effectiveness, deliverability, impact on the burden and equity. Funders were also interested in the clarity and specificity of research ideas, value for money, novelty, international competitiveness of the groups proposing the research, linkages to broader societal issues, and complementarity with other long-term strategic investments that were already made. An important point in the discussion was that researchers and research funders, especially those in the private sector, often speak quite different languages. Researchers need to be clear on what their goals are and communicate these in more readily understood terms. This point is particularly important because it implies that the CHNRI exercises' research priorities that were identified as most likely to generate useful new knowledge may not be considered equally relevant by the funders.
This should certainly be taken into account when presenting and discussing the results of the CHNRI exercises.
Moreover, there seem to be important differences between the categories of funders in the criteria that they use to decide on research priorities. Generally, all investors in health research are concerned with answerability of the proposed research ideas in an ethical way, feasibility and value for money. However, some may be particularly interested in potential for forming partnerships between researchers and industry to increase the translation of findings and their application. Ministries and international organizations appeared more interested in deliverability, affordability and sustainability of the resulting interventions, local and national research capacities to carry out the proposed research ideas, and whether a research question is linked to an ongoing public debate or an important societal issue. Industrial donors may be primarily motivated to generate patents and translate research results into commercial products. Finally, society as a whole may be more concerned with issues of safety and equity issues and ask whether implementation of research results would widen the existing socioeconomic gaps..
Transparency of research priority setting processes must, therefore, begin with those who invest. Perceived returns on investments in health research should be clearly stated at the beginning of the process. They may be defined as reduction in disease burden wherever public money is being invested. Investors from industries may see patentable products as their preferred returns. Non-profit organizations may be primarily interested in increased media attention for their agenda. The context in which investment prioritization takes place is thus primarily defined by expected returns of the funders. Moreover, their investment styles may be balanced and responsible (suggested for those investing public funds), risk-averting (which may be preferred among some industrial partners) or risk-seeking and biased towards high risk -high profit avenues of health research (which may be typical for some industry and notfor-profit organizations).
Apart from funders' perceived returns and their investment styles, the population, geographic area and disease burden of interest, the time frame in which returns are expected is an important defining component of the overall context. Priorities can differ substantially if the overall context is one of great urgency to tackle a problem, or whether decisions are made on very long-term, strategic investments.

CONCLUSIONS
The meeting with research funders organized by the WHO MNCAH department in March 2009 was exceptionally useful in understanding that funders certainly have their own views on what represents an attractive funding option.
In March 2009, WHO officers invited 40 representatives from organizations that provide substantial funding support for global child health research to take part in a consultation process at the WHO. The process aimed to explore funder's perspective in prioritization of health research. Eventually, 16 funders' representatives agreed to take part in the exercise under the condition of anonymity. Participants agreed that the most relevant criteria for prioritisation differed between researchers and funders. Funders are interested in clarity and specificity of research ideas, value for money, novelty, international competitiveness of the groups proposing the research, links to broader societal issues, and complementarity with other long-term strategic investments that they have already made. Some may be particularly interested in the potential for forming partnerships between researchers and industry to improve the translation of findings and their application Those views are not generalizable and may differ between categories of funders. Moreover, funders' perspectives are often quite different from those of researchers, or wider stakeholder groups. It is important to involve funders early in the process of setting research priorities, such as the CHNRI process, to encourage their ownership of the results. Funder-supported criteria must be taken into account, in addition to those preferred by the researchers and wider stakeholders. Otherwise, the outcomes of research prioritization exercises may have very limited impact on funders' decision making.
The key value of the CHNRI method to funders lies in its ability to transparently lay out the potential risks and benefits associated with investing in many competing research ideas, drawing on collective knowledge of the broad research community. Results of the CHNRI process represent an attempt on the part of researchers to communicate their views and opinions to funders in a way that is easily understood, transparent, replicable and intuitive. It provides useful additional information that funders may, or may not take into account when deciding on their own research agenda. From a methodological perspective, finding appropriate and effective ways of involving funders in future CHNRI exercises, communicating the outcomes clearly, and securing their commitment to acknowledge the results of the CHNRI process remain considerable challenges. An even greater challenge in future years will be to develop tools that can detect and evaluate the impact of CHNRI exercises on funder decision making and any change in funding priorities as a direct result of the CHNRI process. This should be particularly relevant to those who make decisions about investing public funds, whose primary agenda should be improving public health in the most cost-effective waya target that CHNRI exercises should serve quite well.