Strategies for Efficient Computation of the Expected Value of Partial Perfect Information

Expected value of information methods evaluate the potential health benefits that can be obtained from conducting new research to reduce uncertainty in the parameters of a cost-effectiveness analysis model, hence reducing decision uncertainty. Expected value of partial perfect information (EVPPI) provides an upper limit to the health gains that can be obtained from conducting a new study on a subset of parameters in the cost-effectiveness analysis and can therefore be used as a sensitivity analysis to identify parameters that most contribute to decision uncertainty and to help guide decisions around which types of study are of most value to prioritize for funding. A common general approach is to use nested Monte Carlo simulation to obtain an estimate of EVPPI. This approach is computationally intensive, can lead to significant sampling bias if an inadequate number of inner samples are obtained, and incorrect results can be obtained if correlations between parameters are not dealt with appropriately. In this article, we set out a range of methods for estimating EVPPI that avoid the need for nested simulation: reparameterization of the net benefit function, Taylor series approximations, and restricted cubic spline estimation of conditional expectations. For each method, we set out the generalized functional form that net benefit must take for the method to be valid. By specifying this functional form, our methods are able to focus on components of the model in which approximation is required, avoiding the complexities involved in developing statistical approximations for the model as a whole. Our methods also allow for any correlations that might exist between model parameters. We illustrate the methods using an example of fluid resuscitation in African children with severe malaria.

Expected value of information methods evaluate the potential health benefits that can be obtained from conducting new research to reduce uncertainty in the parameters of a cost-effectiveness analysis model, hence reducing decision uncertainty. Expected value of partial perfect information (EVPPI) provides an upper limit to the health gains that can be obtained from conducting a new study on a subset of parameters in the costeffectiveness analysis and can therefore be used as a sensitivity analysis to identify parameters that most contribute to decision uncertainty and to help guide decisions around which types of study are of most value to prioritize for funding. A common general approach is to use nested Monte Carlo simulation to obtain an estimate of EVPPI. This approach is computationally intensive, can lead to significant sampling bias if an inadequate number of inner samples are obtained, and incorrect results can be obtained if correlations between parameters are not dealt with appropriately. In this article, we set out a range of methods for estimating EVPPI that avoid the need for nested simulation: reparameterization of the net benefit function, Taylor series approximations, and restricted cubic spline estimation of conditional expectations. For each method, we set out the generalized functional form that net benefit must take for the method to be valid. By specifying this functional form, our methods are able to focus on components of the model in which approximation is required, avoiding the complexities involved in developing statistical approximations for the model as a whole. Our methods also allow for any correlations that might exist between model parameters. We illustrate the methods using an example of fluid resuscitation in African children with severe malaria. Key words: value-of-information; Bayesian methods; cost-effectiveness analysis. (Med Decis Making 2014;34:327-342) V alue of information (VoI) methods provide a coherent decision-theoretic approach to research prioritization. 1,2 When making a decision under imperfect information, it is possible that the expected optimal decision is wrong, and this possibility will be associated with an expected loss. VoI methods define the value of research in terms of the expected reduction in this expected loss resulting from the additional information gained. This is derived from the expected impact of the proposed study on parameter uncertainty. It is therefore natural to take a Bayesian perspective where prior beliefs about parameters are ''updated'' by incorporating new evidence to form posterior beliefs. 3 The expected loss given current information is also known as the expected value of perfect information (EVPI), because it is the amount the decision maker should be willing to pay to eliminate all parameter uncertainty in the decision. 1,2 Where there are multiple sources of uncertainty, it is possible to calculate the EVPI on a subset of focal parameters, termed the expected value of partial perfect information, or alternatively, expected value of perfect parameter information (EVPPI). EVPI and EVPPI provide an upper bound to the value of any proposed future study. Several authors have highlighted the use of these methods in health technology assessment in relation to sensitivity analysis and research prioritization, and we refer interested readers to this literature for a more complete discussion of the role of value-of-information analysis in health technology assessment. 4,5 EVPPI involves an inner expectation nested within an outer expectation, which can be challenging to estimate; a common approach is nested Monte Carlo simulation. 6 There are 3 main issues with the nested simulation approach. First, obtaining estimates of adequate precision can be computationally expensive. 7 Second, estimates will have significant upward bias if insufficient simulation sample sizes are used for the inner loop. 8 Third, it can be difficult to generate unbiased EVPPI estimates when correlations exist between parameters, unless the joint distribution of the correlated parameters takes a known parametric form. 7 Conditions for the net benefit function have been identified that, if satisfied, allow EVPPI to be calculated in a single step by replacing the nonfocal parameters with their expected means. 9 In this article, we extend the work of Ades and others, 9 describing methods that can be used to avoid nested simulation in a broad range of situations, through careful consideration of the structure of the net benefit function and the relationships between its parameters. We illustrate these methods using a case study involving fluid resuscitation in African children with severe malaria, followed by a discussion of the advantages of the methods compared with other approaches and limitations resulting from the assumptions made.

Motivating Example
Malaria accounts for up to 1 million deaths annually in children younger than 5 years living in Sub-Saharan Africa. Children hospitalized with cerebral malaria (coma) have increased case fatality (18%-21%), 10 and survivors are at heightened risk of long-term neurological sequelae (NS). 11,12 There are clinical arguments suggesting that fluid resuscitation could reduce mortality in African children. 13 Furthermore, pilot studies, involving 3 fluids (saline, albumin, and gelofusine), provided limited but promising support for this hypothesis. [13][14][15] Following these studies, a large multinational trial, the FEAST trial, was initiated to provide definitive evidence on the efficacy of fluid resuscitation. 16 We developed an economic model to assess the cost-effectiveness of fluid resuscitation strategies, taking a health services perspective in which the relevant costs and benefits are those relating to the individual patient (irrespective of who actually provides the funds for health care). The model is based on the assumptions given in Table 1, which were informed by consultation with clinical experts advising the FEAST trial. Assumption A6, used to derive the probability of NS in those receiving fluids, requires further explanation. If fluid resuscitation reduces mortality, there will be a subgroup of survivors who would have died without fluids. Assumption A6 allows for the possibility that this subgroup is more severely affected by malaria than those who would A10: Long-term NS is associated with a reduction in quality-adjusted life expectancy of Q S and increased discounted lifetime management costs of C S . These values are assumed to be the same irrespective of fluids received. A11: Parameters are assumed to be independent unless correlations are induced between them through joint estimation from a common data source.
have survived without fluids and therefore are at greater risk of NS. Note that we do not assume this a priori; the model allows for the possibility that fluids increase mortality or that improved survival is associated with reduced NS risk. There are 4 possible outcomes for patients in the model: death, NS-free survival, survival with shortterm NS, and survival with long-term NS. Table 2 lists the probability of each outcome with or without fluid resuscitation, and a full list of model parameters is given in Table 3. To assess the cost-effectiveness of treatment j, we calculate its incremental net benefit B j (u) relative to no fluid, that is, the additional costs incurred and health utility gained above those observed without fluid resuscitation. This includes the cost C F j of fluid j and additional costs and benefits related to changes in the proportion achieving each of the outcomes in Table 2, that is, where u = a; d M j ; q M ; q S ; p L ; d S n o ; W = willingness to pay per quality-adjusted life-year. This model (eq. 1) was designed to estimate the cost-effectiveness of fluids once FEAST trial results were available. However, it can also be used in an EVPPI analysis to provide insight on the potential value of the FEAST trial and other studies that could be undertaken to inform parameters in eq. 1. This requires quantifying the uncertainty around the parameters implied by information available at the time the FEAST trial was designed (Table 3). Values for C H , C F , and q M were based on health economic analysis carried out by several of the authors alongside the FEAST trial. 16 Data on NS-related economic parameters are sparse; our chosen values are toward the upper end of what were considered plausible values, for reasons that are explored in the Discussion section. Table 3 also lists the type of future studies that can provide information on each of the uncertain model parameters. We assumed that baseline mortality in a randomized controlled trial (RCT) population would be different from that in the decision populations, so that a different study type would be needed to provide information on baseline and relative effect parameters. In practice, if the 2 populations are considered sufficiently similar, all of these parameters O3: Survives, NS at 28 d but not long-term could be estimated from an RCT. Below, we present generic methods that can be used to estimate EVPPI in a wide range of circumstances. We then illustrate their application for the fluid resuscitation example by considering the different future studies that we could run (Table 3) and the subset of parameters those studies would directly inform.

METHODS
Let B j ðuÞbe a general form for the incremental net benefit of intervention j given model parameters u and j* be the optimal decision given current information. We define j as a subset of (focal) parameters on which we are considering collecting further information on in a new study, and j C as the set of remaining (nonfocal) model parameters. EVPPI j ð Þ equals the expected gain in net benefit, given current information, from switching to the optimum treatment once j is known with certainty: We describe 2 types of methods that can be used (if necessary in combination) to avoid the inner-simulation step when calculating EVPPI j ð Þ, which all involve finding a solution (exact or approximate) for the inner expectation E  This method is applicable when the net benefit takes the form where j C i ,the ith component of j C , is independent of any of the elements of j in its linear coefficient f i; j j ð Þ. Given this independence, it has been shown 4,9 that EVPPI j ð Þcan now be estimated in a single Monte Carlo simulation by substituting equation 3 into equation 2.

Method 2: Direct Substitution of Means of the Nonfocal Parameters in a Multilinear Net Benefit Function
This method is applicable when the net benefit can be expressed in the form where each function g i; j j C À Á is a product of mutually independent elements of j C and each of the elements of g i; j j C À Á is independent of the elements of j in its linear coefficient f i; j j; j ð Þ.
If these conditions are satisfied, it has been shown 4,9 that the expected value of each function g i; j j C À Á will equal the product of the expectations of its components, so that EVPPI j ð Þcan now be estimated in a single Monte Carlo simulation by substituting equation 4 into equation 2.

Method 3: Reparameterization to Linearize the Net Benefit Function
This method is applicable when the net benefit can be expressed in the form are any functions of the j C parameters and the elements of j C in each b i; j j C À Á are all independent of any of the elements of j in its linear coefficient Method 2 is not applicable if any of the b i; j j C À Á are not products of their components and/or include correlated nonfocal parameters . However, the second condition ensures the independence of each b i; j j C À Á from its linear coefficient f i; j ðjÞ, which implies that EVPPI j ð Þcan now be estimated in a single Monte Carlo simulation by substituting equation 5 into equation 2. This method has been used by Welton and others. 17 Methods Involving Functions Approximating the Conditional Expectations of j C

Method 4: Taylor Series Approximations to the Net-Benefit Function
This method is applicable when net benefit takes the form where f i; j ðjÞare arbitrary functions of the focal parameters and h i; j j C i ; j À Á are arbitrary smooth nonlinear functions of j and a single element are arbitrary functions of the remaining j C parameters (excluding j C i ), and j C i , the elements of j in h i; j j C i ; j À Á , and the elements of j C in b i; j j C Ài À Á are mutually independent.
Because j C i , j C Ài and the elements of j in h i; j j C i ; j À Á are mutually independent, we can decompose the expectation: The expectation of each b i; j j C Ài À Á , if required, can be found using method 3. Because h i; j j C i ; j À Á is nonlinear, its expectation conditional on j will not equal h i; j E j C i À Á ; j À Á , and we cannot use methods 1 to 3. Our strategy is instead to construct a function of E j C i À Á that approximates the required conditional expectation and so can be used to replace the inner simulation step. One approach is to use Taylor series expansions. 9 The nth-order Taylor series expansion of h i; j j C i ; j À Á in the neighborhood of E j C i À Á is given by As long as j C i is independent of the elements of j in h i; j j C i ; j À Á , taking conditional expectations on both sides gives This can be used to give an approximation to the conditional expectation of the net benefit function; for example, a second-order approximation is EVPPI j ð Þcan now be estimated in a single Monte Carlo simulation by substituting equation 6 into equation 2.
The form of the approximation will depend on the functional form of h i; j j C i ; j À Á . For example, net benefit functions commonly consist of an absolute probability that is informed by parameters on the log-odds scale (baseline log-odds plus a log-odds ratio for treatment j), so that h i; j j C i ; j À Á is the inverse-logit function of the form The second-order Taylor series approximation for the inverse-logit function is (see supplementary appendix): Occasionally, the absolute probability is informed by parameters on the cumulative log-log (clog-log) scale, so that function h is the inverse-clog-log function of the form In this case, it can be shown that the second-order Taylor approximation gives (see supplementary appendix) When using Taylor series approximations, it is important to assess the accuracy of the approximation and to consider adaptations that improve accuracy, for example, the addition of higher-order terms. We have developed the following adaptation for this purpose. We split the prior distribution for j C i into quantiles, Q, and find the mean, m Q , of j C i conditional on lying in quantile Q. We then construct separate Taylor series approximations to h i; j ðj i C ; jÞ within each quantile around the quantile-specific mean, m Q . It can be shown that the expected value of the average over this set of Taylor series approximations converges to the required quantity, E j C jj hðj; j C i ; jÞ Â Ã (details and proof in the supplementary appendix). Figure 1 illustrates the accuracy of Taylor series expansions in the case in which j i C is normally distributed, h i; j ðj i C ; jÞ takes the form given in equation 6, and E[j 1 j C i ] = 2. The accuracy of approximations based on Taylor series expansions around E[j C i ] reduces as the variance of j C i increases, and the fourth-order approximation is noticeably more accurate than the second-order approximation. However, averaging over secondorder expansions around the interquartile means is more effective in improving accuracy than increasing the order of the single approximation. The Taylor series gives local approximations whose accuracy diminishes as the distance from the expansion point increases. Therefore, averaging over multiple expansions around a range of values reduces error in the approximation by restricting the range over which each approximation is made, and this is our recommended approach.

Method 5: Spline-Approximation Methods When
There Are Correlations between Parameters This method is applicable when the net benefit takes the form where f i; j ðjÞ are arbitrary functions of the focal parameters, h i; j j C ; j i À Á are arbitrary smooth nonlinear functions of j C and a single element j i of j, and for at least one value of i, j i is correlated with one or more of the elements of j C in h i; j j C ; j i À Á .
Here we can no longer use Taylor series expansions to estimate the conditional expectation of h i; j j C ; j i À Á in terms of the unconditional expectation of j C , because elements of the latter are correlated withj i . Instead, we aim to construct functions j i; j j i ð Þ whose values are close approximations to E EVPPI j ð Þ can then be estimated in a single Monte Carlo simulation by substituting equation 8 into equation 2.
An obvious choice for j i; j j i ð Þis a spline. 18 There are several different types of splines; we describe here the use of restricted cubic splines for this purpose. The restricted cubic spline j i; j j i ð Þ takes the form where t 1 ; . . . t N t f gare the N t knots that define the breakpoints of j i; j j i ð Þ. The parameters g are chosen so that j i; j j i ð Þ is linear for values of j i less than t 1 or greater than t Nt . This requires a minimum of 3 knots. Furthermore, j i; j j i ð Þ must be continuously differentiable, which places further restrictions on the g. To construct these splines, we first require estimates of E for a large set of values for j i spanning a plausible range. Once knots t 1 ; . . . t N t f ghave been chosen, regression analysis can be used to derive coefficients g defining the splines that fit the estimated conditional expectations most closely. This approach can be implemented by the following algorithm.
Step 1: Generate N u samples from the joint distribution of u and estimate h i; j ðj C ; jÞ for each sample and each intervention j. These should be the same set of samples that will subsequently be used to estimate EVPPI j ð Þ.  Taylor series approximations for the expectation of the inverse logit of (j 1 j C i ), where j is known, j C i is normally distributed, and E[j 1 j C i ] = 2. Pluses mark true values for the expectation as the standard deviation of j C i varies, and approximations are based around a single expansion around the mean (second-and fourth-order approximation) and averaging over approximations derived at each interquartile mean (second-order approximation).

EVPPI Calculations for the Fluid Resuscitation Example
We describe below, for the fluid resuscitation example (eq. 1), how EVPPI can be calculated for various different subsets of parameters, representing different potential future studies (Table 3), using the appropriate method described in the Methods section. We assume that we have a set of simulated samples from the joint prior distribution for all of the uncertain parameters.

Example 1: Net Benefit Linear in Nonfocal Parameters
Suppose we wish to evaluate the value of running 2 studies: 1) a 4-arm RCT (such as FEAST) to provide information on relative effects, d M ; d S , and 2) a cohort study with 6-mo follow-up to provide information on a; p B ; p L . In this case, the focal parameters (i.e., those for which we wish to calculate the EVPPI) Method 1 can also be used when j C 5 q M ; p L È É or when j C equals any single element of q M ; q S ; p L È É .

Example 2: Net Benefit Multilinear in Nonfocal Parameters
Suppose we wish to evaluate the value of running 3 studies: 1) a 4-arm RCT (such as FEAST) to provide information on relative effects, d M ; d S , 2) a cohort study with 28-d follow-up (or routine data) to provide information on a; p B , and 3) a long-term cohort study on patients who survive without NS to provide information on q M . In this case, This satisfies the conditions that must be met to use method 2 for single-step Monte Carlo estimation of EVPPI. Therefore, we can apply equations 2 and 4 to restate EVPPI j ð Þ in a form requiring only a single simulation step: Method 2 can also be used when j C 5 q M ; q S ; p L È É .

Example 3: Net-Benefit Linear in Functions of the Nonfocal Parameters
Suppose we wish to assess the value of running a cohort study with long-term follow-up, including patients with and without NS, to estimate persistence rates and long-term quality of life.
Here j 5 q M ; q S ; p L È É , so that j C 5 d M ; a; p B ; d S n o . The net benefit function is no longer multilinear in j C , so neither of methods 1 or 2 is valid. However, by defining the net benefit can be expressed as This satisfies the conditions that must be met to use method 3 for single-step Monte Carlo estimation of EVPPI. Therefore, once the unconditional expectations E b 1; j j C À Á Â Ã and E b 2; j j C À Á Â Ã are known, we can apply equations 2 and 5 to restate EVPPI j ð Þ in a form requiring only a single simulation step: Method 3 can also be used when j is any subset of q M ; q S ; p L È É .

Example 4: Net Benefit Includes Joint Nonlinear Functions of Independent Focal and Nonfocal Parameters
Suppose we wish to evaluate the value of running a 4-arm RCT (such as FEAST) to provide information on relative effects, d M ; d S . Here, j 5 fd M ; d S g, so that j C 5 a; p B ; p L ; q S ; q M È É . In this case, by defining the net benefit can be expressed as Because a and p B are independent and d M and d S are known, the terms h 1 , h 2 , b 1 , and b 2 are mutually independent. Therefore, we can write Ã (a function of of a and d M j ) and E h 2 j; p B À Á Â Ã (a function of p B and d S ) can be approximated using Taylor series expansions, as described in the Methods section (method 4). Method 4 can also be used when j 5 a; p B È É .

Example 5: Net Benefit Includes Joint Nonlinear Functions of Correlated Focal and Nonfocal Parameters
Suppose we wish to evaluate the value of collecting routine data on mortality under current practice (i.e., without fluid resuscitation). Here, j 5 a f g, so that the net benefit can be expressed as Because d M and d S are correlated, none of the methods previously described can be used to estimate can be constructed (method 5) such that Then, EVPPI[ a ] can be approximated by The method described for this example can also be used when j is any single element of d S ; d M ; p B È É .

Evidence Synthesis and Cost-Effectiveness Analysis
We carried out a Bayesian evidence synthesis, using WinBUGS version 1.4.3, 19 to estimate posterior distributions for the treatment effect parameters d M and d S based on the data from the 3 pilot studies (details in the supplementary appendix). A Markov chain Monte Carlo (MCMC) simulation with a burnin of 50 000 (the number of samples required for convergence of multiple chains) followed by simulation of 5 million parameter value sets (chosen based on the memory limits of the computer) was used to represent joint parameter uncertainty, which forms the prior for the EVPPI analysis. Table 4 gives posterior parameter means and correlations, based on the pilot studies listed in supplementary appendix A2. The impact of saving a life on the subsequent risk of NS, d S , is correlated with the effect of each treatment on mortality, d M .
The treatment effects on mortality are also correlated with each other. This is because d S and d M are jointly estimated from the pilot studies, where the observed proportion of NS depends on both d S and d M (supplementary appendix A2). No other correlations exist between model parameters because they are estimated from independent data sources.
Albumin is the only treatment for which the 95% credible interval for the log-odds ratio relative to no fluids does not include 0 (no effect). Appropriate cost-effectiveness thresholds in an African context are less well established than in regions such as the United Kingdom. We therefore explored a range for the willingness-to-pay threshold (WTP) of 0 to 4000 USD/quality-adjusted life-year (QALY) when calculating net benefit. Based on the pilot data and the synthesis model, the most cost-effective option is albumin if WTP is .$269/QALY and gelofusine otherwise (there are no choices of threshold for which saline or no fluid is the optimal treatment). Around the ''switching'' threshold of $269/QALY, there is considerable uncertainty about the most cost-effective treatment. For example, although gelofusine has the highest expected net benefit at a threshold of $250/QALY, there is only an 18% probability that it is the most cost-effective at that threshold, compared with 54% for albumin, 7% for saline, and 21% for no fluids. Results are more certain at higher thresholds; at $1000/QALY, for example, albumin has the highest net benefit with a 92% probability.

Value of Information
All calculations were performed using the software package R (version 2.15). For the effectiveness parameters d M and d S , these samples were generated using MCMC simulation from the evidence synthesis model. For the remaining parameters, samples were generated in R directly. Computations were performed on a PC with an Intel i5 processor and 3 Gb of RAM.
The EVPI is highest around the threshold at which the optimal treatment changes, where the decision is most uncertain and sensitive to additional information ( Figure 2). Another influence on the EVPI/WTP relationship is that, at low threshold values, NS costs are high enough for additional information on NS-related parameters (q S , d S , p L , p B ) to potentially change the decision. This effect can be verified by calculating the EVPPI for different model parameters. Table 5 illustrates EVPPI estimates for different subsets of parameters, each calculated in a single simulation step using the methods described above, assuming a WTP of $250/QALY. Methods that involve replacing nonfocal parameters or functions of them, with their unconditional means in the net benefit function (methods 1-3), can be used to estimate the benefit of knowing individual parameters with certainty in 3 cases: p L , q M , and q S . This benefit is large for the first 2 parameters but 0 for q S (the QALY loss associated with persistent NS).
The accuracy of EVPPI estimated using methods 1 to 3 depends on the precision of the estimates of the unconditional parameter means, which in turn will depend on the size of the sample generated from the posterior distribution. The point at which sample size is sufficient can be assessed by plotting the relationship between the estimate of EVPPI and the sample size used to derive it. Figure 3 illustrates the number of outer simulations needed for EVPPI estimates to converge in the case where j 5 q M and compares nested and single-step simulation. Sample sets of increasing size were used to estimate EVPPI q M À Á using the 1-step approach of method 3 and the 2step approach with 1000 or 2500 inner simulations for each outer simulation. The (upward) bias resulting from an inadequate number of inner simulations   can be clearly seen: with 1000 inner samples per outer simulation, the bias is about 3% of the total EVPPI. Nested simulation is also far slower than 1-step simulation; Table 5 gives computation times using each approach, showing the former to be at least 25 times slower when it is based on 1000 simulations  in the inner loop, and no allowance is made for correlations between parameters (any attempt to incorporate correlations into nested simulation would further increase the speed advantage of the 1-step approach). Methods 4 and 5 involve constructing functions that give approximate values for conditional means. The accuracy of EVPPI estimates will therefore depend on the goodness of fit of the approximating function, which can be explored graphically. Figure  4 illustrates Taylor series approximations for , the expected mortality with gelofusine conditional on perfect information for 1) a known treatment effect d M 4 and 2) a known baseline mortality a. In the first case, the second-order approximation is extremely accurate and cannot be discernibly improved by higher-order terms or the use of interquartile means. In the second case, the second-order approximation is noticeably inaccurate, the fourthorder approximation is still inaccurate but less so, and the approximation based on interquartile means is extremely accurate. However, using the more accurate approximation changes estimated EVPPI by only 1%, from $345 to $342 per person (the latter figure, based on the more accurate approximation, is the one quoted in Table 5). Figure 5 illustrates, in the case where j 5 d S (example 5), how increasing the number of knots in a spline improves its accuracy, although the difference between 10 and 15 knots is negligible. We found that estimates of EVPPI were extremely stable as the number of knots increased; results in Table 5 were calculated using 10 knots, although results with 6 knots differed by less than 1% in all cases.

DISCUSSION
We have presented a range of methods to avoid the need for nested simulation in EVPPI calculations. The advantages of a 1-step simulation strategy for the computation of EVPPI include unbiased estimation and computational efficiency. It is well known that the inner simulation step can be avoided when the net benefit function is linear or multilinear in the focal parameters (methods 1 and 2). 4,9 We have extended these methods to a broader range of net benefit functions by identifying functions of the focal parameters in which the net benefit function is (multi)linear (method 3), which has been used previously, 17 but the conditions in which the method can be used have not been set out before, and it is not commonly employed. This may be because it requires the net benefit function implied by the economic model to be stated explicitly and to be manipulated algebraically, whereas EVPPI analyses commonly treat the model as a ''black box'' generating net benefit estimates. Ades and others 9 proposed the use of Taylor series approximations to the inverse-logit to approximate conditional expectations. We have set out the situations in which a Taylor series approximation can be used in general (method 4), adapted the approximation for the inverse-logit to improve accuracy, and given the formulae for other commonly used functions. We are unaware of any previous work that has proposed using spline techniques (method 5).
Our use of Taylor series expansions (method 4) and splines (method 5) can be seen as conceptually similar to existing statistical meta-modeling approaches that have been advocated in place of nested simulation, such as Gaussian process emulation. 20 Tappenden and others 21 reviewed potential meta-modeling methods and noted that although they may permit flexible estimation of EVPPI with reasonable accuracy, these methods require substantial specialist expertise. The key difference between meta-modeling and our approximation techniques is that we model components of the net benefit function, rather than the entire economic model. As a result, the methods we present require less technical expertise to implement and review. Furthermore, these methods may be useful for models with a large number of parameters, when Gaussian process emulation can be unfeasible. 7 The accuracy of approximations to the conditional expectation of the net benefit function should always be checked by comparing approximate with actual values over a plausible range of the focal parameters, as we illustrate in Figure 4. We found that Taylor series expansion around the unconditional mean of the nonfocal parameter could be markedly inaccurate if that parameter had a high variance, even if higherorder terms were included in the expansion. For such situations, we outline an approach averaging over Taylor series expansions around each of the interquartile means, which greatly improves the accuracy of this approach. We found spline regressions were able to provide close approximations to the conditional mean, particularly if enough knots were chosen. The accuracy of spline approximations is largely driven by the accuracy of the bin means to which they are fitted and the coverage of bin means across the plausible range of the focal parameter. In particular, simulated values will be sparse for values of the focal parameter that are less likely. If the optimal treatment changes for these values, it is particularly important to consider the accuracy of the spline approximation of the expectation conditional on these values. The density of conditional mean estimates across the range of the focal parameter can be increased by using more bins, although this will lead to larger sampling errors for each bin mean if the total number of samples is kept constant. The number of samples N u used for spline estimation should be large enough to support enough bins across the plausible range of the focal parameter while ensuring sufficiently low standard errors for each bin mean.
A particular challenge for EVPPI calculations occurs when correlations exist between components of j C and components of j. 5 In this situation, values at each inner simulation need to be sampled from the conditional distribution of j C ; otherwise, EVPPI estimates will be downwardly biased. One approach would be to carry out a separate MCMC simulation for each realization from the outer simulation. This will give samples of j C from the correct distribution but at considerable computational expense. A more efficient approach may be to assume a parametric joint distribution for the correlated parameters, from which a conditional sampling distribution can be derived at each inner simulation. However, this may introduce an additional source of bias. Onestep estimation of EVPPI using spline approximations allows us to side step this problem. The purpose of VoI is to guide decisions on research prioritization, so that it is necessarily conducted in situations in which current information is inadequate and assumptions may be proven quite wrong by later research. The goal is to reflect as fairly as possible current information, however sparse that might be. The credibility of VoI estimates therefore depends on the credibility of the cost-effectiveness model on which they are based and how accurately prior distributions reflect prestudy parameter uncertainty, which requires guidance from clinicians and decision makers regarding assumptions made in the model and plausible alternatives. Decision makers might require the model to be extended to allow for different severities of NS or the health utility loss due to short-term NS to be included. We have assumed fixed treatment effects across studies and used pilot study data to estimate them. These data were the only evidence available to us and were insufficient to estimate less restrictive models; thus, it is possible that we have underestimated uncertainty in treatment effect estimates. Furthermore, use of pilot study data may introduce bias, as their design and conduct may not be as rigorous as a full RCT. Very little information was available on the long-term costs and health consequences of NS, and the values chosen for this analysis were therefore not evidence based but chosen to ensure nonzero EVPPI estimates for these parameters. When existing evidence is sparse and/or of poor quality, as with long-term NS in our case study, it is important to be able to somehow characterize the extent of the uncertainty, as VOI will be highly sensitive to this. One possibility is to incorporate formal expert elicitation of plausible parameter values, together with opinion on the possible degree of bias in the existing evidence base. 22 Finally, we would advocate the use of sensitivity analyses to explore the sensitivity of VoI results to modeling assumptions, 17 as would be conducted as standard practice for cost-effectiveness analysis. 23 The focus of this article is to illustrate methods for 1-step estimation of EVPPI. Because of the limitations set out above, we do not intend to draw conclusions about research priorities in the area of fluid resuscitation. Nevertheless, the case study illustrates how EVPPI calculations can be informative when considering the relative merits of efficacy trials compared with longer-term epidemiological studies or when selecting outcome measures and treatments to include in a proposed trial. For example, there was considerable debate following the pilot studies around the inclusion of albumin in the FEAST trial, driven by perceptions that this treatment was too costly for use in Africa. Our EVPPI analysis suggests that there was considerable value to including this arm, although the caveats mentioned above apply. The results from the FEAST trial have recently been published, 16 with robust but unexpected findings (harm was demonstrated in the fluid resuscitation arms).
A key limitation of our approach is that the methods rely on being able to explicitly state the net benefit function implied by the model. We would argue that there are clear benefits in explicitly setting out the net benefit function, in terms of transparency, understanding, and communication of assumptions, as well as the reduced computational burden for VoI calculations, and would always recommend doing this unless the decision model is unavoidably too complex to allow it. We have presented the use of univariate splines in our case study, which can be used only if there are no nonlinear functions of multiple correlated focal parameters in the net benefit function. When this is not the case (for example, if we were to consider 2-or 3-arm trials in the fluid replacement example), then the spline approach (method 5) could be implemented through the use of multivariate adaptive regression splines. 7 In this article, we have focused on EVPPI; however, the methods presented here can also be used in the calculation of the expected value of sample information (EVSI). EVSI is more computationally intensive than EVPPI because it requires an extra layer of simulation required to simulate new data from a given new study design. Furthermore, if we wish to optimize over potential new study design factors, then EVSI will need to be evaluated repeatedly, which will substantially increase the computational burden. Ades and others 9 set out algorithms for the estimation of EVSI in a number of situations, all of which require nested Monte Carlo simulation unless the net benefit function satisfies the conditions required for use of methods 1 and 2. The methods presented here extend the circumstances in which we can avoid the inner simulation step for EVSI, hence improving computing times and thereby making the use of EVSI feasible in a wider range of situations.