The effect of scientific evidence on conservation practitioners’ management decisions

A major justification of environmental management research is that it helps practitioners, yet previous studies show it is rarely used to inform their decisions. We tested whether conservation practitioners focusing on bird management were willing to use a synopsis of relevant scientific literature to inform their management decisions. This allowed us to examine whether the limited use of scientific information in management is due to a lack of access to the scientific literature or whether it is because practitioners are either not interested or unable to incorporate the research into their decisions. In on-line surveys, we asked 92 conservation managers, predominantly from Australia, New Zealand, and the United Kingdom, to provide opinions on 28 management techniques that could be applied to reduce predation on birds. We asked their opinions before and after giving them a summary of the literature about the interventions’ effectiveness. We scored the overall effectiveness and certainty of evidence for each intervention through an expert elicitation process—the Delphi method. We used the effectiveness scores to assess the practitioners’ level of understanding and awareness of the literature. On average, each survey participant changed their likelihood of using 45.7% of the interventions after reading the synopsis of the evidence. They were more likely to implement effective interventions and avoid ineffective actions, suggesting that their intended future management strategies may be more successful than current practice. More experienced practitioners were less likely to change their management practices than those with less experience, even though they were not more aware of the existing scientific information than less experienced practitioners. The practitioners’ willingness to change their management choices when provided with summarized scientific evidence suggests that improved accessibility to scientific information would benefit conservation management outcomes. El Efecto de la Evidencia Científica sobre las Decisiones de Manejo de Quienes Practican la Conservación Resumen Una justificación mayor de la investigación en el manejo ambiental es que ayuda a quienes lo practican, aunque estudios previos muestran que rara vez se usa para informar sus decisiones. Probamos si quienes practican la conservación enfocada en el manejo de aves estaban dispuestos a usar una sinopsis de literatura científica relevante para informar sus decisiones de manejo. Esto permitió que examináramos si el uso limitado de información científica en el manejo se debe a una falta de acceso a la literatura científica o si se debe a que quienes practican la conservación no están interesados o no son capaces de incorporar la investigación a sus decisiones. En encuestas en línea les preguntamos a 92 practicantes de la conservación, la mayoría de Australia, Nueva Zelanda y el Reino Unido, que nos proporcionaran opiniones sobre 28 técnicas de manejo que podrían aplicarse para reducir la depredación de aves. Les pedimos sus opiniones antes y después de darles un resumen de la literatura sobre la efectividad de las intervenciones. Calificamos la efectividad general y la certidumbre de la evidencia para cada intervención por medio de un proceso de extracción por expertos – el método Delphi. Usamos las calificaciones de la efectividad para evaluar el nivel de entendimiento y de percatación de la literatura de quienes practican la conservación. En promedio, cada participante de la encuesta cambió su probabilidad de usar 45.7% de las intervenciones después de leer la sinopsis de la evidencia. Fue más probable que implementaran intervenciones efectivas y evitar acciones poco efectivas, lo que sugiere que sus estrategias de manejo futuras puedan ser más exitosas que las de práctica actual. Los practicantes con mayor experiencia tuvieron una menor probabilidad de cambiar sus prácticas de manejo que aquellos con menos experiencia, aunque no estuvieron más conscientes de la información científica existente que quienes tenían menos experiencia. La disponibilidad de los practicantes para cambiar sus opciones de manejo al proporcionárseles evidencia científica resumida sugiere que el acceso mejorado a la información científica podría beneficiar los resultados del manejo de la conservación.

It has been applied in various scientific, political and decision making situations when accurate data are not available, -when expert judgement is the only source of reliable information, and when the true answer to a question is unknown.

Why are we using expert judgement and the Delphi process?
We need a panel of experts for this study to interpret the evidence for the effectiveness of management interventions to reduce bird predation. The average of several experts' opinions is likely to be more reliable and accurate than the opinion of a single expert.
Other methods to elicit expert opinion involve participatory discussion before reaching a consensus. This may bias results towards opinions of the dominant, more experienced members of the expert panel.
Instead, in the Delphi process members of the expert panel remain anonymous throughout scoring. Final results are influenced by the quality of scores and comments made by other experts, rather than by dominant personalities and social judgements.
Using an online platform allows people from around the world to participate in the expert panel, engaging with people from a wide range of backgrounds. There is evidence that including a breadth of backgrounds enhances the benefits of the Delphi process.

What is involved?
The Delphi process improves the judgement of each expert and reduces the variance between the individual scores of the panellists, converging on the 'truth' or the agreed answer.
The Delphi process involves two to three rounds of anonymous scoring. After making an initial judgement, experts are shown a summary of the results and comments from others in the expert panel. They are then asked to reassess their scoring and justify their decisions to change or retain the same scores.

What do we mean by 'effectiveness' of an intervention?
This is a difficult concept, as there are a range of scales and interpretations upon which we could measure the effect of an intervention at reducing predation on birds. For example, effectiveness could be measured by the frequency of it working, the magnitude of positive change, the number of specific species or bird groups it targets and its time and spatial scales.
In addition, the type of impact an intervention has on bird species is important to consider in the effectiveness measure. An intervention that has been shown to increase the population of a species or diversity of a bird community is more 'effective' than an intervention that increases hatchling and reproductive success or decreases rates of predation. We also believe that an intervention is more effective if there is a visible impact on the bird species of concern, rather than a decrease in the predator's population, presence or hunting ability.
We want you to take all of these factors into account and score the effectiveness of an intervention, based on the evidence that is provided in the Bird Conservation Synopsis. The 'effectiveness' scale is from -10 to 100. A rating of -10 is for interventions that are harmful, that cause increases in bird predation or have tradeoffs with other conservation objectives.
You can slide the pointer along the scale to the rating that you think is most accurate (see example below). If the evidence in the synopsis is insufficient to make a decision and you are unable to rate the effectiveness of an intervention, it is perfectly acceptable to tick the 'Uncertain' box at the end of the slider.
It is important to only consider the evidence that is available in the synopsis. Please try to base your judgement on the published literature, while ignoring your past experience. We understand that this may be difficult and slightly unrealistic in a management scenario. However, it is an essential for the purposes of this research. If an important study is missing from the synopsis, please mention it in the comments section (Question 3, see below).

Meaning -10
Harmful or trade-offs 0 Ineffective 25 Slightly effective 50 Moderately effective 75 Very effective 100 Always effective Uncertain Insufficient evidence

What do we mean by 'certainty of evidence?
The 'certainty of evidence' is quite different to the effectiveness measure. It is a rating of how certain you are about the effectiveness score you gave. It is a combination of the quality and quantify of evidence provided for each intervention and the similarity of results across different studies.
The quality of a study is determined by the experiment design. For example, studies that use randomised, replicated controlled experiments with large sample sizes are of higher quality than studies that use before and after comparisons or single site trials.
We will ask you to rate the certainty of evidence for each intervention on a scale of 0-100%. Try to answer the question by thinking about how much more evidence do we need to be 100% certain of the effect of this intervention.
To rate the certainty of evidence for each intervention, move the pointer to the score that you think is most appropriate. If you have comments relating to an intervention, please add them in Question 3 (see below).

Rating Meaning 0%
No useful evidence available 25% Little quality evidence, low certainty 50% Some quality evidence, moderate certainty 75% A lot of quality evidence, high certainty 100% Fully resolved with high quality evidence

Providing useful rational statements and comments
We will then ask you for comments about each intervention. These will form an important component of the summary of results which we give you in the second round of scoring. This is your opportunity to tell others on the expert panel about a relevant piece of information that is not included in the synopsis or to provide a reason for why you scored the way you did. This will be anonymous and your name will not be given out to other experts on the panel.
The comments should be -stated clearly so that others on the expert panel will understand them -based on causal arguments, rather than personal opinion or preference -providing information or other studies that are not captured in the bird synopsis Comments that do not contribute any valuable insight to the debate will not be included in the summary report. This will ensure that experts are not influenced by uninformative comments and encourage experts to fully engage and provide useful justifications, which will ultimately improve the accuracy of the results collected through the Delphi process.
If you rate an intervention higher or lower than the majority of experts, your rational will be used to defend your position in the second round. On the other hand, if your rating for an intervention is within the majority of scores, your rational should be used to influence the experts on the peripheries to move towards the centre.