Improving credibility and transparency of conservation impact evaluations through the partial identification approach
van Wilgen, B.J.
Format Extent777359 bytes
MetadataShow full item record
The fundamental challenge of evaluating the impact of conservation interventions is that researchers must estimate the difference between the outcome after an intervention occurred and what the outcome would have been without it (counterfactual). Because the counterfactual is unobservable, researchers must make an untestable assumption that some units (e.g. organisms or sites) that were not exposed to the intervention can be used as a surrogate for the counterfactual (control). The conventional approach is to make a point-estimate (i.e. single number plus a confidence interval) of impact, using for example regression. Point-estimates provide powerful conclusions, but in non-experimental contexts they depend on strong assumptions about the counterfactual that often lack transparency and credibility. An alternative approach, called partial identification, is to first estimate what the counterfactual bounds would be if the weakest possible assumptions were made. After this, the researcher narrows the bounds by using stronger but credible assumptions that are based on an understanding of why units were selected for the intervention and how they might respond to it. We demonstrate this approach, and show how it compares to conventional approaches, by estimating the impact of a program that removed invasive trees in part of the Cape Floristic Region. We found that even when we used our largest partial identification impact estimate, the program’s control costs were 1.4 times higher than previously estimated. Partial identification holds promise for applications in conservation science because: (1) it encourages researchers to better understand and account for treatment selection biases ; (2) it can offer insights into the plausibility of conventional point-estimate approaches; (3) it could reduce the problem of advocacy science; (4) where impacts are contentious, it might be easier for stakeholders to agree on a bounded estimate than a point estimate; and (5) it only requires basic arithmetic knowledge.