Evaluating the Impact of Environmental Interventions during a Global Quarantine

Introduction

The COVID-19 crisis has turned impact evaluators' lives upside down: suddenly, we can't visit the sites we are responsible for, interview participants in person, or otherwise conduct assessments as we normally would. As with much of the world economy, this necessitates a period of unprecedented creativity to ensure that the critical role impact evaluation serves is not left by the wayside for the duration of the crisis. In this brief overview of an article recently published in the journal Sustainability, we showcase one way we’re engaged with continuing operations while under quarantine by using satellite and other geospatial information, combined with other existing survey data collected by international organizations.  In the era of COVID-19 where field work is halted and working from home is the new normal, methods like those presented in this piece can help evaluators leverage open source and readily available data to determine the impact of projects. Working with geospatial data can help project evaluators maintain flexibility and encourages them to move away from designing new and often expensive data tools for every unique impact evaluation. While the method we explore - Quasi-experimental Geospatial Interpolation (QGI) - may not be a perfect fit for every evaluation, it provides a novel, replicable approach to the estimation of environmental and socioeconomic outcomes.

Case Study - The Global Environment Facility in Uganda

We recently used QGI to measure the impact of GEF Sustainable Forest Management projects on household income in Uganda between 2009 and 2011. QGI is especially useful in not only determining the socioeconomic benefits of GEF projects, but it also provides novel information on the distance from a project that effects might be visible.  For example, in a traditional evaluation an expert might go to the village in which a project was implemented and talk to local stakeholders about the effectiveness of a given intervention.  While this provides critical information, it does not also examine the impact an intervention might have had on neighboring villages, without a great deal of extra work.  QGI seeks to estimate this based on satellite and other sources of geospatial information.

Geospatial Data

In this study, we used geospatial data such as land surface temperature and precipitation, while also integrating household surveys to help fill in data gaps. We used the results of the 2009 and 2011 World Bank Living Standards Measurement Survey to measure the increase in household assets between 2009 and 2011 associated with Sustainable Forest Management project interventions. Our analysis included 33 project locations within Uganda.

Our method is a matching-based approach which sorts households into control and treatment groups based on their distance from a project location. We test the relationship between distance from a project location and change in household assets by iteratively adding households to our treatment group as we increase the distance from a project location. While traditional methods of GIE allow households to switch between treatment and control groups based on the current distance being tested, we established a maximum distance of 50km, and all households more than 50km from a project location became our permanent control group. Every household in the treatment group was coupled with a single household in the control group based on their environmental similarity (i.e. similar annual rainfall, distance to roads, population).

Findings

Using QGI, we were able to determine that GEF Sustainable Forest Management projects were associated with an increase in household assets by approximately $180 dollars between 2009 and 2011. Evidence was strongest for areas between approximately 2 and 7km away from a GEF project location. Areas over 7km away lacked the data needed to draw a reliable conclusion. In total, we tested 480 distance bands between 0 and 5km from project locations.

Conclusion

Geospatial data provides a unique window into the effectiveness of environmental interventions during a time of quarantine: satellites can continue their work with at least 9 feet of separation. Further, combining this with other rich survey data, the relationship between environmental outcomes and other socio- economic variables, can be explored for insights into possible correlates. While there are a great number of questions regarding the relative precision of geospatial approaches to impact evaluation given it’s nascent nature, we argue that this time of quarantine offers a great opportunity to explore their potential further.