Rethinking Evaluation - Sustaining a Focus on Sustainability

Looking back on years of using the sustainability evaluation criterion, one has to ask - how well have we done? The UN Sustainable Development Goals (SDGs) have brought renewed attention to sustainability. Although the DAC evaluation framework includes sustainability as one of its five criteria, looking back on years of using this DAC evaluation criterion, one has to ask - how well have we done? And here I mean in evaluation practice rather than results.

More often than not have I seen sustainability used in different ways than it was originally conceived. The definition - "[s]ustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable" (OECD/DAC key terms for evaluation) - focuses clearly on the outcomes of the intervention and their sustainability.

Many evaluations, however, assess whether the projects themselves will be sustained, often concluding this to be the case when funding is secured from government or another donor. That is right, especially for facilities that continue to be run by the public sector and require government funding. But, the same is true for public-private partnerships (PPPs) where a recent evaluation of ours showed that the impact on government expenditure (in other words: fiscal sustainability) was hardly ever assessed.

Sustainability is also often taken as synonymous with environmental sustainability. When I was leading project evaluations, we hardly ever had the time and resources to assess environmental impacts and whether a project would leave a lasting footprint, positive or negative. At IEG we are in the process of evaluating environmental pollution projects the World Bank Group, which will shed some light on past practices, including data that is available today and remaining gaps.

But, under the SDGs environmental sustainability goes much further than a "simple" question of pollution. It is about the use and depletion of natural resources, about consumption patterns that are out of bounds, and the distribution of consumption patterns. For instance, when we look at access to electricity, our recent evaluation showed how underserved countries especially in Africa are. Sustainable Development Goal 7 is committed to expanding access to affordable and clean energy, increasing renewable energy sources, and attaining energy efficiency as a measure of improved consumption patterns. But, it will not be sufficient for evaluations of the power sector to assess efficiency gains that must be achieved in other parts of the economy.

And, closely linked to the World Bank Group's goals of poverty reduction and greater shared prosperity, there is the question of social sustainability. Upheavals during the past years have often been rooted in growth that has excluded a broader base, where wealth, access, and voice have been captured by the few. The commitment to inclusive growth necessitates that we understand better the distributional effects of interventions, whether they were designed to target groups previously excluded or not. Almost more important for us is to evaluate and understand interventions that we believe to have no distribution effects to shed more light on the actual distribution of results that they have. IEG is in the process of evaluating the World Bank Group's experience in this area to generate some early insights.

Taken together these dimensions of sustainability – economic, fiscal, environmental, and social – are complex. It will be hard and costly to try to address them systematically in all evaluations. At the same time, we evaluators cannot afford turning up with empty hands and concerns about missing data. We need to debate how we would evaluate interventions through these lenses of sustainability, stimulate that the right questions are asked during the design of interventions, and incentivize that relevant data starts being collected.

The blog post originally appeared on the World Bank IEG blog.