Towards evaluation for a sustainable and just future

Over the past eight months, the novel coronavirus pandemic has infected some 20 million people and killed more than 700,000, sparing virtually no country. The economic and social consequences have been devastating. The virus SARS-CoV-2 that caused COVID-19 crossed over from its non-human host, probably a bat, directly or more likely through an intermediate host like a pangolin, to a human in or around the city of Wuhan in China in late 2019. The exact transmission mechanism is still not known but the root causes are clear. The spill-over of zoonotic viruses like SARS-CoV-2 is becoming more common as we come into ever closer contacts with animals, both domesticated and wild.  As human activities extend deeper into undisturbed ecosystems, undiscovered pathogens are released. The destruction is driven by the expansion of agriculture and cattle ranching, logging and deforestation, road construction, mining, new settlements and urban sprawl, making space for the growing human population and its ever increasing demands for raw materials, food stuffs and consumer goods.

Although COVID-19 in itself was not known, the coming pandemic was widely predicted by scientists and there were even government taskforces set out to prepare us for its eventuality. There were precedents—SARS, MERS, H1N1, Zika, Ebola and others—although their impacts were much more modest. COVID-19 spread like a wildfire in a globalized world—there were 3 billion airline trips taken in 2019—due to its characteristics of being airborne and contagious before infected persons become symptomatic.

What does any of this have to do with evaluation, you might ask. In my view, everything. And if not, what is the relevance of evaluation to the real problems of the world? The pandemic is an illustration of the kind of challenges we face today, how interconnected the world is, and how events in one place have global consequences. It also shows how economic development and environmental degradation are intimately intertwined. As we cut down trees, not only do we come into contact with lethal pathogens, but we also undermine the forest’s ability to sequester carbon thereby speeding up global warming. As people get richer, their diets tend to become more meat-based. There are now half a billion cows and 23 billion chicken on the planet. There is a patch the size of Denmark in the Amazon, which has been cleared to grow soy beans to feed pigs in Denmark. Another consequence of the increased meat consumption is higher rates of obesity, diabetes and cardiovascular diseases even in countries that previously didn’t experience them. A recent study by Harvard University provided strongest evidence yet linking air pollution directly to higher mortality. Human health and ecosystem health are inseparable.

The pandemic has affected different groups and communities differently. In the USA, Black, Indigenous and other People of Colour (BIPOC) have been disproportionately hit because they are more likely to be employed in essential jobs that cannot be done remotely, and their living conditions are more cramped. They may also have more pre-existing medical conditions rendering them more vulnerable to the virus. Climate change affects the poor and vulnerable communities hardest, whether it is those living on the low-lying coast of Bangladesh pummelled by more frequent cyclones and sea-level rise or small farmers in African drylands suffering during prolonged droughts.

Many evaluators write about these global challenges, using terms like ‘complex’ and ‘wicked,’ but I am not sure that the practice of evaluation has kept up with the theory. Evaluation as a profession has its roots in social inquiry, where we test the effectiveness of interventions on a well-defined treatment population against a control group. We may use experimental or quasi-experimental tools, or we may lean more towards more participatory and qualitative approaches, but either way the focus is on a single intervention and its effects. Our evaluations test the effectiveness of the intervention in terms of its pre-determined objectives. The desire is to be able to attribute any changes in the outcome to the intervention—or, recognizing the complexity and presence of multiple actors, at least the specific contributions of the intervention.

Apart from being narrowly project-focused, evaluations are still driven by donor concerns for accountability and ‘value for money.’ This treats the central question as a matter of simple accounting instead of a choice between types of intervention or organization that can, say, lift the largest number of people out of poverty with the least amount of money. To make things worse, the accounting in development cooperation is for the purposes of the donors and their priorities, not for the benefit of the claim-holders that the project is intended to benefit. This accounting mentality in evaluation tends to miss the big picture and may end up doing more harm than good.

Seldom do evaluations look at the big picture: Are we actually doing the right thing? Is the intervention that we are promoting meaningful in the larger whole? Is it something that the intended beneficiaries want and need? Is it fixing one part of the problem but creating others elsewhere? Is it having unintended consequences for the environment, for disadvantaged groups, for indigenous peoples, for power relations, etc.?

We must incorporate the environment into our evaluations. Sustainable development lies on social, economic and environmental foundations, yet evaluation—like national accounting—is almost exclusively concerned with the economic and, to a lesser degree, social capital, while natural capital and its depreciation are considered external to the system. According to the World Bank, low-income countries get 47% of their wealth from natural capital. This figure certainly underestimates the value of ecosystem services, in terms of clean water and air, health benefits, recreation, protection against natural hazards, etc. Evaluators must learn how to operate at the nexus of environment and development, which means understanding the interplay between human and natural systems.

Some of these lessons come out clearly in the evaluation of the GEF Global Wildlife Program, which is directly relevant to warding off pandemics such as the current one. The evaluation revealed the need to address the root causes of illegal wildlife trade on multiple fronts while also protecting endangered species in situ. Working with local communities to provide sustainable livelihoods is important, but not sufficient. It is essential to address political will, corruption and demand for wildlife products in the market countries of Asia, Europe and North America. Such interventions—and evaluating them—require holistic perspectives and a broad understanding of the dynamic systems.

For evaluation to remain relevant, it must rise above its project mentality and start looking beyond the internal logic of the interventions that are evaluated. It must systematically search for unintended consequences that may lie outside of the immediate scope of the evaluation. It must expand its vision to encompass the coupled human and natural systems and how they interact. And it must resist focusing on accountability for donors and instead make sure that it contributes to learning, for the wellbeing of the beneficiaries and nature in an equitable manner. If we achieve this, evaluation will be better positioned to contribute to more sustainable and just development in an interconnected world.

This blog post was originally published on the European Evaluation Society (EES) website