Guidelines for Climate Change Mitigation Evaluations: Part 2

Since my last blog post in July - guidelines for climate change mitigation evaluations part I, part II has been completed. So now it's time to summarize what can be found in the guidelines.

In part I, we provided some general considerations about the various scales of climate mitigation interventions and how they interrelate. We also picked up the Theory of No Change (TONC) as a means to evaluate the context of climate mitigation interventions. Climate mitigation interventions consist of a wide range of actors and types of policies, programs and projects (such as energy efficiency, renewable energy, and capacity building), and often such interventions only provide part of the solution. Therefore, the context and nature of each of these partial solutions needs to be integrated into evaluations in order to assess the immediate results of the intervention, as well as its potential deeper impacts.

In part II, we revisit this topic but in a more hands-on manner. For evaluating with the help of the TONC, we collected some indicative indicators from our database of evaluations. Here we ran into some challenges: firstly, we were not able to find indicators that could be applied uniformly across all the interventions we looked at - that would have been nice but would require more work. But also, how useful would such indicators actually be? We will discuss these questions next week during the November 27 webinar!

We encountered two other major measurement challenges which are discussed in part II. These relate to baselines and GHG emission reductions. Indeed, it is hard to find an evaluation in the climate mitigation area that does not discuss these two issues and yet despite all the analysis which has been done, they still prove tricky.

To explain, baselines have to be discussed whenever a climate mitigation project is started in order to prove additionality. In the case of most climate change mitigation interventions, the baseline will not be observable or measurable in reality, but is an imagined counterfactual. To make things even more complex, that counterfactual would not even look the same if you were to compare the ex-ante perspective of the project design with the ex-post perspective after the intervention has taken place. The reason is that (climate mitigation) interventions change only part of the system - the rest of the system has some momentum of its own. The dynamics of the rest of the system will then affect the impact your intervention makes in terms of reducing GHG emissions.

Let's look at an example; a hypothetical intervention tries to add small hydro power plants to a power grid. In the baseline described in the project document it is assumed that all existing coal power plants will keep operating at current levels. The theory goes that the newly built hydropower plants will add capacity and drive down prices for electricity so that previously unserved needs can be satisfied. This in turn will lead to economic development because the SME sector had pent-up demand for electricity. Let us say, however, the project builds its hydro power plant and at the same time an economic crisis, or say, a famine, hits the country.. Production slows, demand for electricity goes down, and one of the coal power plants shuts down because the operators have run into financial difficulty. When the coal plant shuts down, a lot of GHG emissions are saved, but was that down to the project? This is a case of baseline shift on the one hand, the emission levels of the country have changed, but this is mostly unrelated to the project. On the other hand, if the hydro power plants had not been there to pick up the slack, maybe the government would have bailed out the coal power plant operator and kept production going. Who knows?

Typically, however, the evaluator will be asked to look at the hydro power project only. For this case, methodologies have been developed to translate each kWh of emission-free hydro-electricity into "avoided GHG emissions" and we are all now used to that concept. But using an "avoidance" as your main result implies that we are trying to measure something which has not happened. This means climate mitigation evaluations deal with a lot of assumptions.

This issue is compounded when we look at other climate change mitigation intervention and their impact on GHG emissions. For example, I am currently working on a project that tries to shift markets for consumer products so that their production causes fewer GHG emissions. This involves the concept of the carbon footprint, which is very data intensive and loaded also with assumptions. So yes, we are used to measuring the impact of mitigation projects in terms of avoided GHG emissions- but are we satisfied with that?

Let's discuss these questions next week during the November 27 webinar!!