Looking beyond project outputs in climate mitigation evaluation

Kelly Hewitt delivered the following message for the Climate Change M&E Panel of the Evaluation Conclave 2013 in Kathmandu, Nepal:

Please, let me start by saying that though I and my colleague Kapil Thukral, serve on the Evaluation Oversight Committee for the current CIFs global evaluation, it is my separate, independent and individual professional observations as an evaluator that guide my comments this morning. So with that let's move on.

I believe that we all can agree that carbon investment funds aim to bring about additionality, leveraging, and transformation. In the evaluator's world we're speaking to core sub-components of efficiency and sustainability. I had originally planned to talk about how complex it is to assess additionality, leverage, and transformation because of inconsistent definitions and because any assessment involves measuring for the most part capacity building results and real private sector market transformation. Instead, in an attempt to dissolve some of the complexity and to provide some focus to the concept of transformation - I will speak solely to that term.

Most of what we talk about when we discuss additionality, leverage, and transformation in the evaluation context is quite different from the typical private sector definitions of the same terms.

Now it's important to realize that when I speak of private sector definitions, I am not referring to the private sector arm or arms of the five multilateral development banks that design, approve, and implement CIF funded projects. When I use the term private sector, I'm speaking of real markets - supply - product and service quantities, pricing, demand; market entry/access, competition (both firm and product), market penetration, market power - all of which go into real sustainable market transformation.

And despite these real variables, there measurement, and the need to include them in evaluation assessments of publicly funded climate investments, today's evaluation approach of climate investments appears to mention very little if anything at all with regard to the importance of knowing, understanding, and measuring these particular real market variables.

Instead, evaluation has focused on output measurement - for example - How effective have public finance mechanisms to scale up private sector investment in climate solutions been - i.e. number of private sector investments in renewables or energy efficiency before and after an intervention or - another example - number and type of policies produced to incentivize renewable or clean energy investments and/or to create an enabling environment for private investment. And, often, the evaluation stops there.

We can't stop there. Back in the early 90's when we were having the same discussions regarding M&E of energy efficiency initiatives and demand-side management in the US and Europe - there were assumptions made, conclusions drawn regarding outcomes - project contributions to energy use reduction - energy demand - assumptions of outcomes that could not or either were not verified.

Today - we cannot assume that because a project was designed to reduce X amount GHGs annually that because the project output materialized that, indeed, the reduction occurred.

Equally, we cannot assume that because a project was designed to result in market transformation that because the project output of de facto conducive policies and/or private investments materialized - that, indeed, market transformation has occurred.

To measure real transformation we must look at and quantify, among other things, real market variables that provide necessary information on - product and service supply, pricing, competition (both firm and product), market penetration, market power- all of which goes into real sustainable market transformation.

With that I'll stop here hoping that in our interactive discussion we can chat about real word experience with trying to quantify these variables, along with limitations in evaluation scope and application of lessons learned. Thank you.