What Do Evaluations Tell Us about Climate Change Adaptation? A Realist Review
Takaaki Miyaguchi
Ritsumeikan University
Associate Professor
Blog Date:
This is a co-authored piece by the following authors: Takaaki Miyaguchi(Associate Professor, Ritsumeikan University (Japan), takaaki@fc.ritsumei.ac.jp) and Juha I. Uitto(Director, GEF Independent Evaluation Office, juitto@thegef.org)
Hi, I'm Taka, and I?d like to give you a preview of what my co-author Juha and I are going to discuss at the 2nd International Conference on Evaluating Climate Change and Development in November. Our paper analyses Climate Change Adaptation (CCA) programmes and projects implemented by United Nations Development Programme (UNDP) and its partners that have:
- finished initial implementation cycles; and
- been already evaluated.
We have conducted a meta-analysis of the final evaluation reports of the CCA projects from nine countries, i.e. Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania, Turkey and Zimbabwe.
In this analysis, we have adopted a realist approach. The realist approach belongs to the school of theory-based evaluation. It emphasizes the underlying assumptions about the way certain interventions are expected to yield certain outcomes. Theory-based evaluation is guided by a programme theory that intends to explain how an intervention is to achieve its outcomes (Pawson and Tilley, 1997). If a more judgmental approach focusing on quantitative measures and correlations were adopted, this review would not have been able to answer such questions as "for whom a programme/project may work", "under what circumstances", and "how". These considerations are indeed of high relevance for policy makers, programme implementers and donors.
In a realist approach, identification of CMO (context, mechanism, outcome) configurations is vital. We have reviewed all the CCA evaluation reports, and itemized every evaluative remark/mention in those reports by assigning either a positive (+) or negative (-) attribute. This step has at the end generated a total of 577 items (of remarks). By fully utilising these itemized remarks, we have sought to identify the key CMO configurations for the four evaluation criteria, i.e. relevance, efficiency, effectiveness, and sustainability. Here are some of CMO configurations identified through the analysis:
| Context | Mechanism | Outcome | ||
|---|---|---|---|---|
| For Relevance Criterion | ||||
| Host government and line ministries have already identified national and sectoral CCA priorities, and understand programme's objective Local CCA priorities are identified by the local government and local partners |
+ | Close coordination and working relationship with the national and local government enables both partners (government and UN implementing agency) to develop appropriate CCA programme | = | (+) High relevance of programme strategy and intervention components with the national, and global priorities |
| For Efficiency Criterion | ||||
| Sector specific and focused programme design Relevant stakeholders are supportive of UN and well aware of CCA issues and risks |
+ | Partnerships with stakeholders are built at an early stage, where they feel more motivated to participate in the programme | = | (+) High stakeholder involvement |
| For Effectiveness Criterion | ||||
| Specifically identified types of participants are well aware of the climate risks Specific types of skills that they need to acquire are clear to them |
+ | Training and transfer of needed techniques and practices for the relevant people facilitate these skills, techniques and knowledge to be applied and used | = | (+) High level of adaptive capacity (+) High level of utilization of adaptive measures |
| For Sustainability Criterion | ||||
| Relevant stakeholders have strong sense of ownership and have adequate resources and capabilities | + | Introduction of adaptive measures to the stakeholders/institutions with relevant function/mandate enables "rooting" of these measures by the respective stakeholders/institutions | = | (+) High likelihood for initiatives & replications |
During the conference, we will be sharing with you all the identified CMO configurations for further discussion. However, one major limitation we have faced is the fact that such CMO configurations were constructed for each of the four evaluation criteria (i.e. relevance, efficiency, effectiveness, and sustainability) that were pre-set by the CCA evaluation reports. What this means is that the CMO's "O" (outcome) is not consistently treated as the programme?s overall outcome, but as a sub-component that comprises these evaluation criteria. For example, the degree of financial execution as CMO's outcome (as one sub-component that can lead to efficiency) is not a final programme outcome leading to higher level of adaptive capacity or replication effects.
Although we are of the view that applying a realist approach even to these pre-set evaluation criteria has yielded some useful results to the CCA evaluation field and future CCA programming, the questions we want to cast to the reader is what types of methodological challenges you have faced in adopting a realist approach and/or identifying CMO configurations for CCA project(s). We look forward to receiving your comments.
Reference
Pawson, R. & Tilley, N. (1997) Realistic Evaluation. London, Sage.