The Challenge of Evaluation

Ram Chandra Khanal discusses the challenges in the evaluation of climate change based on the climate change adaptation session of the Evaluation Conclave 2013 held on February 26 to March 1 in Kathmandu, Nepal. The article was originally published in the features section of the Climate & Development Knowledge Network.

Evaluation of climate change interventions in South Asia was one of the themes discussed at the recent South Asia Evaluation Conclave held in Kathmandu, Nepal.

Evaluation is a critical part of development interventions ââ?¬â?? for accountability, for learning, to see what worked and what didn't, for influence and so on. Evaluation is particularly critical in the context of South Asia, home to complex social structures, high rates of poverty, gender discrimination, dynamic forces of globalization sweeping traditional societies and numerous development projects for the large populations of this region. Innovative evaluation approaches and practices are particularly important in such complex contexts.

The aim of the conclave was to bring together thinkers, commissioners and practitioners of evaluation in an interactive forum to discuss, deliberate and share advancements in the theory and practice of evaluation in the South Asia region and abroad.

Organized by the Community of Evaluators (CoE) in collaboration with the Community of Evaluators Nepal (CoE-Nepal), this was the second such conclave to be held in South Asia ââ?¬â?? the first was in October 2010 in Delhi. The CoE is a group of evaluators from South Asia (India, Sri Lanka, Bangladesh, Afghanistan, Nepal and Pakistan), working together to strengthen the field of evaluation. This second conclave from 26 February to 1 March was attended by over 300 experts and evaluators from different parts of the world including representatives of the World Bank, Asian Development Bank, IDRC and universities. Robert Chambers, the 'guru' of participatory evaluation, and Michael Quinn Patton gave keynote speeches in the workshop.

Climate change adaptation strategies contain a wide variety of interventions with different scale considerations, processes and types of activities, targeting different sectors and levels. Some focus on building adaptive capacity, others on direct interventions such as development of physical infrastructure; many will include a mix of these elements. Evaluation of climate change related interventions thus poses particular challenges. It can be difficult to distinguish the effects of climate change adaptation strategies from those of other sectoral activities, and thus to attribute outcomes or impact to a single intervention. How then to evaluate and measure impacts of climate change adaptation interventions?

This was the question I attempted to answer in the session on climate change and M&E, organized by SEA Change CoP - an Asian community of practice on the monitoring and evaluation of climate change interventions, with financial support provided by the Rockefeller Foundations and technical and logistical support from PACT. I began by looking at evaluation institutions and processes related to climate change adaptation in Nepal. My conclusion was that there are good mechanisms instituted, but the focus is more on monitoring than evaluation. Moreover evaluation activities by multiple players and at different levels are not well-connected and harmonized in practice.

In the second part of my presentation I used CDKN and the Pilot Programme for Climate Resilience (PPCR) as case studies of evaluation systems for CCA projects. Examples of indicators used by CDKN and PPCR include:

Numbers of people supported by the project to cope with effects of climate change;Degree of integration of climate change in national, including sector, planning;Extent to which vulnerable households, communities, businesses and public sector services use improved (the project) supported tools, instruments, strategies, activities to respond to climate vulnerability and climate change;Evidence of strengthened government capacity and coordination mechanism to mainstream climate resilience;Negotiation skill (on climate change) enhanced in international forums by using knowledge and recommendations generated through the project.

The CDKN and PPCR case studies highlighted the challenges and complexities in evaluating CCA related projects. How to distinguish between development and adaptation? How to take into account the fact that people often respond to CC in autonomous ways? How to measure elements such as adaptive capacity, transformative change and resilience? There has been a move towards result-based and participatory M&E activities, but efforts are fragmented and not sufficient to address current challenges. The CDKN approach of focusing on practical aspects and flexibility were well appreciated and so was that of PPCR.

The main conclusion from the session was that there is no fixed model: evaluation needs to be adaptive, based on the context and project needs. There was also general agreement on the fact that additional precautions (such as focusing on a programmatic approach and building capacity of stakeholders) need to be taken while developing evaluation systems for CCA projects. The complementarities of adaptation vs. development have to be explored. Continuous assessment and sharing learning among stakeholders are also important. Perhaps the main conclusion was that evaluation of CC adaptation projects is evolving and further research is needed.