Draft Study on Evaluation of NRM Interventions Linked to Climate Change - Now Available for Comments

Natural Resource Management (NRM) is at the center of the Climate-Eval Community of Practice's expanded focus for the next phase of engagement. As such the community is looking to explore the issues relevant to evaluation practitioners when climate change aspects are incorporated into NRM interventions. Although the NRM sector is not new, looking at it from the environmental evaluation perspective is somewhat new and unchartered territory. That is why Climate-Eval commissioned a "scoping study" of evaluations of NRM interventions linked to climate change a few months ago (see approach paper). The study was conducted by Frederick Swartzendruber ? Senior Consultant at the GEF IEO, with myself as a Research Assistant. We are proud to say that the draft study is now ready and available for comments and feedback here: Scoping Study.

The scoping study had some interesting findings which give insight into the gaps in research and practice, and ways forward for this nascent field. Below is a brief summary of the key findings, presented here with the intention of proviking comments and reactions from our audience who may be able to relate to some of it with their own experiences. We would love to hear your feedback and any comments received before January 15th, 2015 will be taken into account in the final iteration of the study.

Key Findings:

• NRM Terminology: There is a lack of clarity on the definition of NRM as a term, and this creates confusion. Donor-financed NRM interventions often exclude extractive usage from the definition. For this study, NRM was limited to non-extractive, renewable usages.
• Accountability over Learning: At the project level, most evaluations are terminal studies emphasizing accountability for use of donor resources rather than learning from the lessons of previous evaluations on aspects such as design and implementation.
• Mismatch in Time-Horizons and Scale in Project Evaluations: Time horizon and spatial scale are often the most common and difficult challenges in NRM evaluations, and there is little sign of this scenario changing. The project modality may be largely to blame for this, as NRM projects are usually not longer than 4-6 years, a time too short to realize outcomes such as carbon sequestration or restoration of ecosystem functions. Furthermore, scale-related mismatches can be seen in new approaches such as payments for ecosystem services or landscape approaches to NRM.
• Logframe approach won?t work: The linear cause-and-effect logic of a typical project logframe is ill-suited to NRM interventions. Longer term models incorporating a Theory of Change, experimental and quasi-experimental methods, and adaptive management hold more promise, as the complex interactions of biophysical and human systems need to be identified and new responses put into effect on a rolling basis. The trajectory of outcomes or responses to interventions may also vary, and some processes may also be reversed after a project has ended.
• Some learning-focused program and portfolio evaluations exist, but not enough: For example the GEF IEO?s South China Sea Impact Evaluation; NORAD?s Real-time evaluation of the NICFI program; World Bank/ IFPRI?s impact evaluation in Niger; and USAID?s retrospective stock-taking evaluation of NRM technologies in agriculture and livelihoods in Senegal, Malawi and Namibia. The study did not find many other examples.
• Poor Data Quality: Data quality and credibility remains a challenge, with regards to whether the data being collected are appropriate for supporting conclusions about attribution of project outcomes and impacts. Moreover, adequate baseline data is often not available or unreliable, indicators are poorly selected, there is incomplete monitoring of site-level data capturing biophysical conditions over time, and there are no standards in measurement (?core? indicators).
• Valuation of natural resources is a grey area: Economic valuation of natural resources as well as the ecosystem services that they provide is not only a challenge for measurement but also for policy with respect to global regulatory frameworks (e.g. for REDD+ and PES).

Ways forward:

There is already a significant body of literature discussing the complex ways in which natural resource systems interact with human systems; examples include the two-system evaluand approach; sustainability science; land change or land system architecture; ecohealth; panarchy; and others. Complexity science can be useful in framing the multiple interactions between systems which are unavoidable aspects of NRM.

NRM interventions often take place in nested or interacting systems which function at different scales of time and space, and consideration of these factors in future evaluations is something that needs to be tried and tested over longer time periods. There is some support for theory of change models in accounting for these longer-term processes. Mixed methods as well as carefully selected case studies can also help solve some of the methodological issues involved in evaluating NRM interventions.

Evaluations in this analysis failed to underline the lack of scientific understanding as a weakness in project or program design. Long-term monitoring of closed projects is possible with the in-country monitoring systems such as remote sensing that are now available in many developing countries. One way forward would be to fund such initiatives to better inform the design of future evaluations through improved baseline data and monitoring.

Finally, the lessons learned from evaluations in NRM need to be incorporated into more robust project and program designs in future. USAID, ADB, AfDB, GEF and the World Bank are some organizations that have expressed commitments to strengthen the knowledge management functions, and this bodes well for the future of evaluation overall. The increased interest recently in the paradigms of resilience and adaptation provide a clear entry point for integrating climate change into NRM interventions and evaluations.

Thanks!

On behalf of the author and myself, we would like to thank those of you who supported the study by providing various documents and information, comments and feedback during its formulation. We hope you enjoy reading the study and it provokes some thought on this increasingly pertinent topic in climate change and development evaluation. Have a great holiday season, and happy new year!