Incorporating environmental considerations into UNEG Member Agency Evaluations

My recent work as consultant to the UNEG Working Group on Integrating Environmental and Social Impact into Evaluations (ESI) has shown that UNEG member evaluation offices face challenges in adequately evaluating interventions that are not primarily focused on the environment, but which are likely to have effects on it. I have pulled together below nine specific areas where guidance from ESI would be particularly useful.

Potential areas for coverage by ESI guidance

1. Addressing all dimensions of agency interventions.
   

   Challenge: UNEG members present complex range of interventions for coverage by common guidelines:

   • 50+ agencies
   • Huge range of issues
   • Levels of intervention from community projects to global agreements
   • May be multiple implementers and stakeholders in any intervention

   Response: The ESI guidance should provide considered advice on how to ensure that evaluations adequately assess both the direct and indirect effects of interventions, to ensure that such aspects as institutional development, capacity building and generation of guidelines and policies are not assessed as an "end in themselves”, but as contributions towards actual environmental benefits.

2. Outline a process for agencies to map out potential connections between their major types of intervention and environmental considerations.
   

   Challenge: There is a need to move from the current situation, where each intervention is often evaluated as if it is unique and has completely unpredictable "side-effects" to one where the overall repertoire from which actual unintended consequences may occur is identified in advance.

   Response: The ESI guidance should outline a process through which each agency can collate the environmental considerations arising from its interventions to produce a "map" of common patterns of unintended consequences, which may occur from its different types of activity.

3. Promoting strong coverage of environmental considerations through the use of Theories of Change.
   

   Challenge: Where an intervention has not been built upon an explicit and documented Theory of Change, it can be challenging for evaluators to make explicit the range of targeted causes and effects and their interaction with other factors, including the environment. In many evaluations, one of the first tasks of the evaluators is to hold detailed discussions with project stakeholders to devise a retrospective Theory of Change.

   Response: The ESI guidelines should promote the value of creating a retrospective Theory of Change, for those situations in which this is not present and provide an introduction to the process, with suitable references for more detailed methodological guidance.

4. Emphasis on maximising available evidence through mixed methods and triangulation.
   

   Challenge: the limitations of over-reliance on disconnected data sets. Particularly noted was the use of scattered small-scale case studies inadequately triangulated with other relevant data sets.

   Response: Guidance should cover the types and combinations of evidence, which might provide a basis to raise and address environmental considerations adequately.

5. Ensuring coherent evaluation of environmental considerations through joint evaluations of inter-agency interventions.
   

   Challenge: Many complex interventions are implemented through inter-agency collaboration. Such activities may lead to disconnected approaches, which do not adequately specify or evaluate intended environmental benefits. In many such cases, joint evaluations will be the best option to ensure that the interconnections between social and environmental results chains are suitably assessed.

   Response: ESI guidance could provide an important outline on how to integrate environmental considerations into complex multi-agency interventions, in which no agency is a clear "environmental lead".

6. Channelling attention to timescale and priorities.
   

   Challenge: Environmental considerations are increasingly embedded in complex interventions, spanning different areas of focus (e.g., irrigated agriculture as a means of climate change adaption), as well as in "hybrids, which combine different areas of focus in one development concept (e.g. green economy).

   Response: The ESI guidance should provide advice on how evaluators can best address the long timescales of multifocal and hybrid projects, where the environmental approaches may be implemented as a lower priority than other aspects and where their realisation is therefore likely to be some way along the results chain.

7. Guidance on influencing agencies to ensure that environmental considerations are included in intervention design.
   

   Challenge: It is extremely challenging for an evaluation to make a coherent and evidence-based assessment of environmental effects of an intervention if these were not considered in the original project design. This is because in such situations it is highly unlikely that the institutions involved in implementation will have gathered any coherent evidence, particularly monitoring data, which can be reviewed or used by the evaluators as the basis for findings or conclusions, still less for recommendations.

   Response: Evaluation offices should take a long-term approach by ensuring that environmental considerations are included clearly in their Terms of Reference and that evaluators address these adequately. One means of ensuring quality will be to include this coverage in their evaluation quality rating system.

   Furthermore, evaluation offices should consistently ensure that evaluation recommendations regarding the importance of environmental considerations are entered into the Management Action Record system and from there over time into standard project design procedures.

8. Guidance to agencies on how to systematically consider the extent to which "lessons learned" on the environment are actually used in project design and implementation.
   

   Challenge: In most agencies there are relatively few projects, which cover completely new issues and activities, so it is generally the case that a fairly comprehensive set of lessons could in principle be available to aid project design and implementation. Insofar as previous evaluations have raised environmental considerations, these should therefore be explicit in the designs of those projects that have taken account of lessons. One of the key uses of lessons should therefore be to convert the majority of unintended effects into intended effects and to incorporate them into project designs, building on the accumulated knowledge from previous evaluations.

   Response: ESI guidelines should point evaluators to the importance of exploring the extent to which they have enabled project designs to incorporate available lessons concerning environmental considerations and also to establish how straightforward it is within their agency to locate and utilise those lessons, which are relevant to the type of activity under preparation or evaluation.

9. Match environmental considerations to DAC criteria to raise their profile in evaluations
   

   Challenge: The DAC criteria are at the heart of most evaluation Terms of Reference. Given prevalent restricted resource allocations, particularly for evaluations built around fieldwork, evaluators will tend to address the compulsory elements of ToR first and then move on to other aspects. In most cases, this is likely to reduce attention to environmental considerations, unless these are centrally placed in the ToR.

   Response: ESI could therefore provide guidance on how environmental considerations can be dovetailed into the DAC criteria. This matching process could be an extremely important input of the guidance, although it would still need to be tailored by evaluation offices to the specific types of interventions their agencies practice.