What Would You Do with 200 Million US Dollars?

I recently had an informal meeting with the kind people of the University of Maryland's Earth System Science Interdisciplinary Center, who focus on environmental decision-making under uncertainty. Towards the end of the meeting Dr. Melissa Kenney asked me; "What would you do if you had 200 million US Dollars to improve the monitoring and evaluation of climate change adaptation and sustainable development?"

Hmmm, good question… Wow, that asks for some deeper thinking… Euh, what about investing in the further professionalization of national level M&E departments in developing economies? Improving it through South-South learning. Developing open M&E data portals to feed that information into. Well, that might be one small part of the puzzle, but what else? It basically touches upon the question; "What should monitoring and evaluation look like a decade from now?"

Melissa's question kept me awake that night. The day after I sent it to some more knowledgeable people to see what their answer would be. Interesting discussions were the result and below you will find a recap of some of it.

Deliberate Development instead of Sustainable Development

Looking at the question and considering the next decade, the Sustainable Development Goals (SDGs) are very much on everyone's mind. Though as Sanjay G. Reddy points out in a recent article: "Global goals should be adopted only if we think that having goals does not undermine important procedural values, and if goals will have epistemic, motivational or coordination roles that can lead to improvements in people's lives." UN statistician Howard Friedman also found no statistically significant accelerations of global progress in the Millennium Development Goals (MDG) indicators after the goals were introduced in 2000, except for on debt relief. As such, be conscious and honest about the relative success of global goal setting.

Ms. Anna Williams would prefer to focus on ‘deliberate development' instead of sustainable development. The SDGs might very well provide value by giving the possibility to rally behind a goal, or work towards specific targets. But can we talk about sustainable development if on the local level certain some goals are adopted, while others are ignored? Can we only talk about truly sustainable development if all (or most) of the 17 goals, 169 targets and 300+ indicators are adhered to? On the local level it might be more realistic to focus on a smaller number of relevant goals, targets and indicators. Identify real-world examples of deliberate and durable success, find the common denominators, contrast these successful pathways of change against the not-so-successful elements of the initial theories of change, and develop an overview of what could hold back development – in a sense a theory of no change. The focus should then be on incorporating this knowledge at a program's selection and design stage in a more developmental and formative way.

Ms. Madeleine McKinnon would use tools such as evidence maps and systematic reviews to map our current state of knowledge to know what works and what does not, but also to also to develop a map of the "known unknowns".

High Impact Interventions combined with Adaptive Systems

Blane Harvey would invest in building the sector's capacities to evaluate adaptation and development differently. Blane feels that we need to move away from the idea that deviation from expected outcomes is a failure, because it keeps us stuck in the land of incremental change. The rise of complexity and resilience thinking is pushing us to think very differently about how we support effective adaptation. What we need is adaptation support that is flexible, adaptive to changing circumstances, iterative and informed by ongoing reflection. This means that the end result of a "successful" project may look entirely different than what was envisioned at the start, and we need to get a lot better at evaluating in ways that can take that into account.

Ms. Madeleine McKinnon would focus on those projects that have greatest potential for biggest impact. The potentially higher risks would be managed through adaptive and experimental management and feedback approaches, in which quality data helps to quickly learn and change course as needed.

Mark Trexler would also focus on projects with the potentially biggest impact, but advocates for a modern version of the "Star Chamber", which would look for successes and near-term high-probability bets to dynamically inform funding flows. This should change the current state of play from a chaotic one to a strategic game of climate chess.

Ms. Dunja Krause would launch an initiative to identify the most successful approaches and their contexts, as well as M&E approaches that worked in that context.

An Urban Focus

Most of the adaptation and development interventions over the past decade have focused more on rural areas and the urban periphery, though with 3.9 billion people living in urban areas – and the expectation that by 2050 67% of the world's population will live in urban areas (UN 2014) – there is more and more an urban focus.

Ms. Joyce Coffee would use the funds to start an open data movement for cities, climate profiling cities that have opened their data to the public, developing mentoring and peer-support systems between cities to lynch pin new cities into the movement. The information would feed into an overarching urban open data portal, which would link to an effort to integrate M&E into all urban climate change and sustainability projects. This would also feed into the development of a set of urban impact evaluation guidelines.

Learning

What becomes clear from both Anna's and Blane's views is that we need M&E approaches that place more value on learning and adaptive management.

Patrick Pringle wants to also invest more in understanding things that haven't worked, as too often we are quick to label failures but slow to learn from them. He would want to invest in innovative ways of sharing experiences and exchanging knowledge, as too often knowledge and learning is not mobilized and used to its full extent. This often means 'translating' knowledge so that it is accessible and relevant to different users and communities.

Participation

Glenn O'Neil feels there needs to be an increased effort to develop evaluation in a more participatory way, including both program staff, beneficiaries and other key stakeholders in the entire process because there is a higher utilization of evaluation findings that have been developed in a participatory way.

Ms. Nancy MacPherson feels that we need to invest in letting beneficiaries and developing country evaluators determine the evaluation agenda, and help them lead and participate in evaluations of what matters to them. Evaluation will move beyond aid and beyond our own interests – Broaden the subjects of what gets evaluated to include more politically sensitive subjects and macro issues – not just having M&E respond to donor programs. This demands investments in evaluation capacity development and evaluation leadership in developing settings in order to get away from the current asymmetries of power in evaluation.

Ms. Dunja Krause also feels that plurality of perspectives is key. At current many M&E approaches cannot – for example – encompass diverging views of affected people who might not be part of the target group. Analyzing the diversity of what is out there – based on specific criteria to pinpoint what works and what does not – could contribute to improved M&E that is contextualized and localized, but based on state of the art findings from around the globe.

Ms. Irene Karani would want to focus on the development of national and sub-national M&E systems, and the people in it, to track climate finance in addition to tracking development milestones, in order to be credible and prepared for future climate finance. Ms. Karani agrees with Ms. MacPherson on the need for further development of government capacities in M&E. She would aim for a stepped training and mentoring approach, which would start with the development of a strong set of national-level trainers.

M&E, M&R / E&L, M&MiE, MLE, M…?!

Both Juha Uitto and Patrick Pringle would separate monitoring and reporting (M&R) from evaluation. Patrick explains that monitoring and reporting are essential processes but too often we forget why we do them and become slaves to process. Monitoring and reporting are 'means' not 'ends'. In the end we should focus upon is learning; learning what, works, what doesn't, in what circumstances and why. Investing in learning means changing organizational cultures, improving governance and creating space to reflect on practice. Evaluation and learning need to become part of implementation, not an additional burden or after-thought as it sometimes is today.

Juha also points out that monitoring indicators do not explain the causalities nor the conditions under which certain initiatives and interventions produce desired outcomes, or do not. Evaluation is needed to provide a rigorous analysis of the underlying context, conditions, interlinkages and emerging properties that help us understand what works, how, under what conditions and for whom. At the same time, we must delink evaluation from being seen as purely an accountability tool imposed by donors and others with power. It is essential to promote national and local ownership for evaluation for it to effectively feed into policies, strategies and decision-making.

Ms. Nancy MacPherson advocates for near-real time feedback both for monitoring and evaluation, supported by the power of new ICT technology and big data – though we still need to figure out how to use it. We need to make more good use of monitoring, which should feed into evaluation to understand more in real time and be able to draw conclusions and feed information back into the programs. Hence, a near-real-time monitoring & monitoring-informed evaluation.

Saleemul Huq would want to focus on long-term M&E systems with the initial period being to set up baselines, and in the early years practicing learning-by-doing approaches in which the first few years focusing more on "Learning" rather than "Evaluating". Hence what is needed is a "Monitoring, Learning and Evaluation" (MLE) on a decadal scale basis rather than merely an "M&E" program. A network of existing knowledge institutes could lead the development of the system and initial methodologies, to be linked to national government systems.

End Note

Further M&E capacity development at various levels, the identification of new, innovative and adaptive M&E approaches, development of robust evidence / baseline data and long-term monitoring efforts feeding into learning, were mentioned by many as efforts worth spending part of the 200 million US Dollars on. Though a good number of respondents also agreed that a good chunk of the money needs to be spent on developing an enabling environment for M&E and learning, and incentives for governments and national institutions to adopt new approaches.

My feeling is that a lot of the focus is perhaps on the supply side – how to include people, how to develop, present and share evaluative evidence, whereas we already do generate a lot of evidence and it might be good to look at whether (and if so, why) demand for evidence-based information lags behind supply.

Some wonder whether all of this requires new money, or whether it is more a shift in mind set, organizational practices and the way existing resources are used. The shift in itself will perhaps be harder, though in the long run it will be a prerequisite for the further adoption of new developments.  

You can also wonder whether this should be seen as one big an initiative that would be designed based on the elements pointed out by the respondents, or rather opt for several smaller initiatives.

I think I would opt for a selection of smaller, interrelated initiatives. These should be part of a pathway of change aimed at both capacity development, mind-set shift and a shift in how M&E and learning are financed. So yes, we perhaps need ‘new money' to advance M&E and to foster a necessary mindset shift that will eventually result in smarter use of existing funds.

The reader might have noticed that most of the feedback received came from female colleagues (ie. many good women and a few good men), while the question was sent to an equal amount of male and female M&E professionals. We all know that women make better leaders, and the strategic nature of the question might explain why female colleagues felt more compelled to share their thoughts. It might also be that the answers to the question focuses on transformational change, empowerment and collaboration – traits traditionally associated with women.

Reference: UN, 2014. World Urbanization Prospects.