Evaluation of COVID-19 Impacts: Insights for near real-time monitoring

When COVID-19 was discovered in Malaysia, it affected a GEF project supporting reduction in greenhouse emissions across building infrastructure in the health and hospitality sectors (UNIDO GEF ID:4878). Site visits to industrial facilities were immediately halted as social movement was restricted countrywide. Subsequently in July 2020, restrictions began to ease after the first wave, but the longer-term effects of the economic downturn were beginning to take hold. Enterprises such as hotels were unwilling to invest in demonstration projects with a bleak outlook for economic recovery. Hospitals – another sector targeted by the project – were so overwhelmed dealing with the escalating and devastating health crisis, that capital investments were frozen.

On the other side of the world, case rates in the Congo Basin were low during the pandemic’s first wave in early 2020, yet communities adjacent to protected areas felt the repercussions of international travel bans at once. With loss of tourism revenue, local livelihoods were dramatically affected, and forest resources were at risk of illegal and unsustainable harvesting. Activities and results of an UNDP-implemented GEF project (Sustainable Financing of Protected Area Systems in the Congo Basin, GEF ID 2906), aimed at supporting the long-term financial sustainability of PA systems and associated ecosystems within the Congo Basin, were under threat. The project team had to rapidly re-allocate project resources into small grants to help local communities meet basic needs.

From stories to synthesis

Undoubtedly, many countries and communities have been devastated in significant and potentially irreversible ways. Individual stories of the immediate fallout of COVID-19 pandemic on projects and teams around the world have been widely shared by the international development community in blogs, webinars, and Zoom calls. Yet, over a year on from the first wave of COVID-19 outbreaks and lockdowns, our collective understanding of the pandemic’s implications on project implementation, and ultimately well-being, prosperity, and sustainability are only just starting to emerge.

As part of the GEF’s Annual Performance Review process, we sought to empirically assess impacts of the COVID-19 pandemic on GEF-funded projects under implementation through the fiscal year 2020. The GEF Secretariat already gathered information on effect of COVID-19 on GEF activities from the GEF Agencies and GEF Operational Focal Points. However, the data on GEF projects was anecdotal and did not provide accounts from all the projects that were under implementation. Our review focused largely on effects reported by 846 projects in their annual Project Implementation Reports (PIRs) and Mid-Term Evaluation (MTEs) documents during the first two quarters on 2020 (Jan – June 2020). Thus, it encompasses the earlier stages of the pandemic and first wave of national lockdowns and travel restrictions. It was intended to provide a near real-time synthesis of how projects were affected, and measures taken by projects and agencies to mitigate effects. Through this process, insights emerged on the role and practice of evaluation in supporting COVID recovery and timely management responses.

Widespread yet uneven effects

Of 846 projects covered, Agencies reported COVID-19 pandemic-related effects for 88% of projects. Of all projects covered, 69% reported implementation delay; 34% had activities suspended or put on hold indefinitely. For 9%, at least some of the project activities were cancelled. Delays and postponement were caused by variety of factors including time needed to adapt to working from home, shift in priorities and attention among government partners, restrictions on travel within and between countries, and bans on in-person meeting and suspension of most onsite business operations.

Certain types of projects were disproportionately and immediately affected. These included projects involving physical work like installation, manufacturing and construction or seasonal fieldwork (e.g., planting, breeding). Others were projects directly involving sectors heavily affected by restrictions and global economic instability (e.g., tourism, commercial real estate, travel industry) or reliant on private sector investors for capital investment or co-financing. Some effects were likely under reported or time lag is not yet sufficient to observe. Limited evidence found on how COVID-19 has affected personal well-being and equity considerations among individual staff or target stakeholder groups.

New ways of working

Project teams with support from regional technical advisors and other Agency staff worked to adapt, accelerate, or postpone planned activities. Teams shifted to teleworking and virtual platforms. Yet, many projects and certain processes could not simply move online. Beyond projects involving in situ activities, other barriers to virtual solutions included poor internet connectivity, technical capacity and equipment among certain communities, and administrative processes not yet adapted to online systems (like licensing or approvals by national or municipal government).

The immediate implication of COVID-19 effects is projects requesting extensions (at least 37% of projects covered). Given the slow global vaccine distribution and system-wide effects of COVID-19 on the global economy and international travel, longer-term Implications are likely around project oversight and working arrangements, stakeholder engagement, relevance of particular interventions (particularly market-based strategies), and the role of GEF activities and investments in COVID recovery efforts at project- and systems-level.

Insights for near real-time evaluation

Managing under crisis requires timely and accurate information. Through this process, we made several key decisions in our design and learned valuable lessons in conducting near real-time evaluation. From the outset, we were pragmatic in our evaluation design to deliver findings that could enable rapid learning.

First, we focused on PIRs as our primary data source given the time lag to wait for results from terminal or midpoint evaluations when relevant information would be either outdated or not sufficiently captured. We were further assisted by the newly centralized GEF data portal for managing project information. This enabled us to access the PIRs that covered the period of interest. Second, we used a machine learning assisted online platform – Colandr – to sort and mine text-based data in documents. This tool enabled us to identify, label and manage a vast body of potential evidence in a transparent, efficient, and accurate way. Manual synthesis would have been prohibitively costly in time and resources. Finally, we conducted a purposeful sample of key informant interviews to validate data within documents and provide deeper reflections on trends not commonly reported in project documents (e.g., individual effects, new activities to support COVID-19 response).

The quality and relevance of our review was largely reliant on data reported in project documents. Thus, established agency protocols and explicit COVID-related prompts determined what types of information we could gather. We learned a lot more around effects and responses among UNDP projects because projects were asked to report on this. As projects and agencies continue to contend to the aftershocks of COVID-19 pandemic, structured questions around effects and responses will dramatically facilitate any similar near real-time evaluation efforts. And hopefully prevent extraneous data collection efforts which could be a greater reporting burden on project teams.

Our review only scratched the surface of the effects of the COVID-19 pandemic on GEF projects. Continued tracking of effects for FY2021 (July 2020 to June 2021) is particularly useful. During this period, the effect of COVID-19 deepened in many GEF recipient countries especially South Africa, India, and Brazil. It provides us an opportunity to study the long-term effects on the environmental results that GEF pursues and on the communities that GEF projects seek to assist. GEF Agencies are in the process of preparing PIRs for FY2021. They should be encouraged to report on the effects of the COVID-19 pandemic and ideally in a structured way. With the COVID-19 pandemic affecting GEF projects for more than year, a substantial number of mid-term reviews and terminal evaluations would have covered its effects. These may also prove to be an additional source of information on future work.

Reference

GEF Secretariat, November 2020, "The Impact of COVID-19 on GEF Project Preparation and Implementation" - GEF/C.59/11