Environment, Climate Change and Evaluation: Moving Towards a Holistic View
Juha Uitto is the Director of the GEF Independent Evaluation Office, a GEI Network Partner. He is also a Thought Partner of the Footprint Evaluation Initiative, which is hosted on GEI’s BetterEvaluation knowledge platform. Over the past decade and a half, he has conducted and managed numerous evaluations of international cooperation related to environmental management and the nexus of poverty and the environment. He has published widely on these topics.
We live in an Anthropocene, where human activity is now the most important factor shaping the Earth's natural environment.
This poses new challenges for development policies and programs and their evaluation. Climate change is a reality that we must all live with and try to adapt to. In March 2022, the Intergovernmental Panel on Climate Change (IPCC) released its latest report, which shows that climate damage is happening faster than expected. Heat waves, forest fires, droughts, storms and rising sea levels are increasingly causing damage. But climate change is only one of the many serious environmental problems happening today. Biodiversity – both at the species and ecosystem level – is disappearing faster than ever before in the history of humanity. The world is drowning in waste, which is produced in ever increasing quantities as people become more prosperous. The world's oceans are suffering from plastic waste.
The COVID pandemic is a reminder of how nature and human health are closely interdependent. The breakdown of the balance in nature has led to an increase in zoonotic diseases, where viruses are transmitted from animals to humans. New viruses are taking advantage of the animals that carry them, such as rats and bats, and are thriving around humans. Mosquito-borne diseases, such as malaria, are spreading towards the poles as the climate continues to warm. As human activities encroach deeper into previously untouched nature and forests are cleared for agriculture, mining and construction, we are opening the door to the destruction of the natural world and increasing the threat to humankind, as well.
All of the issues above have an important socio-economic dimension. As inequalities between countries and within societies increase, vulnerable people – especially those in developing countries – are more affected by climate change and environmental degradation.
Evaluation Needs a New Perspective
These interconnected environmental and economic crises have implications for how we evaluate. Traditionally, evaluation looks at interventions from the inside – looking at whether the project has done what was planned and delivered the agreed upon services or products. This approach focuses on assessing the success of the program in isolation from the wider environment and without regard to whether there was an actual change in the situation that was intended to be influenced. The risk is that the evaluation turns out to be only a technocratic tick-box exercise. Achieving real change requires that development and environmental policies and programs focus on the underlying causes of unsustainable development, not just the symptoms. For example, in fragile contexts, vulnerable people and developing countries are more affected by climate change and the environment pollution. Consequently, addressing fragility is necessary to support climate change adaptation.
For example, the Global Environment Facility (GEF) has shifted its strategy to focus on the societal factors that have the greatest impact on the state of the environment. Its new programs revolve around food production, cities and deforestation, rather than just funding environmentally friendly technologies or creating protected areas. Evaluation of these programs will require new approaches. For example, if we look at agricultural production, we cannot focus only on local projects in isolation from the wider picture. Three everyday products – soya beans, palm oil and beef – are responsible for almost 80% of tropical deforestation. Their impact is linked from the local level, through smallholder production, international marketing and transport, to consumption at home. All these levels need to be taken into account in both programming and evaluation.
Figure 1: Understanding the Big Picture
Here are some of my thoughts on what all this means for evaluation. First, evaluation must take a holistic view of its target. Each intervention takes place in a broader context, and that context affects the success of the intervention in many ways. In the GEF's Independent Evaluation Office, we recently conducted three evaluations of the Fund's activities in three different categories of countries: least developed countries, small island developing states, and fragile and conflict-affected countries/situations. The evaluations clearly showed how the context (i.e., fragility, economic conditions) of a country or region can have a statistically significant negative impact on the results of programs. The factors involved can include the security situation, political uncertainty and weak institutions. Each category of countries has its own specific characteristics and within them, each country is different. This calls into question the notion of universal “best practices” for evaluation or program/policy development.
All interventions operate within a broader system where different components – economic. political, social, and other programs – interact with each other. It is therefore not meaningful to look at individual projects in isolation from this. We often see situations where projects have achieved all their objectives, but their impact on the wider system is insignificant because other factors negate the impact. In environmental programs, this is unfortunately often the outcome. Although global environmental funding has risen with increased awareness and international agreements, it cannot keep up with activities that have a negative impact on the environment. Total public funding and private sources for climate change mitigation is currently estimated to be approximately €590 billion per year. According to the International Monetary Fund (IMF), countries spend around €5.4 trillion a year on fossil fuel subsidies alone!
Secondly, it is important to consider both human and natural systems in an evaluation. It is impossible to achieve sustainable development if we focus on one and forget the other. If natural systems break down, we can forget about social and economic development, because they are very much based on natural resources and a stable environment. The “Final Report of the Independent Review on the Economics of Biodiversity” led by Professor Sir Partha Dasgupta, and commissioned by the UK Government makes this clear.
This is not only true for the evaluation of environmental programs, but especially for evaluations of economic development programs. Indeed, it can be stated quite categorically, that all human activity has an environmental impact. The impact can be positive or negative, intended or not. So, here's another reason why evaluation cannot focus only on whether the program has done what it promised to do. Evaluation must identify unexpected effects, not only on the environment, but also on vulnerable groups, minorities, women, Indigenous peoples, and others. It is clear that if these impacts have not been foreseen, they have also not been taken into account in the program's theory of change. Therefore, evaluation must look at the bigger picture. In many cases, it is not possible to find all solutions that benefit both nature and people, and trade-offs are needed. The role of evaluation is not to decide what to do in such cases, but to highlight possible conflicts between, for example, nature conservation and economic development. Evaluation can also shed light on synergistic situations where all the interests of all stakeholders have been maximized, and thus contribute to the knowledge of the issue.
Let Evaluation Questions Guide Method Choice
All of the above requires a multi-method approach to evaluation. There is no "gold standard" where one method is always better than the others. For example, experimental methods that project results for a target group that are then statistically compared with a control group outside the project, are often very difficult to use in real-life situations. Their use may be appropriate for well-defined projects, but not at program level or for understanding broader linkages. Although in some situations statistical data could show differences between the project target group and the control group, it is difficult to explain them using only statistical analysis.
Given the environmental focus of GEF projects, we often use geospatial data methods to evaluate them. Remote sensing helps us to see changes over time in, for example, vegetation productivity, forest cover and types, soil erosion, settlements and the expansion of the road network. We need other sources of information to understand the factors that have influenced changes. Fieldwork and interviews are important, but answers can also be found in the already published literature. For example, our evaluation of GEF forest conservation projects in Cambodia and Vietnam revealed how the same approach led to quite different results in the two countries. Explaining these differences required our evaluators to delve into scientific articles and local research. These revealed the political, social and economic factors that had caused the projects' impacts to differ so significantly.
Evaluation can help us to be more effective in curbing climate change and environmental degradation. But this requires evaluation to take a new, broader perspective. We need to look at programs and projects holistically – from a systems theory perspective, as part of a broader structure. Systems theory thinking allows us to see human and natural systems as an interconnected whole and also to take into account unforeseen impacts. In this way, evaluation is able to produce useful information for more profound change.
(This blog originally appeared as part of the Ministry of Foreign Affairs of Finland’s 2021 Annual Report on Evaluation.)
We welcome you to join the conversation! Comment on this blog or reach out to us on social media: LinkedIn and Twitter. What has been your experience? Share your stories with us.
If you would like to contribute your knowledge to this blog, we would be happy to work with you - please contact us contactgei@globalevaluationinitiative.org.
You can also sign up for our newsletter here.