Footprint Evaluation in Practice: Case Study on Uganda’s Private Sector Development Strategy
Alice Macfarlan is the Knowledge Platform Manager of GEI's BetterEvaluation Knowledge Platform and is part of the Footprint Evaluation Initiative team. This blog draws on the work of Andy Rowe, Kaye Stevens, Patricia Rogers, Dugan Fraser and Laura Mills, authors of the Footprint Evaluation case study report discussed below. Patricia Rogers and Andy Rowe provided review of the blog.
The world is faced with numerous environmental crises with the potential for global catastrophe, including climate change, pollution, loss of biodiversity, ocean warming and acidification, and deforestation. To avoid, mitigate and address these crises, decision-making must be informed by evaluations that explore the actual and likely environmental consequences of all programs, projects, policies, and strategies. However, it is not always clear how to go about this, particularly when evaluating something like a national policy or strategy.
The Footprint Evaluation Initiative is an international collaboration to support evaluators and evaluation managers in considering environmental sustainability in all evaluations, even when environmental sustainability is not a stated goal of the policy or intervention. This is needed so that decision-making can account for the potential and actual impacts of planned interventions (projects, programs, policies) on both the environment and the people that the intervention is intended to benefit. The Footprint Evaluation Initiative is supported by the Global Evaluation Initiative (GEI) and is hosted on GEI’s BetterEvaluation Knowledge Platform.
One of the Footprint Evaluation Initiative’s goals is to demonstrate the feasibility and utility of bringing an environmental sustainability lens into all evaluations. In 2020, with support of the Center for Learning on Evaluation and Results for Anglophone Africa (CLEAR-AA), a GEI Implementing Partner, the Footprint Evaluation Initiative was connected with the Office of the Prime Minister of the Ugandan Government responsible for the midterm review of Uganda’s Private Sector Development Strategy. The Terms of Reference (TOR) for the evaluation asked for an evaluation based on the OECD Development Assistance Committee (DAC) evaluation criteria, with the environment as a cross-cutting theme. This provided a valuable opportunity to demonstrate the Footprint Evaluation approach in an evaluation at a national scale. The primary output of this case study was the addition to the evaluation report of a chapter on the environmental impacts of the strategy, which provided recommendations to inform future revisions of the strategy. The case study also provided a valuable lesson about how to bring considerations of environmental sustainability into an evaluation of a national strategy with an existing TOR, as well as how to identify sources of existing evidence to inform an evaluation.
This blog post explores the key takeaways from the case study.
1. It is feasible and useful to include an assessment of environmental impacts in an evaluation.
While there were constraints on the data that could be collected or retrieved, it was still possible to gather useful evidence for the evaluation, and the evaluation was able to produce recommendations for the revision of the strategy that, if implemented, would reduce future environmental damage, and bring the strategy in-line with environmental commitments and objectives outlined in both international commitments and national development plans.
2. It is necessary to make decisions about the scope and boundaries of systems being evaluated.
In this case study, a decision was made to focus on three exemplars (electrification, business parks and tanneries) rather than trying to assess the environmental impacts of all elements of the strategy. These decisions were based on factors such as limited time and resources, availability of evidence and access to local knowledge, and judgments about political acceptability, legitimacy, and likelihood that messages would be heard.
3. The OECD DAC criteria can provide a framework for addressing environmental sustainability.
These criteria, commonly used in evaluations of international development, include relevance (or appropriateness), effectiveness, impact, sustainability of results (or durability), efficiency, and coherence. They readily accommodate a focus on environmental sustainability (see the Footprint Evaluation webinar on this), with the criteria of coherence, impact and sustainability being particularly relevant for this case study.
4. Publicly available information on national and international commitments can be used to demonstrate the relevance of a focus on environmental impacts.
This information can inform the negotiation of the terms of reference and key evaluation questions when framing an evaluation. For example, in this case an explicit aim of the mid-term review of the national private sector development strategy was to align the strategy with the national development plan, which detailed several environmental objectives and interventions focused on the environment. The World Fact Book can be another useful source of information about national and international commitments related to the environment.
5. Geographic Information Systems (GIS) can be valuable for understanding the site’s geography.
In this case study, GIS mapping of the topography, water flow, prevailing winds, and human population density was used to understand and illustrate the potential environmental risks related to tanneries and business parks.
6. It is likely that additional expertise will be needed to identify and explore a range of possible environmental impacts, but there can be cost-effective ways of including this expertise.
Some negative environmental impacts that might require different types of expertise include the effects of heavy metal toxicity, air and water pollution, land degradation, deforestation, and climate change. In some cases, it can make sense to engage an expert to conduct part of the analysis. For example, this case study engaged an expert in GIS mapping who had expertise in identifying and utilizing multiple databases to construct relevant maps. However, drawing on external expertise does not always require additional contracting of experts. For example, this case study drew on published research into water quality downstream from tanneries.
7. As in any evaluation, It is important to bring together information from a range of different sources.
In this case study, sources of information included:
- Government sources, including relevant planning documents, legislation and regulations, contracts, enquiries, and reviews;
- Academic articles and university theses, for example, Master of Science theses on water quality downstream from tanneries and water treatment practices by tanneries;
- Articles in newspapers and trade association newsletters;
- Google Maps and Google Earth to identify the locations of industrial parks and tanneries; and,
- World Factbook as a valuable source for identifying the international environmental commitments that the country was a signatory to.
8. It is useful to use an iterative process to continue scanning for new information after an initial document review.
For this case study, following an initial document review to provide a background for the evaluation context and inform decisions about scope, the team used a “snowball” method to identify new sources of information as their understanding of key issues developed.
9. Secondary data can be especially important when primary data collection is not possible or is limited.
The resources available for the case study meant that it wasn’t feasible to collect primary data as part of the evaluation of potential and actual environmental impacts. However, it was possible to identify potential environmental impacts by drawing on existing research on the impacts of tanneries and business parks.
10. Strategies and interventions may generate both positive and negative environmental outcomes, depending on contextual factors.
For example, business parks that supplied reliable green energy and better waste management had the potential to reduce greenhouse gas emissions and pollution compared to business parks relying on diesel generators with inadequate waste management. However, depending on the site, the construction of business parks may disrupt wetlands or cause deforestation.
11. It is important to distinguish between reduced negative environmental impacts (compared to a counterfactual, such as “business as usual”) and actual positive (regenerative) environmental impacts.
In this case study, potential environmental benefits took the form of reducing negative impacts (such as replacing the use of fossil fuels to generate electricity) rather than positive environmental impacts such as restoring wetlands. (For more information on distinguishing between impacts, view the Footprint Evaluation webinar: From 'do no harm' to restoration - Working with a typology of Footprint Evaluation)
Read the Report
This case study provides valuable insights into how environmental sustainability can be addressed in evaluations, and how to navigate the challenges that arise when doing so. You can read the full case study here.
Header image courtesy of Footprint Evaluation Initiative by Reinout Dujardin from Pixabay.
We welcome you to join the conversation! Comment on this blog or reach out to us on social media: LinkedIn and Twitter. Have you participated in evaluations on environmental sustainability? Share your stories with us.
If you would like to contribute your knowledge to this blog, we would be happy to work with you - please contact us email@example.com.
You can also sign up for our newsletter here.