Leveraging Evaluations for Sustainable Development: Opportunities, Challenges and a Way Forward

TRANSCRIPT:

[00:33]
Dugan Fraser: 
Ok. Hi, everybody. I'm Dugan Fraser. I'm the Program Manager of the Global Evaluation Initiative. I'm very, very, very happy to welcome Geeta Batra.  She is the Deputy Director of the Independent Evaluation function at the Global Environmental Facility and their Chief Evaluation Officer. Welcome.

Geeta Batra:
Thanks, Dugan. It's a pleasure to be here and have a conversation on this special celebration of IEG's 50th anniversary and especially with the GEI. Thank you.

[01:03]
Dugan:
One of the things that you get to do as part of your role at the GEF, is you review a lot of evaluation reports and you must get a really good sense of what's being done well and where you think people need to improve. What do you think the situation is with the current quality of evaluations being done?

Geeta:
I think, in general, evaluation has come a long way. There are a lot of things that are going well in evaluation.  For one, people are advancing the qualitative and quantitative methods pretty well - investing in that a lot. Involving stakeholders, beneficiaries, focusing on inclusion - I think that’s another area of growth.  Verification and more field work than ever before.  I think that's really valuable in evaluation. But I think that there's always areas one can do more and for one, being from the Global Environmental Facility, I think evaluators can definitely pay more attention to mainstreaming environmental issues in evaluation work. I think another one is probably, you know … we've heard the words “systems thinking” by now quite often in in the evaluation sphere. And I think one area where evaluators can definitely do more is actually applying systems thinking in evaluation work. 

[02:30]
Dugan:
I think this notion of systems is absolutely central to the way the GEI is thinking about strengthening evaluation capacities.  But I want to ask you a question that really plays on my mind, which is how often interventions that achieve their intended objectives get treated as having performed satisfactorily, when in fact their environmental impacts were often negative.  And the problem was that the evaluation didn't look at those impacts.  Is that something you think is a problem?  And if so, what do we do about this?  How do we change this practice?

Geeta:
That's a tough one, but I think about, the age-old saying in evaluation, “what gets measured gets done.”  One of the factors today is that environmental effects have not made their way - mainstreamed - into the indicators that are used often to determine whether projects have met their objectives or not. And I think starting there, it would be nice, especially from an accountability perspective, to make sure that environmental effects are included in the indicators or the metrics that are used to assess effectiveness. That's for starters at the very outset. I think another thing is, another area is, you know, let's look at unintended consequences. I think that's another place where people could easily look at environmental effects. What are the unintended consequences of these for the environment? We're all committed to that, and we can easily mainstream such measures.

[04:14]
Dugan: 
So if you're an evaluator and you've been asked to undertake an evaluation of an intervention, but the commissioner of that evaluation hasn't asked about environmental effects, how does the evaluator go about including those in the study? We need some sneaky tips and tricks.

Geeta:
I think it's fundamentally, you know, starting to demonstrate how you would actually show that these environmental effects are important for the intervention. I think people, once they see things, they start believing in it a little more.  I think you can tell them many stories, but unless you've demonstrated to them that there are environmental effects that actually make a difference, they're not going to do anything about it.  But I think GEI and other trainers are very well-placed to actually work with - whether it's project practitioners or government officials, etc. - to start with theories of change. You’re training and you say, “Why don't you think beyond this? Let’s just look at this theory of change for now and see that it's not as linear in thinking as you think it is. So maybe let's look at, does environment play a role somewhere here? Do you think environment is going to affect some of the outcomes of this project?” I think that's a good starting point. I think GEI and other agencies - and even we're [GEF] are trying to do this - actually train people to think differently.

[05:44]
Dugan:
Of course, training people to think [differently] is really difficult.  I mean, it's easy to train people in how to do things, but to actually revisit the way you actually approach and think about a situation is a challenge. And I think it's something that takes a long time.  You were going to ask me a question?

Geeta:
I was going to say, what do you see as limitations and challenges working with government officials or others in countries to mainstreaming environment or even thinking about environment in their projects?

[06:18]
Dugan:
I think historically people really had to be challenged to take onboard these issues because they would be very centered around their particular sector. So, if it was a transport intervention, they would be thinking about mobility. If it was a health intervention, they would be thinking about health outcomes. But I think over the last year, and maybe slightly longer, it's become impossible not to think about environmental impacts. And so, it's becoming much more widespread that people need to factor in these unintended consequences into their intervention theories of change or logics. And I think that there is a greater willingness to have this conversation.

I think you know the Footprint Initiative, which is something the GEI has helped support, and which is starting to call itself Sustainability-Inclusive Evaluation. What we're really trying to do is to build on this momentum that, I think, is growing in country governments to include environmental considerations into all evaluations. But at the same time, it's hard -  it's very difficult when already an intervention is having a whole range of different outcomes to say to people that this is one of the more important ones. And this is something that requires your attention. And in many instances, it's about your ability to make the case and to be persuasive and to articulate a value proposition that can't be discounted. And that's getting easier as the negative consequences of climate change become so much more evident and visible in people's lives. 

As you think about the evaluations that you've been seeing, the best ones, what do you think are some of the really great practices that are making you happy?

Geeta:
I think people have really started to push the envelope on using a lot of good qualitative analyses, text analytics, quantitative methods, geospatial approaches (which actually tell you where this project is taking place - even evaluators would not know this, right?) So, I think all that is a step in the right direction. Also focusing more on gender inclusion and bringing public and private sector together. This is of particular importance in environmental considerations as it was in infrastructure many years ago - recognizing that this is not a public funding problem alone and you've got to bring the private sector to the table. But trying to understand the contributions and the role that the private sector plays.

There's as much of an educational role that evaluation plays, as much as an accountability role. So, trying to understand those models that work. There was blended finance and climate many years ago.  What works well for biodiversity today? Trying to come up with and understand the context, understand possibilities for solutions and then guide practitioners. So, I think evaluations beginning to play more of a role there and that's really important. I think where evaluators need to be different is taking more risk and innovating a little more.
So, we expect project people and government practitioners, you know, to take on more risk. Is 80% success too high? What about evaluation taking on some of those risks? Everything doesn't work the first time.  
And I think evaluation is a science and an art. I was watching a really interesting film recently. It comes to mind. It was just last week. It was National Science Foundation funded, “Where Are the Blue Whales Today?” It’s about understanding a species, an endangered species. And they were so candid during the beautifully made film about how the scientists were trying to locate and tag this particular blue whale for many days. And yet, after about more than 25 days, they still couldn't tag the whale. Science can make mistakes, too.  And arts, I mean evaluation is a combination of a science and an art. So I think taking some risks, trying to do things differently as opposed to the old ways of doing things, is important. Changing the mindset of an evaluator is as important as trying to expect project practitioners to change their mindsets.

[11:06]
Dugan:
I agree. I think evaluators need to be encouraged to think differently, and that's partly what the GEI is all about.  
In the environmental space, there are more and more commitments that countries are signing up to various conventions, and these require that they report against their adherence to the things that they've agreed to. And this is having the effect of causing a proliferation of monitoring and evaluation systems in a way that I think is not particularly helpful and really taxes many of these institutions. What do you think can be done to help countries avoid the situation where for every new convention they sign, they have to create a new M&E system?

Geeta:
One thing, for example, there's one area of work that the GEF does is through its enabling activities program is actually helping countries on their obligations to the different conventions.  And we recently did an evaluation of that to see exactly this issue, where can things be streamlined?  And our recommendations were centered around this, which is, can you, for example, harmonize some of the reporting requirements?  Those recommendations were actually taken on board and they're trying to harmonize cycles because otherwise you're first reporting to the climate convention, then you're reporting to the chemicals convention - it just is endless for countries.  So that's one and I think conventions are looking to do more of that.  The 2nd is for projects and others, and especially when GEI goes out to train, etc., making sure that the metrics that the countries are actually adopting are well-aligned with these conventions, so that you're not adding even more onerous reporting requirements through indicators, etc.

I wanted to ask you, what do you think could be done in countries to strengthen some of these areas like in monitoring and reporting? Where do you see scope for harmonizing?  Because I think there's a role for GEI here.

[13:24]
Dugan:
Obviously, this is a question that keeps us up at night.  And it relates to the age-old question of how do you herd a massive crowd of cats, right?  Because in many instances, the people who are financing this work - or the people who are implementing this work - want their own data systems.  They want their own mechanisms to tell them how they're doing.  What the GEI offers is a platform or a mechanism for coordination and alignment.  But if people don't want to participate in these kinds of things, you can't make them. And so, what we need to do is we need to prove to people that there is value in the working with us and that by collaborating with us, their lives get easier. And this has to be based on real benefits to the implementers.  They need to start seeing these M&E systems as offering value to them because if they're going to wait five or six years before there's an impact evaluation, they'll be like, “No, I'm not doing this, this is too much work and the returns are too, too poor.”

But the GEI is, I think, making some progress in getting this commitment from the people it works with. And it's a taxing task, which is why we have to limit the number of countries we work in.  And for us, the environmental space is one that by doing work there well, we will do other kinds of work well too. So, if environmental related monitoring and evaluation is successfully done, I think we'll be in a good place to report on fragility and conflict. We'll be in a good space to work on, to report on gender. And, so, this is kind of one of the big reasons that environmental sustainability is a major strategic focus for the GEI. 

As you think about the GEI and the World Bank more broadly, including its close partners like the GEF, what do you think we need to be doing to make the systems that we're working with better able to report on environmental considerations?

Geeta:
That's a tough one.  I think, you know, strengthening technical capability. Because, I think, monitoring and evaluation is definitely not at the level of the advanced economies in the developing countries, right. So, there's a lot of opportunity.  It's getting better.  You're seeing a lot more … the function actually developing a lot more. I think the CLEAR centers under GEI are doing a good job. So, you're getting a lot more evaluation capability in countries now. But I think there has to be more coordination.  And you know, I always call this the not invented here syndrome.  So, if I've not done it, that means it doesn't exist anywhere else.  That just has to be dealt with and overcome, whether it's from an evaluator perspective or a country's perspective. I think a lot more role for sharing of knowledge, taking risks and innovating, but learning from that innovation whether it's in monitoring or in evaluation.  And the other is, I think, making evaluation a little more agile, as well.

You refer to it that you want to give more real time information and I think there's a demand even from donors, from the GEF, I'm sure from the [World] Bank Board, etc., about what can we learn from ongoing programs.  And there's always this dilemma in the case of independent evaluation - you've got to keep the independence, you can't break that down. But I think you can easily inform doing quality-at-entry work to learn from it and say how has the past evidence worked to inform the ongoing or the newly designed programs.  You know, you're not evaluating a new program, but you can definitely say, have lessons been applied to avoid the same mistakes? And I think you can do that for countries, too. But building a platform where people are actually willing to share and willing to take on other people's evaluation findings rather than repeating, reinvesting dollars into the same thing over again, you can start innovating and doing different things right? I think GEI could actually play and forge that role of bringing people together, bringing agencies together.  I think a lot more can be done on that front.

[18:06]
Dugan:
That's music to our ears. And it’s especially music to our ears in relation to what you're saying about knowledge. A lot of what the GEI does produces new and interesting knowledge that we need to be able to share. And the Better Evaluation platform is intended exactly for that. 
We've kind of come to the end of our chat.  

Have you got any other points you'd like to make before we wrap up?

Geeta:
I have one final question for you, Dugan.  How do you see GEI being the glue that brings all of us together?

[18:39]
Dugan:
You know it's the job that we are busy with.  It takes a while.  Strengthening systems is a multi-layer, multi-stakeholder, multi-issue challenge and ultimately really it takes a commitment from a big crew of stakeholders. And that happens over time. What we're really doing is building our brand, creating the relationships. And more than anything, really centering country governments in this conversation because unless these systems are country-owned and country-led, they won't get the momentum that they need.  This requires a bit of humility on our part and a willingness to allow others to lead, which is sometimes a little frustrating because we think we know better, but we seldom do. This is work we're committed to and that we're busy engaging with and we need all the partnerships, all the strong relationships that we can get.  And I think we're on a good track. And one of the partnerships that we value most is our partnership with you guys. So thank you very much for being with us today.

Geeta:
And we look forward to actually engaging with GEI to get our work of mainstreaming environmental evaluation more broadly into country systems and even other evaluation work more broadly.
Thank you.

[20:10]
Dugan:
Yay. Anybody got any fascinating questions? Anyone?

Audience Question: Why should environmental considerations be part of national evaluation systems from your point of view.

[20:24]
Dugan:
So the question is why, very briefly, should environmental considerations be part of evaluations?

Geeta:
So I think for the longest time, there's always been this separation between environment, social, development, etc. And you saw most development practitioners not taking environment into consideration. And vice versa, most of the environmental staff would [say] this is my environmental project - I'm not going to be looking at socio and economic outcomes. Come the SDG movement - which a lot of countries are signed on to - the SDG goals.
The SDG is a perfect cake that actually shows the links finally between environment, social and economic systems, right?  It's so clearly intertwined. It took the pandemic for people to get convinced about this. So, I think there's growing recognition and understanding. This is not a new thing. This has been happening over our lifetimes, but it's more the promoting the understanding of the linkages amongst these three systems and how each one effects each other in the context of a project or an intervention.  And you don't have to give the same level of importance to each of these dimensions in every intervention that you're undertaking. But be cognizant that there could be effects.  You know, let's follow the no harm done to the minimum even if you're not anticipating positive effects. I think it's just paying attention to that at the back of your mind.  And like I said before about the systems approach - we are operating in a system and where everything is linked you can define boundaries for the system.  You don't have to say that this is my first order intervention. I'm going to go way into space to answer, to look at every possible link at every level, but at least surrounding the intervention, what are the factors that interplay?

[22:23]
Dugan: 
For me, life is contradictory and complicated and it's very hard to make sense of things, but if you understand how an intervention is impacting the environment, I think you get a really good insight into what's happening, and you'll learn a lot.  And that's why I think it's really important to put the environmental considerations at the center of any evaluation.