Making Monitoring & Evaluation Systems Effective: A Discussion with Patricia Rogers

TRANSCRIPT:

---

[00:34.59]

Dugan Fraser

Hi, I'm Dugan Fraser. I'm Program Manager of GEI. I'm really pleased to have an exceptional guest with us today. She's a longtime colleague and friend, Patricia J Rogers. Patricia, it's really great to have you with us today.

Patricia J Rogers is the founder and former CEO of Better Evaluation and she was previously the professor of public sector evaluation at RMIT in Melbourne and the Australia and New Zealand School of Government. She's now working as a consultant, including for the GEI and as a member of the footprint evaluation initiative.

So, Patricia, one of the things we are most interested in at the GEI is country monitoring and evaluation systems. And I've been wondering what you think makes them effective. 

[01:21.57]

Patricia Rogers

Well, I guess one of the things for me is thinking about that focus on country-level monitoring and evaluation systems as actually being about multiple systems. When I look at what countries are doing, I see that there's some central monitoring and evaluation system, but there's also a system of supporting monitoring and evaluation in different departments, monitoring and evaluation with different partner organisations they work with. 

So, one of the things I think about what makes them effective is when they're not just thinking about one part of that large interlocked system of systems. And I was really thinking about this recently with a discussion I had a few years ago in Australia where I'm based, where the opposition party at a federal election put forward a proposal to establish the position of an evaluator-general, which made all the evaluators extremely excited at the notion that they would have the same sort of rank as an auditor- general. 

I was very interested in what that would mean, but I was actually quite disappointed with the details of what they were proposing. It was really just having a very small part of that big country-level monitoring and evaluation system. It was really only about conducting four or five very large evaluations each year. 

I mean, that's a really useful function, but it wasn't looking at all at performance monitoring, national reporting. It wasn't looking at supporting good M & E within departments. It wasn't looking at how you engage with statutory authorities and implementing partners. So, I think part of it is to think about the big systems as a whole.

I know that you've done a lot of work working with national government but also thinking about how they interact with subnational governments and that much bigger system. So, the first bit I think is to have that wider view of what it is we're talking about.

Then the second one is really around use. It's got to be about focusing on use and that's like any discrete evaluation or monitoring system or country-level monitoring evaluation system. There's such a risk that we fall into the trap of what we can get data about and how we get that data, and we put all our efforts and focus into that. I think like any evaluation, it's got to be focused on use.  

[03:52.55]

Dugan Fraser

I'm interested in what you're saying about this idea of systems and the demand it places on people to be thinking authentically about systems and understanding a kind of stewardship role really and acting as a traffic conductor who's able to take account of and make sense of all of these different data streams that are operating at so many different levels, which is really I think demanding, right? 

[04:24.30]

Patricia Rogers

Oh, it is and that's a great word, this notion of steward because I think it speaks to what sort of system we are talking about. You could think of an incredibly mechanistic one where you have a central part of the national government setting a whole lot of rules and regulations and indicators and they're all getting aggregated up and reported in some dashboards. There's that version that people often think about with the system.

But if you think about the more organic system, the living systems, then it's much more like stewardship. How do we encourage, nurture, and grow these things, provide the right incentives, provide the capacity so fabulous stuff is happening in all the departments not just doing that aggregating up of the single statistics?

[05:12.76]

Dugan Fraser

When I worked in government and I worked on what we called monitoring and evaluation systems, I came to understand after many years of that type of business that we were really talking about monitoring systems and that the evaluative side of things often got quite underdeveloped. What do you think makes for strong evaluation systems in particular? 

[05:36.50]

Patricia Rogers

I think being purposeful, having a clear direction that evaluations are not just something to be done to be done, to tick off, to be compliant. And that's one of the problems I see with a lot of organisations is that the evaluation system just says, "Do an evaluation." The planning for an evaluation rolls out the same set of questions, the same sort of template, and says, "Go and do it." 

It's a bit like my friend and colleague, Bron McDonald, talked about the way people talk about evaluations like brushing your teeth: "It's just good." "Just do it." "Don't think about it, just go ahead and do it."

And I was thinking about this while brushing my teeth today, but knowing why it's good for you helps you to do it well, even brushing your teeth, even something as simple as that, and doing evaluations even more so. You've really got to think about why are we doing this? Why are we doing this now?

A few years ago, what was then AusAID the Australian International Development Agency had quite a change in their evaluation policy, where they said, "We still require each program area to do an evaluation in the life of a program, but they get to choose when they do it." And they could put a lot of effort upfront because it's a new area and you want to do evaluative thinking about scenarios and options.

We might do a developmental evaluation. We might do a process evaluation. We might do a terminal one to say, "Did it work? Should we keep doing it? But they get to decide where to focus. And that, to me, was an example of an evaluation policy that is focused on use and purpose and that's the requirement. 

[07:18.52]

Dugan Fraser

So, this business of building monitoring and evaluation systems is hard. It takes a long time and I think what you're describing also takes a great deal of care because you really have to think about what it is that you're going to use this information for. Can you think of some ways in which short-term value can be delivered for people involved in building monitoring and evaluation systems that will take long to deliver their intended results? 

[07:48.39]

Patricia Rogers

Yeah, and I still have the image of you in your blog from last year about sitting in the back of a car on a long trip and saying, "Are we there yet?" I just have that image every time you talk about building M & E systems. I think we often take that unhelpful strategy of just saying, "Sit in the back, hang in there. I know it's been a lot of work, but one day this will be useful."

I think that is absolutely not the strategy and particularly when the situation's changing, the interventions are changing, the context is changing, the urgency is changing, and the people are changing. You might have someone who's got a particular focus on something, you get a change of government, you get a change of minister, you get a change of department head, or senior person, or funder and things change.

I think this notion that we'll sit in a corner for a year and build the perfect M & E system and then we'll roll it out is just not the way to do it. I think we have to build in some early wins, and that will do a number of things. It helps people see that there's some value in it after all. It helps them sometimes realize how much work is involved in getting data and how much work needs to be put into making sense of the data.

So, to go around the cycle to say, "Let's get some data that we think is going to be useful. Let's try to make use of it and let's think about what other data we need or how we need to present it, who we need to be involved in those discussions. Let's do a cycle and then let's plan again." 

That notion was buried away in one of Michael Patton's early books where he talked about that as reality testing for planning an evaluation. Again, an evaluation attempts to say what do we need to know? Here's the plan. Here's the data. Here's the report. And he said, "Do a cycle," and it's something he's revisited in the new edition of his book on Utilization-focused Evaluation.

I think it's incredibly powerful because often people don't know what they want until they see something and go, "Well, it's not that." I'm reminded of a few years ago, a government department was building its evaluation capacity and the program managers went to the funders of particular programs and said, "What do you want to have in this evaluation report that you require us to provide?" And largely, there was no idea. They wanted them to do an evaluation but they had no idea what that would be.

And it's actually a risk of shaming people that they then feel they ought to be able to say what it is. But let's take them on that journey, let's sit down with some data and say, "Well, we could report on this. We could do what's useful." It's not a multiple-choice question. It's not a put-you-on-the-spot. It's, "Let's discover this together, what we need to do, what information you need, and what information we need to do the implementation." 

[10:50.53]

Dugan Fraser

One of the things I've always found so revealing is when you work with people to develop key evaluation questions and they really don't know what questions they want answered. It's quite astonishing really how often we drift into evaluations without really knowing what the questions to be answered are. 

[11:12.46]

Patricia Rogers

We have this standard set of questions reliability, relevance, efficiency, which are great criteria. But if you just grind through those and particularly if you have great elaborations on them, and you've got 30 or 50 evaluation questions.

I was looking at a terms of reference just yesterday, where there's something like 40 key evaluation questions. Well, they're clustered but there are still sub-questions.  

[11:37.52]

Dugan Fraser

Yeah, the famous sub-question. 

[11:40.78]

Patricia Rogers

Yeah, the famous sub-question and they're good questions. And there's a couple of months to do this and you're going to go and do a bunch of interviews. And I just think, surely, we're beyond that now. There needs to be a real focus. I like the terms of reference that say, "We are doing this evaluation because here's how we're going to use it." This is the sort of information, who's going to use it, what they need to know, when they need to know it, and who needs to be involved to make that credible. That's a good evaluation terms of reference; they're not just rolling it out. 

I think you're right with the key evaluation questions often reflect that there's a lack of clarity about the purpose. I know several organizations that have tried to have a template as part of the plan where you state what's the purpose of the evaluation, and often people will put "accountability and learning." It just rolls it out and you go, "Well, that's really good. Who's being held accountable, to whom, for what, through what means? Who's going to be doing the learning and what will that mean for them? You need to unpack it and be clear about that and whether a single evaluation can meet these different purposes.
[13:03.96]

Dugan Fraser

So, when you think about designing a monitoring and evaluation system and I mean both monitoring and evaluation system, what are some of the steps you think we should go through? You've talked about doing it quite iteratively. You've described the process of taking care to learn what can be learned. Are there any other things you want to mention?  

[13:29.44]

Patricia Rogers

My experience has really been monitoring and evaluation systems for programs, often quite large programs and for organisations. So I haven't done the work you've done working at a country-level M & E system but working on some of the components of what would be in that system. Well, the means to it, obviously, is doing theories of change and really understanding; how do we understand this works and how good is that understanding? Are there links in the causal chain that are less certain?

I think that's a really helpful thing in terms of the focusing. Are there some long-term outcomes where we need some interim indicators of whether we're going to achieve them? So, I think part of the framing and the shaping of what the M & E system can do and where we need evaluations is part of that. How well do we understand this theory of change?

I mean, if it's well understood, the causal links are well known, the situation's stable. You could tick over a few indicators and that might be sufficient. Then, it's who needs it? How do we do nice dashboards or planning sessions? But if it's really not well understood or it's rapidly changing or might be, then part of your M & E system has to be not just how do we get the data and report it, but how do we build in support for use. 

So, I'm really interested in some of the systems that have been put in place; building it into our regular management processes or implementation processes is part of the M & E system. I think that's the other bit. Not just seeing the M & E person as a technician who scurries away and gets the data that someone's decided they need but supporting those by ensuring it's built into the processes. 

An example that stayed with me for many years was from New South Wales Railways where they had a series of indicators in four areas and they would have weekly meetings at each of their major sites. And they would look at the indicators related to one of those areas at those meetings. We were talking about it and he said, "So, what do you think is the most important thing?" And everyone said that the trains run on time. He said, "Well, punctuality is one but there's three others and that's not the first. The first one is safety, and we're going to look at that and we're going to have indicators."

And obviously, if there's a major event, they jumped to that straight away but they have this and the four other areas on a regular cycle. So they're sort of continuing to look at that and building it into their discussion and their review systems. That's one that's really stuck with me as a model about paying attention to the process and building it into being considered. 

[16:40.10]

Dugan Fraser

We recently had a launch of a GEI product called the Launchpad and I got involved in a bit of a Twitter dialogue with somebody who was very scathing about Monitoring and evaluation and said it was a waste of time and that it's been a big drain on resources in the context he was familiar with. Why do you think people have that attitude towards monitoring and evaluation? And what are some of the practices that have given rise to a negative view of M & E? 

[17:16.59]

Patricia Rogers

Well, often they've had experiences that have demonstrated that in that situation, M & E has been a waste of time and a diversion of resources. Some time ago, there was some discussion about the psychological aspects of M & E and people's resistance to it. I don't think that work paid sufficient attention to the fact that many people have a well-founded concern that a lot of M & E is ineffectual and wasteful at best, and harmful at worst. That it actually diverts efforts not just of the M & E people but of everyone else who's got to run around. It meets some targets that are really inappropriate and unhelpful.

There's a whole literature on the damaging effects of performance indicators, which I don't need to go into. I think part of that is seeing many bad examples and for people to be able to see that it can be different. Part of it is by looking at examples from elsewhere. I think it's really important, and obviously, to have some examples within them.

I think one of the worst moments of my professional life was when I was working with a department where one of the key people was very skeptical about evaluation. Our job was to develop a whole evaluation framework for this major new initiative. It was trying to do all the things we've talked about being useful and embedded. And he was deeply skeptical about the value of it at all.

So, I thought I would in a meeting use an appreciative inquiry approach and say, "Let's talk at an example where it's worked well and why do you think it worked well and how can we do more of that?" So I started, "Could you think of an example where some evaluation has been useful? And there was this long pause then he said, "No, I can't." My whole plan for this wonderful uplifting meeting just went out the window. 

[19:30.96]

Dugan Fraser

I think the only real way to combat that is for people to have the reverse experience. It's for people to actually have an experience of having an important question answered off of going through an evaluative process. 

[19:40.40]

Patricia Rogers

Exactly, which is another reason to do little bits early on.

[19:48.38]

Dugan Fraser

So, what you're really talking about is to be building up a practice that allows people to have good experiences to see the value. Are there any other bits of advice you'd like to share?

[20:00.71]

Patricia Rogers

Well, the other one that's a bit challenging is a lot of this is about being iterative and adaptive, which allows you to respond to new issues and new people and key issues. That's great and it's needed, but it also needs to be balanced with, "Oh my goodness, now we're going to rejig the theory of change again. We're rejigging our indicators. We've just set up our spreadsheets or our dashboards."

So I think there's something about having some constants and some variables and being very thoughtful about the resourcing for that, including people's attention. I think making things as rhythmic and simple and clear as possible but with scope to do something quite different and specific and idiosyncratic, and to try out some things on a small scale.

The other one is about the scalability. If you think about the other Rogers' theory of innovation, he talks about some of the features that help innovation grow. One of them is if it's scalable and the benefits are demonstrable, which brings us back again to the "are we there yet" discussion. If you're just having to hang in there and hope that it's going to be good, and you've had to commit this huge amount of money and resources, IT, and everything to this system, and you're committed, you've got this massive sunk cost, that's a very bad model for building it. 

So, if you can do something that's scalable, do something on a small scale and then you see some benefit and then you grow it. The department in Victoria in my home state that made the most progress early on with evaluation in the early 2000s did that through an iterative process, where they started with a single project and they started with quite a constrained evaluation. How can we make better sense of the existing data we have to tell a story about what we're doing and to understand what we're doing? 

And that was so immensely useful that that project manager who then became the program manager scaled it up to the program. Then when he became part of the division, the bigger unit, scaled it up to that. So, it was very much his personal experience and it was an approach that could be scaled up thoughtfully. He became probably the best advocate for evaluation I've seen. 

The first time I met him, I walked into the open-plan office and this man, who was 6 ft 3, was standing on a desk shouting excitedly at all the staff gathered around him saying, "You've got to do evaluation. It's really important." That's my benchmark of senior management support for evaluation. And it wasn't him just mouthing words; he had seen it. It had been immensely useful for him both within his project and with the partners he was working with and he totally got it. He got the risks in it. He said, "If we run a project and it doesn't work and we're not going to continue it, we need to look after the people (employees) so that they don't just get turfed, that they get reassigned." He thought through this stuff and was personally committed to it. 

[23:17.45]

Dugan Fraser

Patricia, I'm really appreciating the guidance you've given us today in relation to thinking about this as a journey that is best taken in little trips that offer some kind of returns and that don't leave you feeling like you're on an interminable sort of travel across the desert. I think what you're describing is so interesting about starting small and scaling up so that innovation is possible and you're testing and constantly building the system as you go. 

Any last thoughts on what makes M & E systems most useful for public policy and service delivery? I mean, we've been talking about this but just anything you want to add? 

[24:01.62]

Patricia Rogers

I guess it's in that last question. When we talk about public policy and service delivery, I think we often have systems to inform them quite separately. One of the things that's often missing is how do we connect them? How do we connect from public policy, this is a good direction to go, to what does that mean for how we manage implementation and how we do M & E of implementation? And then back the other way; what are we learning from M & E of service delivery? That should inform our public policy.

And the public policy might be of the kind where you say, "We need to do A for everyone, or we need to have a suite of options where we can do A, B, or C because service implementation has shown that we need a range of options." I think that's one of the other connections that we need. 

[24:53.28]

Dugan Fraser

So interesting. I hope this is the first of many conversations we have that we're able to share with people who listen to the GEI podcast. Thank you for joining us today. It's been really good to talk to you and I hope you have a great weekend. 

[25:06.49]

Patricia Rogers

Thank you! Thanks, Dugan. 

[25:08.80]

Dugan Fraser

Bye-bye.