Measuring the Journey: From Theory to Impact

Dugan Fraser is the Program Manager for GEI.

Writing and editing support provided by Maria Fyodorova, Communications Consultant for GEI.

In my last blog, I talked about the key reasons why many governments still lack robust monitoring and evaluation systems to help guide their policy decisions.  Even though there is widespread demand from countries to have these sorts of systems in place, a variety of factors have kept this from happening.  You can read my thoughts on why here. I also talked about the approach that the Global Evaluation Initiative (GEI) is taking to address such challenges.

Today, I’d like to share with you a bit about how GEI will evaluate our progress.

As the GEI community sets off on its journey, a crucial question for us to keep track of is: how will we know if we are making a difference? We know we are on a long, complicated road, with far to go.  As we travel toward our goals, it will be important that we regularly reflect on our progress to improve what we are doing while we are doing it.

Indicators and Tools

The indicators we track are closely tied with GEI’s Theory of Change (a Theory of Change explains how activities produce a series of results that contribute to achieving the final intended impact).  In addition, to adequately monitor outputs – and most importantly, outcomes - from GEI activities, a targeted results framework has been created to ensure that the right data are collected and tracked over the course of the initiative.

Even though measuring outcomes is the priority of our self-evaluation efforts, we still recognize the value of keeping close track of our outputs, which we monitor regularly. In relation to the work we do with countries and their governments, some of the outputs we track include the number of countries where we support M&E diagnostics, the number of countries where we establish partnerships and the number countries where we provide technical assistance.    

GEI interventions also aim to facilitate the development of a supportive national M&E “infrastructure” with a strategic focus on country-owned and -led approaches and indigenous culture and knowledge.  This includes advancing the way civil society, higher education institutions and the private sector participate in and undertake M&E processes. We are also supporting voluntary organizations for professional evaluators (VOPEs) and finding ways to advance the participation of young and emerging evaluators in country systems. Some indicators to help us track this work include the number of people trained, the number of academic institutions that collaborate with us and the number of scholarships awarded.

To help us gather the right data, GEI has established several tools:

• GEI Mapping and Activity Dashboard. GEI has mapped our network’s global presence and has developed a related country analytics dashboard and data visualization tool. Network partners will continue to report on activities using GEI’s Management Information System. This dashboard is currently internal, but will be active here, on the GEI website, in Spring 2022.
• Monitoring and Evaluation Situational Analysis (MESA) tool. The MESA tool is a diagnostics tool that helps analyze and communicate the state of a country’s national M&E system. Repeated MESA exercises over time can help GEI show progress on our core outcome and impact indicators. The MESA tool will be shared publicly in Spring 2022.
• GEI Knowledge Platform. The GEI Knowledge Platform will provide GEI audiences with curated knowledge resources and convene stakeholders to learn, exchange lessons learned, and collaborate on knowledge gaps. The analytics from the Platform will help GEI track indicators on how target audiences are interacting with our knowledge products through metrics like downloads, citations, and comments. The Platform will be released in May 2022.

Innovative & Rigorous Approaches to Tracking our Impact

While it is necessary to monitor outputs, our real interest is in how well we perform in achieving long-term outcomes. We are assembling tools that will allow us to measure early signs that we are triggering changes - if (and how) practices and behaviors are changing around M&E.

We are planning a number of discrete evaluative exercises that include periodic evaluations of our work and case study exercises using rigorous techniques.  Repeated MESA exercises over time, in combination with solid case-based analysis, will help us understand GEI’s causal contribution to change in a particular country. Rigorous case-based methods such as process tracing, Qualitative Comparative Analysis will also be considered.

In addition, the use of other innovative causal inference methods such as conjoint experiments - that test how different bundles of activity (or “treatment combinations”) affect program results - will be considered. This type of evaluative work can be complemented by assessing GEI’s influence through a systematic analysis of textual data across the internet using web-scraping and machine learning techniques.

As we move forward with GEI’s global effort of championing and providing assistance for the use of evidence in public-sector decision-making, we are quite aware that evaluating the success of our work is a difficult challenge. How we measure our additionality relates to the real challenge of managing an evaluation system in a complex environment – an environment characterized by many interconnected (yet always evolving) program components.  This complexity increases as new partners and stakeholders contribute their experiences to GEI. However, for GEI to create the impact it seeks, our approach will always need to evolve, and we will always need to find creative and rigorous ways to track it – to help us improve as we travel forward.

We welcome you to join the conversation! Comment on this blog or reach out to us on social media: LinkedIn and Twitter. What has been your experience? Share your stories with us.
If you would like to contribute your knowledge to this blog, we would be happy to work with you - please contact us contactgei@globalevaluationinitiative.org.