Design, Monitoring and Evaluation for Peacebuilding

You are here

DM&E Tip: When to Use Outcome Mapping

If you read this week’s blog, you’ll already have an introductory grasp of Outcome Mapping methodology.

There are so many evaluative methodologies floating around that deciding which methodology, and which methods, to use can be quite a difficult task. Patricia Rogers, Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology University, is currently developing a website, BetterEvaluation (betterevaluation.org), to help us answer this very question. Until the site is live (later this year?), however, we are left to our own devices.

So, why would one chose to use Outcome Mapping over the traditional summative, mixed-methods evaluation that examines impact and seeks attribution? There are three major categories for you to consider when decided to use Outcome Mapping, and it is important for you to consider if these align with the project you seek to evaluate—if so, Outcome Mapping might be a good fit!

Type of Change

Outcome Mapping is concerned with the outcomes of a project, not impacts (i.e., long-term effects of outcomes, direct and in-direct, intended and un-intended), and specifically outcomes which relate to change in behavior, relationships, activities, actions of a particular group of people, groups, organizations and change.

Hot Tip! Outcome Mapping is not intended for a technical evaluation to assess the relevance of the programming area or an evaluation of the cost-effectiveness of one approach compared to another.1

The approach is meant to measure the process by which change occurs, instead of the end result or impact of the change. This is in contrast to seeking causal relations between the intervention and observable change in the environment, and a key consideration here is whether or not the donor—and for that matter your implementing organization—will accept such an approach. Signs are encouraging: CIDA and DFID both use Outcome Mapping, and indeed, the International Development Research Centre in Canada has found that OM can provide quite complimentary data sets and insights to the traditional evaluative forms that seek attribution and statement on impact.

Whose Change?

This second category relates to whose change you seek to measure. Outcome Mapping is particularly useful when you are trying to measure the changes on direct beneficiaries—what OM refers to as ‘Boundary Partners.’ It is therefore only concerned with the change processes which occurred in those whom the program directly interacts with.

Hot Tip! Outcome Mapping is not useful if you want to measure or understand change through or in in-direct partners.

Furthermore, Outcome Mapping suggests that influence on change is a reciprocal relationship: Boundary Partners are affected by the intervention and in turn affect the intervention. In some ways, this is quite similar to Developmental Evaluation.

Hot Tip! If you are not in a position to change the behavior of the intervention then this approach may not be right for you.

Process-Oriented

As a process-oriented methodology, Outcome Mapping uses graduated Progress Markers in order to determine how close the intervention came to achieving the desired outcome. These Progress Markers then form the basis for monitoring and evaluation; they are developed in collaboration with your Boundary Partners and collaboratively monitored. For more on Progress Markers see this week’s blog post introducing Outcome Mapping methodology.

This can become overwhelming, however, if your project has many Boundary Partners and many Progress Markers. Data overload is not useful, and since Progress Markers are decided on consensus, agreement can be difficult amongst a wide range of actors. Now this does not necessarily discount the use of this methodology; you can, for example, opt to report on Progress Markers through a sample of the Boundary Partners.

Hot Tip! Remember, OM is meant to be a flexible, complimentary approach to traditional M&E methods!

Hot Resources

Outcome Mapping: Building Learning and Reflection into Development Programs by Sarah Earl, Fred Carden and Terry Smutylo

Outcome Mapping: Those Who Dream to Make a Difference by Sarah Earl, Raj Kumar Verma, Adama Ndiaye, Thierry Barreto Fernandes and Kalpana Pant

Introduction to Outcome Mapping Webinar by Outcome Mapping (outcomemapping.ca)

Jonathan White is the Manager of the Learning Portal for DM&E for Peacebuilding at Search for Common Ground. Views expressed herein do not represent SFCG, the Learning Portal or its partners or affiliates

  • 1. Sarah Earl, Fred Carden, Terry Smutylo, Outcome Mapping: Building Learning and Reflection into Development Programs, pp.16.

Hi Jonathan,

Thanks for this great post!  It's a nice sum-up on the strengths and limitations of Outcome Mapping. I would just add that many people tend to use some of the principles of Outcome Mapping. So, for example, they ight use only the concept of "progress marker"  which is in some ways, like an indicator, but is meant to illustrate dynamic movement towards or away from an outcome. It's also meant as an aid to engage in an honest dialogue around what sorts of results are realistic in a given context -- a very important consideration in many unpredictable, peacebuilding contexts!  Many of the people we work with who have used OM, comment that the progress marker concept really helped them to have an honest conversation with their funder around what might be feasible to expect in different situations.  The fact the progress markers also act as a gradient of achievement (e.g. "expect to see, like to see, love to see"), allows users to calibrate funder expectations (and in some cases, their own expectations. We all get carried away with describing what we will acheive..). So - they ARE holding themselves accountable for results, but are doing so in a way that takes into account the non-linear and often unpredicatable nature of change.  Very important for peacebuilding.

All of this to say that we encourage people to take what they find useful from OM. Some people develop entire frameworks, other just use the elements that they find resonate with them or which they can combine with other M&E approaches without too much burden or data overload.

One thing I will say about OM:  You have to be comfortable with the idea of contribution (not attribution).  In theory, this should be easy to do, but in practice, its a huge mental shift for some folks.

Colleen Duggan, IDRC

Thank you very much Colleen for adding your wonderful insights!

I am really taken by the idea of Progress Markers and I would personally like to see this adopted more widely in this type of programming. It would be quite fascinating, I think, to conduct a strategic review of peacebuilding project evaluations utilizing Progress Markers to better understand the percent of projects which achieve one, two or three of the Marker levels. I could see this assisting in developing comparative indices of results across contexts for particular types of programming and holding ourselves more accountable to what is realistic within the timeframe and what is not--of course, timeframe, while related, is another issue entirely!

Best,

Jonathan

Dear Colleen and Jonathan,

I come across your discussion below on Outcome Mapping and it's use in practice. We have experience with Outcome Mapping in peace building and they are mixed. First, a positive aspect is that it really helps us to reflect on social changes we want to achieve. So for formulating outcomes this is very helpful. But then our network members have to report on it - this is the difficult part. I think the main difficulty lies in the 'contribution'. People are hesitant to report on paper to what extent they have contributed to a certain change at a key actor (boundary partner). In our case these can be national governments/state actors, UN bodies and Regional Intergovernmental Organisations (RIGOs). The hesitation can also be of cultural origin; in many cultures or countries there is rather an oral than a written culture. Moreover, especially in countries in conflict written statements can leave a dangerous trail.

What we are now looking for is a more interactive manner of reporting than only on paper, e.g. in an interactive workshop form. We use anyway the learning approaches of "Action Learning" and "Experiential Learning" and these are appropriate for interactive reflection on social changes.

One additional approach that I am exploring now is to make use of "Most Significant Change" evaluation. By asking more questions for stories it can help people to come up with examples from their own experience and reflect what changes they have seen at a given boundary partner, including at a personal level. This can also be considered part of the progress markers. But in our case we have simplified the expect-to-see / like-to-see / love-to-see in the planning phase. It yielded too much information in our global network.

Kind regards, Paul Kosterink, GPPAC 

(PME & Learning at global secretariat of Global Partnership for the Prevention of Armed Conflict)