Design, Monitoring and Evaluation for Peacebuilding

You are here

Theories of Change in Evaluation Approaches

A theory of change that adequately describes the actions, the desired change, and the underlying assumptions or strategy is essential for monitoring and evaluating programmes and projects.  The theory of change will help program staff and evaluators understand what the project is trying to achieve, how, and why.  Knowing this critical information will enable staff and evaluators to monitor and measure the desired results and compare them against the original theory of change.

Hot Resource! Theories of Change and Logic Models: Telling Them Apart by Helene Clark and Andrea A. Anderson

Using theories of change during the monitoring stage of project implementation provides feedback on whether a project, programme or strategy is ‘on track’ to accomplish the desired change and if the environment is evolving as anticipated in the project or programme design. 

While monitoring our assumptions is a critical step of implementation, it is not widely practised.  Nevertheless, the utility of such monitoring should not be discounted.  As our assumptions are monitored, data and perspective can illuminate whether all the design components were adequately taken into account.  This is particularly important in complex environments, where there are a myriad of factors working with and against our attempts to bring about change.  For instance, staff can ask themselves did we consider the right factors and dynamics in the initial analysis?  Has anything unexpected occurred in the environment that was not foreseen, and why wasn’t it foreseen?  Are there gaps in our strategy to bring about change? 

The power of using theories of change in evaluation enables evaluators to ask hard questions about why certain changes are expected, the assumptions of how the change process will unfold, and which outcomes are being selected to focus on and why.  When an evaluation incorporates a theory of change review, each theory should be critically reviewed for its relevance, efficacy and effectiveness as part of the evaluation and covered in the evaluation’s findings, conclusions and lessons learned.  Such analysis will help contribute to an understanding of approaches that work in addressing the underlying factors contributing to violence or working against peace.  

Through an analysis of the accuracy of its underlying theory or theories of change, a programme or project can identify whether a false or incomplete theory may be a key explanatory factor for a programme, project or policy’s failure—and why that theory was false or incomplete.  In contrast, good theories, where the assumptions that underpin them are validated, and where they have guided effective and predictive actions, demonstrate how theories of change can be an essential part of contributing to our understanding of successful interventions seeking to address issues of peace and conflict.

Theories of Change in Evaluation

There are many approaches to evaluation and some lend themselves more readily to the examination of theories of change.1  Listed below are seven potential approaches to evaluation that either emphasize or include an assessment of theories of change.

Theory-based Evaluation – Theory-based Evaluation helps assess whether underlying theories of change or assumptions of a programme are correct by identifying the causal linkages between different variables.  In a broad definition, any evaluation uncovering implicit or explicit assumptions, hypotheses or theories can be categorized as theory-based evaluation.  This approach is particularly useful for learning and accountability as it allows for identifying whether the success, failure or mixed results of the intervention was due to programme theories and assumptions, or implementation.

Most Significant Change – Most Significant Change (MSC) is a participatory monitoring and evaluation technique that provides information on impact and outcomes of an intervention that can be used to assess the performance of the intervention as a whole.  The essence of the MSC approach is a systematic identification and investigation of observed significant changes in the environment.  What differentiates the MSC approach from conventional evaluation approaches is that it uses an inductive approach, whereby intervention participants ‘make sense’ of events after they have happened, while conventional approaches would use predetermined indicators, based on prior conceptions or theories, for the explicitly intended change. 

Hot Resource! The Most Significant Change Technique: A Guide to its Use by Rick Davies and Jess Dart

Developmental EvaluationDevelopmental Evaluation (DE) supports the process of innovation within an organization and its activities by pairing an evaluator with a programme throughout the project cycle for continuous monitoring and evaluation for programme improvement.   It “emerged in response to the need to support real-time learning and adaptation in complex and emergent situations”, and is specifically “designed to capture system dynamics and surface innovative strategies and ideas.”2  Within DE, theories of change can be reconstructed when the evidence suggests that the theory is not working as thought in the design phase.  By comparing older models of change to newer ones within the same programme, one can gain valuable information and insights about how theories and the environment have evolved.

Hot Resource! DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozios, Marc Langlois, Natasha Blanchet-Cohen

Outcome MappingOutcome Mapping (OM) “recognizes that [change] is essentially about people relating to each other and their environment.”  It therefore shifts away from assessing the impacts and/or products of a programme to focusing on changes in behaviour, relationships, actions and activities in the people, groups, and organizations it works with directly (i.e., outcomes).  OM suggests that in order to bring about impact, there must be changes in the behaviour, relationships, actions and activities in the people, groups, and organizations that the intervention works with directly.     

Hot Resource! Outcome Mapping: A Method for Tracking Behavioural Changes in Development Programs by Terry Smutylo

Impact Evaluation – There are several definitions of impact evaluation in the field of development.  The most common definition relates to the OECD DAC criteria:  “Impact evaluation is the systematic identification of the effects—positive or negative, intended or unintended—on individual households, institutions, and the environment caused by a… program or project.”3  It can be an experimental or quasi-experimental design with a mixed methods approach, but frequently emphasizes quantitative data.  An impact evaluation that includes theory of change review (i.e., theory-based impact evaluation) seeks to identify why the change occurred, rather than just if the intervention had an impact.  Six key principles for theory-based impact evaluation are: 4

  1. Map out the causal chain (programme theory)
  2. Understand context
  3. Anticipate heterogeneity
  4. Rigorous evaluation of impact using a credible counterfactual
  5. Rigorous factual analysis
  6. Use mixed methods

Impact Evaluations can also be defined as an evaluation that takes place several months and years after a project has been concluded.  The purpose of conducting an evaluation post-implementation is to determine which changes have become sustainable and produce long-term impact on a complex environment. It seeks to determine “the change in the conflict (or crime) catalysed by the project.”5  This type of approach enables the evaluator to make recommendations on which theories of change in complex environments contributed to the resolution of the conflict or problem and helped to bring about peace. 

Hot Resource! Impact Evaluation: Methodological and Operational Issues by the Asian Development Bank

Empowerment Evaluation – “Empowerment evaluation aims to increase the probability of achieving program success by 1) providing program stakeholders with tools for planning, implementation, and self-evaluation of their program, and 2) mainstreaming evaluation as part of the planning and management of the program/organization.”6  It aims to help improve organizations achieve results by improving their capacity to do evaluation and use evaluation results to improve strategies.  It is a learn-by-doing process that aims for an organization to be able to evaluate its strategies without the assistance of an external empowerment evaluator.  Empowerment evaluation can, therefore, include theories of change at the discretion of the organization utilizing the approach.  A particular potential strength of empowerment evaluation and theories of change is the emphasis on developing internal organizational capacities for conducting future evaluations.   In-house evaluators, technical DME units within INGOs, and donors could conduct evaluations analyse the different components of theories of change.

Whatever approach to evaluation is taken, however, it can be helpful to articulate other, or alternative, theories of change for a programme or project.  The process of identifying other reasons for the change to have occurred or to not have occurred opens up new avenues to explore the theory of change’s validity or lack thereof.  This process can help in assessing and attributing results.

Hot Resource! Contribution Analysis: An Approach to Exploring Cause and Effect by John Mayne

Jonathan White manages the Learning Portal for DM&E for Peacebuilding at Search for Common Ground. Views expressed herein do not represent SFCG, the Learning Portal or its partners or affiliates.

In your article you posit that using theories of change during the monitoring stage is important for practitioners dconducting different types of projects in their fields. Of course, the use and understanding of a theory of change is important in all phases of the process, at it can lead to deeper understanding and further promote change. Throughout the course of my studies we have used theories of change in order to propose project ideas or to conduct an effective, pre-emptive analysis of the conflict situation we are facing. In your opinion, how would practitioners better implement the use of theories of change into their planning stage in order to better organize their intervention efforts? Is there a way to ensure that pracitioners will use theories of change, whether explicit or implicit, to make their programs even more effective?

 

Thank you.

Response to agil's comment:

 

I like the question that you posed on how can practitioners better incorporate theories of change during their planning stage on their intervention efforts. I agree with you that it is important to have this TOC  being make explicit during the analysis prior to carrying out an intervention.  My sense is that most practitioners are well aware of the importance of TOC, mainly because of the strong collaborative and learning network we have. Moreover, I believe that funders for peacebuilding programs  nowadays are  more appreciative of understanding TOC. They can see its relevance in making an impact and subsequently are able to incorporate this importance aspect as part of their grant. Perhaps this is one way practitioners can use theories of change to make their program more effective, by having it as a requirement by the funders

Response to agil's comment:

 

I like the question that you posed on how can practitioners better incorporate theories of change during their planning stage on their intervention efforts. I agree with you that it is important to have this TOC  being make explicit during the analysis prior to carrying out an intervention.  My sense is that most practitioners are well aware of the importance of TOC, mainly because of the strong collaborative and learning network we have. Moreover, I believe that funders for peacebuilding programs  nowadays are  more appreciative of understanding TOC. They can see its relevance in making an impact and subsequently are able to incorporate this importance aspect as part of their grant. Perhaps this is one way practitioners can use theories of change to make their program more effective, by having it as a requirement by the funders