Design, Monitoring and Evaluation for Peacebuilding

You are here

Measuring Social Change: Integrating Equity and Social Justice Issues in Evaluation

In July 2013 the Institute of International Education (IIE) launched a groundbreaking 10-year Alumni Tracking Study to explore and analyze the impact of the Ford Foundation International Fellowships Program (IFP) on its more than 4,300 alumni from 22 countries.  As part of this effort, our evaluation team has discussed the importance of addressing social justice issues and equity in our evaluation design.  

“If successful, IFP would transform a traditional mechanism — an individual fellowship program for graduate degree study — into a powerful tool for reversing discrimination and reducing long-standing inequalities in higher education and in societies at large.”

-- Executive Director Joan Dassin, from Linking Higher Education to Social Change; May 2013.

For the Ford Foundation International Fellowships Program (IFP), the term social justice was linked to the program’s dual vision of 1) providing access and equity in higher education to leaders in marginalized communities and 2) contributing to social justice leadership in diverse communities.  For over a decade (2001 – 2012), IFP enabled emerging social justice leaders from marginalized communities in Asia, Africa, Latin America, the Middle East, and Russia to pursue advanced degrees at more than 600 universities in nearly 50 countries. 

 The IFP Alumni Tracking Study is integrating social justice issues into an evaluation design that has an international and cross-cultural focus.  The study offers a rare opportunity to explore over the long term (2013 – 2023) how an innovative higher education program affected the lives of its beneficiaries and the communities in which they live and work, and to examine whether it shifted the picture of equity and access in developing countries and within underserved populations.  Assessments of graduate scholarship programs have mostly focused on processes and outputs, such as rates of completion and employment. Though this data is critical to understand the short-term outcomes of a program, most methodologies lack a more focused examination of the pathways of program participants and how these relate to equity issues. The importance of measuring program success over time is to understand the “What happened next?” factor of these types of interventions. 

 Transformative Evaluation and Mixed Methods

In order to build an equity-focused evaluation framework, we have explored research around transformative evaluation.  How are we integrating a transformative paradigm in the IFP Alumni Tracking Study?  We are focusing on the importance of mixed methods:  using a combination of quantitative and qualitative methods to understand any equity gaps in results, but also to see whether IFP was able to meet the equity process.  While our quantitative methods will look at themes across the whole population, the qualitative methods will explore the community perspectives, understanding the complexities of the research and the research setting.  As part of this, we need to create a social justice “lens” for our research:  Whose version of reality is being defined?

Further, our own approach to the evaluation is focused on the importance of equity:  in evaluation access, participation and accomplishments.  We position ourselves as equity evaluators by keeping in mind, as we review historic data and reflection data, the equity of historic voice.  In preparation for our primary data collection, we recognize the equity of distribution of voice across the IFP population.  Program research indicates that IFP fellows experienced a range of social injustices at some point in their lives due to poverty, ethnicity, gender, race, religion, sexuality, political discrimination, violence or war, and coming from or living in a rural area or a conflict region.  The study team considers among its key research questions the extent to which IFP alumni, as a result of their fellowship, have addressed and reflected on these injustices not only as individuals, but also as leaders in their communities.  Do they view themselves as marginalized?  Are they still seen as marginalized?  What is the role of IFP in this change?

Challenges

The importance of a rights-based approach to evaluation, specifically an awareness of the complexities surrounding local contexts and subjects, is critical in programs that span several countries and populations. The challenges of this type of evaluation are defining and measuring social change over time, and the political and cultural pressures of local contexts from an international perspective.

Our research indicates that it is difficult to implement equity-focused evaluation and many organizations avoid this type of research. The first reason is the financial burden of additive data collection and design considerations.  For IFP, another challenge of the study is related to the practical challenge of tracing alumni once a program has concluded.  Alumni may move and change their address, change their name in the case of marriage, or simply fall out of contact with the program.  This is further exacerbated when the program concludes and no one maintains contact with the alumni.  

We are also cognizant of the challenges of an evaluation of this size and scope.  While we have opportunities for participatory evaluation, for example, we are limited by the fact that IFP spanned 22 countries and that the country and regional interpretations of the program’s theory of change may vary.  As we identify local researchers in data collection, we are considering the possibility of including IFP alumni and International Partners as local researchers to provide valuable perspectives to the evaluation questions. 

Studying the link between higher education and social justice, and the effect that higher education can have on marginalized populations and leadership, allows us to understand the sustainability of the IFP process.  Our evaluation framework further contributes to social justice because the study itself gives voice and agency to its study participants. 

For more information about the study and to follow our progress, please visit the IIE Website: IFP Alumni Tracking Study

To learn about the IFP program and its alumni, please visit the IFP Legacy Website

See also: The Power Of Many, a video produced by the Ford Foundation:

 

Mirka Tvaruzkova is a Senior Evaluation Officer at the Institute of International Education (IIE) and manages performance and impact evaluations of international fellowship and scholarship programs, including many programs that seek to develop leadership skills or promote access to higher education among underserved populations.  She is the Project Director of the Ford Foundation IFP Alumni Tracking Study.

 

Other resources:

Dassin, J.  (2009). Higher education as a vehicle for social justice: Possibilities and constraints. In Origins, journeys and returns: social justice in international higher education.  New York:  Columbia University Press.

Mertens, D. M. (2008). Transformative research and evaluation. Guilford press.

Mertens, D. M. (2007). Transformative Paradigm Mixed Methods and Social Justice. Journal of mixed methods research, 1(3), 212-225.

Nussbaum, M. (2006). Education and democratic citizenship: Capabilities and quality education. Journal of Human Development, 7(3), 385-395.