MSD competencies - BEAM Exchange (BEAM Exchange)
A catalogue of the full range of knowledge, skills and aptitudes found in high-performing teams that use the market systems approach. Useful for: Practitioners - identify personal training needs and continue your professional development Trainers - diversify your courses and refine your training / capacity-building offers Managers - strengthen your recruitment and induction processes

Appreciative Inquiry - Better Evaluation (Better Evaluation)
Appreciative Inquiry is an approach to organisational change which focuses on strengths rather than on weaknesses - quite different to many approaches to evaluation which focus on deficits and problems. "Appreciative Inquiry is about the coevolutionary search for the best in people, their organizations, and the relevant world around them. In its broadest focus, it involves systematic discovery of what gives “life” to a living system when it is most alive, most effective, and most constructively capable in economic, ecological, and human terms. AI involves, in a central way, the art and practice of asking questions that strengthen a system’s capacity to apprehend, anticipate, and heighten positive potential." (Cooperrider & Whitney 2005, p.3) Appreciative Inquiry is often presented in terms of a 4 step process around an affirmative topic choice: 1. DISCOVER: What gives life? What is the best? Appreciating and identifying processes that work well. 2. DREAM: What might be? What is the world calling for? Envisioning results, and how things might work well in the future. 3. DESIGN: What should be--the ideal? Co-constructing - planning and prioritizing processes that would work well. 4. DESTINY (or DELIVER): How to empower, learn and adjust/improvise? Sustaining the change While Appreciative Inquiry has always had an evaluative focus (working out what is working well and seeking to improve performance and conditions), in recent years there have been explicit efforts to embed AI principles and processes in formal evaluation processes: "Appreciative Inquiry (AI) is a group process that inquires into, identifies and further develops the best of “what is” in organizations in order to create a better future. Often used in the organization development field as an approach to large-scale change, it is a means for addressing issues, challenges, changes and concerns of an organization in ways that builds on the successful, effective and energizing experiences of its members. Underlying AI is a belief that the questions we ask are critical to the world we create." (Preskill & Catsambas 2006 p2)

Case Study - Better Evaluation (Better Evaluation)
A case study focuses on a particular unit - a person, a site, a project. It often uses a combination of quantitative and qualitative data. Case studies can be particularly useful for understanding how different elements fit together and how different elements (implementation, context and other factors) have produced the observed impacts. There are different types of case studies, which can be used for different purposes in evaluation. The GAO (Government Accountability Office) has described six different types of case study: 1. Illustrative: This is descriptive in character and intended to add realism and in-depth examples to other information about a program or policy. (These are often used to complement quantitative data by providing examples of the overall findings). 2. Exploratory: This is also descriptive but is aimed at generating hypotheses for later investigation rather than simply providing illustration. 3. Critical instance: This examines a single instance of unique interest, or serves as a critical test of an assertion about a program, problem or strategy. 4. Program implementation. This investigates operations, often at several sites, and often with reference to a set of norms or standards about implementation processes. 5. Program effects. This examines the causal links between the program and observed effects (outputs, outcomes or impacts, depending on the timing of the evaluation) and usually involves multisite, multimethod evaluations. 6. Cumulative. This brings together findings from many case studies to answer evaluative questions.

Causal Link Monitoring - Better Evaluation (Better Evaluation)
Causal Link Monitoring (CLM) integrates design and monitoring to support adaptive management of projects. CLM helps project planners and managers identify the processes that are required to achieve desired results, and then to observe whether those processes take place, and how. Result-producing processes specify the causal links between results in a logic model or results framework—in other words, the processes between results. CLM focuses on how specific individuals or organizations use one result to achieve another result. Steps in Causal Link Monitoring In CLM, planners start by creating a logic model to help document predictable, agreed-upon elements of the project. Next, they refine the causal links by describing the processes that will transform results at one point in the causal chain to the next. Planners are often less certain about these result-producing processes. Finally, the CLM logic model is enhanced with information about two important sources of uncertainty, contextual factors that may influence the project and diverse perspectives on the problem and its solution. The process can be broken down into seven steps, three for project design, three for monitoring, and a final step in which monitoring data informs redesign: Build a logic model. Identify assumptions about causal links. Enhance the logic model with diverse perspectives and contextual factors. Prioritize areas of observation. Collect monitoring data. Interpret and use monitoring data for adaptive management. Revise the logic model.

Contribution Analysis - Better Evaluation (Better Evaluation)
Contribution Analysis is an approach for assessing causal questions and inferring causality in real-life program evaluations. It offers a step-by-step approach designed to help managers, researchers, and policymakers arrive at conclusions about the contribution their program has made (or is currently making) to particular outcomes. The essential value of contribution analysis is that it offers an approach designed to reduce uncertainty about the contribution the intervention is making to the observed results through an increased understanding of why the observed results have occurred (or not!) and the roles played by the intervention and other internal and external factors. Contribution analysis is particularly useful in situations where the programme is not experimental, i.e. not in trial projects but in situations where the programme has been funded on the basis of a relatively clearly articulated theory of change and where there is little or no scope for varying how the program is implemented. Contribution analysis helps to confirm or revise a theory of change; it is not intended to be used to surface or uncover and display a hitherto implicit or inexplicit theory of change. The report from a contribution analysis is not definitive proof, but rather provides evidence and a line of reasoning from which we can draw a plausible conclusion that, within some level of confidence, the program has made an important contribution to the documented results.

Critical Systems Heuristics - Better Evaluation (Better Evaluation)
Critical System Heuristics (CSH) provides a framework of questions about a program including what is (and what ought to be) its purpose and its source of legitimacy and who are (and who ought to be) its intended beneficiaries. CSH, as developed by Werner Ulrich and later elaborated upon in collaboration with Martin Reynolds, is an approach used to surface, elaborate, and critically consider boundary judgments, that is, the ways in which people/groups decide what is relevant to the system of interest (any situation of concern). CSH is concerned not only with purposive evaluation, where the system or project has a predefined goal and the focus lies in evaluating the means of reaching it, but also more broadly with purposeful evaluation, where both the means and the ends become subjects of inquiry. CSH rests on the foundations of systems thinking and practical philosophy, both of which emphasize the 'infinite richness' of the real world. In this view, understandings of any situation are inherently incomplete, and therefore based on the selective application of knowledge. By systematically questioning the sources of motivation, control, expertise, and legitimation in the system of interest, CSH allows users to make their boundary judgments explicit and defensible. The immediate goal of a CSH evaluation is to elaborate multiple perspectives on a given situation, but the broader aim is to share these perspectives and thereby cut down on actors 'talking past' each other by promoting mutual understanding.

Democratic Evaluation - Better Evaluation (Better Evaluation)
Democratic Evaluation is an approach where the aim of the evaluation is to serve the whole community. This allows people to be informed of what others are doing and sees the evaluator as someone who brokers the process. It generally focuses on inclusive practices which foster participation and collaboration. However it is also used as a means of ensuring public accountability and transparency.

Developmental Evaluation - Better Evaluation (Better Evaluation)
Developmental Evaluation (DE) is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments. DE originators liken their approach to the role of research & development in the private sector product development process because it facilitates real-time, or close to real-time, feedback to program staff thus facilitating a continuous development loop. Michael Quinn Patton is careful to describe this approach as one choice that is responsive to context. This approach is not intended as the solution to every situation. Development evaluation is particularly suited to innovation, radical program re-design, replication, complex issues, crises In these situations, DE can help by: framing concepts, test quick iterations, tracking developments, surfacing issues. This description is from Patton (2010) Developmental Evaluation. Applying Complexity Concepts to Enhance Innovation and Use... "Developmental Evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments. Innovations can take the form of new projects, programs, products, organizational changes, policy reforms, and system interventions. A complex system is characterized by a large number of interacting and interdependent elements in which there is no central control. Patterns of change emerge from rapid, real time interactions that generate learning, evolution, and development – if one is paying attention and knows how to observe and capture the important and emergent patterns. Complex environments for social interventions and innovations are those in which what to do to solve problems is uncertain and key stakeholders are in conflict about how to proceed."

Empowerment Evaluation - Better Evaluation (Better Evaluation)
Empowerment Evaluation is an approach which provides communities with the tools and knowledge that allows them to monitor and evaluate their own performance. First developed by David Fetterman who describe it as “the use of evaluation concepts, techniques, and findings to foster improvement and self-determination.”

Horizontal Evaluation - Better Evaluation (Better Evaluation)
Horizontal evaluation is an approach that combines self-assessment by local participants and external review by peers. Originally developed to evaluate new methodologies for agricultural research and development, horizontal evaluation has wider potential for application. In its original setting, the focus of horizontal evaluation is the actual R&D methodology itself rather than the project per se or the team or organisation that developed it

Innovation History - Better Evaluation (Better Evaluation)
Preparing an ‘innovation history’ is an option for recording and reflecting on an innovation process. People who have been involved in the innovation jointly construct a detailed written account (sometimes referred to as a ‘learning history’) based on their recollections and on available documents. The process of preparing this history stimulates discussion, reflection and learning among stakeholders. Subsequent planning drawing on the innovation history, can (i) build on the lessons learned, (ii) inform a shared vision, (iii) act as a catalyst for change and (iv) improve future performance. An innovation history is developed in stages. Based on the initial detailed account of the innovation process, more concise informational products can be prepared that summarize the innovation process for internal use. Products designed for wider dissemination of findings can help external parties build upon and expand their knowledge and understanding about how innovations are brought about. Such products may include public awareness materials, policy briefs and articles in professional journals. They may be based on the study of an individual case or on reviews that compare and contrast experiences across several cases. Innovation histories are underpinned by two sets of concepts that guide data gathering and analysis. The first set comes from a model of the innovation process called the ‘learning selection’ model (Douthwaite, 2002). The second set is derived from social network analysis (e.g. Cross and Parker, 2004). These concepts help participants involved in creating the innovation history to appreciate innovation as an evolutionary process driven by experiential learning cycles. The experimentation and learning involved in this evolutionary process leads to the generation of novelty, followed by its selection and promulgation. During this process, technologies become ‘fitter’, in the sense that they perform better. Stakeholder group’s social networks influence how this evolutionary learning selection process evolves hence the importance of identifying, tracking and documenting them.

Institutional Histories - Better Evaluation (Better Evaluation)
An institutional history (IH) is a narrative that records key points about how institutional arrangements – new ways of working – have evolved over time and have created and contributed to more effective ways to achieve project or programme goals. An IH is generated and recorded in a collaborative way by scientists, farmers and other stakeholders. A key intention behind institutional histories is to introduce institutional factors into the legitimate narrative of success and failure in research organizations. Histories can be written by using interviews and ‘writeshops’ to construct a timeline, gain a clear understanding of roles and relationships, enquire into what triggers successful innovations and reflect on failures. Lessons drawn from this analysis can be used to improve performance. The dialogue that is promoted between the actors during the preparation of institutional histories can promote learning and capacity building.

Most Significant Change - Better Evaluation (Better Evaluation)
The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why. The are three basic steps in using MSC: Deciding the types of stories that should be collected (stories about what - for example, about practice change or health outcomes or empowerment) Collecting the stories and determining which stories are the most significant Sharing the stories and discussion of values with stakeholders and contributors so that learning happens about what is valued. MSC is not just about collecting and reporting stories but about having processes to learn from these stories – in particular, to learn about the similarities and differences in what different groups and individuals value. It provides some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders. By itself it is not sufficient for impact evaluation as it does not provide information about the usual experience but about the extremes. If you imagine a normal distribution of outcomes for individuals then the stories often come from the extremity of positive change. It can be useful to explicitly add a process to generate and collect stories from the extremity of little or negative change. MSC can be very helpful in explaining HOW change comes about (processes and causal mechanisms) and WHEN (in what situations and contexts). It can therefore be useful to support the development of programme theory (theory of change, logic models).

Participatory Evaluation - Better Evaluation (Better Wvaluation)
Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study. A participatory approach can be taken with any impact evaluation design, and with quantitative and qualitative data. However, the type and level of stakeholder involvement will necessarily vary between different types, for example between a local level impact evaluation and an evaluation of policy changes (Gujit 2014, p.1). It is important to consider the purpose of involving stakeholders, and which stakeholders should be involved how, in order to maximise the effectiveness of the approach.

Participatory Rural Appraisal - Better Evaluation (Better Evaluation)
"Participatory Rural Appraisal (PRA) recently renamed Participatory Learning for Action (PLA), is a methodological approach that is used to enable farmers to analyse their own situation and to develop a common perspective on natural resource management and agriculture at village level. PRA is an assessment and learning process that empowers farmers to create the information base they need for participatory planning and action. Outsiders contribute facilitation skills and external information and opinions. Many different tools have been developed for use in PRA. There are four main classes: tools used in group and team dynamics; tools for sampling; options for interviews and dialogue; and options for visualisation and preparing diagrams. Most countries have had some experience with PRA and local publications are available. IIED regularly reports on new developments in its PLA notes (Pretty et al 1995)." (Bie, 1998)

Positive Deviance - Better Evaluation (Better Evaluation)
Positive Deviance (PD) refers to a behavioral and social change approach which is premised on the observation that in any context, certain individuals confronting similar challenges, constraints, and resource deprivations to their peers, will nonetheless employ uncommon but successful behaviors or strategies which enable them to find better solutions. Through the study of these individuals– subjects referred to as “positive deviants” - the PD approach suggests that innovative solutions to such challenges may be identified and refined from their outlying behavior.

Qualitative Impact Assessment Protocol (QUIP) - Better Evaluation (Better Evaluation)
The QUIP sets out to generate differentiated evidence of impact based on narrative causal statements elicited directly from intended project beneficiaries without use of a control group. Evidence of attribution is sought through respondents’ own accounts of causal mechanisms linking X to Y alongside Z rather than by relying on statistical inference based on variable exposure to X. This narrative data is intended to complement quantitative evidence on changes in X, Y and Z obtained through routine project monitoring.

Randomised Controlled Trial - Better Evaluation (Better Evaluation)
Randomised controlled trials (RCTs), or randomised impact evaluations, are a type of impact evaluation which uses randomised access to social programmes as a means of limiting bias and generating an internally valid impact estimate.

Realist Evaluation - Better Evaluation (Better Evaluation)
Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings. Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?. The complete realist question is: “What works, for whom, in what respects, to what extent, in what contexts, and how?”. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.

Success Case Method - Better Evaluation (Better Evaluation)
The Success Case Method (SCM) involves identifying the most and least successful cases in a program and examining them in detail. This approach was developed by Robert Brinkerhoff to assess the impact of organizational interventions, such as training and coaching, though the use of SCM is not limited to this context. It is a useful approach to document stories of impact and to develop an understanding of the factors that enhance or impede impact. The Success Case Method deliberately looks at the most, and least, successful participants of a program. The purpose is not to examine the average performance - rather, by identifying and examining the extreme cases, it asks: 'When the program works, how well does it work? What is working, and what is not?'.

Utilization-Focused Evaluation - Better Evaluation (Better Evaluation)
Utilization-Focused Evaluation (UFE), developed by Michael Quinn Patton, is an approach based on the principle that an evaluation should be judged on its usefulness to its intended users. Therefore evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance.

Challenge-led Innovation: Organising for Systems Innovation at Scale - Burkett, Ingrid

Not everyone has to play the oboe - forthcoming - Cabaj, Mark (Tamarack Institute)
This resource explores the wide range of roles required in systems change efforts, extending Meg Wheatley’s Two Loop Framework into a richer map of diverse contributions. It underscores that no organization or leader can “do systems change” alone and highlights the complementary—and sometimes contradictory—roles needed to challenge, sustain, disrupt, or reimagine a system. A set of provocative questions helps individuals and teams reflect on how they can contribute meaningfully within the larger ecosystem of actors pushing for and resisting change.

There is No Such Thing as a Fish - Cabaj, Mark
"When it comes to constructing portfolios to address complex challenges, we have more than one option" This paper offers a sharper way to understand the different types of portfolios that changemakers can employ, challenging the common tendency to treat all portfolio approaches as interchangeable. It outlines how distinct portfolio types embody different purposes, theories of change, management practices, and evaluation needs. The piece concludes with a clear call to action: practitioners must be explicit and intentional about the kind of portfolio they are designing, because different portfolio types support different pathways— and possibilities— for transformation.

Equitable Systems Change: Funding Field Catalysts from Origins to Revolutionizing the World - Daniels, Lija McHugh Farnham, Emma Nothmann, Kevin Crouch, Cora (Bridgespan)
The Bridgespan Group's services include strategy consulting and advising, sourcing and diligence, and leadership team support. We take what we learn from this work, identify best practices and innovative ideas, and share them with the social sector.

Development Gateway website - Development Gateway (Development Gateway: an IREX Venture)
We create tools and design processes that help collect, analyze, visualize, and use data to support more effective, open, and engaging institutions.

MEL 360: Monitoring, Evaluation and Learning for Systems Change - Estrella, Andrea; Tolmer, Zazie; Bina, Andrea; Muhereza, Suzane; O'Malley, Charles; Uusikyla, Ida; Fregolent, Federica (UNDP)
MEL 360 offers practical, accessible guidance to help you layer Systems-Informed Monitoring, Evaluation, and Learning (Systems MEL) approaches onto your existing MEL practices

Funding Field Catalysts from Origins to Revolutionizing the World - Farnham, Lija; Nothmann, Emma; Crouch, Kevin; Daniels, Cora (Skoll Foundation & The Bridgespan Group)

A Refresher on Discovery-Driven Planning - Gallo, Amy (Harvard Business Review)

Learning as a Management Strategy - Lowe, Toby; Hesselgreaves, Hannah; Taylor, Abigail (Centre for Public Impact)

Discovery-Driven Planning - McGrath, Rita; MacMillan, Ian (Harvard Business Review)

Three Horizons - Rippel Foundation (The Rippel Foundation)
"Three Horizons" is a framework, developed by Bill Sharpe, as a valuable and intuitive tool for efficiently guiding systems change across various fields The framework itself provides a structured way to think about future change by categorizing developments into three horizons: - Horizon 1 represents the dominant current system, which is increasingly less able to meet emerging needs. - Horizon 2 involves innovations and disruptions that could potentially replace or transform the current system. - Horizon 3 focuses on long-term visions and aspirations for the future, representing transformative changes.

CLARISSA Social Protection Intervention - Evaluation Report - Roelen, Keetie; Howard, Neil; Afroze, Jiniya; Aktar, Afrin; Ton, Giel; Huq, Lopita (Institute of Development Studies)
The CLARISSA Social Protection (SP) Cash Plus intervention represented an innovative social protection scheme for tackling social ills, including the worst forms of child labour (WFCL). The purpose of the intervention was to trial and evidence an innovative social policy intervention for tackling poverty, improving wellbeing, and addressing worst forms of child labour (WFCL). It was a universal and unconditional cash plus programme, combining community mobilisation, case work and cash transfers. It was implemented in a high-density and low-income neighbourhood in Dhaka to build individual, family, and group capacities to meet their needs. This, in turn, was expected to lead to a corresponding decrease in deprivation and community-identified social issues that negatively affect wellbeing, including WFCL.

Storytelling for Systems Change: Listening to Understand - Snow, Thea; Bandaranayaka, Asitha; Fyfe, Rachel; Wolf, Lila (Centre for Public Impact)
The second phase of our Storytelling for Systems Change work, has revealed new insights, raised questions, and offered a range of possibilities.

Impact Management Software & Measurement Platform - Sopact
Impact Management platform is designed for social impact investing funds, grantmakers & public agency to provide comprehensive social impact measurement & impact management.

Two Loops -- Excerpted from - Stilger, Bob