Value for money and adaptive programming - Approaches, measures and management - Laws, Ed; Valters, Craig (ODI, 2021)
- The United Kingdom’s Foreign, Commonwealth and Development Office (FCDO)’s standard economy, efficiency, effectiveness/cost-effectiveness and equity (4E) framework is still relevant for approaching, measuring and managing value for money (VfM) for adaptive programmes. • However, this framework needs to be reframed to capture and incentivise flexibility, learning and adaptation. • VfM appraisal and reporting should be done in a way that draws on beneficiary feedback and informs good decision-making, rather than just being a compliance exercise. • If VfM appraisal and reporting cannot be done appropriately for adaptive programmes, it should be avoided or minimised. There is a risk of diverting time and resources from more suitable tools and methods.

Learning to adapt & adapting to learn - Using elements of outcome mapping in the ‘Resilient Adolescents in the Syria Crisis’ programme - van Ongevalle, Jan; Kvintradze, Ana; Rennesson, Gaël; Miller, David (Save the Children, 2021)
This learning paper highlights how elements of outcome mapping were used by Save the Children Sweden in a project (2018-2020) that supports adolescents, affected by the Syria crisis, to become more resilient. The paper first outlines how the spheres of influence framework has been applied to develop an actor focused theory of change. It then describes how progress markers, as an alternative to SMART indicators, were formulated to monitor the programme’s results. The paper also outlines how long lists of progress markers were categorised in a more realistic and practical results framework. The paper then continues to elaborate how outcome journals, qualitative data analysis techniques and regular review meetings and reflection workshops were utilised for data collection, for collective learning among programme stakeholders and for informing planning and programme adjustment. Various practical guidelines and tips on how to implement elements of outcome mapping are provided. The final part of the paper explores to what extent outcome mapping was able to foster several key enablers of adaptive programme management and highlights some of the challenges that programme stakeholders faced. Practical recommendations towards the use of outcome mapping in future programmes are also proposed.

What have we learned about learning? Unpacking the relationship between knowledge and organisational change in development agencies - Yanguas, Pablo (German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE), 2021)
Development cooperation has spent decades wrangling over the merits, evidence, and implications of what we may term “the learning hypothesis”: the idea that increased knowledge by development organisations must logically lead to increased effectiveness in the performance of their development activities. Organisations of all stripes have built research and monitoring and evaluation (M&E) departments, adopted a multitude of knowledge management systems and tools, and tinkered with different ways to structure their organograms to stimulate knowledge sharing and learning. The topic of organisational learning is particularly significant as the global development community grapples with increasingly complex problems and the aspiration of evidence-based policymaking. This paper presents an analytical framework for interrogating “the learning hypothesis”, breaking it down into causal steps: knowledge causes learning, learning causes organisational change, change causes effectiveness. The framework focuses on the first two sub-hypotheses, mapping out the conceptual space around them by outlining potential relationships between different types of knowledge – tacit and explicit, internal and external – and between different types of learning – operational and strategic. This map provides a foundation for three key research questions: What impact has the rising knowledge agenda had on development organisations? Which factors appear to enable or inhibit organisational learning? What is the relationship between operational and strategic learning and organisational change? A review of available evaluations and studies, including two cases from former UK Department for International Development (DFID) and the World Bank, reveals that there is insufficient evidence to support the causal claim that knowledge leads to learning and thereby to organisational change in development agencies. Sources point to tacit learning prevailing while explicit knowledge management systems flounder, and external advocacy agendas appear more compelling than internal research and evaluation products. It is not entirely clear how, or indeed, whether operational and strategic learning intersect, with delivery-level lessons hardly aggregating into structural or policy shifts. Organisational change – even that aimed at enhancing learning – is rarely based on lessons learned from practice. More research is necessary to fully unpack the learning hypothesis, but what limited evidence is available disproves rather than confirms its central claim. This has significant implications for the future of learning in development agencies as advocated by thought leaders, researchers, and reformers. In particular, the latter should consider an evidence-based reassessment of the function and value-for-money of research and M&E in development practice, and a more critical examination of the politics of external advocacy efforts around innovative aid approaches like thinking and working politically, adaptive management, or results-based management.
Version Number: 1.0

Peer reviews - Guidance for facilitators and participants - Wadley, Ian (Centre for Humanitarian Dialogue, 2021)
Essential points for practitioners and donors • Mediation offers a cost-effective and proven method for resolving armed conflict. Between 1985 and 2015, 75 per cent of armed conflicts in the world were resolved through agreement rather than by force. In most cases these processes will have involved third party facilitation or support. • Professional mediators understand the high stakes involved in their work to prevent, mitigate and resolve armed conflict. In addition, they and their financial supporters are increasingly required to demonstrate ‘value-for-money’ to ensure continued funding. • However, traditional monitoring and evaluation (M&E) methods are not well suited to this task, typically imposing artificially linear project models on a dynamic conflict situation, as well as compliance reporting that moves attention away from real value. • Traditional M&E methods tend to focus on documenting the past and generating vast amounts of data, rather than enabling timely adaptation of the project in the present. • Traditional M&E approaches rely heavily on external evaluation consultants. Even in the best of cases this may interfere with the mediation process and impose a heavy time burden on the project team, leading to low acceptance of traditional M&E approaches by mediation practitioners. • In contrast, an ideal M&E approach for mediation should deliver useful insights in even the most dynamic and sensitive mediation environments, impose a light reporting burden, and be readily accepted by mediation teams. It should protect discretion and trust, enable rapid adaptation, and also provide some assurance that donor funds are being well spent.

Evaluating complex interventions in international development - Masset, Edoardo; Shrestha, S.; Juden, M. (CEDIL, 2021)
This paper reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. Complex interventions are those that are characterised by multiple components, multiple stakeholders, or multiple target populations. They may also be interventions that incorporate multiple processes of behavioural change. While such interventions are very common and receive a large proportion of development aid budgets, they are rarely subject to rigorous evaluations. The CEDIL Methods Working Paper, ‘Evaluating Complex Interventions in International Development’, reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. The paper focuses its attention on methods that address causality and allow us to state conclusively whether an intervention works or not. It shows that several rigorous methods developed in different disciplines can be adapted and used to evaluate complex interventions in international development.
Section: Publications

A Systems Framework for International Development: The Data-Layered Causal Loop Diagram - Blair, Courtney; Gralla, Erica; Wetmore, Finley; Goentzel, Jarrod; Peters, Megan (Production and Operations Management, 2021)
Meeting the United Nations Sustainable Development Goals (SDGs) will require adapting or redirecting a variety of very complex global and local human systems. It is essential that development scholars and practitioners have tools to understand the dynamics of these systems and the key drivers of their behavior, such as barriers to progress and leverage points for driving sustainable change. System dynamics tools are well suited to address this challenge, but they must first be adapted for the data-poor and fragmented environment of development work. Our key contribution is to extend the causal loop diagram (CLD) with a data layer that describes the status of and change in each variable based on available data. By testing dynamic hypotheses against the system's actual behavior, it enables analysis of a system's dynamics and behavioral drivers without simulation. The data-layered CLD was developed through a 4-year engagement with USAID/Uganda. Its contributions are illustrated through an application to agricultural financing in Uganda. Our analysis identified a lack of demand for agricultural loans as a major barrier to broadening agricultural financing, partially refuting an existing hypothesis that access to credit was the main constraint. Our work extends system dynamics theory to meet the challenges of this practice environment, enabling analysis of the complex dynamics that are crucial to achieving the SDGs.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/poms.13492

Applying Evaluation Criteria Thoughtfully - OECD (OECD, 2021)
Relevance, coherence, effectiveness, efficiency, impact, and sustainability are widely used evaluation criteria, particularly in international development co-operation. They help to determine the merit or worth of various interventions, such as strategies, policies, programmes or projects. This guidance aims to help evaluators and others to better understand those criteria, and improve their use. It starts by describing what they are, and how they are meant to be used. Then the definitions and concepts underpinning each criterion are explained. Finally, examples provide the reader with concrete ideas for using them. The criteria were originally laid out in the early 2000s by the Network on Development Evaluation (EvalNet) of the OECD Development Assistance Committee (DAC). Although they have been widely used in evaluation, and beyond, this document is the first to comprehensively explore the concepts in detail, explain their intended use and offer practical guidance. It captures current thinking and best practice in evaluation, drawing on the inputs of internationally renowned evaluation experts from EvalNet and beyond.
Type: doi:https://doi.org/10.1787/543e84ed-en

Complexity-Aware Monitoring and Evaluation - Hertz, Tilman; Brattander, Eva; Rose, Loretta (Journal of MultiDisciplinary Evaluation, 2021)
Background: Addressing today’s sustainability challenges requires adopting a systemic approach where social and ecological systems are treated as integrated social-ecological systems. Such systems are complex, and the international development sector increasingly recognises the need to account for the complexity of the systems that they seek to transform. Purpose: This paper sketches out the elements of a complexity-aware monitoring and evaluation (M&E) system for international development programmes in the area of sustainable development.  Setting: Not applicable. Data Collection and Analysis: The authors draw on existing literature on complexity and evaluation and on their own experience from working in the field of M&E. Findings: An M&E system should not be seen simply as a tool to track compliance against a pre-determined theory of change. Instead, it is most useful as a real-time approach, constantly defining and re-defining narratives for change that help push systems along trajectories of interest. Dealing with complexity involves embracing uncertainty; and this challenges established notions of accountability—something which funders and implementers must begin to redefine together. Keywords: monitoring; evaluation; complexity; social-ecological systems; international development programmes; narratives for change; theory of change
Number: 41

Applying Systems Thinking to Education: The RISE Systems Framework - Spivack, Marla (Research on Improving Systems of Education (RISE), 2021)
Many education systems in low- and middle-income countries are experiencing a learning crisis. Many efforts to address this crisis do not account for the system features of education, meaning that they fail to consider the ways that interactions and feedback loops produce outcomes. Thinking through the feedback relationships that produce the education system can be challenging. The RISE Education Systems Framework, which is sufficiently structured to give boundaries to the analysis but sufficiently flexible to be adapted to multiple scenarios, can be helpful. The RISE Framework identifies four key relationships in an education system: politics, compact, management, and voice and choice; and five features that can be used to describe these relationships: delegation, finance, information, support, and motivation. This Framework can be a useful approach for characterising the key actors and interactions in the education system, thinking through how these interactions produce systems outcomes, and identifying ways to intervene that can shift the system towards better outcomes.

Pushing boundaries: Advocacy evaluation and real-time learning in an HIV prevention research advocacy coalition in sub-Saharan Africa - Dasmariñas, Jules; Schlangen, Rhonda (New Directions for Evaluation, 2021)
Evaluation processes that facilitate learning among advocates must be nimble, creative, and meaningful while transcending putative performance and accountability management. This article describes the experience, lessons, and trajectory of one such approach, Simple, Participatory Assessment of Real Change (SPARC), that a transnational HIV prevention research advocacy coalition pilot-tested in sub-Saharan Africa. Inspired by the pioneering work of the outcome harvesting (OH) and participatory evaluation community, we recuperate advocates' centrality as storytellers, sense-makers, and strategists in advocacy evaluation and describe how we recalibrated SPARC to meet their evaluation and learning needs. This article highlights the normative value of deliberative discourse in evaluation as it contributes to the interpretation of OH and the enrichment of the theory and practice of advocacy evaluation.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20474