Towards evidence-informed adaptive management - Hernandez, Kevin; Ramalingam, Ben; Wild, Leni (ODI/GLAM, 2019)
Development and humanitarian organisations seeking to be adaptive have emphasised the need to be transparent about complexity and uncertainty; to be honest about their inability to control what happens; and to design programmes that change over time to become more appropriate and relevant. At their heart, adaptive management approaches emphasise the ability to lean, 'unlearn' and adapt programming accordingly. The cornerstone of effective learning is the creation, gathering, accumulation, interpretation and use of data and evidence. This working paper provides development professionals with tools, strategies and ideas to help them use evidence for adaptive management in practical and evidence-informed ways. Key messages - Evidence is central to effective and rigorous adaptive management. However, despite this central importance, exactly how evidence has been used to inform decisions for adapting development and humanitarian programmes in the past remains unobservable to many. - There is a need to strengthen and document evidence-informed adaptive management. This working paper proposes a roadmap to do this. - Those seeking to use evidence for adaptive management will need to manage trade-offs between ensuring a rigorous, documented (and auditable) trail of evidence-informed actions, being pragmatic about the time and resources allocated to documentation and recognising that it may be necessary to proceed without rigorous evidence when it is unavailable.

Making adaptive rigour work - Principles and practices for strenghening monitoring, evaluation and learning for adaptive management - Ramalingam, Ben; Wild, Leni; Buffardi, Anne L (ODI/GLAM, 2019)
Adaptive programmes can be accountable, rigorous and high quality in how they use evidence by taking an ’adaptive rigour’ approach. Core development and humanitarian challenges are complex, and require processes of testing, learning and iteration to find solutions – adaptive management offers one approach for this. Yet large bureaucracies and development organisations can have low tolerance for experimentation and learning, and adaptive management can be viewed as an excuse for ‘making things up as you go along’. This briefing from the Global Learning for Adaptive Management (GLAM) initiative argues that adaptive programmes can be accountable, rigorous and high quality in how they use evidence – but this requires rethinking some key assumptions about how they are practised. The paper sets out three key elements of an ‘adaptive rigour’ approach: - Strengthening the quality of monitoring, evaluation and learning data and systems. - Ensuring appropriate investment in monitoring, evaluation and learning across the programme cycle. - Strengthening capacities and incentives to ensure the effective use of evidence and learning as part of decision-making, leading ultimately to improved effectiveness.

Supporting adaptive management. Monitoring and evaluation tools and approaches - Pasanen, Tiina; Barnett, Inka (ODI, 2019)
Key messages • This working paper introduces a set of monitoring and evaluation (M&E) tools and approaches, discussing their potential usefulness in supporting adaptive management in development and humanitarian programmes. It emphasises adaptive programmes characterised by complex aspects, such as: (1) they are innovative; (2) they have uncertain pathways for change; or (3) they operate in uncertain or unstable environments. • The majority of these tools have been used in international development for many years. However, adaptive management brings additional challenges for monitoring and evaluating programmes, as they require intentional M&E design from the start that is oriented towards both learning and accountability. • All of the tools and approaches introduced in this paper have potential to support learning and adaptation, although in various ways and at different stages of a programme. Some tools can support strategic planning and diagnosing throughout a programme – especially during design and inception – while others can help analyse causal relationships at specific points in a programme. It is important to tailor the approach used for its intended purpose. However, whether learning and adaptation happens depends also on factors other than the choice of M&E methods. • For some of these approaches a considerable body of evidence already exists but, for many, more practical examples and systematic analysis is needed. In addition to building the evidence base concerned with which approaches are suitable for different types of adaptive programmes and why, it is also important to improve understanding of the enabling environmental conditions necessary for the tools and approaches outlined here to facilitate and strengthen evaluative thinking, evidence-informed decision-making and ongoing programme iteration.

Monitoring and learning for country-level portfolio decision-making and adaptation - Buffardi, Anne L; Mason, Paige; Hutchings, Claire; Sharp, Samuel (ODI/GLAM, 2019)
Most measurement and adaptive management approaches were developed for and from individual projects. This briefing aims to guide measurement and management of country-level portfolios of work. It identifies potential purposes portfolio-level analyses can fulfil, types of adaptation, and the relative role of monitoring, learning and evaluation (MEL). Drawing on reviews of practice from the United States Agency for International Development (USAID) and the United Kingdom Department for International Development (DFID), guidance notes, and experiences of members of the Global Learning for Adaptive Management (GLAM) initiative, it offers recommendations and considerations that are particularly relevant for this level of analysis and management. Key messages - Portfolio-level analyses can serve eight potential purposes, each of which answers different questions, involves adaptation at different times and levels, and requires different types of evidence. Identifying the purpose(s) and how the component parts relate to each other should guide the development of monitoring and learning systems. - Portfolios are oriented more towards breadth than depth, involve more people with different perspectives, and draw on multiple sources of evidence with potentially greater variation in quality. - In practice, four activities appear to be applied most frequently at a portfolio level: alignment of indicators and aggregation of monitoring data; synthesis of multiple sources and types of information to provide a summary of outputs, outcomes, common observations and trends; periodic review and reflection sessions; and strategic planning, design or refresh of the portfolio strategy. - The extent to which evidence-informed portfolio management is facilitating learning and adaptation has not been well documented to date, and we suggest potential indicators to do so.

Monitoring and Result Measurement for Adaptive Programming - How to Use Data to Manage a MSD Program: Lessons from PRISMA - Khan, Khaled; Seely, Kevin; Ridwan, Mustika; Mulya, Bodhiya (PRISMA, 2019)
Using monitoring data to improve interventions is harder than it seems. Decision-makers are often busy implementing activities, unclear about their roles in data collection and analysis, and uncertain what data matters most or when. PRISMA, an AUD77 million agricultural Market Systems Development (MSD) programme funded by DFAT Australia, has encountered these challenges. With the programme completing its first five year phase, this case study shares ten key lessons divided into three sections: shaping the culture, developing systems, and top management decisions. These lessons aim to help program, sector and intervention managers make better use of monitoring data to improve interventions.

Monitoring And Evaluation Tools and Approaches to Support Adaptive Management - Pasanen, Tiina (GLAM, 2019)

Managing staff transistions through CLA: preserving institutional memory as staff come and go - USAID (USAID LEARN, 2019)
This document is relevant for any position or hiring mechanism. While this document does not explicitly address what happens when someone arrives into a newly-created position, which has its own set of challenges, many of the principles, actions, and resources can be applied in that context. Section 1 offers guidance for how to set up and implement systems at Mission/OU or office to ensure that all staff help preserve institutional memory and enable continuity of relationships. This section is for any staff in a Mission/OU working to build systems that improve handovers and knowledge capture and sharing across the Mission or OU. If a Mission or OU is just beginning to develop a system to manage staff transitions, you may want to start by developing a system within an office or team and then expand it out to other offices or to the Mission/OU writ large. Section 2 includes key principles and actions individuals could consider following when arriving or departing a position, regardless of hiring mechanism and position. This includes staff new to the Mission/OU and staff who are currently in the Mission/OU and are moving into a new position. Section 3 provides a select number of resources to help you take a systematic and comprehensive approach to manage staff transitions as effectively and efficiently as possible at the individual or organizational level.

Paradoxes and partnerships: a study of knowledge exploration and exploitation in international development programmes - Lannon, John; Walsh, John N. (Journal of Knowledge Management, 2019)
Purpose – This paper aims to look at how organisational partnerships balance knowledge exploration and exploitation in contexts that are rife with paradoxes. It draws on paradox theory to examine the partnership’s response to the explore-exploit relationship.

Planning & Navigating Social Change - Tools for Pacific voyagers - Orr, Douglas Epeli; Cavatore, Maria; Aston, Tom (Oxfam, 2019)
Those working to advance social change in the Pacific must understand, work with and respond to the complex and changing relationships and dynamics of power that exist within such networks, and situate their work in the context of decolonisation and self-determination. But the approaches typically used by NGOs to plan our projects and programs have failed to give due attention to these dynamics. The result is that our projects fall short of achieving their promise, despite being technically sound and logical. In this guide, we aim to provide a process and tools that prioritise and draw out the rich, often implicit, knowledge that Pacific Islanders have about our contexts to plan and manage social change initiatives in a manner that values and responds to this ocean of relationships. Drawing from these relationship-based and voyaging traditions, we emphasise the need to: • frame change in terms of change in people; e.g. change in the situation of groups of people and change in terms of the actions of people; • identify a course towards a particular ‘change destination’; • identify the signs to read that will help you determine where you are on that course; • be constantly attentive to the context as you make the journey, and • be ready to adjust your course in response to changes in your context. To support this way of working, we have selected a number of tools from approaches that we believe fit. In particular, we have drawn from the following approaches: • Thinking and Working Politically and other ‘politically aware’ approaches that recognise that the success of a development initiative will be determined by the interplay of relationships of power within a particular context, not just by having the right technical solution; • Outcome mapping because of its focus on people and relationships, and on identifying and supporting needed changes in the behaviour of specific actors in relation to others. This is key, because it’s individuals who deliver services, individuals who decide whether to allocate budget and resources to address a problem, and individuals who make the choice to believe whether a service is relevant to solving their issues; • Adaptive management because of its recognition that development initiatives take place within complex and changing systems and need to adjust to these and that this requires ongoing learning and adaptation.

The Rapid Evaluation, Action, and Learning (REAL) Approach: A toolkit to measure and refine changes and interventions in health campaigns - PATH (PATH, 2019)
Evaluative Tools for Improving Campaign Effectiveness Public health campaigns aim to control disease or deliver essential health services and products through time-limited and periodic channels. Many countries use campaigns to augment or replace routine service delivery, to target certain populations, or to accelerate progress towards coverage targets. The Root Cause Analysis and Rapid Evaluation, Action, and Learning toolkits present a systematic but flexible approach to identifying the root causes of campaign bottlenecks and then designing, testing, and refining solutions to optimize potential impact. These toolkits respond to a growing need for “fit-for-purpose” rapid-testing, adaptive learning approaches to evaluation and the need for a culture shift toward iterative adaptation and improvement that integrates measurement and evidence-informed decision-making into daily practice. The comprehensive package of toolkits, in addition to the individual Root Cause Analysis toolkit and Rapid Evaluation, Action, and Learning toolkit are available below.

Integrating Big Data and Thick Data to Transform Public Services Delivery - Ang, Yuen Yuen (IBM Center for The Business of Government, 2019)
This report explains how public organizations can combine big data and thick data to transform public services delivery—a strategy called mixed analytics. Governments can greatly enhance the value of big data by combining it with “thick” data—rich qualitative information about users, such as their values, goals, and consumption behavior, obtained by observing or interacting with them in their daily lives. Big data holds great promise for improving public services delivery and innovation in government, but they are not a panacea . Having lots of data can be overwhelming or have little utility if the data are “thin”—that is, they lack meaning for users or fail to capture issues that matter most . By yielding insights into what citizens really care about and how they consume services, thick data can inform both the collection and analysis of big data. This report introduces the concept of “mixed analytics,” integrating big data and thick data to transform government decision making, public services delivery, and communication. The report presents three case studies of organizations that employ mixed analytics at the international, federal, and city level, respectively. Together, this research offers a set of transferable lessons for agencies at all levels of government: Lesson 1: Big data is a means to an end, rather than an end. Lesson 2: Thick data can identify unexpected problems or previously unexpressed needs. Lesson 3: Thick data can inform the analysis of big data. Lesson 4: Mixed analytics can offer both scale and depth. Lesson 5: Applying technology is a social activity, not an isolated technical task. Lesson 6: The best solutions are not always high-tech. The report concludes with five actionable recommendations for public managers. Make data and technology relevant to the people who use it. Leverage thick data at appropriate stages of the problem-solving process. Build an interdisciplinary team of quantitative and qualitative experts who work closely with stakeholders. Combine big and thick data to improve communication Improve government agencies’ knowledge of mixed research methods. This report builds on multiple past IBM Center reports about how government can leverage data and analytics to improve decisions, including Data-Driven Government: The Role of Chief Data Officers, by Jane Wiseman; Ten Actions to Implement Big Data Initiatives: A Study of 65 Cities, by Alfred Ho and Bo McCall, and Realizing The Promise of Big Data, by Kevin DeSouza. The report also complements several chapters that assess the past and potential use of data across the public sector in our recent book, Government For The Future: Reflection and Vision For Tomorrow’s Leaders. At a time when the US and other governments continue to enhance their use of data as a strategic asset for transformation, we hope that Professor Ang’s report provides useful insights for government managers and stakeholders.

M&E Universe - intrac (INTRAC, 2019)
The M&E Universe is a free, online resource developed by INTRAC to support development practitioners involved in monitoring and evaluation (M&E). It consists of a series of short papers (2-6 pages) on different subjects related to M&E. It can be explored through an online platform (intrac.org/universe) that is compatible with most web browsers. The M&E Universe is designed for M&E practitioners with different levels of experience and expertise in M&E, from those new to M&E who want an entry level into the subject to experienced practitioners wanting to broaden their knowledge. Initial contact with the Map will provide people with a starting point for understanding different aspects of M&E and how they link to each other. More experienced practitioners can browse the Map to identify more advanced subjects of interest, such as sampling methods, different forms of evaluation, organisational M&E systems and complex methodologies for data collection and analysis.

No Royal Road: Finding and Following the Natural Pathways in Advocacy Evaluation - Coe, Jim; Schlangen, Rhonda (Center for Evaluation Innovation, 2019)
As efforts at driving change become more diffuse, involve more actors, and have more transformational goals, we need a radically different approach to thinking about and assessing effective advocacy. Clear answers, certain judgments, and simple tools won't result in good representations of reality. This brief proposes ideas for adjusting how we approach advocacy monitoring, evaluation, and learning.

Breaking the Mould: Alternative approaches to monitoring and evaluation - Dillon, Neil (ALNAP, 2019)
This paper looks at a range of M&E innovations that are designed specifically to provide input to ongoing iterative decision-making and learning at the project level. It identifies three key areas for potential innovation: 1) timing of M&E data provision; 2) flexibility of M&E frameworks to evolve with programme change; and 3) approaches to integrate diverse perspectives on project implementation in a meaningful way. It then looks at a collection of approaches currently being used in each of these three areas through a series of ‘practice examples’, considering the key lessons learned. Finally, the paper discusses the major opportunities and challenges for applying and scaling up the use of these approaches inside the humanitarian sector.

Knowledge Retention and Transfer (KRT) - Model and Matsurity Matrix - USAID (USAID LEARN, 2019)
The Maturity Matrix is not a standalone tool. It is one of the three major components of the KRT Model. The Maturity Matrix is meant to be used in conjunction with the KRT Toolkit and Implementation Plan.

Evaluation of development programmes - OECD - OECD (2019)
Data & research on evaluation of development programmes inc. paris declaration, budget support, multilateral effectiveness, impact evaluation, joint evaluations, governance, aid for trade

Thinking and Working Politically in Economic Development Programmes – Some Sprints and Stumbles from a DFID Programme in Kyrgyzstan - Koleros, Andrew; Rinnert, David (From Poverty to Power, 2019)
A DFID programme in Kyrgyzstan offers useful insights into how the Thinking and Working Politically approach can escape from its governance ghetto
Keywords: Case Study

Back to the Drawing Board: How to improve monitoring of outcomes - Dillon, Neil; Sundberg, Amelie (ODI/ALNAP, 2019)

Failing Forward: How CARE is focusing on what goes wrong to improve impact - Janoch, Emily (USAID LEARN, 2019)
CARE's Failing Forward initiative is sparking opportunities to showcase the ideas that don't work so we can spend more time implementing the ones that do. It's changing the conversation inside the organization, and leading to changes in the way we design and implement programs. It's also allowing us to make connections across a global portfolio of more than 900 projects, and with new and different partners to learn from their experience.

How We Model Matters: A Manifesto for the Next Generation of Program Theorizing - Whynot, Jane; Lemire, Sebastian; Montague, Steve (Canadian Journal of Program Evaluation, 2019)
In this concluding article, grounded on the exemplary contributions contained in the preceding pages, the guest editors scale the proverbial soapbox and present a manifesto to guide the pursuit and advancement of the next generation of program theorizing. Formulating ten declarations for program theory development and examination, the modest hope of the authors is to motivate and inspire reflective evaluation practitioners to broaden their views, approaches, and techniques for future program theorizing.