MLE or MEL in adaptive programming? - Pellini, Arnaldo (Knowledge counts, 2018)

Citizen engagement: emerging digital technologies create new risks and value - Peixoto, Tiago; Steinberg, Tom (World Bank, 2019)
The recent rapid evolution of digital technologies has been changing behaviors and expectations in countries around the world. These shifts make it the right time to pose the key question this paper explores: Will digital technologies, both those that are already widespread and those that are still emerging, have substantial impacts on the way citizens engage and the ways through which power is sought, used, or contested? The authors address this question both to mitigate some of the World Bank’s operational risks, and to initiate a conversation with peers about how those risks might require policy shifts. The overall framing question also is being explored in case theapproaches to citizen engagement advocated by the World Bank are changing and may require different advice for client countries. Despite the lower technology penetration levels in developing countries, their more malleable governance contexts may be more influenced by the effects of emerging technologies than older states with greater rigidity. Digitally influencedcitizen engagement is, in short, one of those “leapfrog” areas in which developing nations may exploit technologies before the wealthier parts of the world. But countries can leapfrog to worse futures, not just better ones. This paper explores what technology might mean for engagement, makespredictions, and offers measures for governments to consider.

When Does ICT-Enabled Citizen Voice Lead to Government Responsiveness? - Peixoto, Tiago; Fox, Jonathan (IDS Bulletin, 2016)

Crowdsourcing citizen feedback on district development in Ghana using interactive voice response surveys - Pegus, Ciana-Marie (Making All Voices Count Practice Paper, 2016)
Keywords: IMPORTANT

When and how to develop an impact-oriented monitoring and evaluation system - Peersman, Greet; Rogers, Patricia; Guijt, Irene; Hearn, Simon; Pasanen, Tiina; Buffardi, Anne (2016)
This guidance note focuses on: • what an impact-oriented monitoring and evaluation system entails • why an organisation may want to establish such a system • when integrating an impact-orientation into an monitoring and evaluation system is most useful • what should be considered in developing the monitoring and evaluation system, or in tweaking an existing system, to become more impact-focused. The primary audience for this guidance note is internal and external monitoring and evaluation advisors involved in designing and implementing, and/or assessing monitoring and evaluation systems to include a focus on impact. It will also be useful for senior management of organisations who need to know how best to plan for a sustainable monitoring and evaluation system that supports impact assessment or to adapt an existing system to incorporate an impact perspective.

Evaluability assessment for Impact Evaluation: Guidance, checklists and decision support - Peersman, Greet; Guijt, Irene; Pasanen, Tiina (ODI, 2015)
This guidance note focuses on the utility of, and guidance for, evaluability assessment before undertaking an impact evaluation. The primary audience for this guidance note is evaluators conducting an evaluability assessment for impact evaluation. The secondary audience is people commissioning or managing an evaluability assessment for impact evaluation, as well as funders of an impact evaluation. Sections one and two provide an overview of evaluability assessment and how it can be used for impact evaluation. Section three provides guidance for planning to undertake an evaluability assessment for impact evaluation. This is informative for all intended users of the guidance note. Section four includes checklists and decision support for evaluability assessments. The checklist is geared to those conducting the evaluability assessment and can be adapted to suit a particular context or purpose. The decision support provides those conducting an evaluability assessment with evidence-based recommendations for impact evaluation funders and commissioners, about whether, when and how to proceed with the evaluation. Sections five and six provide guidance on what to do after the assessment is concluded, and offer lessons learned from evaluability assessments in practice.

Leveraging Experimental Evaluations for Understanding Causal Mechanisms - Peck, Laura R. (New Directions for Evaluation, 2020)
Experimental evaluations—especially when grounded in theory-based impact evaluation—can provide insights into the mechanisms that generate program impacts. This chapter details variants of experimental evaluation designs and also analytic strategies that leverage experimental evaluation data to learn about causal mechanisms. The design variants are poised to illuminate causal mechanisms related to program implementation and the contribution of selected components of multifaceted programs. The analysis strategies lend themselves to illuminating causal mechanisms related to participants’ responses to program components as well as to the contributions of selected program components themselves. The chapter offers an example from one, theory-based impact evaluation, which embedded both design and analytic strategies to examine the extent to which specific program components and participant experiences might be identified as causal mechanisms. The particular value in using this theory-based experimental strategy is that the results are rigorous and potentially highly relevant to policy and practice.

Measuring Impact. Making Use of the Portfolio: Organizational Learning at USAID - Peabody, Shawn (Measuring Impact, 2015)
This technical analysis explores previous and ongoing social learning efforts, best practices, challenges, and lessons in USAID as a foundation for improving the implementation and design of the Agency’s forestry and biodiversity programs. This analysis is particularly relevant as the Bureau of Economic Growth, Education and the Environment’s Office of Forestry and Biodiversity (E3/FAB) begins to develop a Cross-Mission Learning Program under the Measuring Impact initiative, a five-year activity to promote the adoption of best practices in the USAID program cycle among Missions using biodiversity funds. The findings reported in this document can help inform the overall design and structure of the Learning Program and provide insight into possible challenges and best practices.

Realist Evaluation - Pawson, Ray; Tilley, Nick (2004)
Contents: Introduction 1. The nature of programmes and how they work 2. Basic concepts in the explanation and understanding of programmes 3. Strategies and methods of realist evaluation 4. Realism’s place in the policy cycle: formative, summative and synthetic approaches 5. The nature, presentation and use of findings from realist evaluation 6. Conclusion: strengths, limitations and relationships with other approaches Appendix I – ‘Thinking it through’: an exercise in realist hypothesis making. Appendix II – ‘Varieties of realist evaluation’: pocket illustrations of quantitative, qualitative, formative and synthetic applications. Appendix III – ‘Would it work here?’: a grid to help decide on the feasibility of mounting a programme ‘on your patch’.

Lessons Learned from Applying the Data Powered Positive Deviance - Pawelke, Andreas; Glücker, Andreas; Albanna, Basma; Boy, Jeremy (DPPD, 2021)
This report presents six learnings from four pilot projects conducted by the Data Powered Positive Deviance (DPPD) initiative, a global collaboration between the GIZ Data Lab, the UNDP Accelerator Labs Network, the University of Manchester Center for Digital Development, and UN Global Pulse Lab Jakarta. The pilots seek out grassroots solutions to development challenges that range from the interaction between livestock farming and deforestation to gender-based violence and insecurity in dense urban environments in Ecuador, Mexico, Niger and Somalia. The learnings relate to the early stages of the DPPD method, originally proposed by Albanna & Heeks [1], and focus mainly on the access to, and use of digital data. They are summarized as follows: 1. Remain flexible in the face of data unavailability 2. Leverage existing partnerships for data access 3. Map and fill know-how gaps early 4. Scale with caution 5. Look at deviance over time 6. Look beyond individual or community practices and behavior The report is written for development practitioners, data analysts, domain experts, and more generally anyone interested in using new data sources and technologies to uncover successful local solutions to development challenges.

Reflections on if and how our partnership is working in Bangladesh - Paul, Sukanta; Snijder, Mieke; Apgar, Marina (CLARISSA, 2020)
Effective partnership working is crucial to producing quality results, especially when working with complex problems such as the worst forms...

Utilization-focused evaluation for agricultural innovation - Patton, Michael Quinn; Horton, D (ILAC Brief, 2009)
Utilization-focused evaluation (UFE) is based on the principle that an evaluation should be judged by its utility. So no matter how technically sound and methodologically elegant, an evaluation is not truly a good evaluation unless the findings are used. UFE is a framework for enhancing the likelihood that evaluation findings will be used and lessons will be learnt from the evaluation process. This Brief, based on the book Utilization-focused evaluation, introduces this approach to evaluation, outlines key steps in the evaluation process, identifies some of the main benefits of UFE, and provides two examples of UFE in the context of programmes aimed at promoting agricultural innovation.

Developmental Evaluation - Patton, Michael Quinn (The Nonprofit Quarterly, 2006)

Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use - Patton, Michael Quinn (Guilford Press, 2010)
Developmental evaluation (DE) offers a powerful approach to monitoring and supporting social innovations by working in partnership with program decision makers. In this book, eminent authority Michael Quinn Patton shows how to conduct evaluations within a DE framework. Patton draws on insights about complex dynamic systems, uncertainty, nonlinearity, and emergence. He illustrates how DE can be used for a range of purposes: ongoing program development, adapting effective principles of practice to local contexts, generating innovations and taking them to scale, and facilitating rapid response in crisis situations. Students and practicing evaluators will appreciate the book's extensive case examples and stories, cartoons, clear writing style, "closer look" sidebars, and summary tables. Provided is essential guidance for making evaluations useful, practical, and credible in support of social change.See also Developmental Evaluation Exemplars, edited by Michael Quinn Patton, Kate McKegg, and Nan Wehipeihana, which presents 12 in-depth case studies.
Keywords: Evaluation; Measurement; Organizational effectiveness; Project management

Essentials of Utilization-Focused Evaluation - Patton, Michael Quinn (SAGE Publications, 2011)
Based on Michael Quinn Patton's best-selling Utilization-Focused Evaluation, this briefer book provides an overall framework and essential checklist steps for designing and conducting evaluations that actually get used. The new material and innovative graphics present the utilization-focused evaluation process as a complex adaptive system, incorporating current understandings about systems thinking and complexity concepts. The book integrates theory and practice, is based on both research and professional experience, and offers new case examples and cartoons with Patton's signature humor.
Keywords: Reference / Research; Research

Principles-Focused Evaluation: The GUIDE - Patton, Michael Quinn (Guilford Press, 2017)
How can programs and organizations ensure they are adhering to core principles--and assess whether doing so is yielding desired results? From evaluation pioneer Michael Quinn Patton, this book introduces the principles-focused evaluation (P-FE) approach and demonstrates its relevance and application in a range of settings. Patton explains why principles matter for program development and evaluation and how they can serve as a rudder to navigate the uncertainties, turbulence, and emergent challenges of complex dynamic environments. In-depth exemplars illustrate how the unique GUIDE framework is used to determine whether principles provide meaningful guidance (G) and are useful (U), inspiring (I), developmentally adaptable (D), and evaluable (E). User-friendly features include rubrics, a P-FE checklist, firsthand reflections and examples from experienced P-FE practitioners, sidebars and summary tables, and end-of-chapter application exercises.

Utilization-Focused Evaluation (U-FE) Checklist - Patton, Michael Quinn (Evaluation Checklists Projects, 2013)
Utilization-Focused Evaluation begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use. Use concerns how real people in the real world apply evaluation findings and experience and learn from the evaluation process. The checklist is based on Essentials of Utilization-Focused Evaluation (Patton, 2012, Sage Publications). All references in the checklist to exhibits and menus refer to this book. Step 1 Assess and build program and organizational readiness for utilization-focused evaluation. Step 2 Assess and enhance evaluator readiness and competence to undertake a utilizationfocused evaluation. Step 3 Identify, organize, and engage primary intended users. Step 4 Conduct situation analysis with primary intended users Step 5 Identify primary intended uses by establishing the evaluation’s priority purposes. Step 6 Consider and build in process uses if appropriate. Step 7 Focus priority evaluation questions. Step 8 Check that fundamental areas for evaluation inquiry are being adequately addressed. Step 9 Determine what intervention model or theory of change is being evaluated. Step 10 Negotiate appropriate methods to generate credible findings and support intended use by intended users. Step 11 Make sure intended users understand potential controversies about methods and their implications. Step 12 Simulate use of findings. Step 13 Gather data with ongoing attention to use. Step 14 Organize and present the data for use by primary intended users. Step 15 Prepare an evaluation report to facilitate use and disseminate significant findings to expand influence. Step 16 Follow up with primary intended users to facilitate and enhance use. Step 17 Metaevaluation of use: Be accountable, learn, and improve

Complexity-Aware Monitoring Discussion Note (Screencast) - Patsalides, Melissa; Britt, Heather (USAID, 2014)
This screencast covers information included in the Complexity-Aware Monitoring Discussion Note. The topic is intended for those seeking cutting-edge solutions to monitoring complex aspects of strategies and projects. Most of the principles and promising approaches discussed here have a significant body of theory and practice behind them, but many have not been used to monitor USAID strategies and projects. We have much to learn about whether and how they can be applied successfully in the Agency. USAID's Office of Learning, Evaluation, and Research in the Bureau of Policy, Planning, and Learning has partnered with different groups through monitoring trials to build a body of evidence regarding which complexity-aware M&E approaches are most effective in the contexts of USAID programming.

The Rapid Evaluation, Action, and Learning (REAL) Approach: A toolkit to measure and refine changes and interventions in health campaigns - PATH (PATH, 2019)
Evaluative Tools for Improving Campaign Effectiveness Public health campaigns aim to control disease or deliver essential health services and products through time-limited and periodic channels. Many countries use campaigns to augment or replace routine service delivery, to target certain populations, or to accelerate progress towards coverage targets. The Root Cause Analysis and Rapid Evaluation, Action, and Learning toolkits present a systematic but flexible approach to identifying the root causes of campaign bottlenecks and then designing, testing, and refining solutions to optimize potential impact. These toolkits respond to a growing need for “fit-for-purpose” rapid-testing, adaptive learning approaches to evaluation and the need for a culture shift toward iterative adaptation and improvement that integrates measurement and evidence-informed decision-making into daily practice. The comprehensive package of toolkits, in addition to the individual Root Cause Analysis toolkit and Rapid Evaluation, Action, and Learning toolkit are available below.

The Power of Positive Deviance - Pascale, Richard; Sternin, Jerry; Sternin, Monique (Harvard Business Press, 2010)

Your Company’s Secret Change Agents - Pascale, Richard T.; Sternin, Jerry (Harvard Business Review, 2005)
Somewhere in your organization, groups of people are already doing things differently and better. To create lasting change, find these areas of positive deviance and fan their flames.

We created this MEL system for you, now please own it! | From Poverty to Power - Pasanen, Tiina; Yanguas, Pablo (2021)
External consultants, learning partners or critical friends -whatever we call them- can seldom change the system or organisational (learning) culture from outside. So, how can Monitoring Evaluation and Learning (MEL) consultants support real change instead of creating tools or processes that are quickly forgotten without any real institutional ownership? Consultants and learning partners can seldom change a learning system or organisational culture – it just does not work like that. What they can do is interrogate existing practices, introduce new ideas, and support change processes already underway to ensure lasting, transformative impact. They are a multiplier for homegrown change initiatives, not a substitute. Don’t hire MEL consultants to sell you tools or tell you what to do. Hire them because you need a critical friend along the path of organisational change. Own, then learn – not the other way round.

How to design a monitoring and evaluation framework for a policy research project - Pasanen, Tiina; Shaxson, Louise (Methods Lab, 2016)
Policy research projects face a number of challenges: policy processes are complex, involve multiple actors and often feature a significant time-lag between research and what may or may not happen as a result of it. To complicate matters further, the scope and scale of policy research projects are increasingly moving away from single research studies towards multi-component, multi-site and multi-sector endeavours. These factors mean that developing an overarching monitoring and evaluation framework can be challenging. But it is not impossible. The guidance note aims to support the first steps in designing and structuring the M&E framework (that is, what aspects or areas of policy research projects to monitor and evaluate, why, when and how). It builds on an M&E framework for policy research projects developed and tested by the Research and Policy in Development (RAPID) programme of the Overseas Development Institute, and draws on case studies of how it has been used by RAPID and other research teams.

Supporting adaptive management. Monitoring and evaluation tools and approaches - Pasanen, Tiina; Barnett, Inka (ODI, 2019)
Key messages • This working paper introduces a set of monitoring and evaluation (M&E) tools and approaches, discussing their potential usefulness in supporting adaptive management in development and humanitarian programmes. It emphasises adaptive programmes characterised by complex aspects, such as: (1) they are innovative; (2) they have uncertain pathways for change; or (3) they operate in uncertain or unstable environments. • The majority of these tools have been used in international development for many years. However, adaptive management brings additional challenges for monitoring and evaluating programmes, as they require intentional M&E design from the start that is oriented towards both learning and accountability. • All of the tools and approaches introduced in this paper have potential to support learning and adaptation, although in various ways and at different stages of a programme. Some tools can support strategic planning and diagnosing throughout a programme – especially during design and inception – while others can help analyse causal relationships at specific points in a programme. It is important to tailor the approach used for its intended purpose. However, whether learning and adaptation happens depends also on factors other than the choice of M&E methods. • For some of these approaches a considerable body of evidence already exists but, for many, more practical examples and systematic analysis is needed. In addition to building the evidence base concerned with which approaches are suitable for different types of adaptive programmes and why, it is also important to improve understanding of the enabling environmental conditions necessary for the tools and approaches outlined here to facilitate and strengthen evaluative thinking, evidence-informed decision-making and ongoing programme iteration.

Are we suffering from obsessive measurement disorder? - Pasanen, Tiina (From Poverty to Power, 2019)
ODI’s Tiina Pasanen argues that more data doesn’t necessarily mean we make better decisions, and sets out some ideas for how we count what counts.

Five tips for evaluating your impact in international development - Pasanen, Tiina (The Guardian, 2015)
How can impact evaluations actually be helpful? These top tips will save development professionals time, energy and money. International development can be messy – with uncertain, complex settings, and multiple partners with different interests, goals and capacities. At the same time, we are under increasing pressure to demonstrate impact. We have to show that our projects have made a real change in people’s lives and that donor or taxpayer money hasn’t been wasted. But impact evaluations, which are often seen as a solution to this, aren’t always used. And in some cases they are unhelpful. So how can we improve the quality of impact evaluations, so that they produce results that are useful? This is where evaluability assessments come in. By asking whether we should evaluate a project – and if so, when and how – these assessments can improve the quality of impact evaluations. Here are five other reasons to assess evaluability before starting an impact evaluation.

Monitoring and evaluation: five reality checks for adaptive management - Pasanen, Tiina (ODI Insight, 2017)

Monitoring And Evaluation Tools and Approaches to Support Adaptive Management - Pasanen, Tiina (GLAM, 2019)

The politics of evidence: from evidence-based policy to the good governance of evidence - Parkhurst, Justin O. (Routledge studies in governance and public policy, 2017)
Introduction -- Evidence-based policymaking : an important first step, and the need to take the next -- Bias and the politics of evidence -- The overt politics of evidence : bias and the pursuit of political interests -- The subtle politics of evidence - the cognitive-political origins of bias -- What is "good evidence for policy"? From hierarchies to appropriate evidence -- What is the "good use of evidence" for policy? -- From evidence-based policy to the good governance of evidence

The Politics of Evidence: From Evidence-based Policy to the Good Governance of Evidence - Parkhurst, Justin (Taylor & Francis, 2017)
The Open Access version of this book, available at http://www.tandfebooks.com/, has been made available under a Creative Commons Attribution-Non Commercial-No Derivatives 3.0 license. There has been an enormous increase in interest in the use of evidence for public policymaking, but the vast majority of work on the subject has failed to engage with the political nature of decision making and how this influences the ways in which evidence will be used (or misused) within political areas. This book provides new insights into the nature of political bias with regards to evidence and critically considers what an ‘improved’ use of evidence would look like from a policymaking perspective. Part I describes the great potential for evidence to help achieve social goals, as well as the challenges raised by the political nature of policymaking. It explores the concern of evidence advocates that political interests drive the misuse or manipulation of evidence, as well as counter-concerns of critical policy scholars about how appeals to ‘evidence-based policy’ can depoliticise political debates. Both concerns reflect forms of bias – the first representing technical bias, whereby evidence use violates principles of scientific best practice, and the second representing issue bias in how appeals to evidence can shift political debates to particular questions or marginalise policy-relevant social concerns. Part II then draws on the fields of policy studies and cognitive psychology to understand the origins and mechanisms of both forms of bias in relation to political interests and values. It illustrates how such biases are not only common, but can be much more predictable once we recognise their origins and manifestations in policy arenas. Finally, Part III discusses ways to move forward for those seeking to improve the use of evidence in public policymaking. It explores what constitutes ‘good evidence for policy’, as well as the ‘good use of evidence’ within policy processes, and considers how to build evidence-advisory institutions that embed key principles of both scientific good practice and democratic representation. Taken as a whole, the approach promoted is termed the ‘good governance of evidence’ – a concept that represents the use of rigorous, systematic and technically valid pieces of evidence within decision-making processes that are representative of, and accountable to, populations served.
Keywords: Public policy; Social Policy; Social Services & Welfare

Guide to Evaluating Participatory Processes - Practical guide - Parés, Marc (2013)
Guide to Evaluating Participatory Processes - Practical guide
Keywords: Academia; Biology; Computer Science; Earth Sciences; Geography; History; Law; Math; Medicine; Philosophy; Physics; Political science; Psychology; Religion; Research; economics; universities

Participación Digital y mejora democrática - un camino sin regreso - Palumbo, Gabriel (Luminate, 2020)
Tal vez como nunca en la historia, la democracia está siendo desafiada. A los viejos problemas, muchos no resueltos, se les agregan otros, surgidos de la incorporación de tecnología, de la emergencia de nuevos actores sociales y de una conformación de la subjetividad política cada vez más compleja e impredecible. Esta investigación pretende reactualizar algunos de los debates clásicos sobre la democracia y, al mismo tiempo, sentar una base empírica que permita pensar de qué modo la tecnología puede ayudar a mejorar la calidad democrática. Según nuestro estudio, la participación digital está promoviendo modificaciones importantes en la relación entre la ciudadanía y los decisores políticos, ampliando los espacios de proximidad. Esto sucede, fundamentalmente, porque baja las barreras de acceso y de incentivos haciendo más fácil, más efectiva y más mensurable la participación ciudadana. La participación digital aumenta el períodos electorales y fomenta el asociacionismo cívico como posibilidad cierta y concreta de presionar sobre los decisores políticos. Lo digital habilita, además, participación offline y genera espacios de pedagogía cívica. La capacidad de implantación de temas en el debate público que se puede generar a través de ella mejora la conversación pública, impactando a su vez la calidad de la democracia. Desde Luminate entendemos que favorecer los espacios de rendición de cuentas desde la ciudadanía hacia el poder político es una apuesta al robustecimiento de la experiencia democrática y que las formas digitales están abriendo posibilidades que otras opciones no pudieron lograr. Esperamos que esta investigación sea un aporte para poder seguir reflexionando y avanzando en esta dirección.

MOTION HANDBOOK - Developing a transformative Theory of Change - Palavicino, Carla Alvial; Matti, Cristian; Witte, Jenny (TIPC, Utrecht University, 2022)
A step-by-step guide on how to develop a Transformative Theory of Change, for innovation projects, programmes and organisations working on systems transformation. The MOTION project was initiated with one key question in mind: how can we help projects and organisations be more transformative, using the framework and concept provided by the multi-level perspective? And what kind of tools, methods and frameworks can we co-design that translate scientific concepts into practises relevant for policy practitioners? This led us into a co-creation journey during which researchers from the Transformative Innovation Policy Consortium (TIPC) and EIT Climate-KIC project partners experimented, reflected and learned from each other in building the approach that we share in this handbook. Through this journey, we gained a deeper understanding of what the portfolio approach means in a transformative system change context and which skills and competences are needed to facilitate processes of co-creation in the science-policy-practice interface. We had the opportunity to configure the key building blocks of our theoretical approach, the Transformative Outcomes, into practical insights and actions that can easily be applied by innovation organisations at many levels. This handbook is the culmination of the journey as it translates key learnings from the MOTION project into practical insights that are relevant to practitioners working on systems transformation.

A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes - Pahl-Wostl, Claudia (Global Environmental Change, 2009)
Governance failures are at the origin of many resource management problems. In particular climate change and the concomitant increase of extreme weather events has exposed the inability of current governance regimes to deal with present and future challenges. Still our knowledge about resource governance regimes and how they change is quite limited. This paper develops a conceptual framework addressing the dynamics and adaptive capacity of resource governance regimes as multi-level learning processes. The influence of formal and informal institutions, the role of state and non-state actors, the nature of multi-level interactions and the relative importance of bureaucratic hierarchies, markets and networks are identified as major structural characteristics of governance regimes. Change is conceptualized as social and societal learning that proceeds in a stepwise fashion moving from single to double to triple loop learning. Informal networks are considered to play a crucial role in such learning processes. The framework supports flexible and context sensitive analysis without being case study specific. First empirical evidence from water governance supports the assumptions made on the dynamics of governance regimes and the usefulness of the chosen approach. More complex and diverse governance regimes have a higher adaptive capacity. However, it is still an open question how to overcome the state of single-loop learning that seem to characterize many attempts to adapt to climate change. Only further development and application of shared conceptual frameworks taking into account the real complexity of governance regimes can generate the knowledge base needed to advance current understanding to a state that allows giving meaningful policy advice.
Keywords: Adaptive capacity; Adaptive governance; Climate change adaptation; Complexity; Institutions; Resource management; Social learning

Context-sensitive engagement: Lessons learned from Swiss experiences in South Asia for aid effectiveness in fragile scenarios - Paffenholz, Thania; Jütersonke, Oliver (CCDP, 2008)

Applied political economy analysis for human rights programs and campaigns: A guide for practitioners (second edition) - Pact (Pact, 2023)
This updated guide provides practical guidance to practitioners in the human rights sector and beyond on how to integrate Applied Political Economy Analysis

Organizational Performance Index (OPI) Handbook: A practical guide to the OPI tool for practitioners and development professionals - Pact (Pact, 2015)
This handbook is intended to help development practitioners understand how to use and what can be achieved by Pact’s Organizational Performance Index (OPI) tool. This document will provide practitioners and managers with information on how the Pact OPI works and how best to incorporate it in their country strategy or program.

Illegal Wildlife Trade - Frameworks - Oxford Martin Programme on the Illegal Wildlife Trade (Oxford Martin Programme on the Illegal Wildlife Trade, 2018)
Our frameworks will enable a better understanding of wildlife product consumer preferences and motivations to effectively influence the system.

‘We have the solutions to our own problems’: supporting civil society in Nigeria - Overseas Development Institute (ODI, 2014)
Development assistance programmes that start from careful analysis of what works and what does not in country contexts are still remarkably rare. One reason for this is the lack of well documented examples of ‘politically’ smart aid programming. To this end, this film outlines the experience of the State Accountability and Voice Initiative (SAVI), a DFID-funded programme in Nigeria providing support to civil society, media and governmental organisations.

Outcome Harvesting. Understanding social change results in complex circumstances - outcomeharvesting.net (2018)
This website is a source of applications, events and resources to support the development of a community of practitioners of Outcome Harvesting. BetterEvaluation is one of three sister sites of OutcomeHarvesting.net

Outcome Mapping FAQs - Outcome Mapping Learning Community (Outcome Mapping, 2018)
FAQ resource produced by IDRC's Evaluation Unit, with contributions from Kaia Ambrose, Sarah, Earl, Jan Van Ongevalle and Julius Nyangaga. We've added it here to answer some of your OM related questions.

Knowledge Management strategies and processes in Agile Software Development: a systematic literature review - Ouriques, Raquel Andrade Barros; Wnuk, Krzysztof; Gorschek, Tony; Svensson, Richard Berntsson (CoRR, 2018)
Knowledge-intensive companies that adopt Agile Software Development (ASD) relay on efficient implementation of Knowledge Management (KM) strategies to promotes different Knowledge Processes (KPs) to gain competitive advantage. This study aims to

VOICES - On speaking, mediation, representation and listening - Oswald, Katy (Think Piece, 2014)

Which Voices Are Heard and By Whom? - Oswald, Katy (Making All Voices Count, 2014)

A diagnostic approach for going beyond panaceas - Ostrom, E. (Proceedings of the National Academy of Sciences, 2007)

Monitoring and Adaptive Management - Ossi, Damien (District of Columbia Wildlife Action Plan - 2015, 2015)

Planning & Navigating Social Change - Tools for Pacific voyagers - Orr, Douglas Epeli; Cavatore, Maria; Aston, Tom (Oxfam, 2019)
Those working to advance social change in the Pacific must understand, work with and respond to the complex and changing relationships and dynamics of power that exist within such networks, and situate their work in the context of decolonisation and self-determination. But the approaches typically used by NGOs to plan our projects and programs have failed to give due attention to these dynamics. The result is that our projects fall short of achieving their promise, despite being technically sound and logical. In this guide, we aim to provide a process and tools that prioritise and draw out the rich, often implicit, knowledge that Pacific Islanders have about our contexts to plan and manage social change initiatives in a manner that values and responds to this ocean of relationships. Drawing from these relationship-based and voyaging traditions, we emphasise the need to: • frame change in terms of change in people; e.g. change in the situation of groups of people and change in terms of the actions of people; • identify a course towards a particular ‘change destination’; • identify the signs to read that will help you determine where you are on that course; • be constantly attentive to the context as you make the journey, and • be ready to adjust your course in response to changes in your context. To support this way of working, we have selected a number of tools from approaches that we believe fit. In particular, we have drawn from the following approaches: • Thinking and Working Politically and other ‘politically aware’ approaches that recognise that the success of a development initiative will be determined by the interplay of relationships of power within a particular context, not just by having the right technical solution; • Outcome mapping because of its focus on people and relationships, and on identifying and supporting needed changes in the behaviour of specific actors in relation to others. This is key, because it’s individuals who deliver services, individuals who decide whether to allocate budget and resources to address a problem, and individuals who make the choice to believe whether a service is relevant to solving their issues; • Adaptive management because of its recognition that development initiatives take place within complex and changing systems and need to adjust to these and that this requires ongoing learning and adaptation.

‘Learning journeys’ for adaptive management – Where does it take us? - Ørnemark, Charlotte (GPSA Note 12, 2016)
Keywords: IMPORTANT

What about the results? Lessons from long-term process support to strengthen results-based management (RBM) for Swedish framework NGOs operating in Western Balkans - Ørnemark, Charlotte (NCG, 2012)
The observations and lessons outlined in this report should be seen as a contribution to the on-going learning and reflections in a wider debate on how to assess and monitor results from support to CSOs involved in complex social change processes (as opposed to those involved in more ‘classical’ service delivery and more linear development assistance). These lessons, though directly emerging from the consultancy to provide long-term RBM support to the FOs and their local CSO partners in Western Balkans, also draws on previous experiences and similar lessons from on-going initiatives. In brief, these lessons call for a certain ‘paradigm shift’ in the way we (as development professionals) regard and support RBM for actors involved in less tangible social change processes in highly contested political spaces. Some of these shifts, outlined in more detail in different sections of this report, are summarised below:  From a linear, aggregated cause-and-effect thinking around RBM to one that is linked to human factors and is embedded in systems,  From pushing the burden of reporting down in the system to a clearer division of labour between donors, recipient governments, intermediary agents like framework NGOs (FOs) and implementing CSOs with each actor using its comparative advantage and appropriate role in the RBM process and for its own learning,  From operating in programming and results frameworks where outcomes and impact are largely assumed to be predictable to setting up systems that deal with uncertainty and that capture emerging result patterns through tracking of gradual changes,  From a project/programme perspective to a focus on institutions and systems as actors and arenas for change, where organisations are enabled to act as change agents towards clearly identified processes of social transformation,  From SMART to REAL results frameworks – although SMART (specific, measurable, achievable, realistic and time-bound) principles are always good to keep in mind for the formulation of goals and indicators, the danger of coming up with fictive measuring frameworks calls for a more systematic incorporation of ‘real’ concerns, including making RBM processes rights-based, embedded in local realities (and empowering for those involved), aligned to national reform efforts and national and international human rights commitments, and learning-oriented for all different levels of operations.

“Shared learning” for building urban climate resilience – experiences from Asian cities - Orleans Reed, Sarah; Friend, Richard; Toan, Vu Canh; Thinphanga, Pakamas; Sutarto, Ratri; Singh, Dilip (Environment and Urbanization, 2013)
This paper considers how resilience thinking and, in particular, its emphasis on learning has been applied in 10 cities in Vietnam, India, Thailand and Indonesia. Applying a “shared learning” approach in the Asian Cities Climate Change Resilience Network (ACCCRN) has helped to create or strengthen networks, build appreciation for complexity and uncertainty among stakeholders, provide a space for deliberating concepts such as vulnerability and resilience, and build knowledge and capacities for stakeholders to engage and represent their own interests. Shared learning approaches face considerable challenges navigating politicized urban environments, in which the nature and value of existing systems − and therefore the value of building resilience − are contested. This article suggests that deliberate, strategic intervention by facilitators may contribute to more transformative change on behalf of equitable, socially just outcomes – and thus cautions against seeing urban climate vulnerability as a technical challenge, or shared learning as a “toolkit” for building resilience.

Adaptive management in wildlife conservation - Organ, John; Decker, D.J.; McDonald, J.E.; Riley, Shawn; Mahoney, S.P. (The Wildlife Society Techniques Manual, 2012)
Adaptive management in wildlife conservation emerged from the wildlife profession's search for better solutions to increasingly complex conservation challenges. Adaptive management is an effective process for wildlife managers to employ to (1) deal with uncertainty in the management system, (2) learn from their management actions, and (3) achieve desired results. Being adaptable or flexible in your management approach is not the same as managing adaptively or conducting adaptive management. Adaptive management requires adhering to a stepwise process and fully executing each step. A critical step is rigorous monitoring and assessment of management interventions. Without this, wildlife managers cannot achieve the essence of adaptive management, which is the explicit goal of learning more about the management system after each management action.

Modern Slavery Prevention and Responses in South Asia: An Evidence Map - Oosterhoff, Pauline; Yunus, Raudah; Jensen, Charity; Somerwell, Francis; Pocock, Nicola (DFID, 2018)
The Asia Pacific region has the highest numbers of both slavery and child labour victims in the world. The International Labour Organization (ILO) estimates that 16.5 million and 8.4 million persons were in situations of forced labour and forced marriage respectively in Asia Pacific, among 40.3 million in modern slavery globally (ILO & Walk Free Foundation 2017). Although there is a growing body of research and evaluations on specific sub-sectors and interest in the worst forms of labour exploitation, there has not yet been a systematic scoping or synthesis of studies that would help policymakers understand ‘what works’ to reduce the prevalence of “Modern Slavery”2(MS) in the countries of interest to DFID (India, Nepal, Bangladesh, Pakistan). India alone is estimated to have the largest number of people in modern slavery in the world with nearly 8 million persons living in modern slavery, followed by 3.2 million, 592,000 and 171,000 in Pakistan, Bangladesh and Nepal respectively (Walk Free Foundation 2018). While systematic reviews on interventions to reduce prevalence of trafficking, forced or bonded labour or slavery exist, these have not focused on the target countries of interest to DFID. Previous systematic reviews have focussed on particular types of MS, such as cross border sexual exploitation globally (van der Laan et al. 2011), labour exploitation in Europe (Cockbain et al. 2018) or community based interventions for safer migration programming in low and middle income countries (LMICs) (Zimmerman et al. 2016). Several systematic reviews focus specifically on health needs and post-trafficking care or interventions (Ottisova et al. 2016; Muraya & Fry 2016; Hemmings et al. 2016; Dell et al. 2017), and research methods and tools used in trafficking in persons (TIP) and health research (Cannon et al. 2018; Doherty et al. 2016). A recent global review and evidence map of MS interventions found some interventions in the target countries, but the outcomes were not specified (Bryant & Joudo 2018). In this Evidence Map, we address this gap in evidence for India, Nepal, Bangladesh and Pakistan specifically, scoping the range of Modern Slavery interventions and outcomes for specific target populations (survivors, employers, landlords, services providers, criminal justice officials) and at different levels (individual, community, state). This map is foremost targeted to DFID and its partners in India, Nepal, Bangladesh and Pakistan to inform evidence-based policy making. We hope that it is also useful to academics and practitioners working to address modern slavery, or in intervention areas or locations described.

Using Participatory Statistics to Examine the Impact of Interventions to Eradicate Slavery: Lessons from the Field - Oosterhoff, P.; Bharadwaj, S.; Burns, D.; Raj, A. M.; Nanda, R. B.; Narayanan, P. (2016)
This CDI Practice Paper by Pauline Oosterhoff, Sowmyaa Bharadwaj, Danny Burns, Aruna Mohan Raj, Rituu B. Nanda and Pradeep Narayanan reflects on the use of participatory statistics to assess the impact of interventions to eradicate slavery and bonded labour. It deals with: (1) the challenges of estimating changes in the magnitude of various forms of slavery; (2) the potential of combining participatory approaches with statistical principles to generate robust data for assessing impact of slavery eradication; and (3) the practical and ethical questions in relation to working with people living within a context of modern slavery. The paper draws lessons from the realities of using participatory statistics to support the evaluation of a slavery eradication programme in North India.

Systems Practice - Omidyar Group (The Omidyar Group, 2017)

20 years of Outcome Mapping - OM Learning Community (2021)
Celebrate 20 years of Outcome Mapping with the OMLC Stewards plus special guests in this 90 minute webinar. - Exclusive celebration video with Outcome Blues soundtrack - Fireside chat with the authors of the OM manual, Sarah Earl, Fred Carden & Terry Smutylo, to hear their views on OM’s evolution since the 2001 publication - Launch of our new paper presenting core concepts of OM and guiding practices for transformative change, with special guests Sonal Zaveri & Julius Nyangaga as discussants

20 years of Outcome Mapping - Evolving practices for transformative change - OM Learning Community (Outcome Mapping Learning Community, 2021)
To mark 20 years of Outcome Mapping, this paper presents the core principles and concepts that are foundational to using the approach. It also presents a set of guiding practices to support transformative change. The OMLC Stewards presented the paper at a special webinar on 29th April 2021 - see the link below for the recording.

Locally Led Development: Engaging Local Stakeholders in Building An Evidence Base - OLS (USAID Learning Lab, 2020)
For two months in the Spring of 2020, the Office of Local Sustainability’s Evidence and Learning Team invited staff from across USAID to join us in exploring how the Agency approaches its generation and use of evidence from the perspective of locally led development. Our seven-part Standards of Evidence for Locally Led Development series brought together eight expert presenters and more than 670 participants to engage in conversations ranging from scientific research to complexity-aware monitoring to ethical considerations when conducting research. This blog post outlines some of our key learnings from the series and introduces the topics covered by each presenter. We encourage you to explore the event resources and share your own takeaways in the comment section below. WHAT DID WE LEARN? Three key themes emerged from the series: - Practitioners should draw on a wide range of evidence to inform locally led development programming. - Local actors can provide important contextual knowledge throughout the life of your research programs. - Prioritize local actors as the end users for your evidence to help them build their own self-reliance and increase sustainability of your program’s outcomes.

Evaluating the Design of Behavior Change Interventions: A Case Study of Rhino Horn in Vietnam - Olmedo, Alegría; Sharif, Vian; Milner‐Gulland, E. J. (Conservation Letters, 2018)
Behavioral change interventions are increasingly widely used in conservation. Several projects addressing rhino horn consumption were recently launched in Vietnam. We used key informant interviews, document analysis, and marketing theory to explore their strategies for intervention design, implementation, monitoring, and evaluation. We developed a framework to evaluate whether they followed best practice and identify implementation challenges. Interventions could make greater use of key project design steps, including basing interventions on robust research to understand the behavior in question, identifying the target audience whose behavior interventions aim to change, and developing measures that can provide reliable evidence of success or not. Challenges include the need for law enforcement to complement campaigns; improving cooperation between NGOs; and clearly defining aims of demand-reduction initiatives. Using best practice from other fields and considering demand reduction within the wider context of wildlife, trade policy will help address these challenges.
Keywords: Awareness; Ilegal wildlife trade; Project design; demand reduction; social marketing

The limits of agile - can we apply it to policy making? - Ollerhead, Lisa (2015)
Bringing new policy tools and techniques to the UK Government

Instituciones que aprenden - Informe para la XXVII Cumbre Iberoamericana de Jefes y Jefas de Estado y de Gobierno - Oliván Cortés, Raúl (XXVII Cumbre Iberoamericana, 2020)
La pandemia de la Covid-19 ha brotado súbitamente en un momento de transición para las instituciones y organizaciones sociales de medio mundo. Cuando aún no se había superado la crisis de confianza de la última década entre los ciudadanos y los gobiernos, en un momento de gran impulso de las iniciativas de gobierno abierto, planes de innovación y transformación digital, para hacer más democráticas y eficientes las administraciones a través de programas políticos encaminados a configurar un nuevo contrato social, verde y digital en el marco de la Agenda 2030; el coronavirus ha provocado un shock histórico alterando el decurso del siglo XXI y exigiendo una aceleración de todos los procesos en marcha. La complejidad y dimensión de la pandemia ha puesto de manifiesto la necesidad de arquitecturas institucionales más flexibles, ágiles y resilientes, que sean capaces de incorporar toda la energía civil para aprender de su talento y creatividad, dándole mayor protagonismo a la ciudadanía (activistas, emprendedores, tejido asociativo, academia, makers…) no solo en la toma de decisiones sino también en el diseño e implementación de las estrategias. Conforme se centrifugaba a millones de empleados a teletrabajar desde sus casas, se hacía más poderosa la idea de pensar las organizaciones como flujos más allá de los organigramas estáticos de lugares y personas. Transformar las jerarquías en redes, concebirlas como cuerpos sociales dinámicos, no solo amplía su radio de acción y su conectividad exterior, también reactiva sus fortalezas internas, aflora los liderazgos ocultos, multiplica el valor social producido y maximiza el uso eficiente de los recursos en una época de limitaciones. Los laboratorios de innovación pública, social o ciudadana, o laboratorios de gobierno, junto a otras formas de innovación abierta y diseño social, se reivindican como proyectos inspiradores de un cambio de paradigma: de las instituciones que ordenan a las instituciones que aprenden. Pensar las organizaciones bajo el prisma de la ciencia de redes y la ética de los rizomas –nodos, enlaces, hubs, comunidades…– nos debería permitir una aproximación a la compleja y escurridiza tarea de configurar los ecosistemas de innovación y creatividad en el ámbito de lo público y lo social. Este informe propone un modelo denominado Hexágono de la Innovación Pública (HIP) que promueve un cambio sistémico a través de seis vectores (OPEN_ abierto, TRANS_ transversal, FAST_ ágil, PROTO_ modelado, CO_ colaborativo y TEC_ tecnológico) basados en las propiedades de las redes y en el análisis de 105 metodologías que usan las agencias más innovadoras del mundo. Se incluyen una herramienta de autodiagnóstico y el HIP-SIM, una primera aproximación a un software abierto para visualizar, modelar y simular la creación de ecosistemas innovadores con el que queremos propiciar una comunidad y un debate internacional.

Instituições que aprendem - Relatório para a XXVII Cimeira Ibero-Americana de Chefes de Estado e de Governo - Oliván Cortés, Raúl (XXVII Cimeira Ibero-Americana, 2020)
A pandemia da Covid-19 surgiu de súbito num momento de transição para as instituições e organizações sociais de meio mundo. Antes ainda de se ter ultrapassado a crise de confiança da última década entre os cidadãos e os governos, num momento de grande impulso das iniciativas de governo aberto e de planos de inovação e transformação digital para tornar mais democráticas e eficientes as administrações através de programas políticos encaminhados para configurar um novo contrato social, verde e digital no quadro da Agenda 2030, o coronavírus provocou um choque histórico alterando o curso do século XXI e exigindo uma aceleração de todos os processos em andamento. A complexidade e dimensão da pandemia revelou a necessidade de arquiteturas institucionais mais flexíveis, ágeis e resilientes, capazes de incorporar toda a energia civil para aprenderem com o seu talento e criatividade, dando maior protagonismo à cidadania (ativistas, empreendedores, tecido associativo, academia, makers…) não só na tomada de decisões mas também na configuração e implementação das estratégias. À medida que se afastava do centro a milhões de empregados para teletrabalhar a partir de casa, tornou-se mais poderosa a ideia de pensar nas organizações como fluxos que excedem os organigramas estáticos de lugares e pessoas. Transformar as hierarquias em redes, concebê-las como corpos sociais dinâmicos, não só alarga o seu raio de ação e conectividade externa, mas também reativa as suas forças internas, aflora as lideranças ocultas, multiplica o valor social produzido e maximiza o uso eficiente dos recursos numa época de limitações. Os laboratórios de inovação pública, social ou cidadã, ou laboratórios de governo, a par de outras formas de inovação aberta e configuração social, reivindicam-se como projetos inspiradores de uma mudança de paradigma: de instituições que ordenam para instituições que aprendem. Pensar nas organizações sob a ótica da ciência das redes e da ética dos rizomas – nós, ligações, hubs, comunidades… – deverá permitir-nos uma aproximação à complexa e subtil tarefa de configurar os ecossistemas de inovação e criatividade no âmbito público e social. Este relatório propõe um modelo denominado Hexágono da Inovação Pública (HIP) que promove uma transformação sistémica através de seis vetores (OPEN_ aberto, TRANS_ transversal, FAST_ rápido, PROTO_ modelado, CO_ colaborativo e TEC_ tecnológico) baseados nas propriedades das redes e na análise de 105 metodologias usadas pelas agências mais inovadoras do mundo. Incluem-se uma ferramenta de autodiagnóstico e o HIP-SIM, uma primeira aproximação a um software aberto para visualizar, modelar e simular a criação de ecossistemas inovadores com o qual queremos favorecer a comunidade e o debate internacional.

Modelo HIP - Hexágono de la innovación pública - Oliván Cortés, Raúl (2021)

Lessons Learned From the Use of the Most Significant Change Technique for Adaptive Management of Complex Health Interventions - Ohkubo, Saori; Mwaikambo, Lisa; Salem, Ruwaida M.; Ajijola, Lekan; Nyachae, Paul; Sharma, Mukesh Kumar (Global Health: Science and Practice, 2022)
Introduction:The Most Significant Change (MSC) technique is a complex-aware monitoring and evaluation tool, widely recognized for various adaptive management purposes. The documentation of practical examples using the MSC technique for an ongoing monitoring purpose is limited. We aim to fill the current gap by documenting and sharing the experience and lessons learned of The Challenge Initiative (TCI), which is scaling up evidence-based family planning (FP) and adolescent and youth sexual and reproductive health (AYSRH) interventions in 11 countries in Asia and sub-Saharan Africa. Methods:The qualitative assessment took place in early 2021 to document TCI’s use and adaptation of MSC and determine its added value in adaptive management, routine monitoring, and cross-learning efforts. Focus group discussions and key informant interviews were conducted virtually with staff members involved in collecting and selecting MSC stories. Results:TCI has had a positive experience with using MSC to facilitate adaptive management in multiple countries. The use of MSC has created learning opportunities that have helped diffuse evidence-based FP and AYSRH interventions both within and across countries. The responsive feedback step in the MSC process was viewed as indispensable to learning and collaboration. There are several necessary inputs to successful use of the method, including buy-in about the benefits, training on good interviewing techniques and qualitative research, and dedicated staff to manage the process. Conclusion:Our assessment results suggest that the MSC technique is an effective qualitative data collection tool to strengthen routine monitoring and adaptive management efforts that allows for flexibility in how project stakeholders implement the process. The MSC technique could be an important tool for global health practitioners, policy makers, and researchers working on complex interventions because they continually need to understand stakeholders’ needs and priorities, learn from lessons and evidence-based practices, and be agile about addressing potential challenges.

Open Governance Research Exchange - OGRX (Your hub for quantitative and qualitative research on innovations in governance, 2017)
Your hub for quantitative and qualitative research on innovations in governance.

A Holistic Approach to Evaluating Research - Ofir, Zenda; Schwandt, Thomas; Duggan, Colleen; McLean, Robert (IDRC, 2016)

Minister of International Development and La Francophonie Mandate Letter (November 12, 2015) - Office of the Prime Minister (2015)
Dear Minister:I am honoured that you have agreed to serve Canadians as Minister of International Development and La Francophonie. You will be part of a strong team of ministers led by the Minister of Foreign Affairs.

Quality standards for development evaluation - OECD; Development Assistance Committee (OECD, 2010)
Based on a broad international consultative process, the DAC Quality Standards for Development Evaluation are a reference guide to good practice in development evaluation. With a view to improving the quality of evaluation processes and products, and facilitating collaboration, this reference guide lays out standards for each phase of a typical evaluation process: from defining purpose, to planning, designing, implementing, reporting, and learning from and using evaluation results.

Applying Evaluation Criteria Thoughtfully - OECD (OECD, 2021)
Relevance, coherence, effectiveness, efficiency, impact, and sustainability are widely used evaluation criteria, particularly in international development co-operation. They help to determine the merit or worth of various interventions, such as strategies, policies, programmes or projects. This guidance aims to help evaluators and others to better understand those criteria, and improve their use. It starts by describing what they are, and how they are meant to be used. Then the definitions and concepts underpinning each criterion are explained. Finally, examples provide the reader with concrete ideas for using them. The criteria were originally laid out in the early 2000s by the Network on Development Evaluation (EvalNet) of the OECD Development Assistance Committee (DAC). Although they have been widely used in evaluation, and beyond, this document is the first to comprehensively explore the concepts in detail, explain their intended use and offer practical guidance. It captures current thinking and best practice in evaluation, drawing on the inputs of internationally renowned evaluation experts from EvalNet and beyond.

Case studies of results-based management by providers: Canada - OECD (OECD, 2017)

Case studies of results-based management by providers: New Zealand - OECD (OECD, 2017)

Case studies of results-based management by providers: Sweden - OECD (OECD, 2017)

Case studies of results-based management by providers: Switzerland - OECD (OECD, 2017)

Case studies of results-based management by providers: The Netherlands - OECD (OECD, 2017)

Case studies of results-based management by providers: United Kingdom - OECD (OECD, 2017)

Case studies of results-based management by providers: World Bank Group - OECD (OECD, 2017)

DAC - Mid-term Peer Review of Switzaerland - 2016 - OECD (OECD, 2016)

DAC Peer Review of Switzerland - 2009 - OECD (OECD, 2009)

DAC Peer Review of Switzerland - 2013 - OECD (OECD, 2013)

DAC Peer Review of Switzerland - 2019 - OECD (OECD, 2019)

Evaluation Feedback for Effective Learning and Accountability - OECD (Evaluation and Aid Effectiveness, 2001)
Evaluation is a key tool in improving the quality and effectiveness of development co-operation. The Development Assistance Committee (DAC) Working Party in Aid Evaluation is the only international forum where bilateral and multilateral evaluation experts meet regularly. Its Members meet to improve evaluation practice and to learn lessons from development co-operation activities and experiences. A series "Evaluation and Aid Effectiveness" was launched to share selected pieces of work with a broader public. This publication is composed of two parts: The Workshop Report, based on the fore-mentioned meeting, highlights the various issues raised, topics of discussion and different feedback systems, and outlines the areas identified by participants as most relevant for improving evaluation feedback. The Background Synthesis Report, intended as a starting point for discussion at the workshop, outlines the main concerns and challenges facing evaluation feedback and the means to address these. The report is based on an analysis of questionnaire results, and a review of previous initiatives in this area. The print version of the above report is available free of charge from the Development Co-operation Directorate; please send an email to dac.contact@oecd.org to get a copy.

Evaluation of development programmes - OECD - OECD (2019)
Data & research on evaluation of development programmes inc. paris declaration, budget support, multilateral effectiveness, impact evaluation, joint evaluations, governance, aid for trade

Glossary of Key Terms in Evaluation and Results Based Management - OECD (OECD, 2009)

Innovation for Development Impact: Lessons from the OECD Development Assistance Committee - OECD (The Development Dimension, 2020)
The development co-operation community needs to innovate to meet the global challenges ahead. Although it has an established track record for innovating partnerships, funding instruments and technologies, they are not enough to deliver the Sustainable Development Goals. This report synthesises the lessons emerging from an OECD Development Assistance Committee peer learning exercise on how innovation efforts can be strengthened, individually and collectively, to achieve the 2030 Agenda. The report is organised around three blocks – strategy, management and culture; organisation and collaboration; and, the innovation process – and provides recommendations on how innovation can best benefit poor and vulnerable people around the world.

OECD Development Assistance Committee Peer Review of Canada - OECD (OECD, 2018)

Systems Approaches to Public Sector Challenges: Working with Change - OECD (OECD, 2017)

Doing Development Differently: Who we are, What we’re doing, What we’re learning - ODI (ODI, 2016)
In November 2014, the doing development differently community got together in Harvard to discuss what successful development interventions look like. Two years on, our community is broader than aid. It's broader than donors. It's about all organisations delivering change, producing real solutions to real problems that have real impact. It's about building trust, empowering people and promoting sustainability. Over the past two years, the community has been putting these ideas into practice across the world – being honest about what we are learning, including where we are not getting things right. This document aims to be an entry point for anyone interested in doing development differently. It explores the things that could be stopping you including: You are constrained by a disabling environment. You have strict reporting requirements. People aren't sold on the idea. It's easier to do things the way you've always done it. It's difficult to do in practice. If you are facing one, several, or all of these constraints, this document gives you a place to start. Using 43 case studies from practitoners within donors, governments, implementing organisations and NGOs across the globe, we aim to draw out some key lessons learnt, and give some advice for people considering this approach. These case studies are split across five categories: Swimming against the tide. Working in and with government. Feedback loops and data. Organisational change. Diffusion. Finally, if you want to know more, the document concludes with a suggested list of blogs and reports which explain these concepts in more detail.
Keywords: IMPORTANT

Strategy Development: Most Signficant Change (MSC) Toolkit - ODI (ODI, 2009)
The Most Significant Change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the programme cycle and provides information to help people manage it. MSC contributes to evaluation because it provides data on impact and outcomes which can be used to help assess the performance of the programme as a whole. Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most important of these by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of the reported changes. When the technique is successfully implemented, whole teams of people begin to focus their attention on programme impact. MSC has had several names since it was conceived, each emphasising a different aspect. Examples are: ‘Monitoring-without-indicators’ – MSC does not make use of predefined indicators, especially ones which have to be counted and measured; or the ‘story approach’ – the answers to the central question about change are often in the form of stories of who did what, when and why, and the reasons the

The Methods Lab evaluation toolkit - ODI (2016)
The Methods Lab develops and tests flexible approaches to impact evaluation for interventions that are harder to evaluate because of their complexity. This toolkit brings together analysis, guidance and templates for anyone: planning an impact evaluation designing a monitoring and evaluation system working in a consortium or managing a portfolio of projects

Evaluating Humanitarian Innovation - Obrecht, Alice; Warner, Alexandra; Dillon, Neil (HIF-ALNAP working paper, 2017)

More than just luck: Innovation in humanitarian action - Obrecht, Alice; Warner, Alexandra T. (Humanitarian Innovation Fund, 2016)
In South Sudan, a new water treatment system that provides a community with more clean water at a lower cost. In Sierra Leone, a poster explaining how to prevent the...

Making humanitarian response more flexible: Challenges and questions - Obrecht, Alice; Bourne, S. (ALNAP/ODI, 2018)

Dynamic gridlock: Adaptive Humanitarian Action in the Democratic Republic of Congo - Obrecht, Alice (ALNAP Country Study, 2018)

Key Concepts in Ethnography - O'Reilly, Karen (SAGE Publications, 2009)

Doing legal empowerment differently: Learning from pro-poor litigation in Bangladesh - O'Neil, Tam; Valters, Craig; Farid, Cynthia (ODI, 2015)
Marginalised groups and their allies can, and do, use the law and justice systems, including public interest litigation, to improve their access to rights, goods and services.  Yet there is no automatic link between legal action and improved outcomes for poor people.  Where some minimum conditions are met – a progressive legal framework, a sympathetic judiciary and legal advocacy organisation – pro-poor litigation is a potential tool in disputes over rights and resources.  But concrete benefits for poor people also requires state action to enforce progressive rulings.  Much depends on whether claimants, legal activists and state reformers cooperate around shared interests, and whether activists can negotiate power and interest structures to motivate government action to implement rulings.  Effective strategies are those that link litigation with grassroots legal action and other forms of political and social activism and advocacy.  Donors need to fund legal advocacy organisations in ways that enable them to select social issues that are locally relevant and political feasible – and allows activists and reformers to work in politically smart and adaptive ways.

Using adaptive development to support feminist action - O’Neil, Tam (ODI, 2016)
This paper looks at how adaptive development is being applied by gender programmes and argues that gender and adaptive development communities have much to offer each other.

Mapping Context for Social Accountability: A resource paper - O'Meally, Sion C. (The World Bank, 2013)

Is It Time for a New Paradigm for "Citizen Engagement"? The Role of Context and What the Evidence Tells Us - O'Meally, Simon (People, Spaces, Deliberation, 2013)
The meteoric rise of "citizen engagement" Almost all development agencies promote some form of citizen engagement and accountability, often framed as 'voice', 'demand-side governance', 'demand for good governance' or 'social accountability'. The current World Bank president, Jim Yong Kim, recently put it that, "citizen voice can be pivotal in providing the demand-side pressure on government, service providers, and organizations such as the World Bank that is needed to encourage full and swift response to citizen needs". There has, in turn, been a mushrooming of useful operational guidance on different "tools" for social accountability - i.e. steps, inputs and methodologies - that guide discrete interventions, ranging from citizen score cards to participatory expenditure tracking. One might, however, be forgiven for thinking that some of the debates on citizen engagement need an injection of realism; especially as contextual factors can make or break a "tool's" implementation. A review of experience to date would be one good place to start.

Adaptive management: What it means for CSOs - O’Donnell, Michael (Bond, 2016)
Keywords: IMPORTANT

Developing a Systems Thinking Lens for Collective Leadership - O’Donnell, Joan (Collective Leadership for Scotland, 2023)
Systems thinking approaches are gaining traction as an effective way of understanding and working with increasing complexity. They are being put forward by the Organisation for Economic Co-operation and Development as well as the World Health Organisation as a way to tackle the complex and unpredictable environments we now operate in. As the world has become increasingly interconnected, national or local boundaries cannot isolate and control social problems. The climate emergency, war and political instability have become everyday realities that accompany an ever-widening gap between those who have and control resources, and those who do not have enough to meet basic needs. The usual responses are becoming more and more ineffective; all interventions have unseen consequences or emergent properties that cannot be predicted using an inputoutput outcomes model. Designing interventions therefore requires a fresh lens to manage our way through turbulence and uncertainty. Most complex situations benefit from a complementary mix of a systematic (linear) decisive intervention as much as a systemic (whole-view) understanding of a proposed intervention. In this sense, adopting a systems thinking approach requires both framing the situation using a systems lens, as well as a commitment to take action and learn forward in realtime. When utilised in this way, a systems thinking lens complements rather than replaces traditional management tools. It draws on well-tested concepts and tools and relies on the systems leader to develop their practice, which includes the art of knowing what is needed and when. It is about threading traditional planning methods together with an understanding of the interrelationships, multiple perspectives and boundary judgements that influence the framing of a situation. It is becoming increasingly clear that effective leadership is embedded and invested: objective management science has no place in a world bound by interconnectedness and unintended consequences. Leaders are, as we shall see, an inherent part of both the problem and the proposed intervention. The purpose of this paper is to share some concepts informed by systems thinking to support you, as leaders in your organisations, to bring a systems-informed lens to your work. It puts you as a practitioner-leader at the centre of your own practice and encourages you to reflect critically on your positionality, as well as the lens you use to understand and intervene in complex issues. This paper has been developed on foot of training piloted with public service leaders in Scotland in the summer of 2022, and acts as an introduction to developing your systems literacy skills. While we are all born with a systems sensibility, many of us lose that sense of connectedness and interdependence with the world as we grow up. This paper acts as a prompt to bring a complexity informed approach to your work practice. It also contains some pointers for further resources and tools that may support your learning.

Fractal approaches to scaling transformations to sustainability - O’Brien, Karen; Carmona, Rosario; Gram-Hanssen, Irmelin; Hochachka, Gail; Sygna, Linda; Rosenberg, Milda (Ambio, 2023)
Responses to sustainability challenges are not delivering results at the scale and speed called for by science, international agreements, and concerned citizens. Yet there is a tendency to underestimate the large-scale impacts of small-scale, local, and contextualized actions, and particularly the role of individuals in scaling transformations. Here, we explore a fractal approach to scaling sustainability transformations based on “universal values.” Universal values are proposed as intrinsic characteristics that connect humans and nature in a coherent, acausal way. Drawing on the Three Spheres of Transformation framework, we consider how enacting universal values can generate fractal-like patterns of sustainability that repeat recursively across scales. Fractal approaches shift the focus from scaling through “things” (e.g., technologies, behaviors, projects) to scaling through a quality of agency based on values that apply to all. We discuss practical steps involved in fractal approaches to scaling transformations to sustainability, provide examples, and conclude with questions for future research.
Keywords: Agency; Fractals; Relational paradigms; Scaling; Three Spheres of Transformation; Universal values

Using adaptive monitoring, evaluation and learning in programme design - Nzegwu, Femi (INASP, 2018)

Working the System: What We’ve Learned About Strengthening Accountability - Nugent, Sarah (Chemonics International, 2018)
When designing a new project, how often do we set out to strengthen a particular actor’s or institution’s ability to “hold government to account?” What does that mean exactly? Maybe the better question to ask is: what is the most effective way to strengthen accountability in our development work? And then, what does it look …

Experiential Learning: Fostering lasting behavioural change - Norton, Brittney M. (Prospects practice paper, 4, 2016)
The term ‘experiential learning’ is often mistakenly used to describe any learning experience that involves participatory activities. This paper presents an overview of what true experiential learning is and how this form of learning allows for real-world application - thus promoting lasting behavioural change. Furthermore, it explains how experiential learning was adopted as the methodological core of the Prospects psychosocial programme in Liberia, which builds ‘pre-employment skills’ in 3,500 vulnerable youth aged 15-24 years. The goal of the paper is to provide other youth and education development practitioners with a resource that can be used to strengthen youth and behaviour change programmes through the incorporation of the experiential learning theory. In addition, this paper explains its application in the Prospects programme, before concluding with key considerations of such a design. 1

The Design of Everyday Things, revised and expanded edition - Norman, Donald A. (MIT Press, 2013)
Even the smartest among us can feel inept as we try to figure out the shower control in a hotel or attempt to navigate an unfamiliar television set or stove. When The Design of Everyday Things was published in 1988, cognitive scientist Don Norman provocatively proposed that the fault lies not in ourselves, but in design that ignores the needs and psychology of people. Fully revised to keep the timeless principles of psychology up to date with ever-changing new technologies, The Design of Everyday Things is a powerful appeal for good design, and a reminder of how -- and why -- some products satisfy while others only disappoint.

The Knowledge-Creating Company - Nonaka, Ikujiro (Harvard Business Review, 2007)
Editor’s Note: This 1991 article helped popularize the notion of “tacit” knowledge—the valuable and highly subjective insights and intuitions that are difficult to capture and share because people carry them in their heads. Years later, the piece can still startle a reader with its views of organizations and of the types of knowledge that inform […]
Keywords: Creativity; Innovation; Product development; Research & development

Theory of change in ten steps - Noble, James (NPC, 2019)

On the Right Tack: Reflections on Coalition-Building Initiatives across The Asia Foundation - Nixon, Nicola; Yates, Peter; Saluja, Sumaya; Yi, Su Lae; Lucas, Miranda; Bain, Katherine (The Asia Foundation, 2023)
Coalitions—groups of organizations and individuals that work together to pursue a common policy goal or reform—are crucial to development. Some of The Asia Foundation’s longest-standing and most successful development programs and portfolios have used coalition-building as an implementation modality. This paper examines successful initiatives in Bangladesh, Nepal, Philippines, Sri Lanka, Thailand, and Timor-Leste. By delving into each of these, we shed light on this coalition-building modality, sharing insights about how externally supported coalitions emerge and function and contributing ideas about how such support can be adapted to other contexts. We hope that this paper and these case studies contribute to innovations in contemporary development theory and help practitioners use and move beyond some formulaic and technocratic modalities, such as traditional capacity development, knowledge transfer, and technical assistance. People, relationships, and working partnerships can then take center stage, and sustained cooperation will gain primacy over short-term outputs and deliverables.

Exploring a new governance agenda: What are the questions that matter? - Nixon, Nicola; Verhulst, Stefaan; Matin, Imran; Vermonte, Philips J. (From Poverty to Power, 2021)
The 100 Questions Initiative, pioneered by the GovLab, seeks to overcome the chasm between supply and demand. It begins not by searching for what data is available, but by asking important questions about the biggest challenges societies and countries face, and then seeking more targeted and relevant data solutions. In doing this, it narrows the gap between policy makers and constituents, providing opportunities for improved evidence-based policy and community engagement in developing countries. As part of this initiative, we seek to define the ten most important questions across several domains, including Migration, Gender, Employment, the Future of Work, and—now–Governance. On this occasion, we invited over 100 experts and practitioners in governance and data science –whom we call “bilinguals”– from various organizations, companies, and government agencies to identify what they see as the most pressing governance questions in their respective domains. Over 100 bilinguals were encouraged to prioritize potential impact, novelty, and feasibility in their questioning — moving toward a roadmap for data-driven action and collaboration that is both actionable and ambitious.

From purposeful to meaningful adaptive programming: how about adaptive operations? - Nixon, Nicola (Devpolicy Blog from the Development Policy Centre, 2019)
Embracing adaptive programming approaches but sticking with rigid, traditional systems-based operations? It's time to take a long hard look at all the boring stuff, says Nicola Nixon.

Supporting accountability in fragile settings: A review for the Somalia Implementation and Analysis in Action of Accountability Programme - Nixon, Hamish; Buffardi, Anne; Wales, Joseph; Pasanen, Tiina (ODI, 2017)

Guidance: Rigor and Reproducibility in Grant Applications | grants.nih.gov - NIH (National Institutes of Health, 2018)

The curious case of the realist trial: Methodological oxymoron or unicorn? - Nielsen, Steffen Bohni; Jaspers, Sofie Østergaard; Lemire, Sebastian (Evaluation, 2023)
Realist evaluation and experimental designs are both well-established approaches to evaluation. Over the past 10 years, realist trials—evaluations purposefully combining realist evaluation and experimental designs—have emerged. Informed by a comprehensive review of published realist trials, this article examines to what extent and how realist trials align with quality standards for realist evaluations and randomized controlled trials and to what extent and how the realist and trial aspects of realist trials are integrated. We identified only few examples that met high-quality standards for both experimental and realist studies and that merged the two designs.

A guide to Social Return on Investment - Nicholls, Jeremy; Lawlor, Eilis; Neitzert, Eva; Goodspeed, Tim (The SROI Network, 2012)
There is increasing recognition that we need better ways to account for the social, economic and environmental value that results from our activities. The language varies – ‘impact’, ‘returns’, ‘benefit’, ‘value’ – but the questions around what sort of difference and how much of a difference we are making are the same. Understanding and managing this broader value is becoming increasingly important for the public and private sectors alike. This is true whether it is civil society organisations working to create value, Governments commissioning and investing in activities to create social value, investors seeking to ensure that their investments will make a difference, or private businesses recognising both risk and opportunity in the wider effects of operations. All this means that it is also more important that we have some consistency and a shared language when we talk about value. SROI is the application of a set of principles within a framework that is designed to help bring about that consistency, whilst at the same time recognising that what is of value will be very different for different people in different situations and cultures. The first edition of this guide, which itself built on the work of three earlier SROI guides1, was prepared as part of a three year programme on measuring social value funded in 2008 by the then ‘Office of the Third Sector’ based in the Cabinet Office of the UK Government. This was delivered by a consortium of organisations: the SROI Network, nef (the new economics foundation), Charities Evaluation Services, the National Council for Voluntary Organsations and New Philanthropy Capital. In addition to this programme, the Scottish Government also supported the development of SROI, including a database of indicators to support SROI analysis. The work of the SROI Network now stretches across many different countries and continents, and this second edition of the guide reflects that interest. We have though decided to use only one currency symbol, for reasons of clarity and consistency, and so have continued to use £. However, readers will be able to find examples in various currencies and translations of the Guide on our website. For more information on the developments of SROI, please refer to the SROI Network website: www.thesroinetwork.org

6 Reviewing research evidence - NICE (6. Developing NICE guidelines: the manual, 2012)

Life and death in the App Store - Newton, Casey (The Verge, 2016)
As last year began, the app developer Pixite held its company retreat at a converted Old West movie set outside of Palm Springs. They spent a few days dreaming about the future while eating...

Self Sabotage: Why Doing the Right Thing Results in Failure - Newell, Peter (War on the Rocks, 2020)
In January 2010 improvised explosive device (IED) attacks against dismounted infantry squads in Afghanistan numbered in the single digits — with only two

5 reflections on operationalising CLARISSA to generate evidence - Neupane, Sudarshan (CLARISSA, 2021)
As we start a new year, and especially, as 2021 is the International Year for the Elimination of Child Labour, understanding that the Worst Forms of Child Labour (WFCL) is complex is step one – there are no simple solutions. But we must go deeper than that, we must ensure that the programmes designed to intervene, and shape policy take on that complexity and precariousness. This is where the experience of the Child Labour: Action-Research-Innovation in South and South-Eastern Asia (CLARISSA) programme comes in. It is uniquely set up to generate evidence in innovative ways and to co-develop with our stakeholders’ solutions to the WFCL. This is ambitious, yes, but no one said it was going to be easy.

Designing for Public Services - Nesta (Nesta/IDEO, 2016)

DIY - Development Impact and You: Practical Tools to Trigger and Support Social Innovation - Nesta (NESTA, 2014)
Keywords: IMPORTANT

Skills, atitudes and behaviours that fuel public innovation - a guide to getting the most from Nesta's Competency Framework for Experimenting and Public Problem Solving - Nesta (Nesta, 2018)

States of Change - Learning programme - Nesta (Nesta, 2018)

The Clash of the Counter-bureaucracy and Development - Natsios, Andrew (The Center for Global Development, 2010)
In this essay, Andrew Natsios gives a first-hand account of what he finds most hinders USAID—layers of bureaucracy that misguide and derail development work.

Technical assistance: a practical account of the challenges in design and implementation - Nastase, Alexandra; Rajan, Alok; French, Ben; Bhattacharya, Debarshi (Gates Open Research, 2021)
Technical assistance is provided to country governments as part of international development programmes to support policymaking or strengthen state capability. This article presents the conceptual evolution of ‘technical assistance’ linked to capacity development, starting with programmes aiming exclusively to enhance individual capacity in the 1950s to 1970s and progressing to complex systems approaches in the past ten years. It also presents some of the frequent challenges in designing and implementing technical assistance, drawing from the existing literature and the authors’ experience in international development. The article summarises the latest thinking about delivering more effective development, including the adaptive management practices and the initiatives to strengthen evidence about what works. Finally, we complement this article with a follow-up open letter reflecting on the current policy options and opportunities for change.

Voices of the poor: can anyone hear us? - Naraya, Deepa*Patel (The World Bank, 2000)
This book is the first in a three-part series, about the common patterns that emerged from the poor people's experiences in many different places. Chapter 1 sets out the conceptual framework and methodology. Chapter 2 discusses poverty from the perspective of the poor. Chapter 3 examines poor people's experience with the state, and includes case studies of access to health care and education. Chapter 4 addresses the nature and quality of poor people's interactions with civil society. Chapter 5 considers the household as a key social institution, and discusses gender relations within households and how these relations affect and are affected by larger institutions of society. Chapter 6 focuses on social fragmentation, and includes a discussion of social cohesion and social exclusion. Chapter 7 concludes the analysis and proposes some policy recommendations. The analysis leads to these conclusions: 1) poverty is multidimensional; 2) the state has been largely ineffective in reaching the poor; 3) the role of nongovernmental organizations (NGOs) in the lives of the poor is limited, forcing the poor to depend primarily on their own informal networks; 4) households are crumbling under the stresses of poverty; and 5) the social fabric - poor people's only

Participation and Power in Climate Change Adaptation Policies: Vulnerability in Food Security Programs in Nepal - Nagoda, Sigrid; Nightingale, Andrea J. (World Development, 2017)
The article explores the moments wherein participatory approaches in climate change adaptation (CCA) policies contribute to reinforcing, rather than transforming, the underlying causes of vulnerability. Using the case of food insecure households in the district of Humla in northwestern Nepal, the study demonstrates that the same social and power relations that are driving local vulnerability dynamics, such as caste, gender, and access to social and political networks, also play important roles in shaping the impact of CCA policies. By tracing Nepal’s CCA programs, starting with the local level, through district to international-national level dynamics, the study adds insights into the barriers to exclusion that embed power relations all the way through the chain of policy development. The purpose is to better understand how CCA can perpetuate rather than alleviate the conditions that create differential vulnerability patterns at village level. It raises questions about how whether CCA programs are an adequate response to increasing vulnerability for some of the world’s most marginalized people.

How Does the World Bank Build Citizen Engagement Commitments into Project Design? Results from Pilot Assessments in Mozambique, Myanmar, Nigeria, and Pakistan - Nadelman, Rachel; Le, Ha; Sah, Anjali (Working Paper 525, 2019)
How and to what degree is the World Bank putting its new institutional citizen engagement (CE) commitments into practice? This question guides an independent assessment that the Accountability Research Center (ARC) at American University has undertaken as part of the Institute of Development Studies-led Action for Empowerment and Accountability (A4EA) research programme’s investigation into how external actors can best support local processes of and conditions for empowerment and accountability. This report investigates the World Bank’s incorporation of CE into project design, the critical early stage of donor engagement. To accomplish this, ARC reviewed the World Bank’s fiscal year 2015–17 investment project portfolios for four A4EA priority countries, Mozambique, Myanmar, Nigeria, and Pakistan, which covers 57 projects that range from US$19 million to U$600 million. The analysis determines whether projects commit to seeking a strategic approach to CE, which involves combining multiple tactics so that the whole could be greater than the sum of the parts. This assessment of CE commitments is intended to help to inform possible national, civil society organisation strategies to monitor whether and how these commitments are actually implemented in practice.

TICTeC 2018 - The Making All Voices Count programme: new lessons about donor-funded civic tech - mySociety (2018)
Keywords: Accountability; Civic Tech; IDS; TICTeC; transparency

Addressing challenges in communicating adaptation practices to smallholder farmers in Kenya through a radio intervention - Mwaniki, Fiona Nyawira; Gichuki, Charity; Mwangi, Maina; Mburia, Pamela; Wandago, Benson (Journal of Agriculture and Environment for International Development (JAEID), 2017)

An analysis of factors influencing success of ICT4D projects: a case study of the Schools Computerisation Programme in Mashonaland West Province, Zimbabwe. - Musiyandaka, Donna; Ranga, Gideon; Kiwa, Jacqueline Fungai (The Journal of Community Informatics, 2013)
Keywords: ICT4D programmes; success factors

The open book of social innovation - Murray, Robin; Caulier-Grice, Julie; Mulgan, Geoff (Young Foundation & NESTA, 2010)

The Promise of Lean Experimentation - Murray, Peter; Ma, Steve (Stanford Social Innovation Review, 2015)
By adopting a model from business, nonprofit organizations can launch, test, and implement new programs and services more efficiently. Includes magazine extras.
Keywords: IMPORTANT

Six things that help grant-makers learn and adapt - Murray, Aoife (Itad, 2017)
There’s an emerging body of literature identifying key strategies that can both improve learning and enable adaptive management amongst grant-makers. We recently reviewed a host of publications from eight grant-making organisations. Those organisations have diverse aims and approaches but we’ve identified six common themes that stand out: Strong leadership and facilitation enables successful learning and …

Grounded Theory Method in HCI and CSCW - Muller, Michael J; Kogan, Sandra (IBM Center for Social Software, 2010)

Systems Innovation - Discussion Paper - Mulgan, Geoff; Leadbeater, Charlie (NESTA, 2013)
Introduction Over the past few years there has been growing interest in systemic innovation. We are defining this as an interconnected set of innovations, where each influences the other, with innovation both in the parts of the system and in the ways in which they interconnect. Yet rather than simply theorising, we want to make this practical. We want to explore the potential of systemic innovation to help tackle some of the key challenges the UK currently faces, from supporting an ageing population to tackling unemployment. We would also like to open up the discussion, to engage with the wide and diverse range of experts already working this space to help sharpen up thinking about systemic innovation and influence practical work to advance it. About this paper The two essays contained within this paper are intended to contribute to this debate. The first paper by Geoff Mulgan addresses a number of questions about systemic – or ‘joined–up’ – innovation, asking what it means to truly transform a system, encompassing the means, methods, and actions needed to realise its potential. The second paper from Charlie Leadbeater argues that companies, governments, cities, and entire societies need to move from seeing innovation in products and services as a source of competitive advantage, to focussing on innovation with entire systems. Our next steps: helping make systemic innovation useful and useable This report is just the start. We want to work with academics, practitioners, policymakers, and others in the field, to help realise the great potential of connecting and joining up innovative products, services and processes so that the whole is more than the parts. Over the coming months we will be building upon the rich and diverse literature available (we have start collating this as an annex to this paper), as well as practitioner experience, to ensure we understand systems more rigorously, as well as the skills, capabilities, methods and means needed to help them change for the better.

Collective intelligence and achieving the Sustainable Development Goals - Mulgan, Geoff (nesta, 2018)
Collective intelligence and achieving the Sustainable Development Goals

Supporting the policy environment for economic development (SPEED+) - Strategic Review Report - MSI (USAID, 2019)
Donor-funded policy reform has a long history in Mozambique, with United States Agency for International Development (USAID) efforts dating to the mid-1990s. Many laws and regulations adopted during the past quarter century are the consequence of these efforts. And yet, even with several years of robust economic growth, Mozambique has not experienced the broad economic transformation that policy reforms can trigger: per capita income in 2017 was $519 and more than 80 percent of the country lived on less than two dollars per day. The country ranks 180 out of 189 on the Human Development Index and 135 out of 180 on the World Bank’s Doing Business Index. Supporting the Policy Environment for Economic Development (SPEED+), a $37 million, four-year, USAID-funded program, represents the latest iteration of USAID’s policy reform efforts. SPEED+ began in August 2016 and is scheduled to run through August 2020. As the program passed the midway point, USAID commissioned Management Systems International (MSI) to conduct a strategic review (SR) of the SPEED+ program, with the objectives of advising on: • how to maximize the impact of SPEED+ during its final 18 months; • what type of follow-on program USAID should consider; and • what broader lessons can be drawn from SPEED+’s experience in Mozambique.

Deciding Well in Tumultuous Times - Moss, Ian David (Smarter decisions for a better world, 2020)
Practical advice for donors and institutions responding to COVID-19

Learning how to go local: Lessons from six learning journeys for the Open Government Partnership - Moses, Michael; Soal, Sue (2017)
This brief reviews the evidence from Learning to Make All Voices Count (L-MAVC), a programme funded by Making All Voices Count, and implemented in collaboration with Global Integrity. L-MAVC intended to support six Making All Voices Count grantees, working in five countries, in co-creating and applying a participatory, learning-centred, and adaptive approach to strengthening citizen engagement in governance processes in their contexts, including with respect to the Open Government Partnership (OGP). The evidence from L-MAVC suggests that supporting citizen engagement with, and use of, OGP, and improving the extent to which OGP commitments are shaped by and matter to citizens in subnational districts, is not straightforward. Findings suggest that efforts to broaden and deepen citizen engagement with OGP, including at subnational levels, may be more effective when combined with support that helps local OGP champions iteratively learn and adapt, and discover and apply localisation models that fit best in their contexts. OGP and its partners may strengthen the impact of the initiative, and indeed, the impact of those working to leverage OGP at and below country level, by making structured learning support more available to local reformers. Targeted advocacy, focused on institutionalising collaborative OGP processes, and on linking subnational action with National Action Plans (NAPs), may also be useful.

Learning to Make All Voices Count: Lessons for OGP, donors, and practitioners - Moses, Michael; Soal, Sue (2017)
The Open Government Partnership (OGP), donors and multilaterals, and social accountability practitioners across the world are among various constituencies attempting to harness and actualise emerging insights about the nature of successful governance reform. But each of these groups faces challenges as they do so. This brief reviews the evidence from Learning to Make All Voices Count (L-MAVC), a programme funded by Making All Voices Count, and implemented in collaboration with Global Integrity. L-MAVC intended to support six Making All Voices Count grantees, working in five countries, in co-creating and applying a participatory, learning-centred, and adaptive approach to strengthening citizen engagement in governance processes in their contexts, including with respect to the Open Government Partnership (OGP). Two sets of lessons emerge from the experience of L-MAVC. First, supporting citizen engagement and government accountability in subnational contexts, and localising OGP in ways that matter to citizens, is not straightforward. Doing so successfully entails engaging with, navigating and shaping political and power dynamics in those contexts, and iteratively adapting to emerging lessons and challenges. Second, the effectiveness of adaptive ways of working depends in part on the extent to which they offer opportunities for cross-context peer learning, support the regular collection and use of data, and are themselves adaptive. These lessons have implications for the broader community of actors working to support governance reform, including OGP and its partners, donors and multilateral institutions, and practitioners and policy-makers. If these actors are to contribute more effectively to reforms that affect citizens’ lives, substantial changes – with respect to the nature of support provided to domestic stakeholders, grant-making practices, and practitioner approaches – may be warranted.

Supporting local learning and adaptation: understanding the effectiveness of adaptive processes - Moses, Michael; Soal, Sue (2017)
This brief reviews the evidence from Learning to Make All Voices Count (L-MAVC), a programme funded by Making All Voices Count, and implemented in collaboration with Global Integrity. L-MAVC intended to support six Making All Voices Count grantees, working in five countries, in co-creating and applying a participatory, learning-centred, and adaptive approach to strengthening citizen engagement in governance processes in their contexts, including with respect to the Open Government Partnership (OGP).

Learning to make all voices count: Lessons and reflection on localising the Open Government Partnership - Moses, Michael (2017)
This report reviews the evidence from Learning to Make All Voices Count (L-MAVC), a programme funded by Making All Voices Count and implemented in collaboration with Global Integrity. L-MAVC intended to support six Making All Voices Count grantees, working in five countries, in co-creating and applying a participatory, learning-centred, and adaptive approach to strengthening citizen engagement in governance processes in their contexts, including with respect to the Open Government Partnership (OGP). Two overarching sets of lessons emerge from the experiences of L-MAVC grantees. First, supporting citizen engagement and government accountability in subnational contexts, and localising the OGP in ways that matter to citizens, is not straightforward. Doing so successfully entails engaging with, navigating and shaping political and power dynamics in those contexts, and iteratively adapting to emerging lessons and challenges. Second, the effectiveness of adaptive ways of working depends in part on the extent to which they offer opportunities for cross-context peer learning, support the regular collection and use of data, and are themselves adaptive. These lessons have implications for the broader community of actors working to support governance reform, including the OGP and its partners, donors and multilateral institutions, and practitioners and policy-makers. The evidence from L-MAVC suggests that if these actors are to contribute more effectively to reforms that affect citizens’ lives, substantial changes – with respect to the nature of support provided to domestic stakeholders and to grant-making practices – may be warranted

Rapid experiments with a purpose - Moses, Michael (BEAM Exchange, 2017)
New insights, opinions and perspectives on market systems development, from experts and practitioners.
Keywords: IMPORTANT

Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts - Morrow, Nathan; Nkwake, Apollo M. (Evaluation and Program Planning, 2016)
Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context. We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.
Keywords: Agency; Assumption-aware; Complex context; Complex programs; Participatory approaches; Program design; Program evaluation; Program tools; Theory-based

Conclusion: Agency in the face of complexity and the future of assumption-aware evaluation practice - Morrow, Nathan; Nkwake, Apollo M. (Evaluation and Program Planning, 2016)
This final chapter in the volume pulls together common themes from the diverse set of articles by a group of eight authors in this issue, and presents some reflections on the next steps for improving the ways in which evaluators work with assumptions. Collectively, the authors provide a broad overview of existing and emerging approaches to the articulation and use of assumptions in evaluation theory and practice. The authors reiterate the rationale and key terminology as a common basis for working with assumption in program design and evaluation. They highlight some useful concepts and categorizations to promote more rigorous treatment of assumptions in evaluation. A three-tier framework for fostering agency for assumption-aware evaluation practice is proposed-agency for themselves (evaluators); agency for others (stakeholders); and agency for standards and principles.
Keywords: Agency; Complexity

Six Simple Rules: How to Manage Complexity without Getting Complicated - Morieux, Yves; Tollman, Peter (Harvard Business Review Press, 2014)
New tools for managing complexityDoes your organization manage complexity by making things more complicated? If so, you are not alone.According to The Boston Consulting Group’s fascinating Complexity Index, business complexity has increased sixfold during the past sixty years. And, all the while, organizational complicatedness—that is, the number of structures, processes, committees, decision-making forums, and systems—has increased by a whopping factor of thirty-five. In their attempt to respond to the increasingly complex performance requirements they face, company leaders have created an organizational labyrinth that makes it more and more difficult to improve productivity and to pursue innovation. It also disengages and demotivates the workforce.Clearly it’s time for leaders to stop trying to manage complexity with their traditional tools and instead better leverage employees' intelligence. This book shows you how and explains the implications for designing and leading organizations.The way to manage complexity, the authors argue, is neither with the hard solutions of another era nor with the soft solutions—such as team building and feel-good “people initiatives”—that often follow in their wake. Based on social sciences (notably economics, game theory, and organizational sociology) and The Boston Consulting Group’s work with more than five hundred companies in more than forty countries and in various industries, authors Yves Morieux and Peter Tollman recommend six simple rules to manage complexity without getting complicated.Showing why the rules work and how to put them into practice, Morieux and Tollman give managers a much-needed tool to reinvigorate people in the face of seemingly endless complexity. Included are detailed examples from companies that have achieved a multiplicative effect on performance by using them.It’s time to manage complexity better. Employ these six simple rules to foster autonomy and cooperation and to effectively handle business complexity. As a result, you will improve productivity, innovate more, reengage your workforce, and seize opportunities to create competitive advantage.
Keywords: Complexity; Design Thinking

Embracing the Chaotic: Cynefin and Humanitarian Response - Morealtitude (WanderLust, 2010)
Note: Many thanks to Dave Snowden for his willingness to comment on this article prior to release Because I’m always a little behind the eight-ball, a friend of mine* recently introduced me to the …

Adaptive Leadership in Development course - Moran, Mark; Curth-Bibb, Jodie; Grice, Tim; Glavey, Sarah (edX MOOC, 2017)

Lifestyle politics and the concept of political participation - Moor, Joost de (Acta Politica, 2017)
Van Deth’s comprehensive ‘conceptual map of political participation’ has reinstated a lively debate about the concept of political participation, and provides some compelling solutions to it. However, an important question that has been raised is whether van Deth’s map actually achieves its main goal of unambiguously identifying and classifying emerging, complex types of participation, like online political activism – or lifestyle politics. To contribute to this debate, this article aims to evaluate the usefulness of van Deth’s approach for the analysis of lifestyle politics. Such an evaluation requires a clear classification of lifestyle politics. This, however, is still missing from the literature. The second aim of this article, therefore, is to identify and classify different types of lifestyle politics. On the basis of a literature review, this article argues that lifestyle politics are often enacted throughout different private, public and institutional arenas, and that they are often targeted at various social, economic and political actors at once. Applying van Deth’s conceptual map to these empirical realities, then, suggests that it cannot always account for their complexity sufficiently. Therefore, this article proposes a modification of van Deth’s framework that increases its usefulness for analyzing emerging, complex political participation repertoires.

EPE Week: Lee-Anne Molony on Collaborative Outcomes Reporting - Molony, L A (AEA365 a Tip-a-day by and for Evaluators, 2013)
This AEA365 blog, written by Lee-Anne Molony a Principal Consultant at Clear Horizon, provides a brief overview on Collaborative Outcomes Reporting (COR).

Building intentional networks that drive impact - Mohr, Jeff (In Too Deep, 2016)
These days everyone is familiar with some type of network — whether that’s their professional network on LinkedIn, their social network on…

Building intentional networks that drive impact (part 2) - Mohr, Jeff (In Too Deep, 2016)
This is part 2 of our series on building intentional networks. Make sure you’ve read part 1 before diving into this article!

Quality Flaws: Issues and Challenges in Software Development - Mohan, P.; Shankar, A. Udaya; JayaSriDevi, K. (Computer Engineering and Intelligent Systems, 2012)
A statement “Prevention is better than cure” for illnesses in medical sciences also applies to the software development life cycle in terms of software defects. A defect is a deviation from actual functionality of the application in terms of the correctness and completeness of the specification of the customer requirements. Defective software fails to meet its customer requirements leading to the development of applications with poor quality. Quality is a top priority in every enterprise these days. Organizations struggle in a treadmill race to deliver quality software to stay ahead with new technology, deal with accumulated development backlogs, handle customer issues as software teams work as hard as they can make their organizations stay alive and competitive in the market place. Software companies face an immense pressure to virtually release a bug-free product or a software package. The culture of an organization is a critical success factor in the efforts of process improvement. The paper aims at assessing quality as a function for monitoring and measuring the strength of development processes and any successful application development enterprise requires an unambiguous understanding of customer expectation and maximizing participation of customers in the development activities thereby ensuring that people involved in development activities do the right thing and do the thing right for delivering high quality software . Keywords: Software development, process improvement, software defect, bug-free product, software package

Using elements of DCED Standards for CLA - Mitra, Bilash; Jalil, Mohammad Muaz (2018)
Introduction used in the DCED-BEAM seminar in Nairobi

(Mis)communicating climate change? Why online adaptation databases may fail to catalyze adaptation action - Mitchell, Carrie L.; Burch, Sarah L.; Driscoll, Patrick A. (Wiley Interdisciplinary Reviews: Climate Change, 2016)
Over the last decade a plethora of action-oriented research projects has been conducted in developing countries, exploring how to effectively adapt to the anticipated impacts of climate change. Many intergovernmental agencies and development organizations have chosen to disseminate their research results via online databases. It is unclear, however, whether these databases are useful in terms of actual adaptation planning and implementation. A systematic review of online databases has found at least 64 databases and tools online related to climate change adaptation. Despite the abundance of databases, this analysis reveals that the existing body of online databases generally lack the structure and mechanics to identify, extract, and synthesize both effective and ineffective climate change adaptation practices, projects, programs, and policies. Even relatively basic information, such as identification of projects’ projected versus actual costs is absent, which are crucial decision-making criteria particularly in developing country contexts where resource constraints are significant. In this paper we evaluate these online tools with a focus on identifying features that potentially could contribute to knowledge sharing and successful exchange of climate change adaptation projects and practices within a developing country context. We conclude the paper with recommendations for how to improve efforts to communicate climate change research, such as more nuanced needs assessments of potential users of databases. WIREs Clim Change 2016, 7:600–613. doi: 10.1002/wcc.401 This article is categorized under: Vulnerability and Adaptation to Climate Change > Institutions for Adaptation Climate and Development > Knowledge and Action in Development Social Status of Climate Change Knowledge > Knowledge and Practice

Guidance Note for Adaptive Management - Ministry of Foreign Affairs of Denmark (Government of Denmark, 2022)
This note explains what adaptive aid management is; why and when it should be considered; and how it should be applied. It covers all Danish development support channels and modalities, including bilateral country assistance, assistance to and through civil society, the private sector and to and through multilateral organisations. This guide has three chapters. Chapter 1 provides an executive overview of what adaptive management is. Chapter 2 goes deeper into five key operational principles of adaptive management. Chapter 3 details the main tenets of adaptive management processes during the programming cycle.

Guidelines for Country Strategic Frameworks Programmes and Projects - Ministry of Foreign Affairs of Denmark (Government of Denmark, 2020)

Evidence to Action: Research to Address Illegal Wildlife Trade - Milner-Gulland, E. J.; Cugniere, Laure; Hinsley, Amy; Phelps, Jacob; Rolfe, Michael 't Sas; Veríssimo, Diogo (2018)
Tools and expertise to improve the evidence base for national and international Illegal Wildlife Trade policy already exist but are underutilised. Tapping into these resources would produce substantive benefits for wildlife conservation and associated sectors, enabling governments to better meet their obligations under the Sustainable Development Goals and international biodiversity conventions. This can be achieved through enhanced funding support for inter-sectoral research collaborations, engaging researchers in priority setting and programme design, increasing developing country research capacity and engaging researchers and community voices in policy processes. This briefing, addressed to policy makers and practitioners, is part of the 2018 Evidence to Action: Research to Address Illegal Wildlife Trade event programme, organised by five of the UK’s most active IWT research institutions, to support the London 2018 IWT Conference.

‘You Have to Raise a Fist!’: Seeing and Speaking to the State in South Africa - Mills, Elizabeth (IDS Bulletin, 2016)

Buzzing Communities: How to build bigger, better, and more active online communities - Millington, Richard (Feverbee, 2012)
How to Build Bigger, Better, and More Active Online CommunitiesBuzzing Communities cuts through the fluff to offer a clear process for creating thriving online communities. This book combines a century of proven science, dozens of real-life examples, practical tips, and trusted community-building methods. This step-by-step guide includes a lifecycle for tracking your progress and a framework for managing your organization's community efforts. This Book Will Help You to Understand what the members of your community really want. Dramatically increase the number of newcomers that become regulars. Avoid the mistakes most organizations make when they try to build online communities. Develop a fantastic, user-friendly website for your members. Grow your online community to critical mass and beyond Keep members engaged and active in your community. Measure the community's return on investment and explain the benefits to your organization.

The Indispensable Community: Why Some Brand Communities Thrive When Others Perish - Millington, Richard (Feverbee, 2018)
How to Build Bigger, Better, and More Active Online CommunitiesBuzzing Communities cuts through the fluff to offer a clear process for creating thriving online communities. This book combines a century of proven science, dozens of real-life examples, practical tips, and trusted community-building methods. This step-by-step guide includes a lifecycle for tracking your progress and a framework for managing your organization's community efforts. This Book Will Help You to Understand what the members of your community really want. Dramatically increase the number of newcomers that become regulars. Avoid the mistakes most organizations make when they try to build online communities. Develop a fantastic, user-friendly website for your members. Grow your online community to critical mass and beyond Keep members engaged and active in your community. Measure the community's return on investment and explain the benefits to your organization.

The Relational Work of Systems Change - Millgan, Katherine; Zerda, Juanita; Kania, John (Stanford Social Innovation Review, 2022)
Collective impact efforts must prioritize working together in more relational ways to find systemic solutions to social problems. Sometimes we lose sight of a simple truth about systems: They are made up of people. Despite all of the frameworks and tools at our disposal and all of our learning as a field of practice, purely technical, rational approaches to systems change will not make much of a dent in shifting power or altering our most deeply held beliefs. If most collective impact efforts fall short of supporting people to change in fundamentally consciousness-altering ways, then, the system they are a part of will not significantly change either. However, over the past two decades, the prevailing view among many funders, board members, and institutional leaders has been that only quantifiable and predetermined outcomes can create impact. But if the interrelated, devastating, and deepening crises and divisions over the past two years have taught us anything, it is that complex, adaptive problems defy tidy logic models and reductive technical solutions. It is time to invest our collective energy in more relational and emergent approaches to transforming systems.

Taking Stock of Empowerment Evaluation: An Empirical Review - Miller, Robin Lin; Campbell, Rebecca (American Journal of Evaluation, 2006)
Empowerment evaluation entered the evaluation lexicon in 1993. Since that time, it has attracted many adherents, as well as vocal detractors. A prominent issue in the debates on empowerment evaluation concerns the extent to which empowerment evaluation can be readily distinguished from other approaches to evaluation that share with it an emphasis on participatory and collaborative processes, capacity development, and evaluation use. A second issue concerns the extent to which empowerment evaluation actually leads to empowered outcomes for those who have participated in the evaluation process and those who are the intended beneficiaries of the social programs that were the objects of evaluation. The authors systematically examined 47 case examples of empowerment evaluation published from 1994 through June 2005. The results suggest wide variation among practitioners in adherence to empowerment evaluation principles and weak emphasis on the attainment of empowered outcomes for program beneficiaries. Implications for theory and practice are discussed.

A Pragmatic Approach to Assessing System Change: How to put it into practice - Miehlbradt, Alexandra; Shah, Rachel; Posthumus, Hans; Kessler, Adam (DCED, 2020)
Planning for and assessing system change is a strategic management issue. It is critical for everything from developing a strategy and designing interventions, to adapting strategy, improving implementation and reporting impact. But many programmes get stuck when it comes to assessing system change. The private sector development field has struggled to agree on an approach that programmes can implement and stakeholders can understand. However some mature programmes are starting to assess system change more effectively. Building on these emerging practices, this paper outlines a process that programmes can use to assess system changes regularly and practically. Two complementary papers: Overview and How to put it into practice The Overview summarises the approach and How to put it into practice provides more detailed implementation guidance, worked examples, and useful tips. The Overview explores how to: develop a system change strategy and intervention plans that lay the groundwork for system change assessment, including how to set system boundaries and how to identify the system changes a programme aims to catalyse assess system changes using both: - an intervention lens focused on changes introduced by specific interventions - a helicopter lens that provides a whole system view By analysing findings from both lenses, programmes can improve their strategy and report on their contribution to system change. How to put it into practice uses two case examples for illustration throughout the paper - PRISMA’s work in the maize system in East Java and Indonesia and S4J’s work in the Vocational Education and Training (VET) system in Albania. It targets practitioners responsible for facilitating and/or assessing system change. The paper explains how to: articulate the system changes that a programme aims to catalyse assess those changes use the results to inform decision making and reporting The approach described in the paper builds on the practices outlined in the DCED Results Measurement Standard. The guidance provided has been designed to be useful to programmes that aim to catalyse system changes whether or not they apply the DCED Standard.

A pragmatic approach to assessing system change - Webinar (DCED, 2020)
Few topics inspire as much confusion and debate as systemic change. What is it? How do you measure it? Does it even matter? Assessing changes in systems might be more doable than you think. This webinar explored a back-to-basics approach to assessing system change. In November 2019, thirty results measurement specialists, managers and consultants got together in Bangkok. They took part in workshops on a back-to-basics approach to assessing system changes, applying it to cases from participants’ programmes. Since then, the insights from the workshop have been further developed into a pragmatic approach to assessing system changes that builds on what programmes are actually doing and learning from practice. It can be: applied by programmes using a variety of different systemic change frameworks applied across a variety of sectors implemented with internal resources using familiar methods for information gathering The speakers walked through the approach using examples from the 2019 workshop, including PRISMA in Indonesia and Skills for Jobs (S4J) in Albania.

Building a Learning Culture – The case of the Market Development Facility in Fiji - Miehlbradt, Alexandra (The Donor Committee for Enterprise Development, 2015)
Keywords: Case Report

The behaviour change wheel: A new method for characterising and designing behaviour change interventions - Michie, Susan; van Stralen, Maartje M; West, Robert (Implementation Science : IS, 2011)
Background Improving the design and implementation of evidence-based practice depends on successful behaviour change interventions. This requires an appropriate method for characterising interventions and linking them to an analysis of the targeted behaviour. There exists a plethora of frameworks of behaviour change interventions, but it is not clear how well they serve this purpose. This paper evaluates these frameworks, and develops and evaluates a new framework aimed at overcoming their limitations. Methods A systematic search of electronic databases and consultation with behaviour change experts were used to identify frameworks of behaviour change interventions. These were evaluated according to three criteria: comprehensiveness, coherence, and a clear link to an overarching model of behaviour. A new framework was developed to meet these criteria. The reliability with which it could be applied was examined in two domains of behaviour change: tobacco control and obesity. Results Nineteen frameworks were identified covering nine intervention functions and seven policy categories that could enable those interventions. None of the frameworks reviewed covered the full range of intervention functions or policies, and only a minority met the criteria of coherence or linkage to a model of behaviour. At the centre of a proposed new framework is a 'behaviour system' involving three essential conditions: capability, opportunity, and motivation (what we term the 'COM-B system'). This forms the hub of a 'behaviour change wheel' (BCW) around which are positioned the nine intervention functions aimed at addressing deficits in one or more of these conditions; around this are placed seven categories of policy that could enable those interventions to occur. The BCW was used reliably to characterise interventions within the English Department of Health's 2010 tobacco control strategy and the National Institute of Health and Clinical Excellence's guidance on reducing obesity. Conclusions Interventions and policies to change behaviour can be usefully characterised by means of a BCW comprising: a 'behaviour system' at the hub, encircled by intervention functions and then by policy categories. Research is needed to establish how far the BCW can lead to more efficient design of effective interventions.

Design for Sustainability: Countering the Drivers of Unsustainability in Development Projects - Meyer, Isabel; Marais, Mario Alphonso (The Journal of Community Informatics, 2015)
Development projects aim to make a difference in communities. Funders often require that the intervention needs to last, or that it needs to effect change that will last, over an extended period of time. In short, a sustainable difference needs to be made and this is an important measure of success.  However, there are inherent drivers in the combined system of donors and beneficiaries that counter sustainability. This article explores these drivers, and identifies project design elements that need to be in place in order to promote sustainability by countering the drivers. This approach is applied to an education project, and implications for ICT4D projects are developed.
Keywords: Development; Project design; Systems Approaches; sustainability

Sustainability and Value through Improved Decision-making - Meyer, Isabel; Marais, Mario (Designing and implementing an Information and Communication Technology for Rural Education Development (ICT4RED) initiative in a resource-constrained environment: Nciba school district, Eastern Cape, South Africa, 2014)

Guidance on tasks and deliverables for different evaluation phases - Methods Lab (2015)
This tool describes the five key phases of evaluation, from planning and design, to implementation and communication of results. It provides a list of the main tasks and deliverables for each phase, intended for use by anyone managing an impact evaluation. This tool was developed by Irene Guijt, Simon Hearn, Tiina Pasanen and Patricia Rogers for use in Methods Lab projects. It follows to some extent the BetterEvaluation Rainbow Framework.

Guiding questions to help narrow the scope of an evaluation - Methods Lab (2015)
Time and budget constraints can mean that programmes are not able to assess all possible evaluation questions; this is especially true for multi-component or multi-site programmes operating in challenging environments. This tool identifies areas of enquiry to help programmes prioritise the number of questions and measurement indicators used. This tool was developed by Anne Buffardi for use in in Methods Lab projects.

Report template on integrating impact into an existing monitoring and evaluation system - Methods Lab (2015)
Many development programme staff will commission an impact evaluation towards the end of a project or programme, only to find that the monitoring system did not provide adequate data about implementation, context, baselines or interim results. This tool provides a template outline for a report making recommendations on how to integrate a focus on impact into a programme’s existing monitoring and evaluation system, as the programme moves into a new phase. This template was developed by Anne Buffardi and Tiina Pasanen for use in Methods Lab projects.

Report templates for evaluability assessment - Methods Lab (2015)
An evaluability assessment aims to assess the extent to which, and how best, a project can be evaluated in a reliable and credible fashion. These templates are intended to help anyone conducting an evaluability assessment to structure the final report. This tool was developed by Anne Buffardi and Bronwen McDonald for use in Methods Lab projects. It accompanies The Methods Lab publication ‘Evaluability assessment for impact evaluation: guidance, checklists and decision support’.

Sample agendas for an evaluability assessment stakeholder workshop - Methods Lab (2015)
An evaluability assessment aims to assess the extent to which, and how best, an intervention can be evaluated in a reliable and credible fashion. These sample agendas are intended for people convening key stakeholders (such as project implementation staff and managers, donors and government officials) to discuss the purpose and scope of an impact evaluation and to identify key evaluation questions. This tool was developed by Bronwen McDonald, Anne Buffardi and Irene Guijt for use in Methods Lab projects. It accompanies the Methods Lab publication ‘Evaluability assessment for impact evaluation: guidance, checklists and decision support’.

Sample criteria to select case studies for evaluation - Methods Lab (2018)
Time and budget constraints often mean that organisations are unable to evaluate all of their programmes, and large programmes, operating in multiple locations, are unable to evaluate all project sites. This tool introduces two sets of criteria to support evaluators and programme managers to select case studies or programmes for evaluation: i) information about how relevant or feasible evaluation is for individual programmes, and ii) across the overall portfolio, strategic thinking around what types of cases are most important to understand. This tool was developed by Anne Buffardi, Irene Guijt, Simon Hearn and Tiina Pasanen for use in The Methods Lab projects.

Sample interview questions for evaluability assessment - Methods Lab (2015)

Template concept note for an impact evaluation - Methods Lab (2015)

Agile Software Development: Adaptive Systems Principles and Best Practices - Meso, Peter; Jain, Radhika (Information Systems Management, 2006)
Today's environments of increasing business change require software development methodologies that are more adaptable. This article examines how complex adaptive systems (CAS) theory can be used to increase our understanding of how agile software development practices can be used to develop this capability. A mapping of agile practices to CAS principles and three dimensions (product, process, and people) results in several recommendations for “best practices” in systems development.
Keywords: IMPORTANT

Agile: A New Way of Governing - Mergel, Ines; Ganapati, Sukumar; Whitford, Andrew B. (Public Administration Review, 2021)
The evolving concept of “agile” has fundamentally changed core aspects of software design, project management, and business operations. The agile approach could also reshape government, public management, and governance in general. In this Viewpoint essay, the authors introduce the modern agile movement, reflect on how it can benefit public administrators, and describe several challenges that managers will face when they are expected to make their organizations more flexible and responsive.

Adaptive Management Self-assessment tool - mercy Corps (Mercy Corps, 2016)
The Adaptive Management self-assessment tool has been designed to help teams assess the extent to which they have a supportive environment for adaptive management within their country program. The self-assessment tool helps you think about five different areas that have been identified as important for supporting adaptive management: 1. Culture & leadership 2. Dynamic teams 3. Appropriate analysis 4. Responsive implementation & operations 5. Enabling environment (for example donor funding and relationships)

Adaptive Management Tools: Concept Notes System - Mercy Corps (Mercy Corps, 2015)
Mercy Corps finds adaptive management is supported by four different but interconnected elements: Culture, People & Skills, Tools & Systems, and Enabling Environment. These elements are discussed in the context of one of Mercy Corps’ complex programmes in pastoral Ethiopia.

AdaptScan - Improving your Team's Adaptive Management - Mercy Corps (Mercy Corps, 2020)

Agility and Evolution. How Mercy Corps Adapts at the Organizational Level - Mercy Corps (2017)
How Mercy Corps Adapts at the Organizational Level

How can we be more adaptive? AdaptScan identifies critical areas for improvement - Mercy Corps (Mercy Corps & IRC, 2017)
Keywords: IMPORTANT

Managing Complexity: Adaptive Management at Mercy Corps - Mercy Corps (2015)
Managing Complexity: Adaptive management at Mercy Corps

Webinar: Operationalizing Adaptive Management: What it takes. Why it matters. - Mercy Corps (Mercy Corps, 2015)
Development actors increasingly agree that managing programs adaptively – especially complex interventions – can improve their effectiveness. A growing body of evidence supports this claim. But what does adaptive management look like in practice? What does it require of managers and donors to make happen? How can we reconfigure incentives and success metrics to support adaptation, while remaining compliant?

Building effective research policy networks: linking function and form - Mendizabal, Enrique (ODI, 2006)
This paper looks at the factors that affect the key structural characteristics of research policy networks in an attempt to develop a methodology for studying and understanding what networks do and how. It builds on previous work on the roles and functions that networks carry out and focuses on their structural characteristics. The literature on networks is explored to identify the main and most common factors affecting their structure, and then the paper considers the possible effects of these structural factors on a network's ability to perform one or more of these functions based on some of the lessons learned from a study of seven research policy networks in Peru. The paper provides a menu of factors and characteristics worth considering when intending to build effective networks.

The Alignment, Interest and Influence Matrix (AIIM) guidance note - Mendizabal, Enrique (ODI/RAPID, 2010)
In 2007, on the eve of a workshop to introduce a new version of the RAPID approach to DFID policy teams, Enrique Mendizabal and Ben Ramalingam created the Alignment, Interest and Influence Matrix (AIIM), a stakeholder analysis tool that not only helps to identify key stakeholders, but also suggests a possible course of action towards them.

Understanding Networks: The functions of research policy networks - Mendizabal, Enrique (ODI, 2006)
We are constantly talking about networks. Banks use their networks to offer global services to customers; airlines fly passengers all over the world via their networks of partners; news agencies use media networks to keep us informed every minute of the day; and terrorist networks threaten citizens around the world. The importance of networks extends to the development sector: they organise civil society to advocate for and implement change; they link the local with the global, the private with the public; and they provide spaces for the creation, sharing and dissemination of knowledge. In a way, networks seem to make anything and everything happen. But we have yet to understand what they are and what they can and cannot do. In the development literature, a huge variety of policy and social network concepts and applications exists. This paper attempts to set out a framework to help clarify what research policy networks do.

The Haves and the Have Nots: Civic Technologies and the Pathways to Government Responsiveness - Mellon, Jonathan; Peixoto, Tiago C; Sjoberg, Fredrik M (Policy Research Working Paper, 2022)
“Civic tech” broadly refers to the use of digital technologies to support a range of citizen engagement processes. From allowing individuals to report problems to local government to enabling the crowdsourcing of national legislation, civic tech aims to promote better policies and services – while contributing to more inclusive democratic institutions. But could civic tech affect public issues in a way that benefits some and excludes others? Over the decades, the question of who participates in and who is excluded from participation mediated by technology has been the focus of both civic tech critics and proponents. The latter tend to argue that, by enabling citizens to participate without constraints of time and distance, civic tech facilitates the participation of those who usually abstain from engaging with public issues, leading to more inclusive processes. Critics argue that, given the existing digital divide, unequal access to technology will tend to empower the already empowered, further deepening societal differences. Yet both critics and proponents do tend to share an intuitive assumption: the socio-economic profile of who participates is the primary determinant of who benefits from digitally mediated civic participation. For instance, if more men participate, outcomes will favor male preferences, and if more young people participate, outcomes will be more aligned with the concerns of the youth. In a new paper, we show that the link between the demographics of those participating through digital channels, and the beneficiaries of the participation process, is not necessarily as straightforward as commonly assumed. We review four civic tech cases where data allow us to trace the full participatory chain through: the initial digital divide the participant’s demographics the demands made through the process the policy outcomes We examine online voting in the Brazilian state of Rio Grande do Sul’s participatory budgeting process, the local problem reporting platform Fix My Street (FMS) in the United Kingdom, Iceland’s online crowdsourced constitution process, and the global petitioning platform Change.org. Counterintuitive findings Change.org has been used by nearly half a billion people around the globe. Using a dataset of 3.9 million signers of online petitions in 132 countries, we examine the number of successful petitions and assess whether petitions created by women have more success than those submitted by men. Our analysis shows that, even if women create fewer online petitions than men, their petitions are more likely to be successful. All else equal, when online petitions have an impact on government policy, the agenda being implemented is much closer to the issues women choose to focus on. In Rio Grande do Sul’s digital participatory budgeting (PB), we show that despite important demographic differences between online and offline voters, these inequalities do not affect which types of projects are selected for funding – a consequence of PB’s unique institutional design, which favors redistributive effects. In fact, of all the cases analyzed, none reflect the standard assumption that inequalities in who participates translate directly into inequalities in who benefits from the policy outcomes. Our results suggest that the socio-economic profile of participants predicts only in part who benefits from civic tech. Just as important to policy outcomes is how the platform translates civic participation into policy demands, and how the government responds to those demands. While civic tech practitioners pay a lot of attention to design from a technological perspective, our findings highlight the importance of considering how civic tech platforms function as political institutions that encourage certain types of behavior while discouraging others. Civic tech, it seems, is not inherently good nor bad for democratic institutions. Instead, its effect is a combination of who participates on digital platforms and the choices of platform designers and governments.

The effect of government responsiveness on future political participation - Mellon, Andrew Jonathan; Sjoberg, Fredrik Matias; Peixoto, Tiago Carneiro (The World Bank, 2015)
What effect does government responsiveness have on political participation? Since the 1940s political scientists have used attitudinal measures of perceived efficacy to explain participation. More recent work has focused on underlying genetic factors that condition citizen engagement. The authors develop a ‘calculus of participation’ that incorporates objective efficacy, the extent to which an individual’s participation actually has an impact, and test the model against behavioral data from FixMyStreet.com (n=399,364). The authors find that a successful first experience using FixMyStreet.com (e.g., reporting a pothole and having it fixed) is associated with a 54 percent increase in the probability of an individual submitting a second report. The authors also show that the experience of government responsiveness to the first report submitted has predictive power over all future report submissions. The findings highlight the importance of government responsiveness for fostering an active citizenry, while demonstrating the value of incidentally collected data to examine participatory behavior at the individual level.

Thinking in Systems: A Primer - Meadows, Donella H. (Chelsea Green Publishing, 2008)
In the years following her role as the lead author of the international bestseller, Limits to Growth—the first book to show the consequences of unchecked growth on a finite planet— Donella Meadows remained a pioneer of environmental and social analysis until her untimely death in 2001.Thinking in Systems, is a concise and crucial book offering insight for problem solving on scales ranging from the personal to the global. Edited by the Sustainability Institute’s Diana Wright, this essential primer brings systems thinking out of the realm of computers and equations and into the tangible world, showing readers how to develop the systems-thinking skills that thought leaders across the globe consider critical for 21st-century life.Some of the biggest problems facing the world—war, hunger, poverty, and environmental degradation—are essentially system failures. They cannot be solved by fixing one piece in isolation from the others, because even seemingly minor details have enormous power to undermine the best efforts of too-narrow thinking.While readers will learn the conceptual tools and methods of systems thinking, the heart of the book is grander than methodology. Donella Meadows was known as much for nurturing positive outcomes as she was for delving into the science behind global dilemmas. She reminds readers to pay attention to what is important, not just what is quantifiable, to stay humble, and to stay a learner.In a world growing ever more complicated, crowded, and interdependent, Thinking in Systems helps readers avoid confusion and helplessness, the first step toward finding proactive and effective solutions.

Demanding Good Governance: Lessons from Social Accountability Initiatives in Africa - McNeil, Mary; Malena, Carmen (World Bank Publications, 2010)
Accountability is the cornerstone of good governance. Unless public officials can be held to account, then critical benefits associated with good governance, such as social justice, poverty reduction and development remain elusive. The impacts of non-responsive and unaccountable governance are perhaps most harshly felt by the citizens of Africa, where corruption and governance failures are broadly acknowledged as a principal obstacle to the achievement Over the past decade, a range of social accountability practices―such as participatory budgeting, independent budget analysis, participatory monitoring of public expenditure and citizen evaluation of public services have been experimented with in many Africa countries. Their outcomes and lessons have, thus far, received little attend and documentation. This volume aims to make a contribution towards filling this gap by describing and analyzing a selection of social accountability initiatives from seven Sub-Saharan countries.

Complexity, Organizations and Change - McMillan, Elizabeth (Routledge, 2004)
Complexity science has seriously challenged long-held views in the scientific community about how the world works. These ideas, particularly about the living world, also have radical and profound implications for organizations and society as a whole. Available in paperback for the first time, this insightful book describes and considers ideas from complexity science and examines their use in organizations, especially in bringing about major organizational change. Author McMillan explores how organizations, their design, the way they operate and, importantly, the people who co-create them, are thought of.Explaining the history and development of complexity science in an accessible way for the non-scientific reader, this outstanding book describes key concepts and their use in theory and practice. Illustrated with real-life examples from organizations in the UK, Europe and the USA, the book includes an in-depth case study of an organization which used complexity principles as part of a strategic change intervention. From this, useful models for introducing a complexity-based change process are derived.Complexity, Organizations and Change will appeal to academics, researchers and advanced students who are interested in complexity science and what it means for strategy, organization and management theory and organizational change.

Political economy analysis: Topic Guide - Mcloughlin, Claire (GSDRC, University of Birmingham, 2014)
Political economy analysis (PEA) aims to situate development interventions within an understanding of the prevailing political and economic processes in society – specifically, the incentives, relationships, and distribution and contestation of power between different groups and individuals. Such an analysis can support more politically feasible and therefore more effective development strategies by setting realistic expectations …

Scaling Impact: Innovation for the Public Good - McLean, Robert; Gargani, John (Routledge, 2019)
Scaling Impact introduces a new and practical approach to scaling the positive impacts of research and innovation. Inspired by leading scientific and entrepreneurial innovators from across Africa, Asia, the Caribbean, Latin America, and the Middle East, this book presents a synthesis of unrivalled diversity and grounded ingenuity. The result is a different perspective on how to achieve impact that matters, and an important challenge to the predominant more-is-better paradigm of scaling. For organisations and individuals working to change the world for the better, scaling impact is a common goal and a well-founded aim. The world is changing rapidly, and seemingly intractable problems like environmental degradation or accelerating inequality press us to do better for each other and our environment as a global community. Challenges like these appear to demand a significant scale of action, and here the authors argue that a more creative and critical approach to scaling is both possible and essential. To encourage uptake and co-development, the authors present actionable principles that can help organisations and innovators design, manage, and evaluate scaling strategies. Scaling Impact is essential reading for development and innovation practitioners and professionals, but also for researchers, students, evaluators, and policymakers with a desire to spark meaningful change.
Keywords: Developing & Emerging Countries

Adaptive Management: Learning and Action Approaches to Implementing Norms-shifting Interventions - McLarnon, Courtney; Gayles, Jennifer; Deepan, Prabu (USAID / Passages, 2021)
What Passages has Learned about Adaptive Management: • Be reflective about information that is collected and create a culture of learning. • Be systematic about establishing monitoring and learning systems. • Be strategic about data sources and analysis, prioritizing areas for learning and addressing issues raised. • Be inclusive about information collection: who is collecting what, how, and how is it being used.

Adaptability: the art of winning in an age of uncertainty - McKeown, Max (Kogan Page, 2012)
Keywords: Adaptability (Psychology); Success

Building a culture of learning at scale: learning networks for systems change. - McKenzie, Fiona (Orange Compass & Paul Ramsay Foundation, 2021)
This scoping paper explores the question ‘what would it take to build a culture of learning at scale?’. It focuses on systems-wide learning that can help to inform systems change efforts in complex contexts. To answer this question, literature was reviewed from across diverse disciplines and the realms of education, innovation systems, systems thinking and knowledge management. This inquiry was also supported by in-depth interviews with numerous specialists from the for-purpose sector and the examination of several case studies of learning across systems. The goal was to derive common patterns to inform a ‘learning for systems change’ framework. Learning for systems change is critical when working with complexity. The dynamic nature of complex adaptive systems requires an ability to continually sense and learn from the system and adapt accordingly. This is because the nature of the challenge and ‘what works’ to meet the challenge is continually shifting (Lowe and Plimmer, 2019). This requires an ongoing process of iterative inquiry that draws upon wisdom and insights from across the system. Such learning challenges traditional siloes of expertise and organisational boundaries (Clarke et al., 2019). Learning is not simply a nice to have. It is critical for greater impact and improved outcomes, particularly in mission-driven organisations and initiatives (Price et al., 2019). In this paper, a ‘learning networks’ approach is proposed, one that draws upon individual, group and systems-wide learning to build capacity and resilience for systems change in uncertain environments. This fills a gap in the literature where the focus is largely on learning within organisations. Instead, the focus here is on what is required to support learning to occur across scales and boundaries - from the individual to system-wide. A simple meta-framework for developing learning networks is proposed that includes high level guidance on the enabling conditions - the mindsets, relationships, processes and structures- that would enable learning networks to flourish.

Evaluation Building Blocks: A Guide - McKegg, K.; Oakden, J.; Wehipeihana, N.; King, Julian (Kinnect Group, 2018)
This guide presents the Kinnect Group’s approach to evaluation. Developed through ten years of collaborative work, it has been well tested. While there are many other guides to evaluation and different ways to do it, our approach emphasises the place of evaluative reasoning in evaluation. The guide draws on the work of many evaluation theorists, as well as our practice-based body of knowledge. We hope that using the building blocks as explained in the guide will help you in your endeavour to do credible and useful evaluation. We find this approach works for us. Our clients tell us it gives them meaningful and insightful findings that they can use to take action. Our view to intellectual property is simple. What you read is yours to use. We just ask that you cite the guide when you draw from it. For those of you who engage with this guide and use it in your practice, we look forward to hearing from you. We are particularly interested in feedback where you find aspects of the guide work well for you. In that way we can all continue to build and evolve our practice.

Lessons from a trial of the Success Case Method - McGuinness, Liz (Better Evaluation, 2017)
The Success Case Method approach is useful for documenting stories of impact and for understanding the factors that help or hinder impact. It is particularly useful for uncovering the contextual forces that influence impact. Originally designed for evaluating corporate training programs, the Success Case Method is now being applied to other programs including international development interventions. Last year, I provided technical assistance to a pilot of the Success Case Method as part of the USAID-funded Complexity-Aware M&E Trials.