Development Policy and Impact Evaluation: Learning and Accountability in Private Sector Development - Ton, Giel (Handbook of Development Policy, 2021)
There is broad recognition of the challenges in evaluating policy and programmes on their contribution to sustainable development. Impact evaluations of PSD programmes are carried out at the behest of a particular configuration of interest groups with different expectations. Some groups want to know whether a programme has worked, others want to know how to do these programmes better, others fear that PSD programmes might result in sub-optimal or adverse development outcomes in recipient countries, and some want to be sure that the programme benefits the private sector in the donors' domestic economies. This chapter discusses these with reference to private sector development programmes and explores how contribution analysis, which is a structured stepwise process of theory-based evaluation, can address these challenges and generate findings to improve learning and accountability.
Accepted: 2021-11-05T12:43:15Z Journal Abbreviation: Chapter 33: Development policy and impact evaluation: learning and accountability in private sector development

Learning through and about Contribution Analysis for impact evaluation - Apgar, Marina; Ton, Giel (Institute of Development Studies, 2021)
Over the past five years, colleagues from the Centre for Development Impact (CDI) – a joint initiative between the Institute of Development Studies, Itad and University of East Anglia – have been innovating with and learning how to use Contribution Analysis as an overarching approach to impact evaluation. In this blog series, we share our learning and insights, some of them in raw emergent form, highlight the complexities, nuances, excitements, and challenges of embracing new ways of doing impact evaluation. We begin this series by sharing ideas about ‘causal hotspots’ that first surfaced as ‘aha’ moments in our collaboration on the CDI short course Contribution Analysis for Impact Evaluation – the moments when we found the ways and words to better articulate our ideas and help people navigate the messy realities of theory-based evaluation.

Finding and using causal hotspots: a practice in the making - Apgar, Marina; Snijder, Mieke (Institute of Development Studies, 2021)
This is the second blog in our series ‘Lessons on using Contribution Analysis for impact evaluation’. In our first blog, we introduced Contribution Analysis (CA) as an overarching approach to theory-based evaluation and the idea of causal hotpots as a way to zoom in, unpack and make the hard choices of where to focus evaluation research. Identifying specific links in the theory of change (ToC) with specific evaluation questions enables you to then choose appropriate methods. We have applied the approach in diverse settings, testing how robust Contribution Analysis can really be. Here we walk you through an example.

Exploring a new governance agenda: What are the questions that matter? - Nixon, Nicola; Verhulst, Stefaan; Matin, Imran; Vermonte, Philips J. (From Poverty to Power, 2021)
The 100 Questions Initiative, pioneered by the GovLab, seeks to overcome the chasm between supply and demand. It begins not by searching for what data is available, but by asking important questions about the biggest challenges societies and countries face, and then seeking more targeted and relevant data solutions. In doing this, it narrows the gap between policy makers and constituents, providing opportunities for improved evidence-based policy and community engagement in developing countries. As part of this initiative, we seek to define the ten most important questions across several domains, including Migration, Gender, Employment, the Future of Work, and—now–Governance. On this occasion, we invited over 100 experts and practitioners in governance and data science –whom we call “bilinguals”– from various organizations, companies, and government agencies to identify what they see as the most pressing governance questions in their respective domains. Over 100 bilinguals were encouraged to prioritize potential impact, novelty, and feasibility in their questioning — moving toward a roadmap for data-driven action and collaboration that is both actionable and ambitious.

Lessons for PFM Reform - the PDIA Approach in Africa - Bento, Joana (Public Financial Management Blog - IMF, 2021)
Some important lessons are as follows: First, identifying a policy problem is critical for reforms to get traction both at a political and administrative level. Second, appropriate solutions can emerge from a process of experimentation, iteration, and adaptation. Third, building teams and institutional capabilities is a critical part of solving complex problems in a sustainable way.

Are we there yet? Localisation as the journey towards locally led practice: models, approaches and challenges - Baguios, Arbie; King, Maia; Martins, Alex; Pinnington, Rose (ODI, 2021)
Localisation and locally led international development practice has long been discussed, but has still not been delivered. Systemic barriers have posed challenges, and the term itself is contested. Now, the last tumultuous 18 months could provide a critical juncture to finally move forward with this crucial agenda. The pandemic has highlighted structural inequalities in the global system, and disrupted ways of working in the international development sector. The Black Lives Matter movement has brought conversations about racism and colonialism to the fore. And the climate crisis has highlighted the need for global action on humanity’s challenges that remain rooted in local realities. The emerging analysis in this review aims to set out the key issues in this agenda, building on a wealth of existing knowledge. It aims to span sectors, highlighting many new and existing models and approaches in the humanitarian, development, philanthropic and private sectors. It reviews the barriers and challenges to localisation and locally led practice, with a view to informing a campaign for systemic change to move forward with this agenda. The review is based on: a rapid review of the literature and evidence on localisation and locally led practice; two consultations with over 100 total participants, targeted at Global South actors; and analysis of 28 existing models and approaches.

Building a culture of learning at scale: learning networks for systems change. - McKenzie, Fiona (Orange Compass & Paul Ramsay Foundation, 2021)
This scoping paper explores the question ‘what would it take to build a culture of learning at scale?’. It focuses on systems-wide learning that can help to inform systems change efforts in complex contexts. To answer this question, literature was reviewed from across diverse disciplines and the realms of education, innovation systems, systems thinking and knowledge management. This inquiry was also supported by in-depth interviews with numerous specialists from the for-purpose sector and the examination of several case studies of learning across systems. The goal was to derive common patterns to inform a ‘learning for systems change’ framework. Learning for systems change is critical when working with complexity. The dynamic nature of complex adaptive systems requires an ability to continually sense and learn from the system and adapt accordingly. This is because the nature of the challenge and ‘what works’ to meet the challenge is continually shifting (Lowe and Plimmer, 2019). This requires an ongoing process of iterative inquiry that draws upon wisdom and insights from across the system. Such learning challenges traditional siloes of expertise and organisational boundaries (Clarke et al., 2019). Learning is not simply a nice to have. It is critical for greater impact and improved outcomes, particularly in mission-driven organisations and initiatives (Price et al., 2019). In this paper, a ‘learning networks’ approach is proposed, one that draws upon individual, group and systems-wide learning to build capacity and resilience for systems change in uncertain environments. This fills a gap in the literature where the focus is largely on learning within organisations. Instead, the focus here is on what is required to support learning to occur across scales and boundaries - from the individual to system-wide. A simple meta-framework for developing learning networks is proposed that includes high level guidance on the enabling conditions - the mindsets, relationships, processes and structures- that would enable learning networks to flourish.

Lessons Learned from Applying the Data Powered Positive Deviance - Pawelke, Andreas; Glücker, Andreas; Albanna, Basma; Boy, Jeremy (DPPD, 2021)
This report presents six learnings from four pilot projects conducted by the Data Powered Positive Deviance (DPPD) initiative, a global collaboration between the GIZ Data Lab, the UNDP Accelerator Labs Network, the University of Manchester Center for Digital Development, and UN Global Pulse Lab Jakarta. The pilots seek out grassroots solutions to development challenges that range from the interaction between livestock farming and deforestation to gender-based violence and insecurity in dense urban environments in Ecuador, Mexico, Niger and Somalia. The learnings relate to the early stages of the DPPD method, originally proposed by Albanna & Heeks [1], and focus mainly on the access to, and use of digital data. They are summarized as follows: 1. Remain flexible in the face of data unavailability 2. Leverage existing partnerships for data access 3. Map and fill know-how gaps early 4. Scale with caution 5. Look at deviance over time 6. Look beyond individual or community practices and behavior The report is written for development practitioners, data analysts, domain experts, and more generally anyone interested in using new data sources and technologies to uncover successful local solutions to development challenges.

Twenty years of UK governance programmes in Nigeria - Piron, Laure-Hélène; Cummings, Clare; Williams, Gareth; Derbyshire, Helen; Hadley, Sierd (ODI, 2021)
This Flagship report analyses 20 years of governance programmes in Nigeria funded by the UK Department for International Development (DFID) and the Foreign, Commonwealth and Development Office (FCDO) in the North-western states of Jigawa (since 2001), Kano (since 2005) and Kaduna (since 2006), as well as the North-eastern state of Yobe (since 2011). The report’s main research question is whether, how, under what conditions and for whom UK-funded state-level governance programmes in Nigeria have contributed to sustained changes in governance, and related changes in health and education. ... The report concludes with the following recommendations: To international development partners: 1. Invest for the long term – 10 to 20 years – combining support for both state and nonstate actors. 2. Ensure programmes have the strategic-level mandate, managerial capacity and frontline staff skills to pursue politically savvy opportunities. 3. Take PEA to the next level by unpacking causal mechanisms, understanding incentives and designing interventions to make change happen. 4. Give governance programmes the ability to flex between core governance and service delivery issues. 5. Incentivise greater collaboration between governance and sector programmes. 6. Incentivise greater attention to gender, and to social inclusion beyond disability issues, in governance programming. To FCDO: 7. Empower and resource FCDO teams to enable TWP programmes, ensuring decision-making by country teams to respond to local priorities. 8. Re-imagine TWP for FCDO Nigeria, giving implementers the space to operate in TWP ways. 9. Incentivise stronger collaboration between PERL, Lafiya (health programme) and the Partnership for Learning for All in Nigerian Education. 10. Invest in impact data analysis. To partner governments in Nigeria and beyond: 11. Explicitly set out the objectives for which you would like to receive assistance. 12. Use TWP principles to decide how development partners can support your political objectives and the scope for politically-feasible and mutually-beneficial collaboration. 13. Invest in the coordination of development partners. To non-state partners in Nigeria and beyond: 14. Join coalitions to achieve your priorities. 15. Select development partners which can strengthen your skills, not just fund your activities.

The Systems Work of Social Change: How to Harness Connection, Context, and Power to Cultivate Deep and Enduring Change - Rayner, Cynthia; Bonnici, François (OUP Oxford, 2021)
The issues of poverty, inequality, racial justice, and climate change have never been more pressing or paralyzing. Current approaches to social change, which rely on industrial models of production and power to "solve" social problems, are not helping. In fact, they are designed to entrench the status quo. In The Systems Work of Social Change, Cynthia Rayner and François Bonnici draw on two hundred years of history and a treasure trove of stories of committed social changemakers to uncover principles and practices for social change that radically depart from these approaches. Rather than delivering "solutions," these principles and practices focus on the process of change itself. Through rich storytelling and lucid analysis, Rayner and Bonnici show that connection, context, and power sit at the heart of the change process, ensuring broader agency for people and communities to create social systems that are responsive and representative in a rapidly changing world. Simple yet profound, this book distills a timely set of lessons for practitioners, leaders, scholars, and policymakers.

DPPD Handbook. A step-by-step guide for development practitioners to apply the Data Powered Positive Deviance method - DPPD (DPPD Initiative, 2021)
The Method Positive Deviance (PD) is based on the observation that in every community or organization, there are a few individuals who achieve significantly better outcomes than their peers, despite having similar challenges and resources. These individuals are referred to as positive deviants, and adopting their solutions is what is referred to as the PD approach¹. The method described in this Handbook follows the same logic as the PD approach but uses pre-existing, non-traditional data sources instead of — or in conjunction with — traditional data sources. Non-traditional data in this context broadly refers to data that is digitally captured (e.g. mobile phone records and financial data), mediated (e.g. social media and online data), or observed (e.g. satellite imagery). The integration of such data to complement traditional data sources generally used in PD is what we refer to as Data Powered Positive Deviance² (DPPD). The digital data opportunity Recent developments in the availability of digital data provide an opportunity to look for positive deviants³ in new ways and in unprecedented geographical and on temporal scales. A number of studies⁴ have described the challenges related to the application of the PD approach in development. Given these challenges, there are obvious opportunities for innovation in PD and our particular interest here is in the innovative opportunities offered by non-traditional data, following the increasing “datafication” of development and the growing availability of big datasets in a variety of development sectors⁵. DPPD builds on this and expands our ability to extract value from non-traditional digital data while providing a systematic process for leveraging local know-how and the collective wisdom of communities. Data Powered Positive Deviance The DPPD method described in this Handbook emerged from a process of research and testing and follows the same stages as the PD approach. The difference is that DPPD integrates pre-existing, non-traditional data across the five stages, requiring a series of new and specific methods and practices that are not required in the PD approach. The first stage is also somewhat different because it not only defines the problem, but it also checks if it is suitable and feasible to use the DPPD method for the proposed project. Table 1 lists the five stages of the DPPD method. This Handbook dedicates a section to each stage. Stage 1 Assess problem-method fit Stage 2 Determine positive deviants Stage 3 Discover underlying factors Stage 4 Design and implement interventions Stage 5 Monitor and evaluate

System Innovation on Purpose - Leadbeater, Charles; Winhall, Jennie (The Rockwool Foundation, 2021)
In Building Better Systems, we introduced four keys to unlock system innovation: purpose and power, relationships and resource flows. These four keys make up a set. Systems are often hard to change because power, relationships, and resource flows are locked together in a reinforcing pattern to serve the system’s current purpose. Systems start to change fundamentally when this pattern is disrupted and opened up. Then a new configuration can emerge, serving a new purpose. In this essay series we delve deeper into these four keys and provide practical advice on how they can be put to use. This essay is about the role that purpose plays in orchestrating complex systems and how system innovators can create a new system around a new sense of purpose.

Technical assistance: a practical account of the challenges in design and implementation - Nastase, Alexandra; Rajan, Alok; French, Ben; Bhattacharya, Debarshi (Gates Open Research, 2021)
Technical assistance is provided to country governments as part of international development programmes to support policymaking or strengthen state capability. This article presents the conceptual evolution of ‘technical assistance’ linked to capacity development, starting with programmes aiming exclusively to enhance individual capacity in the 1950s to 1970s and progressing to complex systems approaches in the past ten years. It also presents some of the frequent challenges in designing and implementing technical assistance, drawing from the existing literature and the authors’ experience in international development. The article summarises the latest thinking about delivering more effective development, including the adaptive management practices and the initiatives to strengthen evidence about what works. Finally, we complement this article with a follow-up open letter reflecting on the current policy options and opportunities for change.

Getting Real about Unknowns in Complex Policy Work - Andrews, Matt (Research on Improving Systems of Education (RISE), 2021)
As with all public policy work, education policies are demanding. Policy workers need to ‘know’ a lot—about the problems they are addressing, the people who need to be engaged, the promises they can make in response, the context they are working in, and the processes they will follow to implement. Most policy workers answer questions about such issues within the structures of plan and control processes used to devise budgets and projects. These structures limit their knowledge gathering, organization and sense-making activities to up-front planning activities, and even though sophisticated tools like Theories of Change suggest planners ‘know’ all that is needed for policy success, they often do not. Policies are often fraught with ‘unknowns’ that cannot be captured in passive planning processes and thus repeatedly undermine even the best laid plans. Through a novel strategy that asks how much one knows about the answers to 25 essential policy questions, and an application to recent education policy interventions in Mozambique, this paper shows that it is possible to get real about unknowns in policy work. Just recognizing these unknowns exist—and understanding why they do and what kind of challenge they pose to policy workers—can help promote a more modest and realistic approach to doing complex policy work.

Understanding Governance from the Margins: What Does It Mean In Practice? - Barnes, Katrina, Katrina, Barnes; Anderson, Colin; de Chassy, Stephanie; Ahmed, Affaf; Ali, Mudabbir; Aung, Myo Min; Chaimite, Egidio; Joshi, Anuradha; Khan, Danyal; Loureiro, Miguel; Posse, Lucio; Rowlands, Jo; et al. (Institute of Development Studies (IDS), 2021)
What does governance look like ‘from below’ – from the perspectives of poor and marginalised households? How do patterns of conflict affect that? These were the questions at the heart of the Governance at the Margins research project. Over three years from 2017-2020 we worked to explore this through in-depth study in conflict-affected areas of Mozambique, Myanmar, and Pakistan. Our research teams interviewed the same people regularly over that time, finding out how they resolved problems and interacted with authorities. In this paper we connect what we found to the realities and complexities of development practice, drawing on the input of 20 experienced practitioners working in bilateral and multilateral development agencies and international NGOs, who generously gave their time to help us think through the practical implications of our wealth of findings.

Evaluating complex interventions in international development - Masset, Edoardo; Shrestha, S.; Juden, M. (CEDIL, 2021)
This paper reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. Complex interventions are those that are characterised by multiple components, multiple stakeholders, or multiple target populations. They may also be interventions that incorporate multiple processes of behavioural change. While such interventions are very common and receive a large proportion of development aid budgets, they are rarely subject to rigorous evaluations. The CEDIL Methods Working Paper, ‘Evaluating Complex Interventions in International Development’, reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. The paper focuses its attention on methods that address causality and allow us to state conclusively whether an intervention works or not. It shows that several rigorous methods developed in different disciplines can be adapted and used to evaluate complex interventions in international development.
Section: Publications

Thinking and working politically: What have we learned since 2013 - Teskey, Graham (TWP Community of Practice, 2021)
The Thinking and Working Politically (TWP) Community of Practice (CoP) was established at a small meeting tacked on at the end of a meeting of Governance Advisers working for the United Kingdom’s Department of International Development (DFID) on South and South-East Asian countries, held in Delhi in November 2013. Since then, a number of meetings have been held throughout the world, each addressing different issues; ‘TWP’ has entered the lexicon of mainstream development; the CoP has expanded to more than 300 people; a Washington DC chapter has been established; and the International CoP has been granted modest funding from DFID’s successor, the Foreign, Commonwealth and Development Office (FCDO) and from Australia’s Department of Foreign Affairs and Trade (DFAT). It is legitimate to ask, however, what has been achieved operationally: how have the ideas underpinning TWP affected operational practice? This short paper traces the evolution of the idea and practice of TWP from 2013 to late 2021, and identifies what we have learned. What has been successful, and what has not? I asked in 2017 whether TWP had become a second orthodoxy (Teskey, 2017). Did this represent hubris or was 2017 in some ways the apogee of what might rather grandly be called the TWP ‘movement’?

Inclusive participatory approaches: A facilitator’s guide - Zaremba, Haley; Elias, Marlène; Devi, J. Tulasi; Priyadarshini, Pratiti (2021)
In community engagement and participatory processes, facilitators must make intentional efforts and adopt inclusive strategies to include marginalized and frequently overlooked groups. Yet, there is a lack of guidance on how to inclusively facilitate participatory processes. Facilitators are therefore often poorly prepared to engage with the power relations that underlie these processes, including those between the facilitator and participants and among participants themselves. This guide addresses this shortcoming by presenting strategies that have been shown to enhance the meaningful participation of women and marginalized groups in participatory processes. The aim is to equip facilitators with tools to create inclusive participatory spaces and community engagement. Although it refers to processes related to natural resource management, the guide is also applicable to the facilitation of participatory processes focusing on other issues or fields of significance to communities.
Accepted: 2022-01-12T09:19:12Z

Social Accountability and Service Delivery Effectiveness: What is the Evidence for the Role of Sanctions? Background Paper - Aston, Tom; Zimmer Santos, Grazielli (2022)
Executive Summary Understanding how civil society can get government to respond to their needs, preferences and demands, and deliver goods and services to citizens is a central concern in social accountability initiatives. It is widely argued that sanctions make a key difference to service delivery outcomes, and that without them, transparency and accountability interventions are less likely to be effective and less likely to be sustainable (Anderson et al., 2020; Arugay, 2016; Fox, 2020; Goetz and Jenkins, 2005; Grandvoinnet et al., 2015; McGee and Gaventa, 2011; Molina et al., 2017; Joshi, 2010; Joshi, 2017; Tsai et al. 2019). In this paper, sanctions refer to the threat or imposition of a punishment for transgressing a rule or norm. Yet, what evidence is there to support the claim that sanctions are king? How much do we actually know about social and formal sanctions and their effectiveness in improving service delivery? Looking at 11 meta-reviews and 35 cases, this background paper sheds light on these questions and the conditions under which sanctions promoted within social accountability interventions may have contributed to improved service delivery. Sanctions, both social and formal, feature very prominently in the scholarly literature related to accountability, so much so that some scholars have argued that it has become synonymous with punishment (Mansbridge, 2014; Schedler, 1999). While it may not be true that these concepts are fully synonymous, this trend in scholarship significantly influenced the thinking of donors such as the Department for International Development (DFID) and the World Bank over the last two decades (Grandvoinnet et al., 2015; Malena and McNeil, 2010; Moore and Teskey, 2006; World Bank, 2003). As a result, sanctions also feature in the dominant models, or theories of change, in the social accountability sector. Yet, in reviewing 11 meta-reviews in the transparency and accountability sector we found that while there is plenty of theoretical argument asserting the potential effectiveness of social and formal sanctions, there was limited empirical evidence to support the claim that sanctions were key. Our review of 35 cases revealed five mechanisms of change related to social and formal sanctions. These were: (i) “sticks” – response to punishment; (ii) “big brother is watching” – response to threat of formal punishment; (iii) diagonal accountability – response to threat of formal punishment by horizontal accountability agencies; (iv) litigation – response to legal investigations or lawsuits supported by community paralegals and legal aid organizations and; (v) response to “naming and shaming” by civil society and/or media. We challenge the dominant view in scholarship that harder social sanctions and enforcement of stronger formal sanctions are either necessary to the achievement of higher-level service delivery outcomes or that they will deliver better and more sustainable outcomes. We find that both social and formal sanctions can contribute to improving service delivery outcomes across a variety of country contexts. In half of the 35 cases reviewed, we were able to establish a likely link between social or formal sanctions and intermediate effects over the short term with some degree of confidence. These effects ranged from increased service provider awareness and motivation, increased availability of funding, staff, and materials, to improved infrastructure quality, and in a minority of cases impact level changes such as improved test scores. However, the role of sanctions in delivering outcomes was often unclear, outcomes were almost never sustainable, and in close to half of the 35 cases reviewed there were substantial negative effects. These effects ranged from reducing transparency and funding, to discrediting, relocating, and reprisals for advocates and whistle-blowers, threats of violence to collaborating government actors, damaging staff morale, reducing attendance, and generating conflict among staff and between staff and community members and between staff and patients, damaging trust. There are therefore some serious ethical dilemmas associated with sanctioning efforts which need to be carefully considered. We argue that imposing sanctions without building relationships or systems to promote good behavior is unlikely to improve service delivery outcomes in a sustainable way. Another, perhaps surprising, finding was that there are some actors that are regular targets of sanctions, and in many cases, these actors are a lot weaker than is commonly assumed. Three broad types of actors were the most common targets of punishment: (i) absentee nurses or teachers who had their pay or allowances reduced; (ii) offending officials who were either suspended, relocated, or fired; and (iii) contractors who had to cover the cost of rejected materials or faced lawsuits, alongside civil servants involved in contracting. We suggest that closer relationships may perhaps reduce stakeholders’ appetite to impose sanctions. Particularly in the health sector, we found that more proximate relationships created disincentives for confrontation, and in such circumstances, a “policing” approach to monitoring was also deemed inappropriate and counterproductive. Conversely, it seems that actors generally prefer to sanction “others,” i.e., when an actor/organization was outside the group. Short-term consultants, contractors, and suppliers were easy (and quite vulnerable) targets for harder sanctions. It has also been argued that there may be productive combinations of collaborative and confrontational tactics — i.e., hybrids. We found that many supposed confrontational and collaborative hybrids were, in fact, dislocated in time and space. Many so-called “inside-outside” strategies, therefore, seem to be a potential mischaracterization. We also found, as Fung and Wright (2003) argued nearly two decades ago, that adversarial forms of engagement cannot easily be redeployed for collaborative purposes. It is widely argued that supposedly ‘weaker’ forms of citizen engagement are less effective than those with ‘strong enforceability (McGee and Gaventa, 2011).’ We found no compelling evidence to support this contention. A quarter of cases reviewed were collaborative for certain periods or in certain locations. On average, these were slightly more successful when compared with periods or locations when imposing sanctions was a key strategic emphasis. So, soft power can also be powerful. Rather than one approach necessarily being superior (confrontational, collaborative, or hybrid) however, we argue that the best approach is likely to be the one most appropriate to the context. While we were unable to identify strong trends of contextual factors which enabled social and formal sanctions to play a role in enhancing service delivery, we were able to identify several conditions which we believe offer the greatest promise. These conditions were: (i) supportive leaders who played a role as champions; opening doors or accompanying civil society efforts; (ii) high capacity and legal authority of oversight agencies; (iii) competitive elections, which provided windows of opportunity for CSOs to combine political and social accountability efforts, and; (iv) vulnerable public servants and service providers already in relatively precarious situations and are thus easy targets. Overall, our study finds that sometimes sanctions can be effective, but punishment is not the answer to all the world’s problems. Given these limitations, we recommend that it is time to reconsider “carrots” and enquire further into the enabling conditions for bureaucrats and service providers. Relatedly, scholars, evaluators, and program teams should look more closely at service providers’ or civil servants’ motivations and take context into account more seriously. To uncover these contextual and motivational features, we believe that scholars, evaluators, and prprogrameams should also make better use of theory-based and participatory methods. And perhaps most importantly, donors and practitioners should carefully consider and mitigate the potential for backlash from sanctions-based approaches.

Guidance Note: Practical introduction to adaptive management - DT Global (DT Global, 2022)
DT Global is proud to introduce our new Guidance Note: Practical Introduction to Adaptive Management There is a growing consensus around adaptive management as an effective (even necessary) approach when programs are tackling complex development problems. While there is no standard definition of adaptive management, there is general agreement that such programs need to routinely engage with and respond to program context; constantly test what works in that context; and adjust approaches, plans, and activities based on continuous learning. However, there remains a more limited body of evidence about what this looks like in practice—the enabling conditions, systems, resourcing, skills, and attitudes to effectively operationalise adaptive management. There is also limited guidance around when adaptive management is required, and to what extent—both critical and often overlooked considerations when planning for successful adaptive management. This Guidance Note draws together lessons and good practice in adaptive management from across DT Global’s diverse portfolio of donor-funded programs. It outlines our conceptual framework for adaptive management, with practical guidance on how it can be applied by our program teams. It is also designed to help our teams distinguish adaptive management from good (non adaptive) project management, consider when adaptive management is most useful on a program, and how adaptive a program (or part of a program) should be.

Evaluating programming that thinks and works politically: Challenges and emerging practice - Jacobstein, David; Swift, Sarah (New Directions for Evaluation, 2022)
Issues of power are not new to program evaluation. What is new is a consideration of how programming uses insights into incentives that shape and adapt implementation. How should one evaluate in a way that explicitly assesses the ways in which a program considers power? One of the innovative topics deriving from the democracy and governance space is the approach of thinking and working politically (TWP) which is seeing increased use in development programming. TWP suggests different mental models and practical approaches to achieving development objectives in ways that are more contextually grounded and informed by power dynamics. This article describes several of the core challenges to evaluation of TWP and also a rubric of considerations for more effective evaluation practices in this emerging field.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20527

Critical friends - real time insights for shaping strategy - Johnson, Debbie; Nathan, Geeta; Rahman, Syahirah Abdul (Chapters, 2022)
This chapter focuses on facilitation given by the Innovation Caucus to provide expert academic critique to inform Innovate UK's strategy for UK business innovation. As Innovate UK's strategy developed, it became evident that several policy domains needed critical insights and evidence. An academic critical friend provided the latest academic insights, evidence and wider perspective of the actors in the UK Innovation system, insights of which policy makers often lack. The chapter gave case studies of Innovation Caucus' work with Innovate UK in facilitating extensive research expertise and helping policy makers to open dialogue. It also describes 'The Innovators' Breakfast Club', a webinar series organised by Innovation Caucus, co-developed with Innovate UK, which created live face-to-face dialogue between academic experts and Innovate UK strategy/policy leads. This example showed how critical friends are important in facilitating the initial dialogues needed to build evidence for policy making. The dialogue builds the foundation for a virtuous cycle of learning. The approach provides an opportunity for academic experts to gain insights into the priorities and dilemmas facing policy makers, to shape future academic research and refine the skill set needed to become critical friends.
Publisher: Edward Elgar Publishing

Theory amidst complexity – using process tracing in ex-post evaluations - Krueger, Kate; Wright, Molly (New Directions for Evaluation, 2022)
Evaluators who take a complexity-aware approach must consider tradeoffs related to theoretical parsimony, falsifiability, and measurement validity. These tradeoffs may be particularly pronounced with ex-post evaluation designs in which program theory development and monitoring frameworks are often completed before the evaluator is engaged. In this chapter, we argue that theory-based evaluation (TBE) approaches can address unique ex-post evaluation challenges that complexity-aware evaluation (CAE) alone cannot, and that these two sets of approaches are complimentary. We will outline strategies that evaluators may use to conduct rigorous ex-post evaluations of democracy, human rights, and governance (DRG) interventions that merge CAE's inductive approaches with a theory-testing structure. It will illustrate these strategies with two case studies of ex-post evaluation using process tracing (PT).
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20524

Human Learning Systems: A practical guide for the curious - Lowe, Toby; Padmanabhan, Chandrima; McCart, Des; McNeill, Karen; Brogan, Andy; Smith, Mark (Centre for Public Impact, 2022)
Our new guide provides practical advice to help any organisation working in public service apply the Human Learning Systems approach to their work. In doing so, they will be better equipped to explore, learn and respond to the unique strengths and needs of each person, family and community they serve.

Context-monitoring for adaptive management - Pickwick, Sarah (World Vision, 2022)

Learning to change: The case for systemic learning strategies in the humanitarian sector - Ramalingam, Ben; Mitchell, John (ALNAP/ODI, 2022)
This paper presents the case for systemic organisational change in the humanitarian system. The paper firstly shows that that organisational learning has tended to reinforce existing ways of working and has not been able to shift a culture that values action over reflection. As a result, the rest of the paper asks about the most significant changes in the humanitarian sector

Learning from Life Story Collection and Analysis with Children who Work in the Leather Sector in Bangladesh - Sayem, Mashrique; Paul, Sukanta; Apgar, Marina; Snijder, Mieke (2022)
CLARISSA (Child Labour: Action-Research-Innovation in South and South-eastern Asia) is a participatory evidence and innovation generating programme. We are generating evidence on the drivers of the worst forms of child labour (WFCL) and exploring how to address them through participatory Action Research (PAR) with children and other stakeholders in the leather supply chain in Dhaka, Bangladesh. Our main intervention modality is Systemic Action Research (SAR) (Burns, 2007), of which life story collection and analysis (LSC&A) is the first step in our participatory design that will inform child-led PAR groups which will become the engines of innovative responses to WFCL. The LSC&A methodology is a storytelling and story listening methodology and was chosen because of the universal power of stories to make sense of complex realities and seek new futures. People across the world like to tell their stories; they like to feel listened to and they are interested in how their story connects and compares to others. By collecting and analysing stories from hundreds of children in WFCL we can visualise the bigger system that each individual story is connected to. We hypothesize that through engagement in the process of telling, listening, collecting, and analysing life stories, children engaged in harmful and hazardous work will use their understanding of systems dynamics to move into creating their own solutions to the drivers of WFCL. In 2021 in Bangladesh, we collected 405 life stories from children living in Hazaribagh, Hemayetpur, Lalbagh, and Bhairab in Dhaka, with more than one hundred of these stories collected by children themselves. Following the story collection and transcription, children were supported by the CLARISSA implementation team to collectively analyse the stories through identifying critical ‘factors’ (events which have causes and consequences) and understanding how they causally relate to each other. The analysis of the 405 life stories resulted in the development of large system maps that illustrate all the causal dynamics that underpin lived experiences of WFCL. Based on the systemic analysis process the children identified themes of the PAR to be set up in their localities. Our experience with the LSC&A methodology is the first in the context of children in WFCL. The methodology has been used in one other project in Bangladesh to date. We therefore intentionally aimed to learn from the implementation process and to evaluate if and how the telling of, listening to, collecting and analysing of life stories is empowering and in turn whether it leads to increased ownership of the problems which motivates collective action (Burns, 2021). In this learning note we share our methodological learning and reflect on operational implications for designing and facilitating an LSC&A process with children which we hope will support adaptation and use of the methodology by others working in participatory programming with children.
Accepted: 2022-07-21T08:43:41Z Publisher: Institute of Development Studies

Lives Amid Violence: Transforming Development in the Wake of Conflict - Schomerus, Mareike (2022)
Violent conflict and its aftermath are pressing problems, particularly for international development initiatives. However, the results of development in conflict contexts have generally been disappointing and their preventative potential thus questionable. Available Open Access, Lives Amid Violence argues that this is because practitioners adhere to a mental model that emphasises linearity, certainty, and causality, assuming that violence is best addressed through work plans that deliver state-building, stabilisation and services. Based on ten years of multi-method research from, in, and on conflict-affected countries, this book challenges this approach.Drawing on a significant collaborative body of scholarship, this work puts forward original and generalizable conclusions about how lives amid violence persist, offering an invitation to abandon restricting mental models and to embrace creative ways of thinking and working. These include paying attention to the long-term effects of conflict on individual behaviour and decision-making, the social realities of economic life, the role service delivery plays in negotiations between citizens and states, and to creating meaningful relationships. Transformation also requires reflection and therefore the book concludes with constructive suggestions on how to practice these insights to better support those whose lives are shaped by violence.More details are available at www.transformingdevelopment.orgThe eBook editions of this book are available open access under a CC BY-NC-ND 3.0 licence on bloomsburycollections.com.

Nimble adaptation: Tailoring monitoring, evaluation, and learning methods to provide actionable data in complex environments - Serpe, Lauren; Ingram, Mason; Byom, Kate (New Directions for Evaluation, 2022)
This chapter examines good practices in implementing effective Monitoring, Evaluation, and Learning (MEL) systems within complex international development Democracy, Human Rights, and Governance (DRG) programs, which are characterized by challenges of non-linearity, limited evidence of theories of change, and contextual and politically contingent nature of outcomes. The chapter presents three cases of MEL systems in complex projects implemented by Pact across distinct and diverse operating contexts – Zimbabwe, Cambodia, and Somalia – to illustrate those projects’ MEL approaches that enabled continuous adaptation. The authors analyze the cases to respond to two questions: (1) What are the key elements of effective adaptive management-focused MEL systems in complex environments? (2) What is practical guidance for designing and enabling complexity-responsive and effective adaptive management-focused MEL systems? The case studies illustrate three key elements: (1) Information gathering that closely links context, research, and performance data; (2) Systems for reflection that offer scheduled learning moments of varying frequency and intensity, as well as multiple feedback mechanisms; and (3) Enabling structures that promote adaptive mindsets and attitudes within project teams.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20523

Be a participant not a spectator - Sharp, Cathy (The Evaluator - UK Evaluation Society, 2022)
My focus here is on the realities of evaluating in complexity where ‘nothing is clear, and everything keeps changing’. I outline how I use a series of ‘provocations’ that allow people to choose their own starting point. Sharing those choices fuels conversations that discover, explore, and co-create (rather than manage) our mutual expectations and assumptions and track how these might themselves be influenced by the work as it unfolds. This account draws on a review of literature and my practice experience, including reflections from others brought into local, national, and international conversations about what it means for evaluation to recognise complexity.

Storytelling for Systems Change: insights from the field - Thea Snow; David Murikumthara; Teya Dusseldorp; Rachel Fyfe; Lila Wolff; Jane McCracken (Centre for Public Impact, 2022)
Capturing the impact of community-led work The Centre for Public Impact, Dusseldorp Forum, and Hands Up Mallee have been exploring how stories can be used to more effectively communicate the impact of community-led systems change work. Community-led place based initiatives are modelling new ways of working - shifting away from top down, program-focussed approaches towards an approach grounded in systems thinking and community-led innovations. However, while these stories of change are sitting in communities, they’re often not being told or celebrated. We wanted to understand why this is, and what might be done to better enable these stories to be shared and heard. The story of storytelling We talked to a range of people to uncover the story of storytelling - including collective impact backbone team members, community members, storytelling experts, and those working in and around community-led systems change initiatives across Australia. We explored the roles stories play in different communities; what good storytelling looks like; what barriers to storytelling might be; and what role stories can play in supporting systems change. Our findings We have learned through this project that stories can be used both to change the system and to evaluate, understand and showcase the change that is occurring in communities. We have heard that different stories require different approaches – stories that are seeking to enable change look different to those that are seeking to celebrate change.

What is the Outcomes Star? - Triangle (Triangle, 2022)
The Outcomes Stars are a suite of evidence-based outcomes measurement and keywork tools, which drive an ‘enabling help’ approach to service delivery. They support a person-centred, collaborative and trauma informed approach and give service users, workers, managers and commissioners vital information about needs and progress. Since the first version was published in 2006, the unique features of the Outcomes Star contributed to its popularity and widespread use, with over 1,000 organisations with licenses to use the Star including over 500 charities and 170 local authorities

An introductory systems thinking toolkit for civil servants - UK Government (UK Government Office for Science, 2022)
This document is a an Introductory Toolkit for for civil servants. It is one component of a suite of documents that aims to act as a springboard into systems thinking for civil servants unfamiliar with this approach. These documents introduce a small sample of systems thinking concepts and tools, chosen due to their accessibility and alignment to civil service policy development, but which is by no means comprehensive. They are intended to act as a first step towards using systems thinking approaches to solve complex problems and the reader is encourage to explore the wider systems thinking field further.

The UNDP Accelerator Labs enter a year of maturity: let a thousand flowers bloom! - Annual report 2022 - UNDP accelerator labs (UNDP, 2022)
The UNDP Accelerator Labs were designed as an agile and dynamic Network to allow communication and information transfer between 91 Accelerator Labs in 115 countries, and with the global innovation ecosystem, UNDP as a whole, and thousands of partners including grassroots innovators and their communities. The accumulated knowledge of this Network creates new pathways to the solutions that hold the key to sustainable development problems. What it looked like and how it unfolded last year will be illuminated in this annual report: The UNDP Accelerator Labs enter a year of maturity: let a thousand flowers bloom!

CLA Maturity Tool - Card Deck (Implementing Partners version 1) - USAID LEARN (USAID, 2022)
USAID’s Bureau for Policy, Planning and Learning (PPL) and its support mechanism, LEARN, have developed a Collaborating, Learning, and Adapting (CLA) Framework and Maturity tool to help USAID missions think more deliberately about how to plan for and implement CLA approaches that fit the mission’s context and assist them in achieving their development objectives. While the tool is intended primarily for USAID audiences to be used in participatory self-assessment workshops, the CLA Framework and maturity spectrum are relevant to a wider audience. USAID’s CLA Framework identifies key components and subcomponents of daily work that may be opportunities for intentional, systematic, and resourced CLA. The framework recognizes the diversity of what CLA can look like in various organizations and programs while also giving CLA structure, clarity, and coherence across two key dimensions: • CLA in the Program Cycle: how CLA is incorporated throughout Program Cycle processes, including strategy, project, and activity design and implementation; and • Enabling Conditions: how an organization’s culture, business processes, and resource allocation support CLA integration. Recognizing that CLA is not binary—it’s not an issue of “doing it or not doing it”—PPL and LEARN have developed a spectrum of practice for each of the 16 subcomponents in the framework. The spectrum offers examples of what integration might look like at different stages: Not Yet Present, Emergent, Expanding, Advanced and Institutionalized. The maturity stage descriptions are only illustrative and are intended to spark reflection on current practice and opportunities for improvements. In this resource, each CLA subcomponent page describes the key concepts for that topic and includes a description of the maturity stages. Although the descriptions were originally developed for USAID, the majority of the concepts easily transfer or have equivalents in the partner community. For example, although organizations outside of USAID may not hold “Portfolio Reviews” (part of the Pause & Reflect subcomponent), the majority hold some type of meeting to review programmatic progress. This is the seventh version of the CLA Framework and maturity spectrum. PPL and LEARN will continue reviewed and periodically update them based on user feedback, so if you have comments about the content, please email learning@usaid.gov. We would also love to hear how you’ve used this content with your team or organization.

Collaborating, Learning and Adapting (CLA) Maturity Spectrum (v7) - USAID LEARN (USAID, 2022)
USAID’s Bureau for Policy, Planning and Learning (PPL) and its support mechanism, LEARN, have developed a Collaborating, Learning, and Adapting (CLA) Framework and Maturity tool to help USAID missions think more deliberately about how to plan for and implement CLA approaches that fit the mission’s context and assist them in achieving their development objectives. While the tool is intended primarily for USAID audiences to be used in participatory self-assessment workshops, the CLA Framework and maturity spectrum are relevant to a wider audience. USAID’s CLA Framework identifies key components and subcomponents of daily work that may be opportunities for intentional, systematic, and resourced CLA. The framework recognizes the diversity of what CLA can look like in various organizations and programs while also giving CLA structure, clarity, and coherence across two key dimensions: • CLA in the Program Cycle: how CLA is incorporated throughout Program Cycle processes, including strategy, project, and activity design and implementation; and • Enabling Conditions: how an organization’s culture, business processes, and resource allocation support CLA integration. Recognizing that CLA is not binary—it’s not an issue of “doing it or not doing it”—PPL and LEARN have developed a spectrum of practice for each of the 16 subcomponents in the framework. The spectrum offers examples of what integration might look like at different stages: Not Yet Present, Emergent, Expanding, Advanced and Institutionalized. The maturity stage descriptions are only illustrative and are intended to spark reflection on current practice and opportunities for improvements. In this resource, each CLA subcomponent page describes the key concepts for that topic and includes a description of the maturity stages. Although the descriptions were originally developed for USAID, the majority of the concepts easily transfer or have equivalents in the partner community. For example, although organizations outside of USAID may not hold “Portfolio Reviews” (part of the Pause & Reflect subcomponent), the majority hold some type of meeting to review programmatic progress. This is the seventh version of the CLA Framework and maturity spectrum. PPL and LEARN will continue reviewed and periodically update them based on user feedback, so if you have comments about the content, please email learning@usaid.gov. We would also love to hear how you’ve used this content with your team or organization.

Collaborating, Learning, and Adapting Framework & Key Concepts (Implementing Partner Version 1) - USAID LEARN (USAID, 2022)
Although collaborating, learning, and adapting (CLA) are not new to USAID and its implementing partners, they often do not happen regularly or systematically and are not intentionally resourced. The CLA Framework above identifies components and subcomponents to help you think more deliberately about what approach to CLA might be best tailored to your organizational or project context. The framework recognizes the diversity of what CLA can look like in various organizations and projects while also giving CLA structure, clarity, and coherence across two key dimensions: - CLA in the Program Cycle: how CLA is incorporated into planning and design processes throughout the Program Cycle in order to improve their effectiveness; and - Enabling Conditions: how an organization’s culture, daily operating processes, and resource allocation support CLA integration. Organizations need both integrated CLA practices appropriate for their context and conducive enabling conditions to become stronger learning organizations capable of managing adaptively. The framework stresses the holistic and integrated nature of the various components of CLA to reinforce the principle that CLA is not a separate workstream—it should be integrated into existing processes to strengthen the discipline of development and improve aid effectiveness.

Civil society organizations and managerialism: On the depoliticization of the adaptive management agenda - Gutheil, Lena; Koch, Dirk-Jan (Development Policy Review, 2022)
Motivation In the last decade, a movement formed around making aid delivery more adaptive, relying on principles such as context sensitivity, flexibility, and ownership. The approaches seem promising for civil society organizations (CSOs) to fulfil their mission of fostering social transformation. While several donor agencies have started engaging with such approaches, the authors hardly see their political implications in practice. Purpose The article aims to provide evidence on an adaptive project and demonstrate how the social transformative and political nature of adaptive development management is rendered technical and depoliticized in practice. Methods and approach We use a case study of a development programme based on a social transformative policy framework that is implemented through CSOs in Uganda and Vietnam. Data were collected by means of interviews, participant observation and document analysis. Findings We find that, in practice, the social transformative policy framework is competing with managerial logics. We compare this process with the depoliticization of the Paris Declaration on Aid Effectiveness, finding striking similarities. By using practice theory, we show how managerialism remains the dominant paradigm in the civil society aid sector, fuelling the “anti-politics machine.” Policy implications The article shows that policy frameworks do not always work as intended. Donors should therefore not only change policy frameworks, but also start addressing institutional and operational requirements.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/dpr.12630

Market systems change rubric - Loveridge, Donna (Oxford Policy Management, 2022)
This systems change rubric describes different performance levels according to various systems elements, such as policy (formal rules), practices and relationships and connections. Programmes can use the rubric to assess the performance of systems to help decide where and how to intervene, or during and post-implementation to conduct progress assessments, and assess the effectiveness of interventions and type, breadth and depth of systems change. Each performance level description highlights the type of data and information that needs to be collected. One analysis is completed, users can compare this to the performance descriptions to see which level best matches the analysis. This helps programmes draw conclusions about systems changes. The rubric was developed in 2020 and tested in 2021 and builds on systems change thinking and frameworks from two previous FSG publications. It can be used as: pre-intervention to conduct an assessment during an intervention to conduct progress assessments and reflect on the effectiveness of interventions to change systems and inform decision making post intervention to make judgements about whether interventions were valuable given the resources, time and effort spent Useful for: Implementation managers to determine the effectiveness of interventions, as well as by Monitoring, Evaluation and Learning (MEL) teams to track progress against expected outcomes.

The Relational Work of Systems Change - Millgan, Katherine; Zerda, Juanita; Kania, John (Stanford Social Innovation Review, 2022)
Collective impact efforts must prioritize working together in more relational ways to find systemic solutions to social problems. Sometimes we lose sight of a simple truth about systems: They are made up of people. Despite all of the frameworks and tools at our disposal and all of our learning as a field of practice, purely technical, rational approaches to systems change will not make much of a dent in shifting power or altering our most deeply held beliefs. If most collective impact efforts fall short of supporting people to change in fundamentally consciousness-altering ways, then, the system they are a part of will not significantly change either. However, over the past two decades, the prevailing view among many funders, board members, and institutional leaders has been that only quantifiable and predetermined outcomes can create impact. But if the interrelated, devastating, and deepening crises and divisions over the past two years have taught us anything, it is that complex, adaptive problems defy tidy logic models and reductive technical solutions. It is time to invest our collective energy in more relational and emergent approaches to transforming systems.

MOTION HANDBOOK - Developing a transformative Theory of Change - Palavicino, Carla Alvial; Matti, Cristian; Witte, Jenny (TIPC, Utrecht University, 2022)
A step-by-step guide on how to develop a Transformative Theory of Change, for innovation projects, programmes and organisations working on systems transformation. The MOTION project was initiated with one key question in mind: how can we help projects and organisations be more transformative, using the framework and concept provided by the multi-level perspective? And what kind of tools, methods and frameworks can we co-design that translate scientific concepts into practises relevant for policy practitioners? This led us into a co-creation journey during which researchers from the Transformative Innovation Policy Consortium (TIPC) and EIT Climate-KIC project partners experimented, reflected and learned from each other in building the approach that we share in this handbook. Through this journey, we gained a deeper understanding of what the portfolio approach means in a transformative system change context and which skills and competences are needed to facilitate processes of co-creation in the science-policy-practice interface. We had the opportunity to configure the key building blocks of our theoretical approach, the Transformative Outcomes, into practical insights and actions that can easily be applied by innovation organisations at many levels. This handbook is the culmination of the journey as it translates key learnings from the MOTION project into practical insights that are relevant to practitioners working on systems transformation.

Sensemaking Workshop Preparation Guide and Facilitator Guide and Sensemaking Training - UNDP (UNDP, 2022)
Based on experience from running Sensemaking workshops for UNDP offices and government partners, the Asia-Pacific Regional Innovation Centre developed the Sensemaking Preparation Guide and Facilitator Guide to share its knowledge with teams and organization that are interested in using the Sensemaking process.

Data-powered positive deviance: Combining traditional and non-traditional data to identify and characterise development-related outperformers - Albanna, Basma; Heeks, Richard; Pawelke, Andreas; Boy, Jeremy; Handl, Julia; Gluecker, Andreas (Development Engineering, 2022)
The positive deviance approach in international development scales practices and strategies of positively-deviant individuals and groups: those who are able to achieve significantly better development outcomes than their peers despite having similar resources and challenges. This approach relies mainly on traditional data sources (e.g. surveys and interviews) for identifying those positive deviants and for discovering their successful solutions. The growing availability of non-traditional digital data (e.g. from remote sensing and mobile phones) relating to individuals, communities and spaces enables data innovation opportunities for positive deviance. Such datasets can identify deviance at geographic and temporal scales that were not possible before. But guidance is needed on how this new data can be employed in the positive deviance approach, and how it can be combined with more traditional data to gain deeper, more meaningful, and context-aware insights. This paper presents such guidance through a data-powered method that combines both traditional and non-traditional data to identify and understand positive deviance in new ways and domains. This method has been developed iteratively through six development projects covering five different domains – sustainable cattle ranching, agricultural productivity, rangeland management, research performance, crime control – with global and local development partners in six countries. The projects combine different types of non-traditional data with official statistics, administrative data and interviews. Here, we describe a structured method for data-powered positive deviance developed from the experience of these projects, and we reflect on lessons learned. We hope to encourage and guide greater use of this new method; enabling development practitioners to make more effective use of the non-traditional digital datasets that are increasingly available.

Monitoring and evaluation for thinking and working politically - Aston, Thomas; Roche, Chris; Schaaf, Marta; Cant, Sue (Evaluation, 2022)
This article explores the challenges of monitoring and evaluating politically informed and adaptive programmes in the international development field. We assess the strengths and weaknesses of some specific evaluation methodologies which have been suggested as particularly appropriate for these kinds of programmes based on scholarly literature and the practical experience of the authors in using them. We suggest that those methods which assume generative causality are particularly well suited to the task. We also conclude that factoring in the politics of uncertainty and evidence generation and use is particularly important in order to recognize and value diverse experiential knowledge, integrate understandings of the local context, accommodate adaptation and realistically grapple with the power relations which are inherent in evaluation processes.
Publisher: SAGE Publications Ltd

PFM Reform Through PDIA: What Works and When it Works - Harris, Jamelia; Lawson, Andrew (Public Financial Management Blog - IMF, 2022)

The Difference Learning Makes - Factors that enable and inhibit adaptive programming - Gray, Stephen; Carl, Andy (Christian Aid, 2022)
Executive Summary When Christian Aid (CA) Ireland devised its multi-country and multi-year Irish Aid funded Programme Grant II (2017-2022), they opted to move away from a linear programme management approach and to explore an adaptive one. Across seven countries: Angola, Colombia, El Salvador, Guatemala, Israel and the occupied Palestinian territory, Sierra Leone, and Zimbabwe, CA and partner organisations support marginalised communities to realise their rights, reduce violence and address gender inequality. Since 2019, Adapt Peacebuilding has accompanied CA Ireland, CA country teams and partner organisations as they experimented with using a deliberate adaptive approach. The authors were also asked to follow up on an initial study by CA Ireland and Overseas Development Institute in 2018, which described the rationale for adopting this new approach and included early lessons from its first year of implementation. The aim of this study is to help deepen CA Ireland, CA country teams’ and partners’ understanding of (a) whether their application of adaptive programming has resulted in better development outcomes, and (b) how they can better understand the factors that enabled or inhibited the effectiveness of using this approach. Over the past three years, this study has found evidence and multiple examples that show adaptive programming contributed to better development outcomes. The main reasons cited were that these were made possible both from improvements to programming strategies based on proactive reflection and learning, as well as those that stem from the reactive capacity of adaptive programmes to change course in response to unanticipated changes in operating conditions. This study found that adaptive programming has enabled better development practice where organisations are enhancing their skills to better respond and be flexible to contextual challenges. 72% of partners surveyed described adaptive programming as the most useful approach to programme management that they have used. The programme approach has meant that CA and partner staff were better able to explore the significance of change in the context and their contributions to them. It also enabled spaces for meaningful engagement with communities in learning and programme planning processes and encouraged opportunities for experimentation in programming. The study also found that adaptive programming has supported flexible delivery. This led to better outcomes that would not have been possible were the programme not able to make flexible adjustments. The main focus has been the analysis of nine factors that can determine the effectiveness and impact (or otherwise) of using an adaptive approach, flagging important issues for understanding. These factors are identified as: 1) Leadership; 2) Organisational culture; 3) Conceptual understanding; 4) Staff capacities; 5) Partnership approaches; 6) Participation; 7) Methods and tools; 8) Administrative procedures; and 9) The operating context. Together these can provide an analytical framework for assessing an organisation’s ‘adaptive scope’, which can be used as a tool for better understanding an organisation’s potential to generate improved development outcomes via adaptive programming and how to strengthen them. The study concludes with several recommendations for CA Ireland, all of which have relevance for a broader community of donors and implementing organisations interested in the potential of adaptive programming.

Guidance Note for Adaptive Management - Ministry of Foreign Affairs of Denmark (Government of Denmark, 2022)
This note explains what adaptive aid management is; why and when it should be considered; and how it should be applied. It covers all Danish development support channels and modalities, including bilateral country assistance, assistance to and through civil society, the private sector and to and through multilateral organisations. This guide has three chapters. Chapter 1 provides an executive overview of what adaptive management is. Chapter 2 goes deeper into five key operational principles of adaptive management. Chapter 3 details the main tenets of adaptive management processes during the programming cycle.

The Patterns of Possibility - How to Recast Relationships to Create Healthier Systems and Better Outcomes - Winhall, Jennie; Leadbeater, Charles (The Rockwool Foundation, 2022)
In Building Better Systems, we introduced four keys to unlock system innovation: purpose and power, relationships and resource flows. These four keys make up a set. Systems are often hard to change because power, relationships, and resource flows are locked together in a reinforcing pattern to serve the system’s current purpose. Systems start to change fundamentally when this pattern is disrupted and opened up so that a new configuration can emerge, serving a new purpose. In this article series we delve deeper into these four keys and provide practical advice on how they can be put to use. This article is about relationships. Systems are defined by the patterns of interactions between their parts: their relationships. Those interactions generate the outcomes of the system as a whole. Transforming the outcomes of a system requires remaking its relationships and then unlocking the other keys to system innovation: purpose, power and resources. This shift in relationships allows all those in the system to learn faster, to be more creative. System innovators redesign the relationships in the system to allow dramatically enhanced learning across the system, and thereby generate far better outcomes.

Thinking and working politically: What have we learned since 2013? – FP2P - Green, Duncan (From Poverty to Power, 2022)
It’s always a red letter day when a new paper from Graham Teskey drops. His most recent is Thinking and working politically: What have we learned since 2013? For those that don’t know him, Graham is a consummate insider-outsider within the aid sector – long stints at DFID (UK), DFAT (Australia) and now Abt (Management Consultants). From this vantage point he has been one of the leading proponents of ‘thinking and working politically’, always ready to call out the hand-wavey academics and demand some practical lessons, please. This paper is part biography of an idea, setting out the timeline, moments and key documents and policy wins in the evolution of TWP (which seems to have involved a lot of seminars that I missed due to Oxfam’s meagre travel budget). The other part is, to be honest, a bit of a lament – a study in ‘Why Change Hasn’t Happened’, because TWP has ‘got lost in the maelstrom’ of the wider, largely negative, changes in the aid sector. Overall, it’s a brilliant summary, and one I’ll be recommending to my increasingly long-suffering activism students….

Systems Thinking and Practice: A Guide to Concepts, Principles and Tools for FCDO and Partners - Woodhill, Jim; Millican, Juliet (Institute of Development Studies, 2022)
This guide is a basic reference on systems thinking and practice tailored to the context and needs of the UK Government’s Foreign, Commonwealth & Development Office (FCDO). It is an output of the FCDO Knowledge for Development Programme (K4D), which facilitated a Learning Journey on Systems Thinking and Practice with FCDO staff during 2021 and 2022. The guide offers a common language and shared framing of systems thinking for FCDO and its partners. It explores what this implies for working practices, business processes and leadership. It also offers links to additional resources and tools on systems thinking. We hope it can support systems thinking to become more commonplace within the culture and practices of FCDO and working relations with partner organisations.
Accepted: 2023-02-03T15:47:58Z Publisher: Institute of Development Studies

Theory of Change Workbook: A Step-by-Step Process for Developing or Strengthening Theories of Change - Salib, Monalisa (USAID, 2022)
While over time theories of change have become synonymous with simple if/then statements, a strong theory of change should actually be a much more detailed, context-specific articulation of how we *theorize* change will happen under a program.

Evaluating complexity, simplistically - Aston, Thomas (Medium, 2022)
A critical appraisal's of CEDIL papers on Evaluating Complex Interventions... A study was recently published by the Centre of Excellence for Development Impact and Learning (CEDIL) entitled Evaluating complex interventions in international development. This is the sort of title that raises great expectations. Complexity is a hugely popular theme and many of us are keen to know more about how to evaluate efforts that seek to achieve results amid complexity. In April 2021, CEDIL conducted a webinar on the paper, and in July 2021 CEDIL published a blog. In September 2021, I wrote a blog expressing some reservations regarding the focus of the study; its apparent over-emphasis on interventions, under-emphasis on context, as well as its choice of some supposedly under-used methods. These methods were: (1) factorial designs; (2) adaptive trials; (3) Qualitative Comparative Analysis (QCA); (4) synthetic control; (5) agent-based modelling, and system dynamics.

Lessons Learned From the Use of the Most Significant Change Technique for Adaptive Management of Complex Health Interventions - Ohkubo, Saori; Mwaikambo, Lisa; Salem, Ruwaida M.; Ajijola, Lekan; Nyachae, Paul; Sharma, Mukesh Kumar (Global Health: Science and Practice, 2022)
Introduction:The Most Significant Change (MSC) technique is a complex-aware monitoring and evaluation tool, widely recognized for various adaptive management purposes. The documentation of practical examples using the MSC technique for an ongoing monitoring purpose is limited. We aim to fill the current gap by documenting and sharing the experience and lessons learned of The Challenge Initiative (TCI), which is scaling up evidence-based family planning (FP) and adolescent and youth sexual and reproductive health (AYSRH) interventions in 11 countries in Asia and sub-Saharan Africa. Methods:The qualitative assessment took place in early 2021 to document TCI’s use and adaptation of MSC and determine its added value in adaptive management, routine monitoring, and cross-learning efforts. Focus group discussions and key informant interviews were conducted virtually with staff members involved in collecting and selecting MSC stories. Results:TCI has had a positive experience with using MSC to facilitate adaptive management in multiple countries. The use of MSC has created learning opportunities that have helped diffuse evidence-based FP and AYSRH interventions both within and across countries. The responsive feedback step in the MSC process was viewed as indispensable to learning and collaboration. There are several necessary inputs to successful use of the method, including buy-in about the benefits, training on good interviewing techniques and qualitative research, and dedicated staff to manage the process. Conclusion:Our assessment results suggest that the MSC technique is an effective qualitative data collection tool to strengthen routine monitoring and adaptive management efforts that allows for flexibility in how project stakeholders implement the process. The MSC technique could be an important tool for global health practitioners, policy makers, and researchers working on complex interventions because they continually need to understand stakeholders’ needs and priorities, learn from lessons and evidence-based practices, and be agile about addressing potential challenges.
Publisher: Global Health: Science and Practice Section: ORIGINAL ARTICLE

Key Issues in Digitalisation and Governance - Roberts, Tony; Hernandez, Kevin; Faith, Becky; Prieto Martin, Pedro (SDC, 2022)
Digitalisation is perhaps the most important strategic challenge that governance will face over the coming decade. The process is delivering digital dividends as well as new exclusions and injustices, with the rapid but uneven increase in access to mobile and internet technologies transforming how social and economic life takes place. This report highlights the key opportunities and challenges arising from digitalisation.
Accepted: 2022-07-25T11:31:42Z Publisher: SDC Governance Network

What, so what, now what? - Aston, Thomas (Medium, 2022)
Getting serious about systems change

System Change: A Guidebook for Adopting Portfolio Approaches - Wellsch, Brent (UNDP, 2022)
This guidebook codifies the principles and methods of applying systems change and portfolio approaches to complex development challenges with practical tools and examples. It is based on the empirical learning generated from the collaborative initiatives in UNDP Country Offices in Bhutan, Pakistan, the Philippines, and Viet Nam with support from Regional Innovation Centre for Asia and the Pacific.

Second (and Third) Thoughts on Adaptive Management and Thinking and Working Politically - Green, Duncan (From Poverty to Power, 2022)
Self-Critical reflections on AM and TWP. Linking it with The Hype Cycle - "it feels like we are heading downward to the ‘trough of disillusionment’ form the initial peak of ‘inflated expectations’, but we will bounce back to something more sustained, that becomes a permanent feature of the aid landscape".

The collaborative exploration of alternative futures: A different approach to Theories of Change - CEDIL programme (2022)
ParEvo is a web application that enables the collaborative construction and exploration of a range of alternative futures: likely and unlikely, desirable and undesirable. These are described in the form of a branching narrative structure, developed over a series of iterations involving the interactions of a group of participants. These detailed storylines about the future contrast with optimistic, skeletal and largely singular views of the future found in diagrammatic ToCs often encountered by evaluators. This webinar will describe a recent ParEvo exercise implemented by the Centre for the Study of Existential Risk (CSER) in Cambridge. In this exercise 11 international experts explored ideas about how global risks associated with biotechnology research could be managed, and mismanaged, in the coming four years. Including how these alternative futures were evaluated by participants and the CSER facilitators. Issues to be discussed by the panel, and others, include comparisons with other representations of Theories of Change and other approaches to the exploration of alternative futures, described variously as scenario planning, futures or foresight work. Speakers: Rick Davies, Lara Mani Tom Hobson An overview of the app can be found here: https://mscinnovations.wordpress.com/ The app website is here: https://parevo.org/

Innovating for inclusive rigour in peacebuilding evaluation - Apgar, Marina; Báez-Silva, Ángela Maria; Deng, Ayak Chol; Fairey, Tiffany; Rohrbach, Livia; Alamoussa, Dioma; Bradburn, Helene; Cubillos, Edwin; Gray, Stephen; Wingender, Leslie (Institute of Development Studies, 2022)
Inclusive and rigorous peacebuilding evaluation is both vital and complex. In this blog we share examples of how we are innovating our methodologies to move towards participatory and adaptive practice.

Capacity Development in a Participatory Adaptive Programme: the Case of the Clarissa Consortium - Widmer, Mireille; Apgar, Marina; Afroze, Jiniya; Malla, Sudhir; Healey, Jill; Constant, Sendrine (IDS, 2022)
Doing development differently rests on deliberate efforts to reflect and learn, not just about what programmes are doing and achieving, but about how they are working. This is particularly important for an action research programme like Child Labour: Action- Research-Innovation in South and South-Eastern Asia (CLARISSA), which is implemented by a consortium of organisations from across the research and development spectrum, during a rapidly changing global pandemic. Harnessing the potential of diverse skills and complementary strengths across partners in responding to the complex challenge of the worst forms of child labour, requires capacity to work together in novel ways. This Research and Evidence Paper documents how CLARISSA approached capacity development, and what we learnt from our challenges and successes. From the start, the programme incorporated a capacity development strategy resting on self-assessment of a wide range of behavioural and technical competencies that were deemed important for programme implementation, formal training activities, and periodic review of progress through an after-action review (AAR) process. An inventory of capacity development activities that took place during the first year of implementation reveals a wide range of additional, unplanned activities, enabled by the programme’s flexibility and adaptive management strategy. These are organised into eight modalities, according to the individual or collective nature of the activity, and its sequencing – namely, whether capacity development happens prior to, during, or after (from) implementation. We conclude with some reflections on the emergent nature of capacity development. Planning capacity development in an adaptive programme provides a scaffolding in terms of time, resources, and legitimacy that sustains adaptiveness. We also recognise the gaps that remain to be addressed, particularly on scaling up individual learning to collective capabilities, and widening the focus from implementation teams to individuals working at consortium level.
Accepted: 2022-04-25T10:25:54Z Publisher: Institute of Development Studies

Locally driven development: Overcoming the obstacles - Ingram, George (Brookings Institution, 2022)
Locally led development is a complex process that the development community, in the U.S. and around the world, has spent several decades trying to get right. Yet, despite all the experience and lessons learned, it feels like we are barely beyond the starting line. This publication aims to contribute to the ongoing dialogue on locally led development, especially as to how the United States can address the obstacles posed by U.S. law, regulation, policy, and practice. It consists of two parts: • An essay by George Ingram that notes the path that has taken us to this point, identifies key obstacles, and invites a discussion of how to overcome impediments and move forward. • A set of 15 commentaries written by development experts that add a range of perspectives and nuances to the discussion

Measuring and Monitoring Adaptive Learning: A Landscape Review - LaFond, Anne; Adrian, Haley (USAID MOMENTUM Knowledge Accelerator, 2022)
This landscape review on measuring and monitoring adaptive learning highlights the learning from five adaptive programming guidelines and toolkits and one implementation science framework to inform the monitoring and evaluation of adaptive learning. The introduction of adaptive learning processes and skillsets in global health programming is part of an emerging strategy to advance a learning culture within projects and teams to improve health program performance. The monitoring and evaluation of adaptive learning is an emergent field aiming to monitor how adaptive learning processes have been introduced, how they are used, and whether they are having the intended results. Although there is a growing body of literature on adaptive programming more generally, there is a limited knowledge base on the monitoring and evaluation of adaptive learning interventions and their impacts. Unlike other implementation strategies or program management approaches, there are no standard metrics or a monitoring and evaluation framework to track the integration, implementation, and effectiveness of adaptive learning in health programming.

Gambling on Development: Why Some Countries Win and Others Lose - Dercon, Stefan (C Hurst & Co Publishers Ltd, 2022)

Can PDIA become a regular part of how a government works? - Building State Capability - Harrington, Peter (Building State Capability, 2022)
Institutional change is part of the theory of change of PDIA – scaling through the diffusion of new ways of thinking and greater problem-solving know-how. And once a community of practice reaches critical mass across an eco-system, a tipping point can happen where the eco-system becomes generally more open to novelty, where success is a more effective route to legitimacy, and where leadership is oriented towards value creation.

Seeing the Combined Effects of Aid Programmes - Burge, Richard; Nadelman, Rachel; McGee, Rosie; Fox, Jonathan; Anderson, Colin (IDS, 2022)
Multiple aid agencies often try to support change in the same places, at the same time, and with similar actors. Surprisingly, their interactions and combined effects are rarely explored. This Policy Briefing describes findings from research conducted on recent aid programmes that overlapped in Mozambique, Nigeria, and Pakistan, and from a webinar with UK Foreign, Commonwealth & Development Office (FCDO) advisors and practitioners. The research found three distinct categories of ‘interaction effects’: synergy, parallel play, and disconnect. We explore how using an ‘interaction effects’ lens in practice could inform aid agency strategies and programming.
Accepted: 2022-05-10T09:42:35Z Publisher: Institute of Development Studies

Evaluating complex interventions: What are appropriate methods? - Masset, Eduardo (CEDIL, 2022)
In the CEDIL Methods brief, ‘Evaluating complex interventions: What are appropriate methods?’ we identify four types of complex development interventions: long causal chain interventions, multicomponent interventions, portfolio interventions, and system-level interventions. These interventions are characterised by multiple activities, multiple outcomes, multiple components, a high level of interconnectedness, and non-linear outcomes.
Section: Publications

Integrated Governance: Achieving Governance Results and Contributing to Sector Outcomes - Frazer, Sarah; Granius, Mark; Brinkerhoff, Derick W.; McGregor, Lisa (RTI Press, 2022)
Achieving broad-based socio-economic development requires interventions that bridge disciplines, strategies, and stakeholders. Effective sustained progress requires more than simply an accumulation of sector projects, and poverty reduction, individual wellbeing, community development, and societal advancement do not fall neatly into sectoral categories. However, researchers and practitioners recognize key operational challenges to achieving effective integration that stem from the structures and processes associated with the current practice of international development. Integration calls for the intentional linking of intervention designs, implementation, and evaluation across sectors and disciplines to achieve mutually reinforcing outcomes. In this report, we summarize the results of a study we conducted to explore the challenges facing governance programs that integrate with sector interventions to achieve governance outcomes and contribute to sector outcomes. Through a review of policy documents and project reports from recent integrated governance programs and interviews with donor and practitioner staff, we found three integrated governance programming variants, an emphasis on citizen and government collaboration to improve service delivery, interventions that serve as the glue between sectors, and a balancing act for indicators to measure contribution to sectoral outcomes. Our analysis identified four key success factors: contextual readiness, the application of learning and adapting approaches, donor support, and recognition of the limitations of integrated governance. We then discuss recommendations and implications and for answering the challenge of integrating governance and sector programming to achieve development outcomes.

Rethinking monitoring and evaluation in complex systems — when learning is a result in itself - Innovation, UNDP (Medium, 2022)
By Søren Vester Haldrup, UNDP’s Strategic Innovation Unit

Beyond "Lack of Political Will": Elaborating Political Economy Concepts to Advance "Thinking and Working Politically" Comment on "Health Coverage and Financial Protection in Uganda: A Political Economy Perspective" - Ssennyonjo, Aloysius (International Journal of Health Policy and Management, 2022)
Political economy analysis (PEA) has been advanced as critical to understanding the political dimensions of policy change processes. However, political economy (PE) is not a theory on its own but draws on several concepts. Nannini et al, in concert with other scholars, emphasise that politics is characterised by conflict, contestation and negotiation over interests, ideas and power as various agents attempt to influence their context. This commentary reflects how Nannini et al wrestled with these PEA concepts - summarised in their conceptual framework used for PEA of the Ugandan case study on financial risk protection reforms. The central premise is that a common understanding of the PEA concepts (mainly structure-agency interactions, ideas, interests, institutions and power) forms a basis for strategies to advance thinking and working politically. Consequently, I generate several insights into how we can promote politically informed approaches to designing, implementing and evaluating policy reforms and development efforts.

Redefining rigour: using stories to evaluate systems change? - Lowther, Keira (Centre For Public Impact (CPI), 2022)
What might a different way of understanding rigour for work in complex adaptive systems look like?

Thinking and Working Politically on Health Systems Resilience: Learning from the experience of Cameroon, Nepal and South Africa during COVID-19 - Williams, Gareth (TWP Community of Practice, 2022)

(Re)making the case for adaptive management - Aston, Thomas (Medium, 2022)
Great overview of what to read on adaptive management. It’s a long one, so I’ve split it into two – second installment tomorrow. Christian Aid Ireland’s recent publication The Difference Learning Makes by Stephen Gray and Andy Carl made a bit of a splash. The study found that Christian Aid Ireland’s application of adaptive programming contributed to better development outcomes and supported more flexible delivery. The much vaunted MUVA programme in Mozambique is also coming to a close and presenting its results from using and adaptive approach. So, it struck me that we might be at a critical juncture in the conversation on adaptive management. We’ve had the crashing to earth of inflated expectations in recent misanthropic reflections, [misanthropic, moi? – Duncan] alongside a fragile institutionalisation of adaptive management in donor agencies, NGOs, and private sector organisations. However, I’d argue that we’ve reached the point where adaptive management has passed the proof-of-concept stage.

The Learning Power of Listening - Guijt, Irene; Gottret, Maria Veronica; Hanchar, Anna; Deprez, Steff; Muckenhirn, Rita (PRACTICAL ACTION PUBLISHING, 2022)
Steff had the pleasure to co-author the first SenseMaker Practitioner Guide with a group of friends and colleagues supported and published by Oxfam and CRS. This practical guide is for those who wish to use SenseMaker to conduct assessments, monitor progress, and undertake evaluations or research. Drawing on more than a decade of experience, the authors share dozens of examples from international development, providing practical tips and ideas for context-specific adaptations. They show how the method can be used to for difficult-to-measure outcomes related to poverty reduction, social justice, peacebuilding, resilience, gender norms, behavior change, governance and environmental management. ​ SenseMaker is a unique participatory method of inquiry that encourages and enables novel insights not obtained from conventional quantitative and quantitative and qualitative methods. It is action-oriented and, therefore, well-suited for people needing data- informed insights for adaptive management. "Writing this guide together with Irene, Veronica, Anna and Rita was an enormous learning process in itself and has further shaped our thinking and practice. We hope it will support first-time and experienced users to enhance their practice and that it will inspire people to explore and innovate further with the method."
Pages: 1-188

Using stories in evaluation: Participatory Narrative Inquiry and Sensemaking (2022)
Sensemaking and Participatory Narrative Inquiry (PNI) are similar approaches, based on collecting stories about a real life experience from a large number of stakeholders on a topic of interest to the evaluation, and giving the storytellers themselves the responsibility for analyzing and making sense of the stories. Because a large number of stories are collected and self-interpreted, it is possible to conduct quantitative analysis of recurrent themes, perspectives and feelings emerging in the narratives. By combining elements of qualitative and quantitative research, these approaches help to make sense of complex and evolving realities. This session presented the general approach of Sensemaking and PNI. Panelists discussed their experiences with this method, recently used in IFAD and WFP’s evaluations and invited participants to share their own experiences with using stories in evaluation.

Evidence-led adaptive programming: Lessons from MUVA - Sharp, Samuel; Riemenschneider, Nils; Selvester, Kerry (ODI, 2022)
Calls for more ‘adaptive programming’ have been prominent in international development practice for over a decade. Learning-by-doing is a crucial element of this, but programmes have often found it challenging to become more learning oriented. Establishing some form of reflective practice, against countervailing incentives, is difficult. Incorporating data collection processes that generate useful, timely and practical information to inform these reflections is even more so.This paper explores MUVA - an adaptive female economic empowerment programme in Mozambique. MUVA, we suggest, is atypically evidence-led. It combines systematic, inclusive reflective practice with extensive real-time data collection. We describe the fundamental features of MUVA’s monitoring, evaluation and learning (MEL) approach that supported this. One, how data collection and analysis are synchronised with set cycles for learning and adapting projects. Two, how MEL systems are designed to prioritise actionable learning, with data collection oriented more to the needs of implementing staff than to the reporting requirements of funders.This approach was enabled by building collective ownership over the programme’s objectives and the purpose of MEL from the outset. Implementers are asked about their motivations, and these are related to the programme’s Theory of Change. The evidence culture is supported by the proximity of MEL staff to implementing staff; and through structuring upwards accountability to funders around justifying evidence-based adaptations instead of reporting on more narrow indicators. We conclude by considering the relevance, or not, of MUVA’s approach to programmes in other contexts or issue areas trying to replicate a similarly evidence-informed approach to adaptive management. Key messages Learning-by-doing is essential to adaptive programming, but it can be challenging to establish data collection processes that generate useful, timely and practical information. MUVA – a female economic empowerment programme in Mozambique – has an atypically evidence-led adaptive management approach. This has two fundamental features. One, data collection and analysis are synchronised with set cycles for learning and adapting projects. Two, data collection is oriented more to the needs of implementing staff than to the reporting requirements of funders. Monitoring, evaluation and learning (MEL) systems are designed to prioritise actionable learning. This approach was enabled by building collective ownership over the programme’s objectives and the purpose of MEL from the outset. Implementers are asked about their motivations, and these are related to the programme’s Theory of Change. The evidence culture is supported by the proximity of MEL staff to implementing staff; and through structuring upwards accountability to funders around justifying evidence-based adaptations instead of reporting on more narrow indicators.

Integrating Local Knowledge in Development Programming - Johnson, Madelyn; Maunder, Ishan; Pinga, Andie (USAID, 2022)
The objective of this report, Integrating Local Knowledge in Development Programming is to share knowledge of how development donors and implementing organizations leverage local knowledge to inform programming. In a recent speech at Georgetown University, United States Agency for International Development (USAID) Administrator Samantha Power said, “As Americans with a fraught history living up to our own values, we’ve got to approach this work with intention and humility. But the entire development community needs to interrogate the traditional power dynamics of donor-driven development and look for ways to amplify the local voices of those who too often have been left out of the conversation.” To that end, USAID’s Agency Knowledge Management and Organizational Learning (KMOL) function facilitated conversations with multilateral and bilateral donors and local organizations to understand how organizations define, utilize, and incorporate local knowledge into their programmatic and operational activities. Using qualitative tools to gather data for this report, the research team explored five overarching themes: 1. Local Knowledge Nomenclature and Definitions 2. Best Practices 3. Outcomes 4. Ethics and Power Dynamics 5. Challenges

Achieving disability inclusive employment – Are the current approaches deep enough? - Shaw, Jackie; Wickenden, Mary; Thompson, Stephen; Mader, Philip (Journal of International Development, 2022)
Diverse approaches to promoting disability inclusive employment aim to transform workplaces into truly inclusive environments, usually with intervention strategies targeting two main groups: employers and jobseekers with disabilities. However, they do not always consider other relevant stakeholders or address the relationships and interactions between diverse actors in the wider social ecosystem. These approaches often neglect deeper ‘vexing’ difficulties which block progress towards disability inclusive work environments. Most interventions rightly embrace hegemonic ‘social models of disability’ and use human rights arguments but may neglect entrenched structural factors. Disability inclusive employment is complex, with unaddressed invisible aspects that continue to limit progress. We explore some key relevant disability concepts and then interrogate evidence from the ‘Inclusion Works’ programme working in four middle- and low-income countries, considering some intractable barriers underlying the slow movement towards inclusive employment. Finally, we propose that a more participatory action orientated approach involving disabled people and others is needed to both generate deeper understanding and provide pathways towards new solutions to obstinate problems through progressive action learning processes in context. Programmatic interventions that work across the levels of the ecosystem and address power relations and interactions between stakeholders could lead to more substantial forms of disability inclusive employment.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/jid.3692

Where have we got to on Thinking and Working Politically? Update and a Mildly Heretical Thought. – FP2P - Green, Duncan (From Poverty to Power, 2022)
Headed off recently to discuss the state of Thinking and Working Politically within the aid sector. This is a loose network of aid wonks that came together to try and move aid from a pure focus on technical issues, towards taking account of power and politics and why they can facilitate/frustrate attempts to make change happen in any given context. It was great to be in a room with others (50/50 in person and online) – the neurons fire in a way that just doesn’t happen online (but I also need to brush up on my meeting skills, as when I accidentally clicked on a random video in my timeline and it started playing at full volume. Super awkward). On to the content (Chatham House rule, so no names or institutions). Some observations about the evolution of a movement I’ve been connected with for over a decade. I got a lot of pushback from participants on an earlier draft and have made quite a few changes, but they should definitely feel free to set me straight in comments!

How Beijing Commands: Grey, Black, and Red Directives from Deng to Xi - Ang, Yuen Yuen (The China Quarterly, 2022)

How to develop capacity in the international development sector - Prieto Martín, Pedro (CLARISSA, 2022)
In her acclaimed study of effective cross-organizational teamwork, Harvard Business School Professor Amy C. Edmonson concluded that “trying things that...

Designing Contribution Analysis of Participatory Programming to Tackle the Worst Forms of Child Labour - Apgar, Marina; Snijder, Mieke; Prieto Martin, Pedro; Ton, Giel; Macleod, Shona; Kakri, Shanta; Paul, Sukanta (Institute of Development Studies, 2022)
This Research and Evidence Paper presents the theory-based and participatory evaluation design of the Child Labour: Action-Research- Innovation in South and South-Eastern Asia (CLARISSA) programme. The evaluation is embedded in emergent Participatory Action Research with children and other stakeholders to address the drivers of the worst forms of child labour (WFCL). The report describes the use of contribution analysis as an overarching approach, with its emphasis on crafting, nesting and iteratively reflecting on causal theories of change. It illustrates how hierarchically-nested impact pathways lead to specific evaluation questions and mixing different evaluation methods in response to these questions, critical assumptions, and agreement on causal mechanisms to be examined in depth. It also illustrates how realist evaluation can be combined with contribution analysis to deeply investigate specific causal links in the theory of change. It reflects on learning from the use of causal hotspots as a vehicle for mixing methods. It offers considerations on how to navigate relationships and operational trade-offs in making methodological choices to build robust and credible evidence on how, for whom, and under what conditions participatory programming can work to address complex problems such as child labour.
Accepted: 2022-08-18T11:05:51Z Publisher: Institute of Development Studies

The Haves and the Have Nots: Civic Technologies and the Pathways to Government Responsiveness - Mellon, Jonathan; Peixoto, Tiago C; Sjoberg, Fredrik M (World Bank, 2022)
“Civic tech” broadly refers to the use of digital technologies to support a range of citizen engagement processes. From allowing individuals to report problems to local government to enabling the crowdsourcing of national legislation, civic tech aims to promote better policies and services – while contributing to more inclusive democratic institutions. But could civic tech affect public issues in a way that benefits some and excludes others? Over the decades, the question of who participates in and who is excluded from participation mediated by technology has been the focus of both civic tech critics and proponents. The latter tend to argue that, by enabling citizens to participate without constraints of time and distance, civic tech facilitates the participation of those who usually abstain from engaging with public issues, leading to more inclusive processes. Critics argue that, given the existing digital divide, unequal access to technology will tend to empower the already empowered, further deepening societal differences. Yet both critics and proponents do tend to share an intuitive assumption: the socio-economic profile of who participates is the primary determinant of who benefits from digitally mediated civic participation. For instance, if more men participate, outcomes will favor male preferences, and if more young people participate, outcomes will be more aligned with the concerns of the youth. In a new paper, we show that the link between the demographics of those participating through digital channels, and the beneficiaries of the participation process, is not necessarily as straightforward as commonly assumed. We review four civic tech cases where data allow us to trace the full participatory chain through: the initial digital divide the participant’s demographics the demands made through the process the policy outcomes We examine online voting in the Brazilian state of Rio Grande do Sul’s participatory budgeting process, the local problem reporting platform Fix My Street (FMS) in the United Kingdom, Iceland’s online crowdsourced constitution process, and the global petitioning platform Change.org. Counterintuitive findings Change.org has been used by nearly half a billion people around the globe. Using a dataset of 3.9 million signers of online petitions in 132 countries, we examine the number of successful petitions and assess whether petitions created by women have more success than those submitted by men. Our analysis shows that, even if women create fewer online petitions than men, their petitions are more likely to be successful. All else equal, when online petitions have an impact on government policy, the agenda being implemented is much closer to the issues women choose to focus on. In Rio Grande do Sul’s digital participatory budgeting (PB), we show that despite important demographic differences between online and offline voters, these inequalities do not affect which types of projects are selected for funding – a consequence of PB’s unique institutional design, which favors redistributive effects. In fact, of all the cases analyzed, none reflect the standard assumption that inequalities in who participates translate directly into inequalities in who benefits from the policy outcomes. Our results suggest that the socio-economic profile of participants predicts only in part who benefits from civic tech. Just as important to policy outcomes is how the platform translates civic participation into policy demands, and how the government responds to those demands. While civic tech practitioners pay a lot of attention to design from a technological perspective, our findings highlight the importance of considering how civic tech platforms function as political institutions that encourage certain types of behavior while discouraging others. Civic tech, it seems, is not inherently good nor bad for democratic institutions. Instead, its effect is a combination of who participates on digital platforms and the choices of platform designers and governments.

How do we know we are doing good work? - Sharp, Cathy; McLaughlin, Dot; Whitley, Janet; Lawson, Karen (Collective Leadership for Scotland, 2022)
Collective leadership feels timely and important in an uncertain, fast changing, and challenging world. This report comes at this heightened moment of urgency and appetite for renewal, bringing potential to do things differently in public services and communities. The opportunities and challenges of true collaboration-in-practice, in the absence of blueprints, mean that it has never been more necessary to change ways of working and foreground learning. As outcomes remain important, and time and resources are scarcer than ever, the approach of collective leadership makes the creation of impact a shared, conscious, and actionable choice. A pathway to ultimate impact at scale is co-created through dialogue about expectations and contributions, and by design, not assuming change will happen because we have good intentions. Collective Leadership for Scotland (CLfS) has a strong vision and enjoys continuing active interest, drawing together participants from a variety of agencies working in public services. People are looking for fresh thinking, space and time for reflection, connection with others, a chance to think about how to tackle difficulties, and to test out what it takes to do, and continue to do, the work of collaborative public service. These motivations are deepened and brought into sharper focus by the pandemic, with an added interest in developing skills in online facilitation. CLfS contributes to building a critical mass for system change, to help to sustain the ambitions of the Christie Commission and the delivery of the National Outcomes for Scotland. There remains further potential to realise wider and deeper impact amongst organisations, communities, and wider systems. The conclusions of this report are likely to have wider resonance beyond interests in the CLfS programmes. This report deepens understanding of some of the challenges of commissioning, convening, and the scope for deeper impact through building reflective and relational leadership practices. It also outlines social and experiential sensemaking and facilitation practices to strengthen the action inquiry approach as a deliberate learning strategy, building cultures that support new forms of collaborative inquiry and systemic action research.

Digital authoritarianism in Bangladesh: Weaponising a draconian law to silence dissent in the pandemic era | Association for Progressive Communications - Siddiki, Zayed (2022)

Thinking and Working Politically in the Land Sector in Mekong Region - Burns, Tony; Ingalls, Micah; Rickersey, Kate (2022)
In recent decades, the World Bank and many bilateral development partners have provided funding to support land administration reform. Traditional land administration reform projects focus on the economic and technical design of interventions based on a library of best practice, commonly avoiding the “messy politics” typically involved in land governance. Experience and lessons from land administration reform initiatives have been documented and a recurrent theme is that many projects fail to create effective, transformative change and gain the critical mass, and the community participation, necessary to ensure the sustainability of land administration reform. Over the last decade there have been concerted efforts to develop more politically informed ways of thinking and working using a range of methodologies referred to, variously, as Thinking and Working Politicallyi (TWP), Problem Driven Iterative Adaptation (PDIA) and Doing Development Differently (DDD). There is little evidence that these different approaches have been applied in the land sector.

Learning from Life Story Collection and Analysis With Children Who Work in the Worst Forms of Child Labour in Nepal - Karki, Shanta; Apgar, Marina; Snijder, Mieke; Sharma, Ranjana (Institute of Development Studies, 2022)
The CLARISSA Nepal team collected and analysed 400 life stories of children and young people engaged in or affected by the worst forms of child labour (WFCL), particularly in the “Adult Entertainment” sector in Nepal, which includes children working in Dohoris (restaurants playing folk music), dance bars, spa-massage parlours, khaja ghars (tea/snack shop) and guest houses. Stories were also collected from children in CLARISSA’s focus neighbourhoods, children in this category include street connected children and those working in transportation, party palaces, domestic labour and construction sites. Of the 400 stories collected, 350 were collected by adult researchers and 50 were collected by children themselves.
Accepted: 2022-11-04T12:22:13Z Publisher: Institute of Development Studies

How to do Process Tracing: A Method for Testing “How Change Happened” in Complex and Dynamic Settings - Lynn, Jewlya; Stachowiak, Sarah; Beyers, Jennifer (ORS Impact, 2022)
Process tracing is a causal methodology that can help people understand how a particular large-scale change actually happened within a complex dynamic environment. Much of the existing literature provides important information about the method; we wrote this brief to help more people operationalize the concepts and learn about practical steps for using this method more easily, with quality, and toward a more equitable world. This piece was written based on our experiences implementing process tracing when our experience showed that existing materials on the method had a lot more conceptual than practical information. We’ve approached this as people with some successful (and some unsuccessful) experience with the method itself, alongside deep experience in evaluating initiatives and strategies in complex and dynamic settings. We focus not on the Bayesian side of process tracing but rather on how this can be implemented in a way that’s more participatory and lifts up the experiences and wisdom of those closest to the work and the problems being tackled. We hope this contributes to and helps make more approachable the important work of political scientists and methodologists upon which this work sits.

UNDP Digital Leadership Learning Modules - UNDP (United Nations Development Programme, 2022)
In order to support the digital transformation of government operations Digital Learning Modules for Civil Servants are available, an off-the-shelf package of capacity development in form of replicable training modules to empower public servants at both the local and central government level to be leaders of digital transformation for delivering better public services. The modules cover a multitude of fundamental areas: comprehending digital government and services, human-centered design for inclusivity and agile learning cycles; feature the importance of security and privacy, the value of data and how to manage data and technology related risks; spotlight the key role of supportive leadership and offer practical tools for assessing and overcoming main barriers to ensure a successful digital transformation journey.

Local Capacity Strengthening Policy - USAID (USAID, 2022)

The Art and Craft of Bricolage in Evaluation - Aston, Thomas; Apgar, Marina (Institute for Development Studies, 2022)
This CDI Practice Paper by Tom Aston and Marina Apgar makes the case for ‘bricolage’ in complexity-aware and qualitative evaluation methods. It provides a framework based on a review of 33 methods to support evaluators to be more intentional about bricolage and to combine the component parts of relevant methods more effectively. It discusses two cases from practice to illustrate the value added of taking a more intentional approach. It further argues that navigating different forms of power is a critical skill for bricolage, and that doing so can help to ensure rigour.
Accepted: 2022-10-14T10:56:39Z Publisher: Institute of Development Studies

Everyday patterns for shifting systems - GCSI (GCSI, 2022)

Everyday patterns for shifting systems - Right scaling - GCSI (GCSI, 2022)

Beneficiary feedback mechanisms - UK Aid (FCDO, 2022)
This guidance seeks to ensure that UK Aid Direct applicants and grant holders understand what the Foreign, Commonwealth & Development Office (FCDO) means by beneficiary feedback mechanisms, and more specifically, that they: • Understand the terms used that relate to beneficiary feedback mechanisms in UK Aid Direct guidance and templates • Understand beneficiary feedback mechanisms and why they are a useful tool for project monitoring and learning • Learn how to use beneficiary feedback mechanisms during project implementation. • Can demonstrate that using beneficiary feedback mechanisms can lead to greater accountability.

Transforming M&E for Uncertain and Complex Contexts: The UNDP’s Innovation Sandbox Approach - Fraser, Dugan; Haldrup, Søren (Powered by Evidence Podcast, 2022)
How can we transform monitoring and evaluation (M&E) into a more adaptive, emergent process to address uncertainty and complexity in todays’ world? How do we move from compliance and accountability to learning – to support better, more timely, decisions? Join GEI Program Manager, Dugan Fraser, as he discusses these questions and others with Special Guest, Søren Haldrup, from UNDP's Strategic Innovation Unit where he manages UNDP's innovation facility and leads a new initiative called the M&E Sandbox.

How and Why Practitioners Think and Work Politically - Evidence from Chemonics Programming Across Sectors - Kantelberg, Renee; Swift-Morgan, Jennifer; Watson, Bryce (Chemonics International, 2022)
Most development practitioners have long recognized that deep contextual knowledge is crucial to understanding how projects interact with their local systems and, in turn, to navigating these systems. Moreover, this knowledge must complement projects' technical solutions, or they will fall flat and may even undercut project objectives as they clatter down. What, then, explains practitioners' particular interest in TWP as an explicit strategy and more than just "doing good development"? This report responds to that question and to the many calls for a more comprehensive picture of TWP by presenting new evidence of the various forms that TWP may take in practice. The evidence comes from a 2022 study that Chemonics undertook to foster more robust learning about TWP. Specifically, we closely examined Chemonics implemented projects that used or are using various forms of TWP in nine countries: Bangladesh, Iraq, Mozambique, the Philippines, Pakistan, Syria, Timor-Leste, and Tunisia. In conducting the study, we interviewed multiple staff from these projects. We complemented what we learned from these projects with a review of eight additional Chemonics- implemented projects applying TWP that had received dedicated support from Chemonics’ Center for Politically Informed Programming (the Center). We consider these findings alongside those of the recent (2022) USAID-Chemonics study on political economy analysis (PEA) usage to identify and articulate what is different and more effective about PEA processes and TWP practices that have received more support.

How Political Contexts Influence Education Systems: Patterns, Constraints, Entry Points - Levy, Brian (Research on Improving Systems of Education (RISE), 2022)
This paper synthesises the findings of a set of country studies commissioned by the RISE Programme to explore the influence of politics and power on education sector policymaking and implementation. The synthesis groups the countries into three political-institutional contexts: - Dominant contexts, where power is centred around a political leader and a hierarchical governance structure. As the Vietnam case details, top-down leadership potentially can provide a robust platform for improving learning outcomes. However, as the case studies of Ethiopia, Indonesia, Nigeria, and Tanzania illustrate, all-too-often dominant leaders’ goals vis-à-vis the education sector can veer in other directions. - In impersonal competitive contexts, a combination of strong formal institutions and effective processes of resolving disagreements can, on occasion, result in a shared commitment among powerful interests to improve learning outcomes—but in none of the case studies is this outcome evident. In Peru, substantial learning gains have been achieved despite messy top-level politics. But the Chilean, Indian, and South African case studies suggest that the all-too-common result of rule-boundedness plus unresolved political contestation over the education sector’s goals is some combination of exaggerated rule compliance and/or performative isomorphic mimicry. - Personalised competitive contexts (Bangladesh, Ghana, and Kenya for example) lack the seeming strengths of either their dominant or their impersonal competitive contexts; there are multiple politically-influential groups and multiple, competing goals—but no credible framework of rules to bring coherence either to political competition or to the education bureaucracy. The case studies show that political and institutional constraints can render ineffective many specialised sectoral interventions intended to improve learning outcomes. But they also point to the possibility that ‘soft governance’ entry points might open up some context-aligned opportunities for improving learning outcomes. In dominant contexts, the focus might usefully be on trying to influence the goals and strategies of top-level leadership. In impersonal competitive contexts, it might be on strengthening alliances between mission-oriented public officials and other developmentally-oriented stakeholders. In personalised competitive contexts, gains are more likely to come from the bottom-up—via a combination of local-level initiatives plus a broader effort to inculcate a shared sense among a country’s citizenry of ‘all for education’.

Development Delusions and Contradictions: An Anatomy of the Foreign Aid Industry - Sims, David (Palgrave Macmillan, 2022)
This book analyses the shortcomings of the Western development aid programme. Through exploring the evolution of aid over more than seven decades, development is examined as an industry with a variety of motives and actors. The driving forces and dynamics in the relationship between aid and economic development are highlighted in relation to faulty development structures and misaligned aims. With a particular focus on Egypt, radical questions are posed on how global aid and development can be improved, including how it can respond to the COVID-19 pandemic.This book aims to present an alternative aid framework to help overcome the dysfunctionality of the current international development system. It will be of interest to researchers and policymakers working within development economics and development policy.

Creative Activities for Work Teams and Communities of Practice. A guide for designing participatory, productive, and engaging sessions - AKF (2023)
This free guide aims to build the skills and confidence of people designing and facilitating participatory, productive, and engaging workshop sessions for work teams and communities of practice by providing them with a series of creative activities that can be run in-person or virtually. By the time they have gone through this guide, session leads will be able to: Describe ways in which creative activities can be of value to teams and communities of practice, Design both in-person and virtual workshop sessions that incorporate creative activities, Facilitate participatory, productive and engaging workshop sessions that incorporate one or several creative activities, and Lead an effective debrief following a creative activity. This toolkit was developed by the Aga Khan Foundation.

Virtual Facilitation Techniques for Work Teams and Communities of Practice. A guide for designing and running dynamic and engaging remote sessions and meetings - AKF (2023)
This free guide aims to build the skills and confidence of people designing and facilitating virtual meetings, including online community of practice sessions. By the time they have gone through this guide, community of practice facilitators will be able to: Design an engaging and participatory virtual session, Develop an effective, varied and realistic agenda for a virtual session, Apply a wide range of virtual facilitation techniques, and Using technology to facilitate virtual sessions. This toolkit was developed by the Aga Khan Foundation.

Si Guides on System Innovation - Colchester, Joss; Si Network (Si Network, 2023)
So you have just hear about systems thinking and systems innovation and are keen to know more, Ok sparky let's get started.We have structured all the content into four main areas that we think you ...

Setting new standards for better MEL. Lessons for grantees & funders - Colnar, Megan; Azevedo, Andrea; Tolmie, Courtney; Caddick, Hannah (Better Evaluation, 2023)
How can donors and grantees work together to create effective monitoring, evaluation, and learning (MEL) practices that drive field-wide transformation? The Open Society Foundation’s Fiscal Governance Program found success by focusing on six key approaches, including empowering grantees and relinquishing power. In 2021, an external close-out evaluation by Intention to Impact of the program (which ran for 7 years and gave over $150 million in grants) revealed something pretty remarkable—the program’s deliberate focus on strengthening field-wide monitoring, evaluation, and learning practices was a success. Substantial capacity increases were observed across key institutions and grantees, new complexity-sensitive practices and methods were being actively championed and deployed, and a growing community of better-connected practitioners were exchanging tips and tricks on how to apply smart, context-specific MEL across fiscal governance issues. What’s more, in this evaluation, most grantees gave high praise to these efforts. So, how did this come about? We detail the six different approaches we used in our new publication Setting new standards for better MEL: Lessons for funders and grantees. The approaches range from checking power dynamics to growing skills for evaluative thinking and seeding peer learning and field-wide research. The publication is paired with a toolkit and showcases resources we used and iterated on across the various approaches.

Two decades of public sector innovation: building an analytical framework from a systematic literature review of types, strategies, conditions, and results - Criado, J. Ignacio; Alcaide-Muñoz, Laura; Liarte, Irene (Public Management Review, 2023)
Scholarly attention in innovation in the public sector is growing rapidly, provoking analytical complexity. We developed a systematic literature review about Public Sector Innovation (PSI), analysing 169 articles published between 2001 and 2021, using PRISMA. We present a comprehensive approach to PSI testing and empirically develop an analytical framework based on the most common combinations of the studied dimensions. Additionally, we propose three main research avenues for the future of PSI: (1) studying PSI in different contexts, (2) expanding the analysis of configurations in PSI initiatives, and (3) analysing ambidextrous strategies to support the practical implementation of PSI.
Publisher: Routledge _eprint: https://doi.org/10.1080/14719037.2023.2254310

Hacking by the prompt: Innovative ways to utilize ChatGPT for evaluators - Ferretti, Silva (New Directions for Evaluation, 2023)
“Hacking by the prompt”—writing simple yet creative conversational instructions in ChatGPT's message window—revealed many valuable additions to the evaluator's toolbox for all stages of the evaluation process. This includes the production of terms of reference and proposals for the dissemination of final reports. ChatGPT does not come with an instruction book, so evaluators must experiment creatively to understand its potential. The surprising performance of ChatGPT leads to the question: will it eventually substitute for evaluators? By describing ChatGPT through four personality characteristics (pedantic, “I know it all,” meek, and “speech virtuoso”), this article provides case examples of the potential and pitfall of ChatGPT in transforming evaluation practice. Anthropomorphizing ChatGPT is debatable, but the result is clear: tongue-in-cheek personality characteristics helped hack ChatGPT more creatively while remaining aware of its challenges. This article combines practical ideas with deeper reflection on evaluation. It concludes that ChatGPT can substitute for evaluators when evaluations mostly focus on paperwork and conventional approaches “by the book” (an unfortunate trend in the sector). ChatGPT cannot substitute engagement with reality and critical thinking. Will ChatGPT then be a stimulus to rediscover the humanity and the reality we lost in processes?
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20557

Designing & Facilitating Collaborative Learning Networks. A toolkit - Folsom, Amanda (Results for Development, 2023)
Collaborative Learning — an approach which brings together people who face common challenges to share knowledge and jointly problem-solve — is a powerful way to support locally-led development and achieve impact. It provides a structured process in which change agents determine priorities, set the learning agenda, work together to identify strategies to address complex challenges, and provide ongoing implementation support to one another. Collaborative Learning — unlike traditional approaches to technical assistance — centers the expertise of local change agents and captures the valuable tacit knowledge of practitioners to advance systems change. This toolkit synthesizes lessons, tips, and tools accumulated from more than a decade of experience designing and facilitating over 20 Collaborative Learning Networks. The lead author was Amanda Folsom, Senior Program Director and Collaborative Learning Practice Lead, Results for Development (R4D). The toolkit benefited from the contributions of many R4D colleagues, including Katie Bowman, Cheryl Cashin, Tanya Jones, Gina Lagomarsino, Agnes Munyua, Maria Jose Pastor, and Abeba Taddese. R4D has not done this alone. The lessons and examples are drawn from our work with many network partners from the Joint Learning Network for Universal Health Coverage (JLN), Linked Immunisation Action Network (Linked), Health Systems Strengthening Accelerator, Strategic Purchasing Africa Resource Center (SPARC), the Primary Health Care Performance Initiative (PHCPI), the Partnership for Evidence and Equity in Responsive Social Systems (PEERSS), and the School Action Learning Exchange (SALEX), to name a few. We are grateful for the thought partnership and support of our partners and funders, including the Bill & Melinda Gates Foundation, Gavi, Hewlett Foundation, Jacobs Foundation, Rockefeller Foundation, USAID, and World Bank.

The Art of Scaling Deep - Research in Summary - Fraser, Tatiana (The Systems Sanctuary, 2023)
Over the last 15 years the concept of scale has become a foundational part of the apparatus of the social and environmental change sector. A business mindset of growth has been seamlessly transferred to the social and environmental problems we are collectively trying to shift in the world. Scaling up, (influencing policy) has been considered the strategic pathway to systems change. Scaling out (spreading new models) is seen as a pathway to success. The allure of these scaling theories lies, in part, in their tangibility, and the easy way in which they can be measured. However this focus on growth has shifted our attention away from a series of messy truths. Sometimes bigger isn't better. Endless growth is not sustainable and our urgency to try to fix the problem and seek solutions may be part of the crisis we are in. One unintended consequence of this has been that another type of scale has been devalued and as a result, under-resourced. It’s a scale that values the slow steady work of deepening relationships. It recognizes the significance of context, building connections that bridge diverse communities and it prioritizes inner work and healing as integral components of the scaling process. We call this type of scale ‘Scaling Deep’ and we believe that adequately supporting it, and funding it, holds the greatest potential for long lasting systemic change. The purpose of this research has been to delve deeper into the art and craft of Scaling Deep. Ultimately, our goal is for it to become firmly ingrained within the recognized realm of social change, alongside the well-established concepts of scaling up and scaling out. We want practitioners who are Scaling Deep to be able to harness the wisdom and power of this work and to talk about it openly, with confidence and credibility. We want understanding of this approach to flourish and evolve and for it to be appropriately celebrated and supported. We want decision-makers to be informed and inspired by the principles and practices of scaling deep. For it to be embedded in theories of change as an essential component of decision-making processes within the wider change discourse. Importantly we want leaders who are Scaling Deep to have access to sufficient resources and to receive the care from the field that they need to thrive. We would like to see organisations that have the power to invest, to align their efforts with the profound impact that scaling deep aspires to cultivate in the world. Before we begin, let us be crystal clear. We are not opposed to scaling up and out as strategies for change. Innovation and scaling what is working is part of how we evolve as humanity. As social entrepreneurs ourselves, we have both done this twice over. We value it and we know it’s important. We recognize these are strategies for creating widespread impact and effecting positive change. Our intention here is not to dismiss or undermine the value of scaling up and out, but rather to encourage a broader and more holistic perspective that includes other dimensions of scale. There is a need to understand how the different scaling approaches can work together, rather than seeing them as hierarchical and disconnected. Our ultimate goal is to equip the change sector with a more inclusive, expansive, and powerful approach to tackling the myriad challenges we face, by exploring the potential of Scaling Deep as a transformational strategy for systems change.

Building a team culture for Adaptive Management in MSD: 5 Strategies MEL Managers Say Work - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (USAID, 2023)
Keywords: Bolder Meghan; Gover Dun; Henao Lina; Nasution Zulka; Okutu David

Enhancing partner and system-level learning: 8 Tips from MEL Managers - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (USAID, 2023)
Effective learning is a key driver of market systems change, with the potential to enhance system competitiveness, resilience, and inclusiveness. Shifting the Locus of Learning: Catalyzing Private Sector Learning to Drive Systemic Change recently outlined a rationale for enhancing the scale and quality of learning in a system and identifying 10 strategies programs can contextualize to catalyze learning. These strategies are also backed with robust examples from 13 programs doing this work across 11 countries. To deepen insights on what MSD Monitoring, Evaluation and Learning (MEL) Managers have experienced in putting several of those strategies into practice, the Feed the Future Market Systems and Partnerships (MSP) Activity convened a series of peer discussions as part of a larger initiative (see Figure 1). This brief shares the collective learning and experience on this topic of three senior MEL Managers who were interested in and had experience with this topic. The group represented full-time, program-based MEL Leads working on MSD programs funded by USAID and DFAT, based in Fiji, Albania, and Kosovo working for Adam Smith International, SwissContact, and DT Global, respectively. From those discussions, this paper synthesizes eight tips from MEL Managers for practically enhancing partner and system-level learning: 1. Identify the right decision-maker(s) at potential partners. 2. Use a co-creation process to identify learning opportunities. 3. Use diagnostics and assessments to strengthen partner and system capacity for actionable learning. 4. Use a phased capacity strengthening process tied to behavior change. 5. Measure partners’ continued investment in and use of learning—not the continuation of specific learning activities. 6. Work with sector-level institutions for scale but be aware of risks. 7. Leverage informal communities of practice to share learning. 8. Use the right terminology to talk about partner and system-focused learning.

Practioners Guide to Assessing Systems Change - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (USAID, 2023)

The Dynamics of Team Learning: Harmony and Rhythm in Teamwork Arrangements for Innovation - Harvey, Jean-François; Cromwell, Johnathan R.; Johnson, Kevin J.; Edmondson, Amy C. (Administrative Science Quarterly, 2023)
Innovation teams must navigate inherent tensions between different learning activities to produce high levels of performance. Yet, we know little about how teams combine these activities—notably reflexive, experimental, vicarious, and contextual learning—most effectively over time. In this article, we integrate research on teamwork episodes with insights from music theory to develop a new theoretical perspective on team dynamics, which explains how team activities can produce harmony, dissonance, or rhythm in teamwork arrangements that lead to either positive or negative effects on overall performance. We first tested our theory in a field study using longitudinal data from 102 innovation teams at a Fortune Global 500 company; then, we replicated and elaborated our theory in a study of 61 MBA project teams at an elite North American university. Results show that some learning activities can occur within the same teamwork episode to have harmonious positive effects on team performance, while other activities combine to have dissonant negative effects when occurring in the same episode. We argue that dissonant activities must be spread across teamwork episodes to help teams achieve a positive rhythm of team learning over time. Our findings contribute to theory on team dynamics, team learning, and ambidexterity.

Digitally-induced change in the public sector: a systematic review and research agenda - Haug, Nathalie; Dan, Sorin; Mergel, Ines (Public Management Review, 2023)
Digital transformation has become a buzzword that is permeating multiple fields, including public administration and management. However, it is unclear what is transformational and how incremental and transformational change processes are linked. Using the PRISMA method, we conduct a systematic literature review to structure this growing body of evidence. We identified 164 studies on digitally-induced change and provide evidence for their drivers, implementation processes, and outcomes. We derive a theoretical framework that shows which incremental changes happen in public administrations that are implementing digital technologies and what their cumulative, transformative effects are on society as a whole.
Publisher: Routledge _eprint: https://doi.org/10.1080/14719037.2023.2234917

Large language model applications for evaluation: Opportunities and ethical implications - Head, Cari Beth; Jasper, Paul; McConnachie, Matthew; Raftree, Linda; Higdon, Grace (New Directions for Evaluation, 2023)
Large language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text-based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep-learning AI techniques, where language models are trained on extremely large text data from the internet and then re-used for downstream tasks with limited fine-tuning required. They offer exciting opportunities for evaluators to automate and accelerate time-consuming tasks involving text analytics and text generation. We estimate that over two-thirds of evaluation tasks will be affected by LLMs in the next 5 years. Use-case examples include summarizing text data, extracting key information from text, analyzing and classifying text content, writing text, and translation. Despite the advances, the technologies pose significant challenges and risks. Because LLM technologies are generally trained on text from the internet, they tend to perpetuate biases (racism, sexism, ethnocentrism, and more) and exclusion of non-majority languages. Current tools like ChatGPT have not been specifically developed for monitoring, evaluation, research, and learning (MERL) purposes, possibly limiting their accuracy and usefulness for evaluation. In addition, technical limitations and challenges with bias can lead to real world harm. To overcome these technical challenges and ethical risks, the evaluation community will need to work collaboratively with the data science community to co-develop tools and processes and to ensure the application of quality and ethical standards.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20556

Assessing Value for Money: the Oxford Policy Management Approach - King, Julian; Daniel Wate; Esther Namukasa; Alex Hurrell; Frances Hansford; Patrick Ward; Shiva Faramarzifar (Oxford Policy Management, 2023)
This document offers practical guidance for assessing the Value for Money (VfM) of government- and donor-financed programmes and policy interventions. In line with OPM’s focus and mission, it has been predominantly applied in the international development sector, but the approach upon which it is based is also used in the context of domestic public policy and programmes.1 There is increasing scrutiny on VfM in international development, but a lack of appropriate methods to support its assessment. There is a risk of reaching invalid conclusions if VfM evaluation is tied to a narrow set of indicators devoid of any evaluative judgement—for example, by emphasising the most readily quantifiable measures rather than the most important (but harder to quantify) aspects of performance, or by focusing on the quantification of outputs and outcomes at the expense of more nuanced consideration of their quality, value, and importance. The approach presented in this guide combines theory and practice from evaluation and economics to respond to requirements for accountability and good resource allocation, as well as to support reflection, learning, and adaptive management. It involves developing and implementing a framework for: • organising evidence of performance and VfM; • interpreting the evidence on an agreed basis; and • presenting a clear and robust performance story. This guide sets out a framework for making and presenting judgements in a way that opens both the reasoning process and the evidence to scrutiny. The approach is designed to be used in alignment with broader monitoring, evaluation, and learning (MEL) systems—both for efficiency's sake, and to ensure conceptual coherence between VfM evaluation and wider MEL work. The VfM framework achieves these aims by: • using explicit criteria (aspects of performance) and standards (levels of performance) to provide a transparent basis for making sound judgements about performance and VfM; • combining quantitative and qualitative forms of evidence to support a richer and more nuanced understanding than can be gained from the use of indicators alone; • accommodating economic evaluation (where feasible and appropriate) without limiting the analysis to economic methods and metrics alone; and • incorporating and building on an approach to VfM evaluation which is familiar to international aid donors.

Adaptive Programming and going with the grain: IMAGINE's new water governance model in Goma, DRC - Kirk, Tom; Green, Duncan; Stys, Patrycja; Mosquera, Tom (Development Policy Review, 2023)
Motivation This paper explores adaptive approaches to development programmes that aim at improving service provision in underperforming sectors in fragile and conflict affected states (FCAS). It does this through a case study of the Integrated Maji Infrastructure and Governance Initiative for eastern Democratic Republic of the Congo's (IMAGINE) public-private partnership model for water provision. Purpose The processes and decisions that culminated in IMAGINE's model emphasize the need for programming that is culturally and politically aware, responsive to events, learns in real-time, is entrepreneurial, and works with the grain of local institutions to support change. Detailed case studies of such ways of working are crucial for programmes that seek to challenge and reform the status quo in FCAS. Methods and approach The paper is based on 42 semi-structured interviews conducted in the summers of 2019 and 2020. They reflect the broad spectrum of actors – individuals, public authorities, and organisations – involved in IMAGINE's evolution. Findings The narrative focuses on IMAGINE's attempts to professionalise and commercialise Goma's water sector. It shows how as IMAGINE repeatedly adapted to ground realities, it took on the characteristics of a public authority, thereby, engendering backlashes that threated its longer-term goals. However, by revisiting its initial values and logics it was able to get things done and achieve it aims. Policy implications IMAGINE's story suggests that adaptive programmes should put politically savvy local development professionals in key positions and enable them to carefully construct coalitions of allies across the systems they aim to disrupt. This may also require them to revisit and adapt their initial ideas, guiding principles and values as greater understandings of development problems are gained. A pubic authorities lens, attuned to the logics programmes seek to address and introduce to FCAS, may help analysts to foreground the implications of such adaptations.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/dpr.12691

Laudes Rubrics - Measurement and Learning Approach - Laudes Foundation (Laudes Foundation, 2023)
How can we measure and learn when promoting systems change? It’s a challenge that has inspired Laudes Foundation to develop a rubrics-based methodology to help us, our partners and the wider field of philanthropy, understand our contribution to change, while learning and adapting to new and unforeseen circumstances. Change cannot be captured by numbers alone because metrics put the focus on what can be counted, not always what’s most important. Rubrics are a framework that set a standard for what ‘good’ looks like – and create a shared language for describing and assessing it using both quantitative and qualitative evidence. At Laudes Foundation, the rubrics are integrated into our grantmaking processes – from the design phase through to measurement, evaluation and learning. The four Laudes Foundation rubrics categories Laudes Foundation has developed 21 rubrics that work across different levels, from processes to long-term impact. When measuring a specific initiative, a smaller set of relevant rubrics are chosen and assessed on a rating scale from ‘harmful’ to ‘thrivable’. The 21 rubrics are categorised into four groups, with some natural overlap between categories B, C and D. Category A focuses on the process-related aspects of initiatives, including design, implementation, monitoring, communication and learning, and organisation and network effectiveness. Category B focuses on the early and later changes that need to happen to create the right conditions to achieve the 2025 outcomes. Category C captures the 2025 outcomes, focusing on how policymakers, financiers, business leaders, and workers and producers behave. Category D captures the 2030 impacts, describing the new reality created as a result of sustained efforts.

How to Sharpen a Nonviolent Movement - McClennen, Sophia; Popovic, Srdja; Wright, Joseph (Journal of Democracy, 2023)
In the past three decades nonviolent social protest has become the most reliable path to democracy. However, not all nonviolent mobilization campaigns succeed. To examine why some nonviolent campaigns are more successful than others, we analyze the use of a particular type of activist campaign tactic, the "dilemma action." The dilemma action is a nonviolent civil-disobedience tactic that provokes a "response dilemma" for the target. Collecting original data on dilemma actions during nonviolent activist campaigns, we find that roughly one-third of mass nonviolent campaigns in the past century deploy this strategy. We theorize four mechanisms linking dilemma actions to nonviolent activist campaign success: facilitating group formation, delegitimizing opponents, reducing fear, and generating sympathetic media coverage. Finally, we assess whether dilemma actions increase campaign success rates, finding that dilemma actions are associated with an increase of 11–16 percent in activist-campaign success.
Publisher: Johns Hopkins University Press

On the Right Tack: Reflections on Coalition-Building Initiatives across The Asia Foundation - Nixon, Nicola; Yates, Peter; Saluja, Sumaya; Yi, Su Lae; Lucas, Miranda; Bain, Katherine (The Asia Foundation, 2023)
Coalitions—groups of organizations and individuals that work together to pursue a common policy goal or reform—are crucial to development. Some of The Asia Foundation’s longest-standing and most successful development programs and portfolios have used coalition-building as an implementation modality. This paper examines successful initiatives in Bangladesh, Nepal, Philippines, Sri Lanka, Thailand, and Timor-Leste. By delving into each of these, we shed light on this coalition-building modality, sharing insights about how externally supported coalitions emerge and function and contributing ideas about how such support can be adapted to other contexts. We hope that this paper and these case studies contribute to innovations in contemporary development theory and help practitioners use and move beyond some formulaic and technocratic modalities, such as traditional capacity development, knowledge transfer, and technical assistance. People, relationships, and working partnerships can then take center stage, and sustained cooperation will gain primacy over short-term outputs and deliverables.

Developing a Systems Thinking Lens for Collective Leadership - O’Donnell, Joan (Collective Leadership for Scotland, 2023)
Systems thinking approaches are gaining traction as an effective way of understanding and working with increasing complexity. They are being put forward by the Organisation for Economic Co-operation and Development as well as the World Health Organisation as a way to tackle the complex and unpredictable environments we now operate in. As the world has become increasingly interconnected, national or local boundaries cannot isolate and control social problems. The climate emergency, war and political instability have become everyday realities that accompany an ever-widening gap between those who have and control resources, and those who do not have enough to meet basic needs. The usual responses are becoming more and more ineffective; all interventions have unseen consequences or emergent properties that cannot be predicted using an inputoutput outcomes model. Designing interventions therefore requires a fresh lens to manage our way through turbulence and uncertainty. Most complex situations benefit from a complementary mix of a systematic (linear) decisive intervention as much as a systemic (whole-view) understanding of a proposed intervention. In this sense, adopting a systems thinking approach requires both framing the situation using a systems lens, as well as a commitment to take action and learn forward in realtime. When utilised in this way, a systems thinking lens complements rather than replaces traditional management tools. It draws on well-tested concepts and tools and relies on the systems leader to develop their practice, which includes the art of knowing what is needed and when. It is about threading traditional planning methods together with an understanding of the interrelationships, multiple perspectives and boundary judgements that influence the framing of a situation. It is becoming increasingly clear that effective leadership is embedded and invested: objective management science has no place in a world bound by interconnectedness and unintended consequences. Leaders are, as we shall see, an inherent part of both the problem and the proposed intervention. The purpose of this paper is to share some concepts informed by systems thinking to support you, as leaders in your organisations, to bring a systems-informed lens to your work. It puts you as a practitioner-leader at the centre of your own practice and encourages you to reflect critically on your positionality, as well as the lens you use to understand and intervene in complex issues. This paper has been developed on foot of training piloted with public service leaders in Scotland in the summer of 2022, and acts as an introduction to developing your systems literacy skills. While we are all born with a systems sensibility, many of us lose that sense of connectedness and interdependence with the world as we grow up. This paper acts as a prompt to bring a complexity informed approach to your work practice. It also contains some pointers for further resources and tools that may support your learning.

Re-institute website - RE!NSTITUTE (Re!nstitute, 2023)
We are a dynamic non-profit organization with a unique approach to managing change and catalyzing innovation to support global systems transformation. We know that to achieve equal, just, and safe communities, we must collectively challenge and change the systems that people rely on — especially those most vulnerable. Together, we must RE!BUILD a system's resilience and capacity to take on our biggest generational challenges. Justice, housing, health, and other systems in every country are often struggling to come together, across institutions and organizations, to unite under a common vision for change, and implement those changes in effective and innovative ways. Our work and methodology — the 100-Day Challenge — are designed to support these systems to achieve extraordinary results. Our approach is rooted in the philosophy that frontline staff, supported by leadership, can unleash innovation and achieve incredible results by creating new relationships, experimenting with fresh ideas from across their system, and building authentic engagement with those with lived experience.

Using AI to disrupt business as usual in small evaluation firms - Sabarre, Nina R.; Beckmann, Blake; Bhaskara, Sahiti; Doll, Kathleen (New Directions for Evaluation, 2023)
While many knowledge workers may fear that the rise of artificial intelligence (AI) will threaten their jobs, this article argues that small evaluation businesses should embrace AI tools to increase their value in the marketplace and remain relevant. In this article, consultants from a research, evaluation, and strategy firm, Intention 2 Impact, Inc., make a case for using AI tools to disrupt business as usual in evaluation from theoretical and practical perspectives. Theoretically, AI may be another example of technology that was initially feared but is now ubiquitous in society. Using concrete examples, the authors describe how businesses and evaluators have evolved to keep up with changes in supply and demand. Lastly, it is posited that embracing AI will save time for those working in small businesses, which can ultimately increase added value and profitability.
_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ev.20562

Propel - Strhive (2023)
Learn from experience. Together. In the complex landscape of international development, organisations need a way to learn from their experiences and build on what works. Propel is the software solution that revolutionises the way organisations capture, access, and reuse learnings to adapt, innovate, and create lasting change on a global scale. Let's stop reinventing the wheel and build on what works, together.

MSD Competency Framework - BEAM Exchange (BEAM Exchange, 2023)
MSD Competency Framework A catalogue of the full range of knowledge, skills and aptitudes found in high-performing teams that use the market systems approach. Useful for: Practitioners - identify personal training needs and continue your professional development Trainers - diversify your courses and refine your training / capacity-building offers Managers - strengthen your recruitment and induction processes MSD competencies Competency is a mix of knowledge, skills and attitudes demonstrated through concrete behaviours. This section explores the 17 competencies used by high-performing MSD teams. Arranged in three groups they relate to how we understand the world, make decisions and interact with others. Each competency is defined and has links to useful resources that explain or illustrate the knowledge or skills that feed it. Where available, sources of teaching and learning for that competency are included. Teaching and learning modes An exploration of the most common modes of teaching and learning that support practitioners to acquire knowledge, skills and attitudes. Some modes of learning are more relevant to certain types of competency - as indicated by the box colours. Each page has a summary of the teaching mode, guidance for team leaders and trainers, and examples of how to use this mode for developing specific competencies. Assessment modes Assessing competency is important for recruitment, performance appraisal and continued professional development These pages describe distinct approaches to evaluating individuals. They include a summary of the evaluation mode, guidance for assessors, and examples tailored to specific competencies.

Holistic Flexibility for Deploying Systems Thinking as a Cognitive Skill - Chowdhury, Rajneesh (Systemic Practice and Action Research, 2023)
Considering Systems Thinking (ST) as a cognitive skill can create greater acceptability of and openness to the discipline from practitioners and researchers outside operations research and management science. Rather than associating ST with frameworks and methodologies, ST as a cognitive skill can help popularize and democratize the discipline. This paper highlights how the conceptual lens of Holistic Flexibility can help practitioners deploy ST as a cognitive skill without the application of any traditional systems methodology. Holistic Flexibility is defined as the dynamic interplay between a state of mind that has the ability to absorb systemic complexity and a state of practice that has the ability to embrace flexibility, both in intent and in form. Through two case-studies, discussions in this paper highlight how Holistic Flexibility can serve as a conceptual lens for systems practitioners. The case-studies demonstrate the importance of a practitioner’s ability to seamlessly manage and work with multiple variables, stakeholders, and factors to deliver responsible outcomes with the aid of learning loops. The main contribution of this paper lies in the case-studies and analyses presented that provide use cases for Holistic Flexibility in ST, which will help address recent calls in the discipline for ST to be considered as a cognitive skill.

Strengthen Your Team’s CLA Practices: Introducing the CLA Maturity Tool for USAID Implementers - Social Impact , Inc (2023)
The Collaborating, Learning & Adapting (CLA) Maturity Tool has been used by USAID operating units since 2015. The tool enables staff to self-assess their current CLA practice and plan ways to improve their internal and external collaboration, organizational learning efforts, and adaptive management practices. Recently, SI built on this resource to create the CLA Maturity Tool for implementing partners (IPs) to provide a version that speaks more directly to the experience of international and local organizations. Learn more here.

Introducing the CLA Maturity Tool for Implementing Partners - Salib, Monalisa; Ziegler, Jessica (Social Impact, 2023)
On December 13, 2022, Social Impact hosted a webinar in our Evidence for Impact series, "Strengthen Your Team's CLA Practices: Introducing the CLA Maturity Tool for USAID Implementers." Through the session, we shared more about the origin and history of the USAID tool upon which this version is based, why and how we updated
Section: Uncategorized

A Better Approach to After-Action Reviews - Fletcher, Angus; Cline, Preston B.; Hoffman, Matthew (Harvard Business Review, 2023)
In the decades since the Army created the After Action Review (AAR), businesses have embraced the practice as a way of learning from both failure and success. But all too often the practice gets reduced to nothing more than a pro forma exercise. The authors of this article describe the history and philosophy of the original AAR, debunk three myths about the practice that impede its proper use, and finally suggest three improvements that can help business leaders make the most of it.
Section: Collaboration and teams
Keywords: Collaboration and teams; Crisis management; Project management

Innovative M&E from the Sandbox and beyond - Vester, Søren; Tran, Samuel (Medium, 2023)
In this blog we are sharing a digest of some of the many useful and innovative monitoring, evaluation and learning resources and efforts that have come through the M&E Sandbox in 2022. A lot of these resources have been shared by our community in response to the overwhelmingly positive feedback from the launch of the Sandbox (please keep them coming!). We hope you find it useful. We have grouped these efforts and resources under six broad questions: - How do we measure systems transformation? - How do we know if we are on track? - How do we rethink complexity and independence in evaluation? - Why, how and for whom do we measure? - How do we generate insights and learn? - How do we make decisions and adapt?

Two Loops Guide - Colchester, Joss (Si Network, 2023)
The Two Loop model is a nonlinear theory of change based upon the ideas of living systems created by Margaret Wheatley & Deborah Frieze. It provides us with insight into the simultaneous growth and decline process that are underway within a system during a transition period. In this way, it provides a heuristic for us to better try and recognize and connect the past to the future during a change process. It likewise gives us the possibility to try and come to some consensus about where we might be as a group in this process of change and the best actions to take at different stages. This guide will be of relevance for anyone involved in a complex organizational change process.

Understanding Political Economy Analysis and Thinking and Working Politically - Whaites, Alan; Piron, Laure-Hélène; Menocal, Alina Rocha; Teskey, Graham (FCDO and TWP CoP, 2023)
This guide is adapted from work by the UK Foreign, Commonwealth and Development Office (FCDO) with inputs from members of the Thinking and Working Politically Community of Practice (TWP CoP). It outlines how to understand and use a set of analytical tools that are collectively known as Political Economy Analysis (PEA). The guide aims to equip practitioners to act in an informed manner, given that development objectives are invariably politically complex, and entail engaging with counterparts’ political incentives and preferences. The guide summarises different types of tools – from very light-touch to more in-depth approaches – and provides advice on how development professionals can decide what is most appropriate in a given context, with illustrations based on the experiences of teams working on these issues. This guide will help development professionals and others to make use of PEA and to apply it to their own specific needs. The first part of the guide offers a general picture of the approach. The second part provides more specific guidance for those who are tasked with deploying a PEA. Contents –> Main audience What is PEA, its role and purpose (Section 2) –> General information for all readers The main elements of PEA (Section 3) Thinking and Working Politically (Section 4) –> Core information for teams planning and using PEA How to ensure quality (Section 5) –> Essential reading for those directly responsible for a PEA Important concepts and terminology (Annex) –> General information for all readers

Celebrating Adaptive Delivery: A View from the Frontline in Myanmar - Barnes, Katrina; Lonsdale, Jane (IDS, 2023)
The conversation on adaptive management has grown fast amongst development actors. These conversations often focus on designing, commissioning, and managing large-scale development programmes. Exactly how this impacts the frontline, the implementers, and day-to-day project delivery is still being debated. Yet, perspectives drawn directly from practice are often largely missing within these debates. This paper is written by two development practitioners. Through this paper, we reflect on the difference between adaptive management and adaptive delivery, and how this interacts with risk and aid accountability, particularly in contexts of fragility. Drawing on examples of Oxfam in Myanmar work and our personal insights in relation to delivering programming across humanitarian, peace-building, and development, we suggest that in complex, conflict-affected, and highly political environments adaptive delivery already happens far more regularly than is currently recognised, as a necessity to get activities delivered. However, it happens despite the system, not because of it, and is therefore often hidden and carried out ‘under the radar’ rather than celebrated as a success in difficult environments. This paper was written in 2019, before the military seized control of Myanmar in February 2021. Whilst it draws on examples from pre-2021 Myanmar to illustrate real life cases, it is a contribution to a broader global debate on adaptive management in practice, specifically in fragile contexts. This is not specifically aimed at practitioners working in Myanmar at present, who are now working in a protracted crisis. This paper makes tangible recommendations on steps that donors, international non-governmental organisations, local staff, and partners could take to promote a system of encouraging and celebrating adaptability in programme delivery in fragile contexts.
Accepted: 2023-02-02T14:48:03Z Publisher: Institute of Development Studies

Don't Lead Alone: Think Like a System, Act Like a Network, Lead Like a Movement! - Justis, Cleveland; Student, Daniel (Fast Company Press, 2023)
Think. Act. Lead.It seems simple enough. But understanding your desired impact and how it fits into a larger picture, connecting your work to others and finding new collaborators, and bringing those collaborators together and moving them in a unified direction is never easy.Governments, businesses, and nonprofits all have unique approaches and ideas that many of us learn through our work. Yet, we rarely consider the skills needed to create and maintain the partnerships between them. Most of us learn those skills through trial, error, and often, failure. Worse, we typically stay in our self-reinforcing silos, sharing perspectives and frustrations with like-minded people, limiting our vision of what our work can become. By partnering with other sectors, we combine and adapt approaches to solve complex problems, and leaders in any industry can create large-scale change.Cleveland Justis and Daniel Student share a road map for effective partnerships that increase impact and profitability. Using real-life examples and practice exercises, the authors teach how to acquire and use skills to solve complex problems and propel your organization forward by combining a multitude of perspectives, split into three sections:Think Like a SystemAct Like a NetworkLead Like a MovementIt’s time to get out of our silos. Don’t lead alone.

How Big Things Get Done: The Surprising Factors Behind Every Successful Project, from Home Renovations to Space Exploration - Flyvbjerg, Bent; Gardner, Dan (Macmillan, 2023)
World expert Bent Flyvbjerg and bestselling author Dan Gardner reveal the secrets to successfully planning and delivering ambitious projects on any scale.Nothing is more inspiring than a big vision that becomes a triumphant new reality. Think of how Apple’s iPod went from a project with a single employee to an enormously successful product launch in eleven months. But such successes are the exception. Consider how London’s Crossrail project delivered five years late and billions over budget. More modest endeavours, whether launching a small business, organizing a conference, or just finishing a work project on time, also commonly fail. Why?Understanding what distinguishes the triumphs from the failures has been the life’s work of Oxford professor Bent Flyvbjerg. In How Big Things Get Done, he identifies the errors that lead projects to fail, and the research-based principles that will make yours succeed:- Understand your odds. If you don’t know them, you won’t win.- Plan slow, act fast. Getting to the action quick feels right. But it’s wrong.- Think right to left. Start with your goal, then identify the steps to get there.- Find your Lego. Big is best built from small.- Master the unknown unknowns. Most think they can’t, so they fail. Flyvbjerg shows how you can.Full of vivid examples ranging from the building of the Sydney Opera House to the making of the latest Pixar blockbusters, How Big Things Get Done reveals how to get any ambitious project done – on time and on budget.

The Big Con: How the Consulting Industry Weakens our Businesses, Infantilizes our Governments and Warps our Economies - Mazzucato, Mariana; Collington, Rosie (Allen Lane, 2023)
There is an entrenched relationship between the consulting industry and the way business and government are managed today which must change.Mariana Mazzucato and Rosie Collington show that our economies' reliance on companies such as McKinsey, Boston Consulting Group, Bain & Company, PwC, Deloitte, KPMG and EY stunts innovation, obfuscates corporate and political accountability and impedes our collective mission of halting climate breakdown.The 'Big Con' describes the confidence trick the consulting industry performs in contracts with hollowed-out and risk-averse governments and shareholder value-maximizing firms. It grew from the 1980s and 1990s in the wake of reforms by both the neoliberal right and Third Way progressives, and it thrives on the ills of modern capitalism, from financialization and privatization to the climate crisis. It is possible because of the unique power that big consultancies wield through extensive contracts and networks - as advisors, legitimators and outsourcers - and the illusion that they are objective sources of expertise and capacity. To make matters worse, our best and brightest graduates are often redirected away from public service into consulting. In all these ways, the Big Con weakens our businesses, infantilizes our governments and warps our economies.Mazzucato and Collington expertly debunk the myth that consultancies always add value to the economy. With a wealth of original research, they argue brilliantly for investment and collective intelligence within all organizations and communities, and for a new system in which public and private sectors work innovatively for the common good. We must recalibrate the role of consultants and rebuild economies and governments that are fit for purpose.

Reflections on Ten Years of USAID’s Experience with Political Economy Analysis and Thinking and Working Politically - Brinkerhoff, Derick; Cassidy, Marc (TWP CoP, 2023)
The global Thinking and Working Politically (TWP) Community of Practice (CoP), the Washington DC TWP CoP, and USAID organised a webinar on 5 December 2022 to take stock of how USAID and its partners have used PEA to inform programme strategy, design and implementation, and support TWP. This paper synthesises the key points arising from the webinar, including observations on the impacts, opportunities, challenges, and prospects for PEA/TWP to become more deeply adopted and sustained as a development methodology and approach across sectors. It starts by defining key concepts. It then highlights insights from the discussions of the impact of the application of PEA and TWP principles across sectors. The paper concludes by looking at progress achieved to date, as well as constraints and opportunities to increase the uptake of both thinking and working politically in USAID-sponsored programming.

Adaptive Evaluation: A Complexity-based approach to Systematic Learning for Innovation and Scaling in Development - Gokhale, Siddhant; Walton, Michael (Center for International Development, Harvard University, 2023)
Nearly all challenges in international development tend to be complex because they depend on constantly evolving human behaviour, systems, and contexts, involving multiple actors, entities, and processes. As a result, both the discovery and scaling of innovations to address challenges in development often involve changes in system behaviour or even system-level transformation. This is rarely a linear process over time and can result in unexpected outcomes. Existing evaluation techniques commonly used in international development, including Randomized Control Trials (RCT) and quasi-experimental methods, are good at assessing specific effects of interventions but are not designed for the change processes inherent to innovation and scaling within a system. There is a need to reconstruct how we use existing measurement tools, techniques, and methodologies so that they capture the complexity of the environment in which an intervention or change occurs. We introduce Adaptive Evaluation, designed to learn at various levels of complexity while supporting the transformation needed to foster sustainable change. An Adaptive Evaluation uses three main approaches to work with complex questions—systems diagnosis, theorybased assessment of change processes, and iterative designs. An Adaptive Evaluation typically builds hypotheses from field-based interactions, emphasizes learning over testing, advocates open-mindedness with techniques, and appreciates the value of dialogue and participation in navigating complex processes. It can use RCT or similar techniques to analyse specific processes within a system or a development cycle, but these are embedded in a broader approach to assessment and interpretation. It is designed to be flexible and adjust to shifting contexts. Finally, an Adaptive Evaluation can be applied at any stage in a complex intervention's lifecycle, from the interpretation of the system and change processes to rapid experimentation, prototyping, and testing of select interventions, and then adaptation to different settings for impact at scale. This paper provides the theoretical basis for an Adaptive Evaluation—the main approaches, core ideology, process, and applications.

Glass Half Full: Civic Space and Contestation in Bangladesh, Sri Lanka and Nepal - Surie, Mandakini D.; Saluja, Sumaya; Nixon, Nicola (The Asia Foundation, 2023)
The past decade has witnessed a surge of interest in and concern over the global trend toward democratic regression. In South Asia, regulatory and institutional frameworks have become increasingly restrictive, curbing the ability of citizens and civil society organizations to occupy and use civic spaces to organize, express themselves, and participate in decisions that affect the lives of people whose interests they serve. Of course, this is only one half of the story. The other half is how citizens, collectives, and organizations adapt by carving out spaces where they can maintain—or even expand—the boundaries of their engagement in local and national civic spaces. In this paper, we examine how civic spaces are evolving in three South Asian countries—Bangladesh, Sri Lanka, and Nepal—drawing on the experiences and perspectives of civil society representatives. The paper concludes with recommendations for donors and development practitioners on how they can make relevant investments that will strengthen civic spaces and support democratic resilience in the region.

The Secret Sauce of Development Professionals: Tools for Assessing TOR Potential to Source Scalable Learning Interventions - Barton, Adam (Research on Improving Systems of Education (RISE), 2023)
Terms of reference (TORs) play an outsized role in driving scalable educational programming. These procurement documents shape, constrain, and signal programme priorities and possibilities. Successful funders and implementers across the globe hold rich processual knowledge about this documentation, which they use to draft and assess TORs. This project explores such best-practice knowledge around TOR review, seeking to support the design and implementation of educational programmes that can improve learning at scale in developing contexts.

Process Tracing Innovations in Practice: Finding the Middle Path - Aston, Thomas; Wadeson, Alix (Institute of Development Studies, 2023)
Evaluation practitioners in the international development sector have given considerable attention in recent years to process tracing as a method for evaluating impact, including discussion of how to assess the relative importance of causal factors. Despite the increasing interest, there is a relative dearth of examples of practical learning and evidence of applying process tracing in practice. This CDI Practice Paper draws on comparative learning from applying three different types of process tracing in international development initiatives. It argues in favour of a ‘middle path’ of applying evidence tests and rubrics to structure evaluative judgements rather than formal Bayesian updating or looser forms of process tracing. It also calls attention to the potential added value of taking a participatory approach, offering practical recommendations for how to do this effectively.
Accepted: 2023-03-27T15:20:09Z Publisher: Institute of Development Studies

The Last 10 Per Cent: Why the World Needs a Leaner, More Innovative and Pragmatic Development Sector, Today - Harper, Erica (Routledge India, 2023)
Criticism that the development sector has not delivered in terms of eliminating extreme poverty, fast-tracking growth and preventing conflict, is neither new nor surprising. In fact, it may be the one thing that scholars, donors and practitioners agree on. While many of these concerns are valid, this book makes a case that the sector is closer to unlocking the gates to more effective and efficient development outcomes than is popularly believed. Specifically, it argues that by overturning a few myths, making better use of evidence and employing some different rules, practitioners, policy specialists and donors can foster the changes in the development architecture that are needed to reach the 10 percent of the world’s population still living in extreme poverty.Engaging, provocative and clear sighted, the book provides insight into interventions around democratic governance, refugee response, counterterrorism, gender mainstreaming, environmental protection and private sector engagement. It is instructive reading for professionals across the development sector, think tanks and NGOs.

Berkana Two Loop Model: A quick introduction to an accessible model for systems change - Proud, Emma (LinkedIn - Emma Proud, 2023)
A model I've been using a lot recently, to bring systems thinking to life, is the Berkana Two Loop model. It doesn't describe complexity or systems thinking. But it does describe systems change in a way that's simple and oriented to action.

Revealing the Relational Mechanisms of Research for Development Through Social Network Analysis - Apgar, Marina; Fournie, Guillaume; Haesler, Barbara; Higdon, Grace Lyn; Kenny, Leah; Oppel, Annalena; Pauls, Evelyn; Smith, Matthew; Snijder, Mieke; Vink, Daan; Hossain, Mazeda (The European Journal of Development Research, 2023)
Achieving impact through research for development programmes (R4D) requires engagement with diverse stakeholders across the research, development and policy divides. Understanding how such programmes support the emergence of outcomes, therefore, requires a focus on the relational aspects of engagement and collaboration. Increasingly, evaluation of large research collaborations is employing social network analysis (SNA), making use of its relational view of causation. In this paper, we use three applications of SNA within similar large R4D programmes, through our work within evaluation of three Interidsiplinary Hubs of the Global Challenges Research Fund, to explore its potential as an evaluation method. Our comparative analysis shows that SNA can uncover the structural dimensions of interactions within R4D programmes and enable learning about how networks evolve through time. We reflect on common challenges across the cases including navigating different forms of bias that result from incomplete network data, multiple interpretations across scales, and the challenges of making causal inference and related ethical dilemmas. We conclude with lessons on the methodological and operational dimensions of using SNA within monitoring, evaluation and learning (MEL) systems that aim to support both learning and accountability.

Evaluating Research for Development: Innovation to Navigate Complexity - Apgar, Marina; Snijder, Mieke; Higdon, Grace Lyn; Szabo, Sylvia (The European Journal of Development Research, 2023)
Large publicly funded programmes of research continue to receive increased investment as interventions aiming to produce impact for the world’s poorest and most marginalized populations. At this intersection of research and development, research is expected to contribute to complex processes of societal change. Embracing a co-produced view of impact as emerging along uncertain causal pathways often without predefined outcomes calls for innovation in the use of complexity-aware approaches to evaluation. The papers in this special issue present rich experiences of authors working across sectors and geographies, employing methodological innovation and navigating power as they reconcile tensions. They illustrate the challenges with (i) evaluating performance to meet accountability demands while fostering learning for adaptation; (ii) evaluating prospective theories of change while capturing emergent change; (iii) evaluating internal relational dimensions while measuring external development outcomes; (iv) evaluating across scales: from measuring local level end impact to understanding contributions to systems level change. Taken as a whole, the issue illustrates how the research for development evaluation field is maturing through the experiences of a growing and diverse group of researchers and evaluators as they shift from using narrow accountability instruments to appreciating emergent causal pathways within research for development.

Theory of Change in complex Research for Development programmes: challenges and solutions from the Global Challenges Research Fund - Chapman, Sarah; Boodhoo, Adiilah; Duffy, Carren; Goodman, Suki; Michalopoulou, Maria (The European Journal of Development Research, 2023)
The United Kingdom Research and Innovation (UKRI) Global Challenges Research Fund (GCRF) aimed to address global challenges to achieve the United Nations (UN) Sustainable Development Goals through 12 interdisciplinary research hubs. This research documents key lessons learned around working with Theory of Change (ToC) to guide Monitoring, Evaluation and Learning (MEL) within these complex research for development hubs. Interviews and document reviews were conducted in ten of the research hubs. The results revealed that only one hub invested in an explicit visual system mapping approach, and that funder timelines, budget constraints and issues with capacity and expertise limited the application of these approaches across all hubs. In contrast, many hubs attempted to deal with visual complexity by means of ether constructing multiple, nested ToCs, or a conscious simplification of complexity through reducing their ToC towards a straightforward and uncomplicated chain model or spherical model. While the former approach had some value, most hubs struggled to find capacity to support the full articulation of nested ToCs. In contrast, the latter approach resulted in ToCs which lacked detail or mechanism articulation, but which nevertheless were often ‘fit for purpose’ in ensuring effective communication and coherence across diverse stakeholders and sub-projects. We conclude that in instances where the reporting, funding and management cycles of complex research for development programmes cannot be adapted to properly support learning-based approaches to ToC development, imposing simplicity in the ToC might be fit for purpose. This might also be preferable to more complex visual approaches that are only partially realised.

Cost–Benefit Analysis (CBA) or the Highway? An Alternative Road to Investigating the Value for Money of International Development Research - Peterson, Heidi (The European Journal of Development Research, 2023)
Research for development (R4D) funding is increasingly expected to demonstrate value for money (VfM). However, the dominance of positivist approaches to evaluating VfM, such as cost-benefit analysis, do not fully account for the complexity of R4D funds and risk undermining efforts to contribute to transformational development. This paper posits an alternative approach to evaluating VfM, using the UK’s Global Challenges Research Fund and the Newton Fund as case studies. Based on a constructivist approach to valuing outcomes, this approach applies a collaboratively developed rubric-based peer review to a sample of projects. This is more appropriate for the complexity of R4D interventions, particularly when considering uncertain and emergent outcomes over a long timeframe. This approach could be adapted to other complex interventions, demonstrating that our options are not merely “CBA or the highway” and there are indeed alternative routes to evaluating VfM.

How are Research for Development Programmes Implementing and Evaluating Equitable Partnerships to Address Power Asymmetries? - Snijder, Mieke; Steege, Rosie; Callander, Michelle; Wahome, Michel; Rahman, M. Feisal; Apgar, Marina; Theobald, Sally; Bracken, Louise J.; Dean, Laura; Mansaray, Bintu; Saligram, Prasanna; Garimella, Surekha; et al. (The European Journal of Development Research, 2023)
The complexity of issues addressed by research for development (R4D) requires collaborations between partners from a range of disciplines and cultural contexts. Power asymmetries within such partnerships may obstruct the fair distribution of resources, responsibilities and benefits across all partners. This paper presents a cross-case analysis of five R4D partnership evaluations, their methods and how they unearthed and addressed power asymmetries. It contributes to the field of R4D partnership evaluations by detailing approaches and methods employed to evaluate these partnerships. Theory-based evaluations deepened understandings of how equitable partnerships contribute to R4D generating impact and centring the relational side of R4D. Participatory approaches that involved all partners in developing and evaluating partnership principles ensured contextually appropriate definitions and a focus on what partners value.

Laying the Foundations for Impact: Lessons from the GCRF Evaluation - Vogel, Isabel; Barnett, Chris (The European Journal of Development Research, 2023)
Research for development (R4D) aims to make a tangible difference to development challenges, but these effects typically take years to emerge. Evaluation (especially impact evaluation) often takes place before there is evidence of development impact. In this paper, we focus on opportunities for assessing the potential for impact at earlier stages in the research and innovation process. We argue that such a focus can help research programme managers and evaluators learn about the pre-conditions for impact and adjust accordingly. Using the Global Challenges Research Fund (GCRF) as a large-scale case of R4D evaluation, we identify and explore some of the building blocks that can increase impact potential. Guided by GCRF’s theory of change, we explore emerging evidence that highlights the importance of ways of working that supports positioning for impact. We conclude by drawing out a unifying construct around standards of development excellence; to sit alongside notions of scientific excellence for research intended to have an impact. Standards can help programme managers, researchers and evaluators learn and adapt to increase the likelihood of impact.

Building Capacity for Strategic Innovation: an Emerging Competency Framework for Portfolio Work - Begovic, Milica; Colville, Jenniffer; Quaggiotto, Giulio; Vester, Søren; Naatujuna, Deborah; Oprunenco, Alex; Sadiku; Lejla; Uusikyla, Ida; Uriartt, Simone; Zorouali, Kawtar (Medium, 2023)
In a previous post we reflected on how a key learning from our “project to portfolio” journey to date is that it is ultimately about mustering the organizational will to transform. If the early days of our innovation work were about demonstrating results quickly and creating space for experimentation, now the challenge is of a different order. Eventually this means helping UNDP transition to a different value proposition and business model, as eloquently articulated by Gerd Trogemann: “No matter how well conceived and relevant in their own right, projects tend to pursue single point rather than systemic solutions, limit strategic space and the ability to adapt continuously and to connect the dots systemically. Systemic solutions need adaptive ways of working, strategic space, iterative learning, and radical collaboration.”

A new pathway: how can funders support meaningful monitoring, evaluation, and learning practice in the field? - Blog post on Better Evaluation - Acevedo, Andrea; Colnar, Megan (2023)
How can donors and grantees work together to create effective monitoring, evaluation, and learning (MEL) practices that drive field-wide transformation?

Participatory action research - Cornish, Flora; Breton, Nancy; Moreno-Tabarez, Ulises; Delgado, Jenna; Rua, Mohi; de-Graft Aikins, Ama; Hodgetts, Darrin (Nature Reviews Methods Primers, 2023)
Participatory action research (PAR) is an approach to research that prioritizes the value of experiential knowledge for tackling problems caused by unequal and harmful social systems, and for envisioning and implementing alternatives. PAR involves the participation and leadership of those people experiencing issues, who take action to produce emancipatory social change, through conducting systematic research to generate new knowledge. This Primer sets out key considerations for the design of a PAR project. The core of the Primer introduces six building blocks for PAR project design: building relationships; establishing working practices; establishing a common understanding of the issue; observing, gathering and generating materials; collaborative analysis; and planning and taking action. We discuss key challenges faced by PAR projects, namely, mismatches with institutional research infrastructure; risks of co-option; power inequalities; and the decentralizing of control. To counter such challenges, PAR researchers may build PAR-friendly networks of people and infrastructures; cultivate a critical community to hold them to account; use critical reflexivity; redistribute powers; and learn to trust the process. PAR’s societal contribution and methodological development, we argue, can best be advanced by engaging with contemporary social movements that demand the redressingl of inequities and the recognition of situated expertise.
Number: 1 Publisher: Nature Publishing Group

The adoption of innovation in international development organisations: lessons for development co-operation - Kumpf, Benjamin; Jhunjhunwala, Parnika (OECD Publishing, 2023)
Addressing 21st century development challenges requires investments in innovation, including the use of new approaches and technologies. Currently, many development organisations prioritise investments in isolated innovation pilots that leverage a specific approach or technology rather than pursuing a strategic approach to expand the organisation’s toolbox with innovations that have proven their comparative advantage over what is currently used. This Working Paper addresses this challenge of adopting innovations. How can development organisations institutionalise a new way of working, bringing what was once novel to the core of how business is done? Analysing successful adoption efforts across five DAC agencies, the paper lays out a proposed process for the adoption of innovations. The paper features five case-studies and concludes with a set of lessons and recommendations for policy makers on innovation management generally, and adoption of innovation in particular.

USAID ADS 201 - Operational Policy for the Program Cycle (Update 05/22/2023) - USAID (USAID, 2023)
The Program Cycle is USAID’s operational model for planning, delivering, assessing, and adapting development programming in a given region or country to advance U.S. foreign policy. It encompasses guidance and procedures for: 1) Making strategic decisions at the regional or country level about programmatic areas of focus and associated resources; 2) Designing supportive projects and/or activities to implement these strategic plans; and 3) Learning from performance monitoring, evaluations, and other relevant sources of information to make course corrections as needed and inform future programming.

CLARISSA Cash Plus: Innovative Social Protection in Bangladesh - Roelen, Keetie; Howard, Neil; Afroze, Jiniya; Aktar, Afrin; Ton, Giel; Huq, Lopita (Institute of Development Studies, 2023)
Social protection, and cash transfers especially, have been found to have many positive impacts on families’ lives and are now widely recognised as a cornerstone of any prosperous, fair society. The CLARISSA Cash Plus intervention is an innovative social protection scheme for tackling social ills, including the worst forms of child labour (WFCL). Combining community mobilisation, case work and cash transfers, it aims to support people in a poor neighbourhood in Dhaka to build their individual, family, and group capacities to meet their needs. An increase in capacities is expected to lead to a corresponding decrease in deprivation and community-identified social issues that negatively affect wellbeing, including WFCL.
Accepted: 2023-06-30T11:14:55Z Publisher: Institute of Development Studies

Participation for Humanitarian Innovation - Background Paper - Smith, Amy; Thompson, Martha; Saida Benhayoune; Crespo Cardona, Omar (Elrha, 2023)
A resource designed to help organisations, teams and individuals manage innovation journeys responsibly and successfully. We have partnered with MIT D-Lab to develop a new resource to drive greater diversity and inclusion within project design and implementation. The Participation for Humanitarian Innovation (PfHI) toolkit sets out a robust approach to setting expectations for and monitoring the degree of participation within research and innovation projects for, with, and by people affected by crisis. The PfHI toolkit is composed of five tools: Opportunity Adviser: Identify and prioritise the desired benefits of participation while weighing the potential barriers. Participation Matrix: Agree on the precise degree of participation to target at a given project stage. Resource Navigator: Select tools and processes to address the needs of stakeholders, the project objectives and context. Quality Guidance: Ensure engagements are delivered to the highest standard. Assessment Matrix: Collectively evaluate the degree of participation achieved during an activity or project phase, learn and adapt. By applying these tools before, during and after a research and/or innovation project, implementors can ensure that stakeholders/end-users are included and participating at the highest possible degree. We encourage users of the PfHI toolkit to consider how else to integrate the tools into existing practices. For instance, the Evaluation Matrix could be used to supplement existing MEAL activities to: Establish baselines to measure changes in participation over time. Track the degree of participation/engagement of stakeholders. Regularly assess the degree of participation. Seek feedback from participants about the degree of their participation. Monitor the progress of activities related to stakeholder engagement. Our Participation for Humanitarian Innovation toolkit represents our ongoing commitment to responsible research and innovation across our portfolio of grants and for the humanitarian sector more broadly. We hope you will consider downloading and using the toolkit on your next project.

Participation for Humanitarian Innovation - Toolkit - Smith, Amy; Thompson, Martha; Saida Benhayoune; Crespo Cardona, Omar (Elrha, 2023)
A resource designed to help organisations, teams and individuals manage innovation journeys responsibly and successfully. We have partnered with MIT D-Lab to develop a new resource to drive greater diversity and inclusion within project design and implementation. The Participation for Humanitarian Innovation (PfHI) toolkit sets out a robust approach to setting expectations for and monitoring the degree of participation within research and innovation projects for, with, and by people affected by crisis. The PfHI toolkit is composed of five tools: Opportunity Adviser: Identify and prioritise the desired benefits of participation while weighing the potential barriers. Participation Matrix: Agree on the precise degree of participation to target at a given project stage. Resource Navigator: Select tools and processes to address the needs of stakeholders, the project objectives and context. Quality Guidance: Ensure engagements are delivered to the highest standard. Assessment Matrix: Collectively evaluate the degree of participation achieved during an activity or project phase, learn and adapt. By applying these tools before, during and after a research and/or innovation project, implementors can ensure that stakeholders/end-users are included and participating at the highest possible degree. We encourage users of the PfHI toolkit to consider how else to integrate the tools into existing practices. For instance, the Evaluation Matrix could be used to supplement existing MEAL activities to: Establish baselines to measure changes in participation over time. Track the degree of participation/engagement of stakeholders. Regularly assess the degree of participation. Seek feedback from participants about the degree of their participation. Monitor the progress of activities related to stakeholder engagement. Our Participation for Humanitarian Innovation toolkit represents our ongoing commitment to responsible research and innovation across our portfolio of grants and for the humanitarian sector more broadly. We hope you will consider downloading and using the toolkit on your next project.

Mechanisms of Techno-Moral Change: A Taxonomy and Overview - Danaher, John; Sætra, Henrik Skaug (Ethical Theory and Moral Practice, 2023)
The idea that technologies can change moral beliefs and practices is an old one. But how, exactly, does this happen? This paper builds on an emerging field of inquiry by developing a synoptic taxonomy of the mechanisms of techno-moral change. It argues that technology affects moral beliefs and practices in three main domains: decisional (how we make morally loaded decisions), relational (how we relate to others) and perceptual (how we perceive situations). It argues that across these three domains there are six primary mechanisms of techno-moral change: (i) adding options; (ii) changing decision-making costs; (iii) enabling new relationships; (iv) changing the burdens and expectations within relationships; (v) changing the balance of power in relationships; and (vi) changing perception (information, mental models and metaphors). The paper also discusses the layered, interactive and second-order effects of these mechanisms.

How to Build Movements with Cyclical Patterns in Mind - Non Profit News | Nonprofit Quarterly - Algoso, Dave; Guerzovich, Florencia; Gattoni, Soledad (Nonprofit Quarterly, 2023)
The world changes too much for anyone who is invested in social change work to imagine that this work is linear and predictable. Opportunities come and go, whether caused by a pandemic or political shifts. This much most social movement leaders and activists intuitively understand. But what can be done with this realization? How might movement groups better prepare for moments of opportunity? We want to explore how we can create the changes we want to see by responding to the changes that are outside our control.

When Complex is as Simple as it Gets: Guide for Recasting Policy and Management in the Anthropocene - Roe, Emery (Institute for Development Studies, 2023)
Many readers recognise and understand that complex is about as simple as it gets for major policy and management. This guide is for those unwilling in the Anthropocene to shrink back into the older platitudes about ‘keep it simple’ and ‘not to worry, we’ll scale up the analysis later on’. This guide offers key concepts, methods, counternarratives, and analogies that recast major policy and management issues in ways that do not deny their complexity but help render them more tractable for action.
Accepted: 2023-06-06T09:51:20Z ISSN: 2040-0209

Evaluative reasoning in complexity - King, Julian (Evaluation and Value for Investment, 2023)

Value for Investment: Application and Insights. Youth Primary Mental Health and Addictions Evaluation - King, Julian; Crocket, Alicia; Field, Adrian (Dovetail Consulting, 2023)
What’s in the guide? This new document provides: An overview of the VfI approach, including the conceptual principles and processes underpinning it A worked example of the approach in action: evaluating the Youth Primary Mental Health and Addictions initiative in New Zealand Transferrable learning for others considering the use of the VfI approach.

Practitioners Guidance to Assessing Systems Change: Co-Authors Preview - Gover, Dun; Nasution, Zulka; Okutu, David; Bolder, Meghan; Henao, Lina (USAID, 2023)
Check out this video to see what’s inside our new resource: Practitioners' Guidance to Assessing Systems Change, developed by MEL Managers for MEL Managers. (Check out the Guidance here https://bit.ly/MSPMELClinics.) Hear from the authors about which parts they love the most and how this guide challenges MEL managers to assess systems change as an ongoing aspect of implementation, generating feedback that teams need to better understand and catalyze change, for more impact.

Guide to Innovation Partnerships – A United Nations Ghide - Halse, Michelle; Ginsberg-Fletcher, Gabriella; Balbi, Luisa (UN Global Pulse, 2023)

Applying Adaptive Management in a Fragile Context – Case Study - Lonsdale, Jane; Green, Duncan; Robertson, Kelly (DT Global, 2023)
DT Global is proud to introduce our new Guidance Note: Practical Introduction to Adaptive Management There is a growing consensus around adaptive management as an effective (even necessary) approach when programs are tackling complex development problems. While there is no standard definition of adaptive management, there is general agreement that such programs need to routinely engage with and respond to program context; constantly test what works in that context; and adjust approaches, plans, and activities based on continuous learning. However, there remains a more limited body of evidence about what this looks like in practice—the enabling conditions, systems, resourcing, skills, and attitudes to effectively operationalise adaptive management. There is also limited guidance around when adaptive management is required, and to what extent—both critical and often overlooked considerations when planning for successful adaptive management. This Guidance Note draws together lessons and good practice in adaptive management from across DT Global’s diverse portfolio of donor-funded programs. It outlines our conceptual framework for adaptive management, with practical guidance on how it can be applied by our program teams. It is also designed to help our teams distinguish adaptive management from good (non adaptive) project management, consider when adaptive management is most useful on a program, and how adaptive a program (or part of a program) should be.

Applied political economy analysis for human rights programs and campaigns: A guide for practitioners (second edition) - Pact (Pact, 2023)
This updated guide provides practical guidance to practitioners in the human rights sector and beyond on how to integrate Applied Political Economy Analysis

Knowledge Retention and Transfer (KRT) - Model overview presentation - USAID (USAID, 2023)
What problem is the KRT model trying to solve? For workforces that experience continuous staff turnover, the lack of systematic knowledge transfer can often lead to: - Loss of programmatic momentum, - Duplication of efforts and frustration, and - Wasted time and resources. The Knowledge Retention and Transfer (KRT) model provides tools, processes, and practices to individuals, teams, offices, and organizations to improve knowledge handover, which in turn improves efficiency and programmatic and operational learning.

Theory of change software - Better Evaluation (2023)
There are a number of options when it comes to using software to help create a logic model. These range from generic word processing tools (Word, Powerpoint, or their Google Doc or Mac equivalents), to software that has been specifically tailored for visualising Theories of Change, like TOCO or Miradi. You should consider what resources you have to invest in software, both in terms of cost and in time to learn and use the features. If you only have a short timeframe and have simple needs, then a basic tool may suit you better than some of the more complex software available. It's important to investigate a few options and see what is going to be best for you.

Fast Forwarding to Systems Maps of Corruption: Getting to Usable Analysis More Quickly - Woodrow, Peter (CJL, 2023)
For AC practitioners, systems mapping is essential but also difficult. To lessen obstacles, Peter Woodrow proposes a scaffolded approach. An Experiment in “Fast Forwarding” Drawing upon our near-decade work on corruption, we recently decided to try an experiment: we would present “common patterns” of corruption as tentative models to adapt and add to—rather than try to teach people to do systems mapping from scratch. In this teaching experiment, each common pattern would function as a kind of “scaffolding” or framework that participants can build on to generate a cogent systems map as the basis for identifying possible points of intervention and subsequent program planning.

Using a ‘Partnership Rubric’ in Participatory Evaluations - Snijder, Mieke; Hicks, Jacky; Paul, Sukanta; Arulanantham, Amit; Apgar, Marina; Afroze, Jiniya; Karki, Shanta; Mareschal, Sophie; Prieto Martín, Pedro; Uddin, Forhad; Veitch, Helen (Institute of Development Studies, 2023)
Programmes that aim to tackle complex societal issues, such as the worst forms of child labour, require rich partnerships that bring together different perspectives. CLARISSA’s consortium partnership adopts an empowerment approach to the interventions we deliver and our ways of working together. Part of this approach involves ongoing reflection and learning about how we work together in our partnership, and how this can be adapted if needed. This learning note focuses on a method used in CLARISSA to both reflect on and strengthen how we work in partnerships – the partnership rubric. We found that using the rubric flexibly was key to mitigating some of the challenges of such a complex consortium. This included using it in different sizes of forum, with different levels of preparation. Periodically adapting it for country context and as new partners came on board also helped ensure a shared sense of our preferred ways of working as the project progressed.
Accepted: 2023-07-13T10:20:02Z Publisher: Institute of Development Studies

Agile on steroids through the RenDanHeYi - Quintarelli, Emanuele (Boundaryless, 2023)
This article offers initial insights into the overlap, positioning, shared values, and convergences between RenDanHeYi compared to agile practices.

Does the whole add more than the SUM of its parts? - Guerzovich, Florencia (Medium, 2023)
This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots.

Different kinds of rubrics - King, Julian (Evaluation and Value for Money, 2023)
Generic, analytic, holistic, orbic, asterisc, escalieric, cephalopodic

Lasting Roots: Naatal Mbay and the Integrated Finance Model in Senegal - Fowler, Ben; Courbois, Laura (USAID, 2023)
This report addresses the well-recognized evidence gap1 on the longer-term impacts created by marketdriven programming; specifically, programming influenced by market systems development (MSD) principles. It does so by presenting the findings of an ex-post study conducted three and a half years after the close of USAID’s Feed the Future Senegal Naatal Mbay Activity (hereafter Naatal Mbay) in 2019. It examines the scale and sustainability of changes resulting from Naatal Mbay’s introduction of an integrated finance model (IFM) – described in Error! Reference source not found. below – in the domestic rice sector. This study is one in a series of ex-post evaluations that are being conducted between 2023-2026 on USAID-funded MSD interventions around the world. This study focused on four questions, noted below in Figure 1. These were addressed using a mix of desk research, 122 key informant interviews with market actors and other stakeholders remotely and in Senegal, focus group discussions with 26 rice producers networks in Senegal, and a validation workshop with USAID/Senegal, implementing partner staff and market actors. Findings were analyzed leveraging the Disrupting System Dynamics (DSD) framework (see Figure 4 in the body of the report) as an analytical tool for understanding systems change.

Three out of Four Practical Lessons about right fitting MEL for Additive Effects - Guerzovich, Florencia (Medium, 2023)
This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots.

From mouthset to mindset shifts in co-creating systems change - Blomkamp, Emma; Snow, Thea; Burkett, Ingrid (Griffith Centre for Systems Innovation, 2023)

Adaptive Management Across Project Cycles: Look into Coherence in Time - Guerzovich, Florencia (Medium, 2023)
This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots. ...

Gestión de la complejidad para el impacto sistémico: respuestas a entornos VUCA y BANI - Peña-López, Ismael (SociedadRed, 2023)
Research on the Information Society, the Digital Divide and Information and Communication Technologies for development

Applying the River of Life Method to Support Reflection and Learning in Terre des hommes Nepal - Karki, Shanta; Giri, Roju; Neupane, Sudarshan; Snijder, Mieke; Apgar, Marina (Institute of Development Studies, 2023)
The RoL method is a visual narrative method that helps people tell stories of the past, present, and future. Individuals can use this method to introduce themselves in a fun and descriptive way. A group can use it to understand and reflect on the past and imagine the future of a project. Besides, it can also be used to build a shared view of a process over time while acknowledging different and perhaps contradictory perspectives. The method uses drawings rather than text, making it useful in groups that do not share a common language. Metaphors from a river are used to explore aspects of a story – such as whirlpools depicting challenges or lakes suggesting a sense of calm etc. When used in a group, it is an active method, engaging people in the process of storytelling and listening through visualising their experiences and using metaphors to explore in depth. In CLARISSA, we adapted the RoL method to document our collective understanding of the story of implementation of the programme as part of the programme’s monitoring, evaluation and learning component. The purpose was to surface the details of our process of the systemic Action Research that we are undertaking with children in the worst forms of child labour and business owners. We used the same river metaphors as is often applied when the method is used with individuals.
Accepted: 2023-09-18T10:13:55Z Publisher: Institute of Development Studies

Is the problem driven iterative adaptation approach (PDIA) a panacea for public financial management reform? Evidence from six African countries - Lawson, Andrew; Harris, Jamelia (World Development Perspectives, 2023)
This article assesses the application of the problem driven iterative adaptation (PDIA) approach to public financial management reform in six African countries. It draws on primary data collected using a mix of interviews, overt participation observations and a short survey. PDIA responds to shortcomings in orthodox approaches to reform and technical assistance in developing countries. It stresses local solutions to local problems, achieved through experimentation and adaptation. The principles of PDIA are appealing, but its empirical assessment is in its infancy. This study aims to fill part of this gap. Findings show that PDIA delivers results in the short-term, particularly in cases where there is an influential authorising agent and dedicated team. Progress was less forthcoming for reforms that required high level political buy-in from senior officials. The approach does exceptionally well to develop staff capability, transferable skills, and local empowerment to solve local problems, thus potentially benefitting future reforms.

Fractal approaches to scaling transformations to sustainability - O’Brien, Karen; Carmona, Rosario; Gram-Hanssen, Irmelin; Hochachka, Gail; Sygna, Linda; Rosenberg, Milda (Ambio, 2023)
Responses to sustainability challenges are not delivering results at the scale and speed called for by science, international agreements, and concerned citizens. Yet there is a tendency to underestimate the large-scale impacts of small-scale, local, and contextualized actions, and particularly the role of individuals in scaling transformations. Here, we explore a fractal approach to scaling sustainability transformations based on “universal values.” Universal values are proposed as intrinsic characteristics that connect humans and nature in a coherent, acausal way. Drawing on the Three Spheres of Transformation framework, we consider how enacting universal values can generate fractal-like patterns of sustainability that repeat recursively across scales. Fractal approaches shift the focus from scaling through “things” (e.g., technologies, behaviors, projects) to scaling through a quality of agency based on values that apply to all. We discuss practical steps involved in fractal approaches to scaling transformations to sustainability, provide examples, and conclude with questions for future research.
Keywords: Agency; Fractals; Relational paradigms; Scaling; Three Spheres of Transformation; Universal values

Sense-making with stakeholders and rubrics - King, Julian (Evaluation and Value for Investment, 2023)
Principles and processes

Causal mapping for evaluators - Powell, Steve; Copestake, James; Remnant, Fiona (Evaluation, 2023)
Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally i...
Publisher: SAGE PublicationsSage UK: London, England

The curious case of the realist trial: Methodological oxymoron or unicorn? - Nielsen, Steffen Bohni; Jaspers, Sofie Østergaard; Lemire, Sebastian (Evaluation, 2023)
Realist evaluation and experimental designs are both well-established approaches to evaluation. Over the past 10 years, realist trials—evaluations purposefully combining realist evaluation and experimental designs—have emerged. Informed by a comprehensive review of published realist trials, this article examines to what extent and how realist trials align with quality standards for realist evaluations and randomized controlled trials and to what extent and how the realist and trial aspects of realist trials are integrated. We identified only few examples that met high-quality standards for both experimental and realist studies and that merged the two designs.
Publisher: SAGE Publications Ltd

Foreign Aid and Its Unintended Consequences - Koch, Dirk-Jan (Routledge, 2023)
Foreign aid and international development frequently bring with it a range of unintended consequences, both negative and positive. This book delves into these consequences, providing a fresh and comprehensive guide to understanding and addressing them. The book starts by laying out a theoretical framework based on complexity thinking, before going on to explore the ten most prevalent kinds of unintended effects of foreign aid: backlash effects, conflict effects, migration and resettlement effects, price effects, marginalization effects, behavioural effects, negative spillover effects, governance effects, environmental effects, and ripple effects. Each chapter revolves around a set of concrete case studies, analysing the mechanisms underpinning the unintended effects and proposing ways in which policymakers, practitioners, and evaluators can tackle negative side effects and maximize positive side effects. The book also includes personal testimonies, a succinct overview of unintended effects, and suggestions for further reading. Providing a clear overview of what side effects to anticipate when planning, executing, and evaluating aid, this book will be an important resource for students, development practitioners, and policymakers alike.

Why are effective feedback mechanisms in cash transfers so important? - Jannat, Mahiratul (CLARISSA, 2023)
The CLARISSA Social Protection Intervention was set us as an innovative social policy intervention for tackling social ills, with a...

Participatory evaluation design and sense-making - making meaning through co-constructed lenses - King, Julian; Elliott, Kerry; Hollingsworth, Hilary (Evaluation and Value for Investment, 2023)
Based on a presentation at Australian Evaluation Society International Evaluation Conference, Brisbane, 28 September 2023. The importance of participatory approaches to evaluation is well-understood. Involving the right people and balancing voices at the stakeholder table can help to make evaluations more valid, credible, useful, and actually used. Not all evaluations, however, are done in participatory ways. This presentation shares the experience of three evaluators in two recent projects with clients and stakeholders who were unaccustomed to participatory evaluation. In both cases, new ways of working were introduced, requiring participants to come into the evaluation tent, collaborate and learn with us. Engagement processes deliberately focused on incrementally building deep stakeholder input into evaluation design - including key evaluation questions, context analysis, program theory, rubrics, methods of data gathering, and reporting formats. The resulting evaluation frameworks represent an agreed, shared set of lenses for making sense of the evidence. Through this process, relationships and trust were cemented, stakeholders understood the evaluation process and endorsed the resulting frameworks. The client and sector have become more interested, more engaged stakeholders and more savvy evaluation participants and consumers. As the evaluations progress, the evaluation design builds in ongoing stakeholder involvement in reviewing emergent footprints of system change and making evaluative judgements. Participatory approaches are many, varied, and flexible. We will share our reflections and generalisable learning from the participatory approaches we applied in these evaluations. We will invite the audience to reflect on their own practices and experiences, and provoke them to consider whether participatory evaluation is 'just' a methodological option or a foundational element of good evaluation.

How adaptive M&E from the peace sector can help demonstrate the value of aid - Kratzer, Sebastian (2023)
Over the last decade, the peace sector has been developing and adapting Monitoring and Evaluation (M&E) systems and tools to fit their contexts and ways of working. This evolution may hold some insights for the aid community in how to go beyond more traditional, backwards-looking M&E to navigate today’s volatile, interest-based world of politics and aid.

Changing theories - Aston, Thomas (Medium, 2023)
You may, or may not, be surprised to hear that many theories of change lack what we might generally understand as a theory.

Storytelling and Evaluation: Can Pathways be the Theory Missing in Explanatory Narratives? - Guerzovich, Florencia (Medium, 2023)
Florencia Guerzovich

Anatomy of Social Accountability 3.0: A working framework for moving forward - Guerzovich, Florencia; Aston, Tom (Florianopolis, SC, Brazil: Grupo Politeia, ESAG, University of the State of Santa Catarina and act4delivery, 2023)
This note presents the key components of a potential next generation of social accountability programming. The key aspects outlined emerge from a review of over 150 cases as well as tacit knowledge from dozens of practitioners (Guerzovich and Aston, 2023).The main thread of social accountability 3.0 and what distinguishes it from previous generations is a focus on its contribution towards more responsive systems and accountable social contracts. In particular, social accountability should be considered as an operational means to rework social contracts.Over the last two decades, the field has been defined by a supposed dichotomy between minimalist (i.e., “widget” based) approach or a maximalist (“go big or go home”) approaches. Yet, most practice sits somewhere between these extremes. The paper presents various options derived from practice; practitioners should choose whichever best fits their own particular circumstances.Those hoping to improve and expand their practice with a systems-lens may benefit from considering how they approach the key elements of social accountability 3.0.
Keywords: accountability; development; responsiveness; social accountability; social contract; systems

Hidden Potential: The Science of Achieving Greater Things - Grant, Adam (WH Allen, 2023)
#1 New York Times bestseller "This brilliant book will shatter your assumptions about what it takes to improve and succeed. I wish I could go back in time and gift it to my younger self. It would've helped me find a more joyful path to progress."-Serena Williams, 23-time Grand Slam singles tennis championThe #1 New York Times bestselling author of Think Again illuminates how we can elevate ourselves and others to unexpected heights.We live in a world that’s obsessed with talent. We celebrate gifted students in school, natural athletes in sports, and child prodigies in music. But admiring people who start out with innate advantages leads us to overlook the distances we ourselves can travel. We can all improve at improving. And when opportunity doesn’t knock, there are ways to build a door.Hidden Potential offers a new framework for raising aspirations and exceeding expectations. Adam Grant weaves together groundbreaking evidence, surprising insights, and vivid storytelling that takes us from the classroom to the boardroom, the playground to the Olympics, and underground to outer space. He shows that progress depends less on how hard you work than how well you learn. Growth is not about the genius you possess – it’s about the character you develop. Grant explores how to build the character skills and motivational structures to realize our own potential, and how to design systems that create opportunities for those who have been underrated and overlooked.This book reveals how anyone can rise to achieve greater things. The true measure of your potential is not the height of the peak you’ve reached, but how far you’ve climbed to get there.

Challenge-led Innovation Workbook. Organising for Systems Innovation at Scale - Burkett, Ingrid (Griffith Centre for Systems Innovation, 2023)
Organising for Systems Innovation at Scale Our team at Griffith Centre for Systems Innovation have been experimenting with and evolving a Challenge-led Innovation Approach (based on Mission-oriented approaches developed by Mariana Mazzucato at UCL IIPP and others internationally). We are using this approach to guide the way we work internally and engage with our systems innovation partners. We’ve facilitated intensive Re:Treats, worked with government bodies, businesses and civic organisations, and engaged deeply with others exploring this work. We have a bias for developing and testing HOW such approaches could be applied to respond to both local and global challenges rather than getting too caught up in the what and why of such approaches. We decided to openly share our learnings and thinking to date in this workbook, to spark conversations and innovation in both practice and thinking amongst those exploring how we work, and to learn together to address complex systems and challenges. We see this booklet as a first step in a longer learning journey. In it we share an overview of: the principles and processes that sparked our evolution to a Challenge-led Innovation framework (from Mission-oriented). examples of our learnings from other system innovators who are experimenting. an adaptable process to help guide the learning journey. learning tools and canvases to catalyse thinking, practice, and further adaptations. Part One sets out some foundations we’ve identified as important to Challenge-led Innovation. If you want to jump straight into the mapping process, we suggest you skip to Part Two. The final section, Part Three, focuses on what we have learnt about the conditions needed and how to get started on a Challenge-led initiative.

Scaling Up Development Impact - Guerrero, Isabel; Gokhale, Siddhant; Fahsbender, Jossie (Bowker, 2023)
Today, nearly one billion people lack electricity, over three billion lack clean water, and 750 million lack basic literacy skills. Many of these challenges could be solved with existing solutions, and technology enables us to reach the last mile like never before. Yet, few solutions attain the necessary scale to match the size of these challenges. Scaling Up Development Impact offers an analytical framework, a set of practical tools, and adaptive evaluation techniques to accompany the scaling process. It presents rich organizational experiences that showcase real-world journeys toward increased impact. From the people from IMAGOgg.org

Life Stories From Children Working in Bangladesh’s Leather Sector and its Neighbourhoods: Told and Analysed by Children - Sayem, Mashrique; Sayed, Sayma; Maksud, A. K. M.; Reaz Hossain, Khandaker; Afroze, Jiniya; Burns, Danny; Raw, Anna; Hacker, Elizabeth (Institute of Development Studies, 2023)
CLARISSA (Child Labour: Action-Research-Innovation in South and South-Eastern Asia) has a participatory and child-centred approach that supports children to gather evidence, analyse it themselves and generate solutions to the problems they identify. The life story collection and collective analysis processes supported children engaged in the worst forms of child labour in Bangladesh to share and analyse their life stories. Over 400 life stories were collected from children who worked in the leather supply chain, or who lived and worked in leather sector neighbourhoods. Using causal mapping, 53 children who were engaged in or had experience of the worst forms of child labour collectively analysed the data. This resulted in children’s life stories becoming the evidence base for revealing macro‑level system dynamics that drive the worst forms of child labour. This paper is a record of the children’s analysis of the life stories and key themes they identified, which formed the basis of a series of seven child-led Participatory Action Research groups.
Accepted: 2023-11-07T09:14:57Z Publisher: Institute of Development Studies

3 things that are enabling the UNDP’s shift to portfolios - Begovic, Milica (Medium, 2023)
By Millie Begovic, Head of the UNDP Strategic Innovation Unit

How AI Fits into Lean Six Sigma - Holweg, Matthias; Davenport, Thomas H.; Snyder, Ken (Harvard Business Review, 2023)
AI already is being used in some areas of process improvement, and the usage of this technology — including generative AI — promises to grow. That’s because it can perform tasks faster and much less expensively than humans alone. But it will never fully replace people — and that poses management challenges.
Section: Operations and supply chain management

Insights for Influence: Understanding Impact Pathways in Crisis Response - Clark, Louise; Carpenter, Jo; Taylor, Joe (Institute of Development Studies, 2023)
The Covid-19 Responses for Equity (CORE) programme was a three-year initiative funded by the Canadian International Development Research Centre (IDRC) that brought together 20 projects from across the global South to understand the socioeconomic impacts of the Covid-19 pandemic, improve existing responses, and generate better policy options for recovery. The research covered 42 countries across Africa, Asia, Latin America, and the Middle East to understand the ways in which the pandemic affected the most vulnerable people and regions, and deepened existing vulnerabilities. Research projects covered a broad range of themes, including macroeconomic policies for support and recovery; supporting essential economic activity and protecting informal businesses, small producers, and women workers; and promoting democratic governance to strengthen accountability, social inclusion, and civil engagement. The Institute of Development Studies (IDS) provided knowledge translation (KT) support to CORE research partners to maximise the learning generated across the research portfolio and deepen engagement with governments, civil society, and the scientific community. As part of this support, the IDS KT team worked with CORE project teams to reconstruct and reflect on their impact pathways to facilitate South-South knowledge exchange on effective strategies for research impact, and share learning on how the CORE cohort has influenced policy and delivered change. This report presents an overview of these impact pathways and the lessons learnt from a selection of the projects chosen to represent the diversity of approaches to engage policymakers, civil society, and the media to generate and share evidence of the effect of the pandemic on diverse vulnerable groups.
Accepted: 2023-11-10T12:56:06Z Publisher: Institute of Development Studies

Unconditional cash support sparks transformation: A story from CLARISSA Social Protection Intervention in Dhaka - Aktar, Afrin; Debnath, Raj; Kashfi, Sharmin (CLARISSA, 2023)
The CLARISSA Social Protection (SP) intervention provided six months of unconditional cash transfers to every household in the Gojmohol neighbourhood,...

Disentangling Government Responses: How Do We Know When Accountability Work is Gaining Traction? - Fox, Jonathan; Halloran, Brendan; Fölscher, Alta; McGee, Rosie (Accountability Research Center, 2024)
Advocacy for public accountability aims to produce certain reactions from government officials or service providers. However, the reactions can be many and diverse, and it is not always clear to advocates how to interpret them and decide on next steps—whether to intensify efforts or back off; continue the same strategy or make adjustments. This paper presents a framework to help accountability advocates and practitioners interpret government reactions to their efforts and move forward appropriately. The framework arises from learning and reflection in the context of the International Budget Partnership (IBP)’s Strengthening Public Accountability with Results and Knowledge (SPARK) program. SPARK seeks to bolster the collective agency of marginalized communities and coalitions to advance democratic and equitable fiscal governance systems that channel public resources to services that address the priority needs of these historically excluded groups. The paper: · unpacks the broad umbrella category of government responses to citizen-led accountability initiatives, discerning within it three overlapping categories of responses, responsiveness, and accountable responsiveness (RRA), and pinpointing what distinguishes them; · maps out the public financial management (PFM) system as the critical conduit for any public service to progress from non-response through these three categories, reflecting the terrain and priorities of the SPARK program and IBP more broadly. Looking at RRA in the context of this PFM map offers key insights for the twin tasks of tracing and unblocking bottlenecks in resource management systems that hinder responsiveness and accountability, as well as demanding specific reforms that make the fiscal governance system more open to citizen engagement; and · discusses how the RRA framework can be used to take stock of partial or incremental responses and assess whether change is moving towards accountable governance, by reference to the SPARK program and other IBP efforts. The RRA framework supports accountability-claiming practice by facilitating close attention to the actions of and influences on key government stakeholders, careful interpretation of these, and consequential adjustments to advocacy strategies. Its implications for monitoring, evaluating, and learning (MEL) from practice are also significant. MEL has been evolving in the accountability field in line with the thinking and practical challenges behind the development of the RRA framework. The framework adds to a range of emerging approaches that support the application theories of change, evidence-based adaptive programming, and politically aware systems thinking in the complex task of building accountable responsiveness towards constituencies historically excluded from public goods and services, and ultimately equitable development outcomes from an intersectional perspective.
Artwork Size: 936692 Bytes
Keywords: Development studies; Public policy; Social policy

Scaling up Social Accountability in Complex Governance Systems: A Relational Approach for Evidencing Sustainability - Guerzovich, Florencia; Wadeson, Alix (World Bank, 2024)
When social accountability interventions scale up and their sustainability depends on the interactions of many agents and system components, related results are rarely observable at the end of an intervention. The 2019 OECD Development Assistance Committee’s (OECD DAC) revamped evaluations criteria for assessing sustainability acknowledges that such results are often emergent, and should be monitored and evaluated with this in mind. It therefore emphasizes a turn towards assessing complex processes prospectively. It also asks evaluations to consider how likely it is that these results are evident at the time they are monitored or evaluated. However,the social accountability field continues to have gaps regarding doing this effectively in practice. This paper presents and provides evidence from testing an innovative operational approach that has promising potential to support this aim - a sequential, relational rubric. This approach can support practitioners to monitor, evaluate and learn about the causal processes of scale up of social accountability interventions with an eye towards sustainability i.e., considering prospective sustainability. It is grounded in systems thinking, co-production and social learning theory, as well as links with collective governance and social contract theory for development. Evidence yielded from the authors’ testing of this approach on a sample of diverse projects from the Global Partnership for Social Accountability (GPSA) program revealed that the alleged ‘absence of evidence’ dilemma of social accountability scale up is due to ill-fitting concepts and methods for assessment. It challenges existing assumptions and findings that claim that social accountabilityprocesses do not scale and are unsustainable. The authors propose that by using fit-for-purpose concepts and methods with a focus on social learning and compromise – also called a ‘resonance pathway to scale’ which this paper discusses in detail – it is possible to observe loosely coordinated scale up processes at work in many (but not all) social accountability interventions and identify tangible evidence of prospective sustainability. An important caveat is that these processes, the outcomes they generate, and the corresponding evidence often look qualitatively different than the original intervention design and predictions for scale-up at that point in time. This is because the process of deliberation and compromise inherent to social accountability work in dynamic local systems introduces changes and new conditions for uptake by diverse actors in the public sector, civil society, and donor institutions. The paper concludes that even relatively small-scale localized projects of three to five years with budgets of less than one million USD, across different contexts and sectors can produce processes and outcomes which contribute to many forms of sustainability, including via scaleup.Furthermore, the cross-fertilization of learning and aggregation of results for scale-up across projects within and beyond the GPSA (and other programs) can help monitoring evaluation and learning (MEL) and social accountability practitioners alike to deliver on a program’s mandate. Doing so can also create new knowledge for the wider social accountability field that siloed interventions, lacking suitable concepts and methods for assessing scale-up and prospective sustainability, often fail to produce. The paper ends with recommendations for taking forward this approach and the associated benefits, implications and required investments.

Breaking free from the theory of change straight jacket - Apgar, Marina; Snijder, Mieke; Paul, Sukanta (CLARISSA, 2024)
The CLARISSA Social Protection Intervention was set us as an innovative social policy intervention for tackling social ills, with a...

Social Accountability 3.0: Engaging Citizens to Increase Systemic Responsiveness. Background Paper - Guerzovich, Florencia; Aston, Tom (Grupo Politeia, ESAG, UDESC and act4delivery, 2024)
Social accountability has become increasingly important to development programming since the 2004 World Development Report spearheaded its first generation of programming. In 2016, Thomas Carothers synthesized global experts’ perspectives on what became known as the second generation. With an evolving global context, new research and evaluations and tacit knowledge on how those assumptions played out in practice, the time is ripe for asking what the evidence since 2004 can tell us about what the social accountability’s next generation might look like? This meta-analysis takes stock of that knowledge through a systems lens and presents a possible “social accountability 3.0.” It draws on a database of 157 cases of initiatives implemented by different civil society organizations around the world and the lived experience of practitioners. Scholarly debates present practitioners’ options in binary terms, contrasting 1.0 minimalist and 2.0 maximalist options. The review finds that this binary is inaccurate and misleading; it masks practitioners’ diverse interpretations of 2.0 and range of feasibility considerations. Instead, work in the last decade looks more like a mosaic. By opening the analysis to the broader range of evidence, and reflecting more critically on scholars’ assumptions, a more adaptable middle path for social accountability 3.0 emerges. Its key outcomes include contributing to systemic responsiveness and an operational mechanism to rework the social contract.

Addressing the evaluation-of-sustainability-paradox: A relational rubric for evidencing… - Guerzovich, Florencia (Medium, 2024)
Florencia Guerzovich and Alix Wadeson