TY - GEN TI - 10 Things to Know About Evaluation AU - Buffardi, Anne AU - Hearn, Simon AU - Pasanen, Tiina AU - Price, clare AU - Ball, Louise AB - Evaluation is essential to good development. But there are still many myths and misconceptions about what it is - and how it should be used. ODI's Research and Policy in Development Programme (RAPID) has many years' experience supporting evaluation in complex development contexts. In support of the International Year of Evaluation 2015, we've put together our essential 'things to know' about evaluation in 10 infographics. Available in English and French. DA - 2015/06// PY - 2015 PB - RAPID UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9685.pdf Y2 - 2018/11/10/00:00:00 ER - TY - VIDEO TI - 10 years of Outcome Mapping AU - Smith, R AU - Mauremootoo, John AU - Ambrose, K. AU - Hearn, Simon AB - This webinar from the Outcome Mapping Learning Community (OMLC) presents the key findings from research conducted into the extent of Outcome Mapping use and the support required for its implementation. C5 - Vimeo Video DA - 2012/09// PY - 2012 UR - https://vimeo.com/channels/outcomemapping Y2 - 2018/10/18/00:00:00 ER - TY - ELEC TI - 111 Evaluation Cartoons for Presentations and Blog Posts AU - Lysy, Chris DA - 2020/// PY - 2020 UR - https://freshspectrum.com/evaluation-cartoons/ Y2 - 2023/09/28/10:52:18 ER - TY - CHAP TI - 17. Shifting perceptions, changing practices in PRA: from infinite innovation to the quest for quality AU - Cornwall, Andrea AU - Guijt, Irene T2 - Participatory Learning and Action 50: Critical reflections, future directions A2 - Kenton, N. T3 - PLA notes AB - In the beginning, there were methods. For many of us in the circle of enthusiasts of participatory approaches in the early 1990s, maps and models, calendars and Venn diagrams, matrices and rankings and the interactions and insights they produced defined what we did and what we had in common. It was this, too, that made participatory rural appraisal (PRA) – and rapid rural appraisal (RRA) before it – something that was very different from anything we’d known before. PRA bridged barriers that might otherwise have kept a social anthropologist and an irrigation engineer like us apart. And it brought us together with dozens of others, from a constellation of disciplines and professions, who shared our excitement about an approach that seemed to offer much for ‘doing development’ differently. In 1995, we co-edited PLA Notes 24 on Critical reflections on practice, in which we sought to engage practitioners and advocates in debate about the looming crises of quality that were to become so much a feature of PRA practice in the later 1990s. In this paper, we look back over more than a decade of engagement with PRA as ‘critical insiders’. Participatory Learning and Action has, naturally enough, served more as a vehicle for practitioners to share their successes and innovations than their critical reflections. Accordingly, we draw here on sources that go beyond it, including reflections from the Pathways to Participation project (see Cornwall and Pratt, 2003a, in PLA Notes 47, and contributions to Cornwall and Pratt 2003b), from work with gender and participatory development (Welbourn, 1992; Guijt and Kaul Shah, 1998; Cornwall 2000), and from the lively debates that we have had for more than a decade with colleagues the world over. These thoughts are our personal reflections, from standpoints associated with the two institutions – IIED and IDS – that were so much part of early efforts to promote and institutionalise PRA in international development practice. Our account is, therefore, very much a partial one. We offer it here as a means of locating some of the threads that have run through debates about PRA since the first issues of Participatory Learning and Action, and some of the challenges that practitioners of participatory learning and action methodologies continue to face. In it, we reflect on distinct phases in the development of PRA (see Figure 1), during which a series of issues emerged as themes for critical reflection. The phases indicated in the diagram relate generally to the prevailing sentiment and practice. Clearly there are exceptions – there have been critical voices and some were using PRA to address issues of power from day one, just as there is still innovation and excitement in some quarters today. DA - 2004/// PY - 2004 PB - IIED SN - 978-1-84369-526-4 UR - https://books.google.co.uk/books?id=-Im4wDpECt0C ER - TY - VIDEO TI - 20 years of Outcome Mapping AU - OM Learning Community AB - Celebrate 20 years of Outcome Mapping with the OMLC Stewards plus special guests in this 90 minute webinar. - Exclusive celebration video with Outcome Blues soundtrack - Fireside chat with the authors of the OM manual, Sarah Earl, Fred Carden & Terry Smutylo, to hear their views on OM’s evolution since the 2001 publication - Launch of our new paper presenting core concepts of OM and guiding practices for transformative change, with special guests Sonal Zaveri & Julius Nyangaga as discussants DA - 2021/04/29/ PY - 2021 UR - https://www.youtube.com/watch?v=Cf01oIUKtZE Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - 20 years of Outcome Mapping - Evolving practices for transformative change AU - OM Learning Community AB - To mark 20 years of Outcome Mapping, this paper presents the core principles and concepts that are foundational to using the approach. It also presents a set of guiding practices to support transformative change. The OMLC Stewards presented the paper at a special webinar on 29th April 2021 - see the link below for the recording. CY - OM Resources: Key Community Documents DA - 2021/03// PY - 2021 PB - Outcome Mapping Learning Community UR - https://www.outcomemapping.ca/download/en_20%20years%20of%20OM.pdf Y2 - 2022/09/30/11:11:16 ER - TY - CHAP TI - 3. Reflections and directions: a personal note AU - Chambers, Robert T2 - Participatory Learning and Action 50: Critical reflections, future directions A2 - Kenton, N. T3 - PLA notes DA - 2004/// PY - 2004 PB - IIED SN - 978-1-84369-526-4 UR - https://books.google.co.uk/books?id=-Im4wDpECt0C ER - TY - MGZN TI - A Better Approach to After-Action Reviews AU - Fletcher, Angus AU - Cline, Preston B. AU - Hoffman, Matthew T2 - Harvard Business Review AB - In the decades since the Army created the After Action Review (AAR), businesses have embraced the practice as a way of learning from both failure and success. But all too often the practice gets reduced to nothing more than a pro forma exercise. The authors of this article describe the history and philosophy of the original AAR, debunk three myths about the practice that impede its proper use, and finally suggest three improvements that can help business leaders make the most of it. DA - 2023/01/12/T13:15:17Z PY - 2023 DP - hbr.org SN - 0017-8012 UR - https://hbr.org/2023/01/a-better-approach-to-after-action-reviews Y2 - 2024/01/12/15:42:52 KW - Collaboration and teams KW - Crisis management KW - Project management ER - TY - RPRT TI - A Brief Introduction to Critical Systems Heuristics (CSH) AU - Ulrich, Werner DA - 2005/// PY - 2005 SP - 15 UR - http://wulrich.com/downloads/ulrich_2005f.pdf ER - TY - RPRT TI - A developmental Evaluation Primer AU - Gamble, Jamie A.A. CY - Montreal DA - 2008/// PY - 2008 PB - J.W. McConnell Family Foundation UR - https://mcconnellfoundation.ca/wp-content/uploads/2017/07/A-Developmental-Evaluation-Primer-EN.pdf Y2 - 2019/05/16/00:00:00 ER - TY - RPRT TI - A guide to monitoring and evaluating policy influence AU - Jones, Harry DA - 2011/02// PY - 2011 SP - 12 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6453.pdf Y2 - 2018/12/10/00:00:00 ER - TY - BOOK TI - A guide to Social Return on Investment AU - Nicholls, Jeremy AU - Lawlor, Eilis AU - Neitzert, Eva AU - Goodspeed, Tim AB - There is increasing recognition that we need better ways to account for the social, economic and environmental value that results from our activities. The language varies – ‘impact’, ‘returns’, ‘benefit’, ‘value’ – but the questions around what sort of difference and how much of a difference we are making are the same. Understanding and managing this broader value is becoming increasingly important for the public and private sectors alike. This is true whether it is civil society organisations working to create value, Governments commissioning and investing in activities to create social value, investors seeking to ensure that their investments will make a difference, or private businesses recognising both risk and opportunity in the wider effects of operations. All this means that it is also more important that we have some consistency and a shared language when we talk about value. SROI is the application of a set of principles within a framework that is designed to help bring about that consistency, whilst at the same time recognising that what is of value will be very different for different people in different situations and cultures. The first edition of this guide, which itself built on the work of three earlier SROI guides1, was prepared as part of a three year programme on measuring social value funded in 2008 by the then ‘Office of the Third Sector’ based in the Cabinet Office of the UK Government. This was delivered by a consortium of organisations: the SROI Network, nef (the new economics foundation), Charities Evaluation Services, the National Council for Voluntary Organsations and New Philanthropy Capital. In addition to this programme, the Scottish Government also supported the development of SROI, including a database of indicators to support SROI analysis. The work of the SROI Network now stretches across many different countries and continents, and this second edition of the guide reflects that interest. We have though decided to use only one currency symbol, for reasons of clarity and consistency, and so have continued to use £. However, readers will be able to find examples in various currencies and translations of the Guide on our website. For more information on the developments of SROI, please refer to the SROI Network website: www.thesroinetwork.org DA - 2012/03// PY - 2012 PB - The SROI Network UR - https://socialvalueselfassessmenttool.org/wp-content/uploads/intranet/758/pdf-guide.pdf Y2 - 2020/10/09/00:00:00 ER - TY - RPRT TI - A Holistic Approach to Evaluating Research AU - Ofir, Zenda AU - Schwandt, Thomas AU - Duggan, Colleen AU - McLean, Robert DA - 2016/// PY - 2016 DP - Zotero SP - 28 LA - en PB - IDRC UR - https://www.idrc.ca/sites/default/files/sp/Documents%20EN/Research-Quality-Plus-A-Holistic-Approach-to-Evaluating-Research.pdf ER - TY - RPRT TI - A leaders's guide to after-action reviews AU - U.S. Army T2 - Training Circular AB - An after-action review (AAR) is a professional discussion of an event, focused on performance standards, that enables soldiers to discover for themselves what happened, why it happened, and how to sustain strengths and improve on weaknesses. It is a tool leaders and units can use to get maximum benefit from every mission or task. It provides- • Candid insights into specific soldier, leader, and unit strengths and weaknesses from various perspectives. • Feedback and insight critical to battle-focused training. • Details often lacking in evaluation reports alone. Evaluation is the basis for the commander's unit-training assessment. No commander, no matter how skilled, will see as much as the individual soldiers and leaders who actually conduct the training. Leaders can better correct deficiencies and sustain strengths by carefully evaluating and comparing soldier, leader, and unit performance against the standard. The AAR is the keystone of the evaluation process. Feedback compares the actual output of a process with the intended outcome. By focusing on the task's standards and by describing specific observations, leaders and soldiers identify strengths and weaknesses and together decide how to improve their performances. This shared learning improves task proficiency and promotes unit bonding and esprit. Squad and platoon leaders will use the information to develop input for unittraining plans. The AAR is a valid and valuable technique regardless of branch, echelon, or training task. Of course, AARs are not cure-alls for unit-training problems. Leaders must still make on-the-spot corrections and take responsibility for training their soldiers and units. However, AARs are a key part of the training process. The goal is to improve soldier, leader, and unit performance. The result is a more cohesive and proficient fighting force. Because soldiers and leaders participating in an AAR actively discover what happened and why, they learn and remember more than they would from a critique alone. A critique only gives one viewpoint and frequently provides little opportunity for discussion of events by participants. Soldier observations and comments may not be encouraged. The climate of the critique, focusing only on what is wrong, prevents candid discussion of training events and stifles learning and team building. CY - Washington DC DA - 1993/07// PY - 1993 PB - Department of the Army SN - 25-20 UR - https://www.hsdl.org/?view&did=775082 Y2 - 2024/01/12/00:00:00 ER - TY - RPRT TI - A mini-primer of critical systems heuristics. AU - Ulrich, W. AB - "Critical Systems Heuristics," also just called "Critical Heuristics" or "CSH," is a framework for reflective practice based on practical philosophy and systems thinking. The basic idea of CSH is to support boundary critique – a systematic effort of handling boundary judgments critically. Boundary judgments determine which empirical observations and value considerations count as relevant and which others are left out or are considered less important. Because they condition both "facts" and "values," boundary judgments play an essential role when it comes to assessing the meaning and merits of a claim. Their systematic discussion can help bridge differences of perspectives across disciplines and between experts and non-experts. They also lend themselves to a specific critical employment, called emancipatory boundary critique, against claims that do not uncover their underlying boundary assumptions. CSH can thus serve as a tool for coproducing knowledge as well as for critical and emancipatory purposes on the part of people concerned by, but not necessarily involved in, the definition of relevant facts and values. DA - 2005/// PY - 2005 UR - http://wulrich.com/csh.html Y2 - 2018/10/13/00:00:00 ER - TY - BLOG TI - A new pathway: how can funders support meaningful monitoring, evaluation, and learning practice in the field? - Blog post on Better Evaluation AU - Acevedo, Andrea AU - Colnar, Megan AB - How can donors and grantees work together to create effective monitoring, evaluation, and learning (MEL) practices that drive field-wide transformation? DA - 2023/04/10/ PY - 2023 LA - en M3 - Better Evaluation ST - A new pathway UR - https://www.betterevaluation.org/blog/new-pathway-how-can-funders-support-meaningful-monitoring-evaluation-learning-practice-field Y2 - 2023/07/04/09:46:07 ER - TY - RPRT TI - A Political Economy Analysis Framework for EdTech Evidence Uptake AU - Pellini, Arnaldo AU - Nicolai, Susan AU - Magee, Arran AU - Sharp, Sam AU - Wilson, Sam DA - 2021/02/13/ PY - 2021 PB - The EdTech Hub ER - TY - RPRT TI - A problem-focused approach to violence against women: The political-economy of justice and security programming AU - Denney, Lisa AU - Domingo, Pilar AB - The United Nations Commission on the Status of Women ended its 57th session on 15 March 2013 with an outcome document affirming the importance of eliminating violence against women (VAW). The Commission was unable, however, to achieve consensus on a global action plan. The negative reaction of some UN member states to an action plan is a worrying reminder of ongoing resistance to reform. These persistent challenges highlight the continuing struggle to gain a serious global commitment to address VAW and recognise it as a breach of women’s fundamental human rights. Engaging in this struggle, many donors have put addressing VAW generally, and in fragile and conflictaffected situations (FCAS) specifically, at the top of the development agenda and made it a major priority of international policy. But in practice progress remains difficult, not least due to entrenched resistance and discriminatory socio-political norms and gender relations that persist in many societies. The problem of violence against women therefore needs to be addressed from the perspective of the concrete socio-political and cultural conditions that shape its particular features and the relevant context specific dynamics of conflict, post-conflict patterns of violence and fragility. International efforts to support reform in the area of VAW in FCAS need to go beyond prescriptive approaches that focus on what access to protection, justice and redress should look like. We propose here an approach that engages with the specificities of the problem – paying attention to context, and the concrete political-economy dynamics of the drivers of VAW – and takes account of the real options that DA - 2013/03// PY - 2013 SP - 12 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8325.pdf Y2 - 2018/12/10/00:00:00 ER - TY - JOUR TI - A reliability and validity study of the Dynamic Indicators of Basic Early Literacy Skills—Modified AU - Elliot, Jacquelyn AU - Lee, Steven W AU - Tollefson, Nona T2 - School Psychology Review AB - Examined the psychometric properties of a set of preliteracy measures modified from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) with a sample of 75 kindergarten students. The modified battery (called DIBELS—M) includes measures of Letter Naming Fluency, Sound Naming Fluency, Initial Phoneme Ability, and Phonemic Segmentation Ability. These measures were assessed through repeated administrations in 2-wk intervals at the end of the kindergarten year. Results indicate interrater reliability estimates and coefficients of stability and equivalence for 3 of the measures ranged from .80 to the mid .90s with about one-half of the coefficients above .90. Correlations between DIBELS—M scores and criterion measures of phonological awareness, standardized achievement measures, and teacher ratings of achievement yielded concurrent validity coefficients ranging from .60 to .70. Hierarchical regression analysis showed that the 4 DIBELS—M measures accounted for 73% of the variance in scores on the Skills Cluster of the Woodcock-Johnson Psychoeducational Battery—Revised. The contributions of the study, including psychometric analysis of the DIBELS—M with a new sample and formation of composite scores, are discussed in relation to the extant literature. DA - 2001/// PY - 2001 DO - 10.1080/02796015.2001.12086099 VL - 30 IS - 1 SP - 33 EP - 49 UR - http://www.iapsych.com/wj3ewok/LinkedDocuments/elliott2001.pdf Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - A study of the sustained uptake developmental evaluation: How does developmental evaluation work in the USAID context, what factors help and hinder its success, and what is its value to stakeholders? AU - Baylor, Rebecca AU - Fatehi, Y. K. AU - Esper, H. DA - 2019/04// PY - 2019 PB - USAID UR - https://pdf.usaid.gov/pdf_docs/PA00TNRP.pdf Y2 - 2019/05/17/11:44:46 ER - TY - JOUR TI - A Systems Framework for International Development: The Data-Layered Causal Loop Diagram AU - Blair, Courtney AU - Gralla, Erica AU - Wetmore, Finley AU - Goentzel, Jarrod AU - Peters, Megan T2 - Production and Operations Management AB - Meeting the United Nations Sustainable Development Goals (SDGs) will require adapting or redirecting a variety of very complex global and local human systems. It is essential that development scholars and practitioners have tools to understand the dynamics of these systems and the key drivers of their behavior, such as barriers to progress and leverage points for driving sustainable change. System dynamics tools are well suited to address this challenge, but they must first be adapted for the data-poor and fragmented environment of development work. Our key contribution is to extend the causal loop diagram (CLD) with a data layer that describes the status of and change in each variable based on available data. By testing dynamic hypotheses against the system's actual behavior, it enables analysis of a system's dynamics and behavioral drivers without simulation. The data-layered CLD was developed through a 4-year engagement with USAID/Uganda. Its contributions are illustrated through an application to agricultural financing in Uganda. Our analysis identified a lack of demand for agricultural loans as a major barrier to broadening agricultural financing, partially refuting an existing hypothesis that access to credit was the main constraint. Our work extends system dynamics theory to meet the challenges of this practice environment, enabling analysis of the complex dynamics that are crucial to achieving the SDGs. DA - 2021/// PY - 2021 DO - 10.1111/poms.13492 DP - Wiley Online Library VL - 30 IS - 12 SP - 4374 EP - 4395 LA - en SN - 1937-5956 ST - A Systems Framework for International Development UR - https://onlinelibrary.wiley.com/doi/abs/10.1111/poms.13492 Y2 - 2022/07/01/08:55:09 ER - TY - RPRT TI - A Theory of Change of citizen participation: an update AU - Peña-López, Ismael AB - when it was reduced to a subsidiary internal service lacking all kind of political attributions. The work done in those years had been formidable, but too many things had passed since, especially the 15M Spanish Indignados Movement, the raise of technopolitics… and the raise of populism and fascism all across Europe. We urgently needed a theoretical framework in which to substantiate our political strategy, so I came up with a Theory of Change of citizen participation (see Figure 1) which defined four expected impacts of our political action: 1. Efficiency, efficacy and legitimacy of public decisions improves. 2. Populism has decreased in institutions and the public sphere. 3. Citizens understand the complexity of public decision-making. 4. Citizen participation and political engagement clearly shifts towards a technopolitical paradigm. These impacts were expected to be achieved after some outcomes resulting from some outputs grouped in five programmes: 1. Programme of citizen participation. 2. Programme of internal participation. 3. Programme of collaboration. 4. Programme of intermediaries, facilitators and infomediaries. 5. Programme of e-participation, e-voting and technopolitics. 20 months after, the Theory of Change of Citizen Participation has worked quite well. But it does have some limitations, especially at the operational level-which is what the whole thing was about, to help in putting some order in our daily work. CY - Barcelona DA - 2020/// PY - 2020 DP - www.academia.edu LA - en PB - ICTlogy ST - A Theory of Change of citizen participation UR - https://www.academia.edu/42805069/A_Theory_of_Change_of_citizen_participation_an_update Y2 - 2020/10/15/10:02:18 ER - TY - BOOK TI - A trainer's guide for participatory learning and action T2 - IIED Participatory methodology series A3 - Pretty, Jules A3 - Guijt, Irene CY - London DA - 2002/// PY - 2002 DP - K10plus ISBN ET - Reprint SP - 267 LA - eng SN - 978-1-899825-00-4 ER - TY - CHAP TI - Accountability and Learning: Exploding the Myth of Incompatibility between Accountability and Learning AU - Guijt, Irene T2 - NGO Management - The Earthscan Compendium A2 - Fowler, Alan A2 - Malunga, Chiku AB - When accountability is understood as reporting on pre-deined deliverables, it is often considered to be irreconcilable with learning. This conventional wisdom inhibits an appreciation of their connection. In this chapter, Irene Guijt exposes the laws and traps in reasoning that keep accountability and learning apart. She provides practitioners with principles and basic good ideas that open up prospects for accountability and learning to complement each other. CY - London DA - 2010/// PY - 2010 DP - www.taylorfrancis.com SP - 339 EP - 352 LA - en PB - Routledge SN - 978-1-84977-542-7 UR - https://www.taylorfrancis.com/chapters/edit/10.4324/9781849775427-36/accountability-learning-exploding-myth-incompatibility-accountability-learning-irene-guijt Y2 - 2023/01/24/10:09:58 ER - TY - RPRT TI - Adapting data collection and utilisation to a Covid-19 reality: Monitoring, evaluation and learning approaches for adaptive management AU - Ziegler, Jessica AU - Mason, Paige T2 - Briefing Paper AB - This briefing note focuses on the remote collection and use of data for adaptive management during the Covid-19 pandemic, setting out key considerations to help practitioners think through a transition from more ‘traditional’ monitoring, evaluation and learning (MEL) to MEL for adaptive management (MEL4AM) that reflects the unique data collection challenges presented by Covid-19. The brief provides an overview of some key considerations in remote data collection, when this is required, and identifies other sources that address these issues in more detail. It concludes with a discussion of how to bring the information resulting from remote monitoring into decision-making to enable adaptive management. Key messages - When planning for remote data collection during the Covid-19 pandemic, first determine what information is still necessary, because data needs may have changed, e.g. if programming has pivoted or needs to pivot due to Covid-19. Then identify how the programme’s information needs align with existing data sources and what gaps remain, which will guide the need for remote data collection. - Also consider what data is ‘good enough’ for current decision-making needs in order to provide sufficient information to the right people at the right time to an acceptable standard of rigour. - There may be pragmatic reasons to reduce the number or scope of MEL activities, such as logistical constraints or ethical considerations introduced by the pandemic. - MEL activities should be accompanied by frequent feedback loops and pause points to reflect on emergent needs and challenges, information needs that have been met, and contextual changes that may affect MEL. - Be clear with decision-makers about the assumptions and gaps in the data, including proxies used and their limitations, sampling changes, and how these changes and assumptions may affect the decisions/options being discussed. DA - 2020/// PY - 2020 DP - Zotero SP - 11 LA - en PB - ODI/GLAM ER - TY - RPRT TI - Adapting development: improving services to the poor AU - Wild, Leni AU - Booth, David AU - Cummings, Clare AU - Foresti, Marta AU - Wales, Joseph AB - On current trends, it will take decades or longer to bring basic services to the world’s most disadvantaged people. Meeting this challenge means recognising the political conditions that enable or obstruct development progress - a radical departure from the approach of the Millennium Development Goals. CY - London DA - 2015/02// PY - 2015 PB - ODI ST - Adapting development UR - https://www.odi.org/publications/8125-adapting-development-service-delivery-sdgs Y2 - 2016/05/11/12:37:16 KW - Practice ER - TY - RPRT TI - Adaptive Evaluation: A Complexity-based approach to Systematic Learning for Innovation and Scaling in Development AU - Gokhale, Siddhant AU - Walton, Michael T2 - CID Faculty Working Paper AB - Nearly all challenges in international development tend to be complex because they depend on constantly evolving human behaviour, systems, and contexts, involving multiple actors, entities, and processes. As a result, both the discovery and scaling of innovations to address challenges in development often involve changes in system behaviour or even system-level transformation. This is rarely a linear process over time and can result in unexpected outcomes. Existing evaluation techniques commonly used in international development, including Randomized Control Trials (RCT) and quasi-experimental methods, are good at assessing specific effects of interventions but are not designed for the change processes inherent to innovation and scaling within a system. There is a need to reconstruct how we use existing measurement tools, techniques, and methodologies so that they capture the complexity of the environment in which an intervention or change occurs. We introduce Adaptive Evaluation, designed to learn at various levels of complexity while supporting the transformation needed to foster sustainable change. An Adaptive Evaluation uses three main approaches to work with complex questions—systems diagnosis, theorybased assessment of change processes, and iterative designs. An Adaptive Evaluation typically builds hypotheses from field-based interactions, emphasizes learning over testing, advocates open-mindedness with techniques, and appreciates the value of dialogue and participation in navigating complex processes. It can use RCT or similar techniques to analyse specific processes within a system or a development cycle, but these are embedded in a broader approach to assessment and interpretation. It is designed to be flexible and adjust to shifting contexts. Finally, an Adaptive Evaluation can be applied at any stage in a complex intervention's lifecycle, from the interpretation of the system and change processes to rapid experimentation, prototyping, and testing of select interventions, and then adaptation to different settings for impact at scale. This paper provides the theoretical basis for an Adaptive Evaluation—the main approaches, core ideology, process, and applications. CY - Boston DA - 2023/03// PY - 2023 DP - Zotero LA - en PB - Center for International Development, Harvard University SN - 428 ER - TY - BLOG TI - Adaptive Management Across Project Cycles: Look into Coherence in Time AU - Guerzovich, Florencia T2 - Medium AB - This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots. ... DA - 2023/08/08/T10:27:08.510Z PY - 2023 LA - en ST - Adaptive Management Across Project Cycles UR - https://medium.com/@florcig/adaptive-management-across-project-cycles-look-into-coherence-in-time-ab99caa3a9e5 Y2 - 2023/08/10/09:00:46 ER - TY - RPRT TI - Adaptive Programming in Fragile, Conflict and Violence-Affected Settings, What Works and Under What Conditions?: The Case of PERL, Nigeria AU - Punton, Melanie AU - Burge, Richard T2 - Action for Empowerment and Accountability Research Programme AB - This paper examines adaptive approaches to aid programming in Nigeria. Through field research and desk reviews, we have investigated some of the assertions around the ‘adaptive management and programming’ approach, which has arisen in recent years as a response to critiques of overly rigid, pre-designed, blueprint and linear project plans. This is the second of three case studies in a series which explore if and how adaptive approaches, including rapid learning and planning responses, are particularly relevant and useful for promoting empowerment and accountability in fragile, conflict and violence-affected settings (FCVAS). This case study focuses on PERL (Partnership to Engage, Reform and Learn) in Nigeria, a five-year governance programme (2016-21) with a total budget of £100 million. It aims to promote better service delivery through bringing together government and citizens groups to collectively address governance challenges. PERL is viewed by DFID as the final stage of a 20-year investment, building on learning, experience and partnerships from 15 years of DFID-funded governance programming in Nigeria. It was designed to incorporate learning and adaptation through a ‘living’ theory of change, continuous political economy analysis at different levels, adaptive implementation by location-based delivery teams who are encouraged to be flexible and let partners take the lead, regular learning and reflection, and adaptive resourcing, HR and administrative systems. The case study draws on a conceptual framework (the ‘adaptive triangle’) that looks at three types of adaptation – adaptive management, adaptive programming and adaptive delivery – and the interconnections and tensions between them. CY - Brighton DA - 2018/11/26/ PY - 2018 DP - opendocs.ids.ac.uk LA - en PB - Itad, Oxfam and IDS UR - https://opendocs.ids.ac.uk/opendocs/handle/123456789/14148 Y2 - 2019/02/04/00:00:00 KW - A4EA KW - Adaptive Development KW - Economy KW - Fishery ER - TY - RPRT TI - Adaptive Programming in Fragile, Conflict and Violence-Affected Settings, What Works and Under What Conditions?: The Case of Pyoe Pin, Myanmar AU - Christie, Angela AU - Green, Duncan T2 - Action for Empowerment and Accountability Research Programme AB - This paper examines adaptive approaches in aid programming in a fragile, conflict and violence-affected setting (FCVAS), namely Myanmar. A combination of desk review and field research has been used to examine some of the assertions around the ‘adaptive management’ approach, which has arisen in recent years as a response to critiques of overly rigid, pre-designed, blue-print and linear project plans. This paper explores if and how adaptive approaches, including rapid learning and planning responses (fast feedback loops and agile programming) are particularly relevant and useful for promoting empowerment and accountability in such ‘messy places’. This case study focuses on Pyoe Pin (‘Young Shoots’), a DFID-funded, British Council managed governance programme, which has been running since 2007. CY - Brighton DA - 2018/07/12/ PY - 2018 DP - opendocs.ids.ac.uk LA - en PB - Itad, Oxfam and IDS ST - Adaptive Programming in Fragile, Conflict and Violence-Affected Settings, What Works and Under What Conditions? UR - https://opendocs.ids.ac.uk/opendocs/handle/123456789/13888 Y2 - 2018/08/02/09:48:28 KW - A4EA KW - Adaptive Development KW - Economy KW - Fishery ER - TY - JOUR TI - Addressing attribution through contribution analysis: Using performance measures sensibly AU - Mayne, John T2 - The Canadian Journal of Program Evaluation AB - The changing culture of public administration involves accountability for results and outcomes. This article suggests that performance measurement can address such attribution questions. Contribution analysis has a major role to play in helping managers, researchers, and policymakers to arrive at conclusions about the contribution their program has made to particular outcomes. The article describes the steps necessary to produce a credible contribution story. Contents Accountability for Outcomes The Problem of Attribution The Case of Performance Measurement Recognizing the Limits of Measurement Two Uses of Performance Measurement: Understanding and Reporting Approaches to Attribution: Contribution Analysis Techniques for Strengthening Your Performance Story DA - 2001/// PY - 2001 VL - 16 IS - 1 SP - 1 EP - 24 SN - 0834-1516 UR - https://www.researchgate.net/publication/279533461_Addressing_Attribution_Through_Contribution_Analysis_Using_Performance_Measures_Sensibly ER - TY - RPRT TI - Addressing gender in impact evaluation: What should be considered? AU - Fletcher, Gillian AB - Gender and sexuality are intimately entwined; we must not lose sight of the ways in which gender affects non-heterosexual people, transgender people and people who do not identify as either male or female. • Gender and gender-related injustice is a feature of all interventions, whatever the focus, be it agriculture, capacity building, disaster management, education, health, peace building, water, sanitation and hygiene, or other. • Showing an increase in the number of women participants in an intervention is not the same as demonstrating gender impact. An ‘add women and stir’ approach is not good enough. • A good intervention design will identify critical inequalities and conduct a needs assessment that clearly identifies gender-related issues. If this needs assessment feeds directly into the programme theory, it will facilitate assessment of the intervention’s gender-related impact and will be more likely it is to have positive gender-related impact. DA - 2015/10// PY - 2015 SP - 24 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9934.pdf Y2 - 2018/11/10/00:00:00 ER - TY - BLOG TI - Addressing the evaluation-of-sustainability-paradox: A relational rubric for evidencing… AU - Guerzovich, Florencia T2 - Medium AB - Florencia Guerzovich and Alix Wadeson DA - 2024/02/13/T12:15:56.628Z PY - 2024 LA - en ST - Addressing the evaluation-of-sustainability-paradox UR - https://medium.com/@florcig/addressing-the-evaluation-of-sustainability-paradox-a-relational-rubric-for-evidencing-001683a36398 Y2 - 2024/02/13/13:07:44 ER - TY - RPRT TI - An introductory systems thinking toolkit for civil servants AU - UK Government AB - This document is a an Introductory Toolkit for for civil servants. It is one component of a suite of documents that aims to act as a springboard into systems thinking for civil servants unfamiliar with this approach. These documents introduce a small sample of systems thinking concepts and tools, chosen due to their accessibility and alignment to civil service policy development, but which is by no means comprehensive. They are intended to act as a first step towards using systems thinking approaches to solve complex problems and the reader is encourage to explore the wider systems thinking field further. CY - London DA - 2022/// PY - 2022 LA - en PB - UK Government Office for Science UR - https://www.gov.uk/government/publications/systems-thinking-for-civil-servants/toolkit Y2 - 2023/02/08/15:56:28 ER - TY - RPRT TI - An inventory and review of countering violent extremism and insurgency monitoring systems AU - Carter, Lynn AU - Phyllis, Dinino CY - Washington DC DA - 2012/12/10/ PY - 2012 SP - 92 PB - USAID UR - https://pdf.usaid.gov/pdf_docs/PA00JSKQ.pdf Y2 - 2019/09/17/00:00:00 ER - TY - RPRT TI - Appendix 3. ToC Examples AU - Vogel, Isabel AU - Stephenson, Zoe DA - 2012/// PY - 2012 PB - DFID ST - DFID UR - https://assets.publishing.service.gov.uk/media/57a08a66ed915d622c000703/Appendix_3_ToC_Examples.pdf Y2 - 2020/10/15/10:19:46 ER - TY - BOOK TI - Applying Evaluation Criteria Thoughtfully AU - OECD AB - Relevance, coherence, effectiveness, efficiency, impact, and sustainability are widely used evaluation criteria, particularly in international development co-operation. They help to determine the merit or worth of various interventions, such as strategies, policies, programmes or projects. This guidance aims to help evaluators and others to better understand those criteria, and improve their use. It starts by describing what they are, and how they are meant to be used. Then the definitions and concepts underpinning each criterion are explained. Finally, examples provide the reader with concrete ideas for using them. The criteria were originally laid out in the early 2000s by the Network on Development Evaluation (EvalNet) of the OECD Development Assistance Committee (DAC). Although they have been widely used in evaluation, and beyond, this document is the first to comprehensively explore the concepts in detail, explain their intended use and offer practical guidance. It captures current thinking and best practice in evaluation, drawing on the inputs of internationally renowned evaluation experts from EvalNet and beyond. CY - Paris DA - 2021/03// PY - 2021 PB - OECD UR - https://www.oecd-ilibrary.org/content/publication/543e84ed-en ER - TY - CHAP TI - Applying Power Analysis: Using the ‘Powercube’ to explore forms, levels and spaces AU - Gaventa, John T2 - Power, Empowerment and Social Change A2 - McGee, Rosie A2 - Pettit, Jethro AB - In a complex, globalised and rapidly changing world, power dynamics are multidimensional, constantly evolving, and full of complexity. The ‘powercube’ (Gaventa, 2006) is an approach to power analysis which can be used to examine the multiple forms, levels and spaces of power, and their interactions. Building on earlier work on power, and elaborated and popularised in collaboration with other colleagues through the web site powercube.net and numerous other resources, the powercube has been widely used around the world for analysis of power, education and awareness building, context analysis, programme and strategy development, and monitoring and evaluation. This article briefly outlines the evolution of the powercube, and provides examples of the issue areas in which it has been used, and for what purposes. Drawing on these, we then offer eight lessons of how to apply the powercube framework for analysing and transforming power relations. CY - London ; New York DA - 2019/11/08/ PY - 2019 LA - Inglés SN - 978-1-138-57531-8 UR - https://www.routledge.com/Power-Empowerment-and-Social-Change-1st-Edition/McGee-Pettit/p/book/9781138575318 ER - TY - RPRT TI - Applying Rigorous Impact Evaluation in GIZ Governance Programmes: Results of a GIZ Initiative on Impacts in Governance AU - von Schiller, Armin AB - Pressure is mounting on international development cooperation agencies to prove the impact of their work. Private and public commissioners as well as the general public are increasingly asking for robust evidence of impact. In this context, rigorous impact evaluation (RIE) methods are increasingly receiving attention within the broader German development system and in GIZ. Compared to other implementing agencies such as DFID or USAid, the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH has so far relatively little experience in systematically applying rigorous methods of impact evaluation. This is particularly true in the governance sector. In order to gain more experience and to understand which methods and formats are best suited for GIZ governance programmes, the Governance and Conflict division and the Africa department launched the ‘Impact Initiative Africa’ in 2016, a cooperative effort with several programmes in Africa. The Initiative set out to apply the experiences from GIZ governance programmes to design and conduct RIEs, and to use the results to steer programme implementation. Initially, the Initiative included three countries: Benin (Programme for Decentralisation and Local Development), Malawi (Support to Public Financial and Economic Management) and Mozambique (Good Financial Governance in Mozambique). During its implementation, the Initiative also benefitted from the experience of two additional governance programmes which had already undertaken RIEs, namely Peru (Citizen-oriented State Reform Programme) and Pakistan (Support to Local Governance Programme II). This report summarizes the insights gained from these experiences and discusses opportunities and limitations regarding the use and usability of RIEs in GIZ governance programmes as well as proposals on how to organise RIEs to maximise learning potential and benefits for the specific programmes and the GIZ Governance sector at large. CY - Bonn DA - 2020/// PY - 2020 PB - GIZ GmbH UR - https://www.idos-research.de/uploads/media/giz2021-0020en-rigorous-impact-evaluation-giz-governance-programmes-results.pdf Y2 - 2023/03/28/09:50:39 ER - TY - RPRT TI - Applying Systems Thinking to Education: The RISE Systems Framework AU - Spivack, Marla AB - Many education systems in low- and middle-income countries are experiencing a learning crisis. Many efforts to address this crisis do not account for the system features of education, meaning that they fail to consider the ways that interactions and feedback loops produce outcomes. Thinking through the feedback relationships that produce the education system can be challenging. The RISE Education Systems Framework, which is sufficiently structured to give boundaries to the analysis but sufficiently flexible to be adapted to multiple scenarios, can be helpful. The RISE Framework identifies four key relationships in an education system: politics, compact, management, and voice and choice; and five features that can be used to describe these relationships: delegation, finance, information, support, and motivation. This Framework can be a useful approach for characterising the key actors and interactions in the education system, thinking through how these interactions produce systems outcomes, and identifying ways to intervene that can shift the system towards better outcomes. DA - 2021/05/26/ PY - 2021 DP - DOI.org (Crossref) LA - en PB - Research on Improving Systems of Education (RISE) ST - Applying Systems Thinking to Education UR - https://riseprogramme.org/publications/applying-systems-thinking-education-rise-systems-framework Y2 - 2021/12/16/15:08:08 ER - TY - JOUR TI - Applying Systems Thinking via Systemigrams™ for Defining the Body of Knowledge and Curriculum to Advance Systems Engineering (BKCASE) Project AU - Squires, Alice AU - Pyster, Art AU - Sauser, Brian AU - Olwell, David AU - Enck, Stephanie AU - Gelosh, Don AU - Anthony, Jim T2 - INCOSE International Symposium AB - Systems thinking is commonly accepted as the backbone of a successful systems engineering approach. As such, the Body of Knowledge and Curriculum to Advance Systems Engineering (BKCASE) team chose to leverage a systems thinking based tool, called Systemitool, to describe our project to the vast audience that would potentially become involved directly or indirectly in the success of the project. This paper describes the process and steps used by the authors and the BKCASE team to develop the project's systemic diagram, or Systemigram™, and the story behind the project, the products, and the vision of the BKCASE project. The goal of the paper is to provide guidance so that readers can leverage the lessons learned from this effort to successfully develop their own project definitions and stories. DA - 2010/// PY - 2010 DO - 10.1002/j.2334-5837.2010.tb01101.x DP - Wiley Online Library VL - 20 IS - 1 SP - 739 EP - 753 LA - en SN - 2334-5837 UR - https://apps.dtic.mil/dtic/tr/fulltext/u2/a522654.pdf Y2 - 2020/10/09/08:43:12 ER - TY - RPRT TI - Applying the River of Life Method to Support Reflection and Learning in Terre des hommes Nepal AU - Karki, Shanta AU - Giri, Roju AU - Neupane, Sudarshan AU - Snijder, Mieke AU - Apgar, Marina T2 - CLARISSA Learning Note AB - The RoL method is a visual narrative method that helps people tell stories of the past, present, and future. Individuals can use this method to introduce themselves in a fun and descriptive way. A group can use it to understand and reflect on the past and imagine the future of a project. Besides, it can also be used to build a shared view of a process over time while acknowledging different and perhaps contradictory perspectives. The method uses drawings rather than text, making it useful in groups that do not share a common language. Metaphors from a river are used to explore aspects of a story – such as whirlpools depicting challenges or lakes suggesting a sense of calm etc. When used in a group, it is an active method, engaging people in the process of storytelling and listening through visualising their experiences and using metaphors to explore in depth. In CLARISSA, we adapted the RoL method to document our collective understanding of the story of implementation of the programme as part of the programme’s monitoring, evaluation and learning component. The purpose was to surface the details of our process of the systemic Action Research that we are undertaking with children in the worst forms of child labour and business owners. We used the same river metaphors as is often applied when the method is used with individuals. CY - Brighton DA - 2023/09// PY - 2023 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 4 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/18112 Y2 - 2023/10/05/09:07:58 ER - TY - BLOG TI - Appreciative Inquiry AU - Better Evaluation T2 - Better Evaluation AB - Appreciative Inquiry is an approach to organisational change which focuses on strengths rather than on weaknesses - quite different to many approaches to evaluation which focus on deficits and problems. "Appreciative Inquiry is about the coevolutionary search for the best in people, their organizations, and the relevant world around them. In its broadest focus, it involves systematic discovery of what gives “life” to a living system when it is most alive, most effective, and most constructively capable in economic, ecological, and human terms. AI involves, in a central way, the art and practice of asking questions that strengthen a system’s capacity to apprehend, anticipate, and heighten positive potential." (Cooperrider & Whitney 2005, p.3) Appreciative Inquiry is often presented in terms of a 4 step process around an affirmative topic choice: 1. DISCOVER: What gives life? What is the best? Appreciating and identifying processes that work well. 2. DREAM: What might be? What is the world calling for? Envisioning results, and how things might work well in the future. 3. DESIGN: What should be--the ideal? Co-constructing - planning and prioritizing processes that would work well. 4. DESTINY (or DELIVER): How to empower, learn and adjust/improvise? Sustaining the change While Appreciative Inquiry has always had an evaluative focus (working out what is working well and seeking to improve performance and conditions), in recent years there have been explicit efforts to embed AI principles and processes in formal evaluation processes: "Appreciative Inquiry (AI) is a group process that inquires into, identifies and further develops the best of “what is” in organizations in order to create a better future. Often used in the organization development field as an approach to large-scale change, it is a means for addressing issues, challenges, changes and concerns of an organization in ways that builds on the successful, effective and energizing experiences of its members. Underlying AI is a belief that the questions we ask are critical to the world we create." (Preskill & Catsambas 2006 p2) UR - https://www.betterevaluation.org/en/plan/approach/appreciative_inquiry Y2 - 2018/12/10/00:00:00 ER - TY - RPRT TI - Appreciative Inquiry: An approach for learning and change based on our own best practices​ AU - Acosta, Anne AU - Douthwaite, Boru T2 - ILAC Brief AB - Since it was conceptualized in the late 1980s as a research methodology and change paradigm, the technique of ‘appreciative inquiry’ (AI) has proved to be highly effective for capturing the positive features of an organization or social system and energizing the members to strive for higher levels of performance. This Brief outlines the basic principles and methods of AI, describes various domains in which it has been undertaken and provides a recent example of its use in a centre affiliated with the Consultative Group on International Agricultural Research (CGIAR). DA - 2005/07// PY - 2005 PB - Institute of Learning and Change SN - 6 UR - https://www.betterevaluation.org/sites/default/files/ILAC_Brief06_inquiry.pdf Y2 - 2018/12/10/00:00:00 ER - TY - ELEC TI - Appreciative Inquiry Australia AU - Appreciative Inquiry Australia AB - The site provides news and information on AI events and learning opportunities in Australia and also provides an archive of articles and discussions centred on the use of AI. DA - 2015/// PY - 2015 UR - http://appreciativeinquiry.com.au/ Y2 - 2018/12/10/00:00:00 ER - TY - CHAP TI - Appreciative Inquiry: Organization Development and the Strengths Revolution AU - Stavros, Jacqueline M. AU - Godwin, Lindsey N. AU - Cooperrider, David L. T2 - Practicing Organization Development A2 - Rothwell, William J. A2 - Stavros, Jackie A2 - Sullivan, Roland L. AB - A ppreciative Inquiry (AI) is a theory and practice of inquiry-and-changethat shifts the perspective of organization development (OD) methodsby suggesting that the very act of asking generative questions has pro-found impact in organizational systems. Inquiry and change are not separatemoments. Our questions focus our attention on what is “there” to be noticed.Reflecting its social constructionist roots (Cooperrider, Barrett, and Srivastva1995; Gergen 1995), which suggest that words create worlds, AI offers a newchange imperative by suggesting that we be aware of the negativity bias thatpervades our investigations into organizational life and instead shift our focusto the good, the better, and the possibilities that often go undernoticed in oursystems. Building on Gergen (1995) and Cooperrider and Avital (2003), Cooper-rider and Godwin (2012) summarize, “AI posits that human systems move in thedirection of the questions they most frequently and authentically ask; knowl-edge and organizational destiny are intimately interwoven; what we know andhow we study it has a direct impact on where we end up” (740).Leveraging the power of generative questions, AI changes the focus of whatwe typically study in organizational life, questioning the prevailing mindset that“organizations are problems to be solved,” (Cooperrider and Srivastva 1987).Instead, AI suggests that “organizations are mysteries and miracles of humanrelatedness; they are living systems, alive and embedded in ever-wideningwebs of infinite strength and limitless human imagination. Organizations, ascenters of human connectivity and collaboration, are ‘universes of strengths,’” (Cooperrider and Godwin 2010, 10). AI invites change agents to look intotheir organizations with “appreciative eyes”—scanning the system for thingsfor which to be grateful, seeking out what is next and what is possible, andfocusing on valuing those things of value worth valuing. AI theorists posit thatsuch a shift in our approach to organizational change is needed if we are toinspire our imaginative capacities to their fullest potential. DA - 2015/10/19/ PY - 2015 DP - DOI.org (Crossref) ET - 1 SP - 96 EP - 116 LA - en PB - Wiley SN - 978-1-118-94770-8 978-1-119-17662-6 ST - Appreciative Inquiry UR - https://onlinelibrary.wiley.com/doi/10.1002/9781119176626.ch6 Y2 - 2023/10/17/11:04:35 ER - TY - BLOG TI - Appreciative Inquiry Principles AU - Coaching Leaders AB - This article, written by Emotional Intelligence Coach Andy Smith, describes the anticipatory principle which is one of the underpinnings of Appreciative Inquiry (AI). It argues that it is easier and more effective to move towards a positive imagined future than away from a negative one. It is principle number 4 in a series of principles outlined. More principle are listed below. DA - 2015/// PY - 2015 UR - https://coachingleaders.co.uk/category/appreciative-inquiry/appreciative-inquiry-principles/ Y2 - 2018/10/13/00:00:00 ER - TY - JOUR TI - Appreciative Inquiry: Three decades of generative impact AU - Godwin, Lindsey N. T2 - AI Practitioner DA - 2016/02/01/ PY - 2016 DO - 10.12781/978-1-907549-26-7-3 DP - DOI.org (Crossref) VL - 18 IS - 1 SP - 24 EP - 29 J2 - AIP SN - 17418224 ST - Appreciative Inquiry UR - https://aipractitioner.com/product/appreciative-inquiry-three-decades-of-generative-impact/ Y2 - 2023/10/17/11:03:30 ER - TY - CONF TI - Approaches for addressing complexity in programme theory: Experiences in applying the Actor-based Change (ABC) Framework to a variety of international development programmes and evaluations AU - Koleros, Andrew AU - Oldenbeuving, Mark T2 - UK Evaluation Society 2018 Annual Evaluation Conference DA - 2018/05/03/ PY - 2018 DP - Zotero SP - 15 LA - en ER - TY - BLOG TI - Are we suffering from obsessive measurement disorder? AU - Pasanen, Tiina T2 - From Poverty to Power AB - ODI’s Tiina Pasanen argues that more data doesn’t necessarily mean we make better decisions, and sets out some ideas for how we count what counts. DA - 2019/08/15/T06:00:36+00:00 PY - 2019 LA - en-GB UR - https://oxfamblogs.org/fp2p/are-we-suffering-from-obsessive-measurement-disorder/ Y2 - 2020/10/15/11:18:09 ER - TY - RPRT TI - Assessing Impact in Dynamic and Complex Environments: Systemic Action Research and Participatory Systemic Inquiry AU - Burns, Danny T2 - CDI Practice Paper AB - This CDI Practice Paper is about the uses of Systemic Action Research (SAR) and Participatory Systemic Inquiry (PSI) for impact assessment (Burns 2006, 2007, 2011, 2012, 2013; Wadsworth 2001, 2010). CY - Brighton DA - 2014/// PY - 2014 PB - IDS ST - Assessing Impact in Dynamic and Complex Environments UR - http://www.ids.ac.uk/publication/assessing-impact-in-dynamic-and-complex-environments-systemic-action-research-and-participatory-systemic-inquiry Y2 - 2017/01/18/15:37:48 ER - TY - RPRT TI - Assessing Innovation Potential for Social Impact: Overview for Social Sector Leaders AU - Global Knowledge Initiative DA - 2016/// PY - 2016 SP - 26 UR - http://globalknowledgeinitiative.org/wp-content/uploads/2016/10/AIIP-Toolset-Overview-With-Tools_GKI-FINAL.pdf Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - Assessing Rural Transformations: Piloting a Qualitative Impact Protocol in Malawi and Ethiopia AU - Copestake, James AU - Remnant, Fiona T2 - Bath Papers in International Development and Wellbeing Working Paper DA - 2014/// PY - 2014 SP - 30 PB - Centre for Development Studies (CDS) SN - 35 UR - http://www.bath.ac.uk/cds/publications/bpd35.pdf Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - Assessing Value for Money: the Oxford Policy Management Approach AU - King, Julian AU - Daniel Wate AU - Esther Namukasa AU - Alex Hurrell AU - Frances Hansford AU - Patrick Ward AU - Shiva Faramarzifar AB - This document offers practical guidance for assessing the Value for Money (VfM) of government- and donor-financed programmes and policy interventions. In line with OPM’s focus and mission, it has been predominantly applied in the international development sector, but the approach upon which it is based is also used in the context of domestic public policy and programmes.1 There is increasing scrutiny on VfM in international development, but a lack of appropriate methods to support its assessment. There is a risk of reaching invalid conclusions if VfM evaluation is tied to a narrow set of indicators devoid of any evaluative judgement—for example, by emphasising the most readily quantifiable measures rather than the most important (but harder to quantify) aspects of performance, or by focusing on the quantification of outputs and outcomes at the expense of more nuanced consideration of their quality, value, and importance. The approach presented in this guide combines theory and practice from evaluation and economics to respond to requirements for accountability and good resource allocation, as well as to support reflection, learning, and adaptive management. It involves developing and implementing a framework for: • organising evidence of performance and VfM; • interpreting the evidence on an agreed basis; and • presenting a clear and robust performance story. This guide sets out a framework for making and presenting judgements in a way that opens both the reasoning process and the evidence to scrutiny. The approach is designed to be used in alignment with broader monitoring, evaluation, and learning (MEL) systems—both for efficiency's sake, and to ensure conceptual coherence between VfM evaluation and wider MEL work. The VfM framework achieves these aims by: • using explicit criteria (aspects of performance) and standards (levels of performance) to provide a transparent basis for making sound judgements about performance and VfM; • combining quantitative and qualitative forms of evidence to support a richer and more nuanced understanding than can be gained from the use of indicators alone; • accommodating economic evaluation (where feasible and appropriate) without limiting the analysis to economic methods and metrics alone; and • incorporating and building on an approach to VfM evaluation which is familiar to international aid donors. CY - Oxford DA - 2023/// PY - 2023 LA - en PB - Oxford Policy Management UR - http://www.opml.co.uk/publications/opm%E2%80%99s-approach-assessing-value-money Y2 - 2018/03/16/09:58:46 ER - TY - JOUR TI - Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts AU - Morrow, Nathan AU - Nkwake, Apollo M. T2 - Evaluation and Program Planning AB - Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context. We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts. DA - 2016/12/01/ PY - 2016 DO - 10.1016/j.evalprogplan.2016.05.011 DP - ScienceDirect VL - 59 SP - 141 EP - 153 J2 - Evaluation and Program Planning SN - 0149-7189 UR - http://www.sciencedirect.com/science/article/pii/S0149718916301057 Y2 - 2018/02/04/17:23:29 KW - Agency KW - Assumption-aware KW - Complex context KW - Complex programs KW - Participatory approaches KW - Program design KW - Program evaluation KW - Program tools KW - Theory-based ER - TY - JOUR TI - Assumptions, conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development AU - Archibald, Thomas AU - Sharrock, Guy AU - Buckley, Jane AU - Cook, Natalie T2 - Evaluation and Program Planning AB - Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming. DA - 2016/12/01/ PY - 2016 DO - 10.1016/j.evalprogplan.2016.05.015 DP - ScienceDirect VL - 59 SP - 119 EP - 127 J2 - Evaluation and Program Planning SN - 0149-7189 ST - Assumptions, conjectures, and other miracles UR - http://www.sciencedirect.com/science/article/pii/S0149718916301021 Y2 - 2018/02/04/17:17:42 KW - Assumptions KW - Community development KW - Critical thinking KW - Evaluation capacity building KW - Evaluative thinking KW - International Development KW - Theory of change ER - TY - RPRT TI - Back to the Drawing Board: How to improve monitoring of outcomes AU - Dillon, Neil AU - Sundberg, Amelie T2 - ALNAP Paper CY - London DA - 2019/// PY - 2019 DP - Zotero SP - 43 LA - en PB - ODI/ALNAP ER - TY - MGZN TI - Be a participant not a spectator AU - Sharp, Cathy T2 - The Evaluator - UK Evaluation Society AB - My focus here is on the realities of evaluating in complexity where ‘nothing is clear, and everything keeps changing’. I outline how I use a series of ‘provocations’ that allow people to choose their own starting point. Sharing those choices fuels conversations that discover, explore, and co-create (rather than manage) our mutual expectations and assumptions and track how these might themselves be influenced by the work as it unfolds. This account draws on a review of literature and my practice experience, including reflections from others brought into local, national, and international conversations about what it means for evaluation to recognise complexity. DA - 2022/// PY - 2022 IS - Spring UR - https://research-for-real.co.uk/wp-content/uploads/2022/06/Cathy-Sharp-Be-a-participant-not-a-spectator-Winner-Dione-Hills-Tavistock-UKES-prize.pdf Y2 - 2023/05/22/14:20:32 ER - TY - BOOK TI - Being Spherical: Reshaping Our Lives and Our World for the 21st Century AU - Lawson, Phil AU - Lindstrom, Robert L. AB - In this concise, illustrated work exploring the dynamics of living in the 21st century, Phil Lawson and Robert L. Lindstrom show us how we individually and collectively shape the future - a future no longer dominated by traditional hierarchies and concentrated power centers. The authors explain why, in this dawning era of interconnection and interdependency, we require a dramatic transformation in the way we see, think and act. Using mythological imagery in the form of The Sphere and an innovative and adaptive vocabulary, they introduce us to concepts and processes for reshaping our lives, our organizations and our world for the better. In addition, they introduce the Spherical Modeling Tool (SMT), a practical hands-on awareness application. BEING SPHERICAL GUIDES US AS WE LEARN TO: * Prepare for a future we cannot even vaguely predict. * Raise our children well in a climate of moral uncertainty. * Create viable organizations in an era of extreme complexity. * Come to the aid of an abused and ailing planet. AS WE LEARN TO SEE THE SPHERE WE: * Understand why we do what we do and what to do differently. * Discover creative solutions to longstanding problems. * Connect and reconnect within ourselves and with others. * Ready ourselves of quantum leaps in inspiration and awareness. THE GREATER OUR SPHERICAL INTEGRITY THE BETTER WE: * Flex with internal and external pressures. * Adapt to disorienting and accelerating change. * Respond to unpredictable and uncontrollable conditions. * Roll through hard times and bounce forward from adversity. Excerpted from Being Spherical: Reshaping Our Lives and Our World for the 21st Century by Phil Lawson, Robert L. Lindstrom. Copyright 2004. Reprinted by permission. All rights reserved. DA - 2004/04/15/ PY - 2004 DP - Amazon ET - 1st edition SP - 183 LA - English PB - Sphericity Press SN - 978-0-9761910-0-1 ST - Being Spherical ER - TY - RPRT TI - Beneficiary Assessment: An Approach Described AU - Salmen, Lawrence T2 - Social Development Papers AB - Beneficiary assessment is a tool for managers who wish to improve the quality of development operations. This is an approach to information gathering which assesses the value of an activity as it is perceived by its principal users. The approach is qualitative in that it attempts to derive understanding from shared experience as well as observation, and gives primacy to the centrality of the other person’s point of view. As the Bank and others engaged in development activities seek to do their work better, one key indicator will need to be how the ultimate customer, or intended beneficiary, assesses the value of this work, project or policy, as it affects his or her life. The illumination of how an intended beneficiary appreciates a planned or ongoing developmental activity is the primary objective of this approach. Beneficiary assessment is a systematic inquiry into people’s values and behavior in relation to a planned or ongoing intervention for social and economic change. This method draws heavily from the tradition in social science known as "qualitative research...that fundamentally depends on watching people in their own territory and interacting with them in their own language, on their own terms" (Kirk and Miller). Yet beneficiary assessment also includes direct observation, incorporating simple counting, and is expressed in quantitative terms. The ultimate goal of beneficiary assessment is to reveal the meaning people give to particular aspects of their lives so that development activities may better enhance people’s ability to improve their own living conditions, as they see fit. This demands close rapport between the practitioner of this approach, the beneficiary and the development manager. The beneficiary assessment approach is not intended to supplant the questionnaire survey but to provide reliable qualitative, indepth information on the socio-cultural conditions of a beneficiary population which is intended to be of immediate use to managers and policymakers responsible for improving people’s lives. DA - 2002/08// PY - 2002 SP - 29 PB - World Bank SN - 10 UR - http://siteresources.worldbank.org/INTPCENG/1143333-1116505682469/20509250/BAAPProach.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Beneficiary Feedback Mechanisms AU - Hayman, Rachel AB - There is a growing emphasis among NGOs and donors on ensuring that the voices of beneficiaries are heard. A range of beneficiary feedback mechanisms (BFMs) exist which enable beneficiary perspectives and suggestions to be gathered and used within an M&E system. BFMs are tools designed to enable a continuous cycle of interaction between those receiving and those delivering aid-funded interventions. DA - 2020/// PY - 2020 PB - INTRAC ER - TY - RPRT TI - Beneficiary feedback mechanisms AU - UK Aid AB - This guidance seeks to ensure that UK Aid Direct applicants and grant holders understand what the Foreign, Commonwealth & Development Office (FCDO) means by beneficiary feedback mechanisms, and more specifically, that they: • Understand the terms used that relate to beneficiary feedback mechanisms in UK Aid Direct guidance and templates • Understand beneficiary feedback mechanisms and why they are a useful tool for project monitoring and learning • Learn how to use beneficiary feedback mechanisms during project implementation. • Can demonstrate that using beneficiary feedback mechanisms can lead to greater accountability. CY - London DA - 2022/11// PY - 2022 DP - Zotero LA - en PB - FCDO ER - TY - JOUR TI - Between logframes and theory of change: reviewing debates and a practical experience AU - Prinsen, Gerard AU - Nijhof, Saskia T2 - Development in Practice AB - Theory of Change (ToC) is an emerging methodology in the practice of development programmes, often contrasted with the dominant logical framework. This article reviews current debates around ToC before identifying five aspects that are appreciated in practice. It appears that these aspects mostly cover areas where the logical framework is not – or is no longer – meeting the needs of practitioners. Subsequently, the article analyses experiences in ToC training for NGO staff and concludes that ToC can address shortcomings of the logical framework – if only by going back to some of the roots of the logical framework. DA - 2015/02/17/ PY - 2015 DO - 10.1080/09614524.2015.1003532 DP - Taylor and Francis+NEJM VL - 25 IS - 2 SP - 234 EP - 246 SN - 0961-4524 ST - Between logframes and theory of change UR - https://doi.org/10.1080/09614524.2015.1003532 Y2 - 2023/01/12/12:26:31 ER - TY - CHAP TI - Beyond Logframe: Critique, Variations and Alternatives AU - Hummelbrunner, Richard T2 - Beyond Logframe; Using Systems Concepts in Evaluation A2 - Fujita, Nobuko AB - Over the last decades, the Logical Framework Approach (LFA) has become universally known and has assumed a key role for planning and managing development interventions. LFA, however, is not uncontroversial and the approach has been subject to criticism, concerning both its theoretical foundations and practical use. Despite these criticisms LFA’s position has not been fundamentally weakened and while many donors acknowledge its limits and weaknesses, they maintain (some would say impose) its use as a planning and monitoring tool. This chapter reviews some of the experience gained with LFA and outlines major attempts to develop variations – or move beyond it altogether. The first section briefly describes the LFA concept and summarizes the main points of critique. Section 2 then explores some variations which have been developed in response to this critique and to improve LFA as a management tool. Section 3 proposes a systemic alternative to logframe and Section 4 outlines alternatives to LFA which have recently been introduced in German development aid. CY - Tokyo DA - 2010/// PY - 2010 SP - 1 EP - 33 PB - Foundation for Advanced Studies on International Development. UR - http://www.perfeval.pol.ulaval.ca/sites/perfeval.pol.ulaval.ca/files/publication_129.pdf#page=8 Y2 - 2022/01/28/00:00:00 ER - TY - RPRT TI - Beyond Logframes AU - Simister, Nigel AB - The logical framework, otherwise known as a logframe, is a commonly used planning tool. Initially designed for use within simple projects, it is increasingly being applied to complex programmes and organisations. In these cases the logical framework has limitations. There are several options that can help overcome these limitations. These include expanding the logframe, and using multiple logframes. DA - 2017/01// PY - 2017 PB - Intrac UR - https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Beyond-Logframes.pdf Y2 - 2023/01/12/12:31:28 ER - TY - JOUR TI - Beyond the log frame: a new tool for examining health and peacebuilding initiatives AU - Grove, Natalie J. AU - Zwi, Anthony B. T2 - Development in Practice AB - How do we move from identifying ethical principles to enhancing development practice? How can donors and NGOs move beyond the reporting of technical outputs to explore less tangible aspects of their health projects: contributions to rebuilding trust, promoting social cohesion, and enhancing good governance at community level? This article considers these questions in relation to health and peace-building activities in conflicted settings. It describes difficulties facing practitioners and donors seeking to undertake health and peace work, in particular focusing on the lack of appropriate tools for screening, monitoring, and evaluating projects. It critiques the logical framework, a tool commonly used in project planning, monitoring, and evaluation, and considers it alongside a new tool, the Health and Peace Building Filter, which has been designed to reflect on health programming in fragile or conflicted settings. The authors argue that such tools can help to move us beyond focusing on inputs and outputs to examining processes, relationships, and the indirect consequences of aid programmes. DA - 2008/02/01/ PY - 2008 DO - 10.1080/09614520701778850 DP - Taylor and Francis+NEJM VL - 18 IS - 1 SP - 66 EP - 81 SN - 0961-4524 ST - Beyond the log frame UR - https://doi.org/10.1080/09614520701778850 Y2 - 2023/01/12/11:01:39 ER - TY - JOUR TI - Biased Policy Professionals AU - Banuri, Sheheryar AU - Dercon, Stefan AU - Gauri, Varun T2 - The World Bank Economic Review AB - Although the decisions of policy professionals are often more consequential than those of individuals in their private capacity, there is a dearth of studies on the biases of policy professionals: those who prepare and implement policy on behalf of elected politicians. Experiments conducted on a novel subject pool of development policy professionals (public servants of the World Bank and the Department for International Development in the UK) show that policy professionals are indeed subject to decision-making traps, including the effects of framing outcomes as losses or gains, and, most strikingly, confirmation bias driven by ideological predisposition, despite having an explicit mission to promote evidence-informed and impartial decision making. These findings should worry policy professionals and their principals in governments and large organizations, as well as citizens themselves. A further experiment, in which policy professionals engage in discussion, shows that deliberation may be able to mitigate the effects of some of these biases. DA - 2019/06/01/ PY - 2019 DO - 10.1093/wber/lhy033 DP - academic.oup.com VL - 33 IS - 2 SP - 310 EP - 327 J2 - World Bank Econ Rev LA - en SN - 0258-6770 UR - https://academic.oup.com/wber/article/33/2/310/5530388 Y2 - 2019/08/12/22:46:05 KW - bias ER - TY - RPRT TI - Big Data to Data Science - Moving from “What” to “How” in the MERL Tech Space AU - Bertermann, Kecia AU - Robinson, Alexandra AU - Bamberger, Michael AU - Higdon, Grace Lyn AU - Raftre, Linda AB - This paper probes trends in the use of big data by a community of early adopters working in monitoring, evaluation, research, and learning (MERL) in the development and humanitarian sectors. Qualitative analysis was conducted on data from MERL Tech conference records and key informant interviews. Findings indicate that MERL practitioners are in a fragmented, experimental phase, with use and application of big data varying widely, accompanied by shifting terminologies. We take an in-depth look at barriers to and enablers of use of big data within MERL, as well as benefits and drawbacks. Concerns about bias, privacy, and the potential for big data to magnify existing inequalities arose frequently. The research surfaced a need for more systematic and broader sharing of big data use cases and case studies in the development sector. DA - 2020/07// PY - 2020 DP - Zotero SP - 20 LA - en PB - MERL Tech ER - TY - RPRT TI - Biodiversity How-To Guide 2: Using Results Chains to Depict Theories of Change in USAID Biodiversity Programming AU - USAID AB - The Biodiversity How-To Guide 2: Using Results Chains to Depict Theories of Change in USAID Biodiversity Programming is the second in a series of three guides that provide in-depth guidance on key tools and practices to support design teams as they design and manage biodiversity programs within the Program Cycle and in accordance with the USAID Biodiversity Policy. It builds off the situation model guide to help design teams clearly state the expected results and assumptions behind the proposed strategic approaches that make up the program’s theory of change. The situation model provides the foundation to identify necessary results to reduce threats; brainstorm and prioritize strategic approaches; and then develop results chains to clarify assumptions behind selected approaches. CY - Washington DC DA - 2016/08// PY - 2016 LA - en PB - USAID ST - Biodiversity How-To Guide 1 UR - https://rmportal.net/biodiversityconservation-gateway/resources/projects/measuring-impact/how-to-guides-for-usaid-biodiversity-programming/biodiversity-how-to-guide-1-developing-situation-models-in-usaid-biodiversity-programming Y2 - 2019/03/22/11:39:14 ER - TY - SLIDE TI - Bradford Hill Criteria for Causal Inference T2 - 2015 ANZEA Conference A2 - King, Julian AB - We think we’re good at determining causality, but we suck at it One of the great challenges in evaluation is determining whether the results we’re seeing are because of the program we’re evaluating, some other influences out there in the big world, or random chance. At one level, this is an everyday, common sense task. As a species we’ve been making judgments about causation for a million years or so. Unfortunately, though, the way we are wired does not predispose us to logical thinking. We are inclined to be led astray by all sorts of biases and heuristics. Over time, the rocket science for dealing with causation has become more sophisticated – a key example being the experimental study design or randomised controlled trial (RCT). And our evidence base about what works has been enriched as a result. But deep down we’re still biased, heuristical beings and not very good at thinking things through. We’ve become so enthusiastic about experimental designs we’re a little inclined to think they are the only way to determine whether A causes B. Such a rigid view is no much use in the real world, where there are all sorts of ethical, conceptual, practical and economic barriers that mean we can’t always conduct RCTs. Even where technically possible, they are not necessarily the best tool for every job. CY - Auckland DA - 2015/// PY - 2015 UR - https://www.julianking.co.nz/wp-content/uploads/2014/09/150602-BHC-jk5-web.pdf Y2 - 2023/09/29/10:41:30 ER - TY - RPRT TI - Breaking the Mould: Alternative approaches to monitoring and evaluation AU - Dillon, Neil AB - This paper looks at a range of M&E innovations that are designed specifically to provide input to ongoing iterative decision-making and learning at the project level. It identifies three key areas for potential innovation: 1) timing of M&E data provision; 2) flexibility of M&E frameworks to evolve with programme change; and 3) approaches to integrate diverse perspectives on project implementation in a meaningful way. It then looks at a collection of approaches currently being used in each of these three areas through a series of ‘practice examples’, considering the key lessons learned. Finally, the paper discusses the major opportunities and challenges for applying and scaling up the use of these approaches inside the humanitarian sector. CY - London DA - 2019/// PY - 2019 PB - ALNAP UR - https://reliefweb.int/report/world/breaking-mould-alternative-approaches-monitoring-and-evaluation Y2 - 2023/10/17/00:00:00 ER - TY - RPRT TI - Bridging the Gap: Synthesising Evidence from Secondary Quantitative and Primary Qualitative Data AU - Roelen, Keetie T2 - CDI Practice Paper DA - 2016/02// PY - 2016 PB - IDS SN - 15 UR - http://cdimpact.org/blog/%E2%80%98your-story%E2%80%99-versus-%E2%80%98my-story%E2%80%99-finding-truth-disagreement Y2 - 2016/05/12/10:46:20 ER - TY - RPRT TI - Broadening the range of designs and methods for impact evaluations AU - Stern, Elliot AU - Stame, Nicoletta AU - Mayne, John AU - Forss, Kim AU - Davies, Rick AU - Befani, Barbara CY - London DA - 2012/04// PY - 2012 DP - Crossref LA - en PB - DFID UR - http://repository.fteval.at/id/eprint/126 Y2 - 2018/09/26/10:20:40 ER - TY - RPRT TI - Budgeting for Developmental Evaluation: An Interview with Michael Quinn Patton AU - Better Evaluation DA - 2012/04// PY - 2012 SP - 2 UR - https://www.betterevaluation.org/sites/default/files/Budgeting_for_Developmental_Evaluation.pdf Y2 - 2023/03/28/00:00:00 ER - TY - RPRT TI - Building a global learning alliance on adaptive management AU - Wild, Leni AU - Ramalingam, Ben CY - London DA - 2018/09// PY - 2018 DP - Zotero LA - en PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/12327.pdf Y2 - 2018/12/21/00:00:00 ER - TY - RPRT TI - Building a team culture for Adaptive Management in MSD: 5 Strategies MEL Managers Say Work AU - Gover, Dun AU - Nasution, Zulka AU - Okutu, David AU - Bolder, Meghan AU - Henao, Lina T2 - MSD in MEL Brief DA - 2023/// PY - 2023 DP - Zotero LA - en PB - USAID SN - 2 UR - https://agrilinks.org/sites/default/files/media/file/MSD%20in%20MEL%20Brief%202_Building%20Culture_508.pdf Y2 - 2023/10/02/09:38:55 KW - Bolder Meghan KW - Gover Dun KW - Henao Lina KW - Nasution Zulka KW - Okutu David ER - TY - RPRT TI - Building Better Systems - An introduction to System Innovation AU - Winhall, Jennie AU - Leadbeater, Charles AB - This paper lays out a series of steps people can take to create the new systems we need to meet shared, public challenges. Systems are ubiquitous and powerful. We rely on them to support our daily lives: every time we turn on a tap, flick a switch for electricity, drop our child at school, jump on a bus or visit a doctor we rely on a wider system. There is a widespread sense, among decision makers and citizens that in the coming decades society will need not just new products, software and services, but new systems for living sustainably in a socially inclusive society. The need for better, different systems will be heightened by the impact and lessons of the Covid-19 pandemic. Systems are productive precisely because they are more than standalone products. A system pulls together all the different ingredients needed to meet a need or to produce an outcome: the shipping container is a product, containerisation is a system; a contactless payment card is a product which only works as part of a payments system; an operation in a hospital can only take place because it is part of a wider health system. To understand how a system works it has to be seen as a whole, from the macro policy frameworks of social security systems right down to how a citizen goes about finding a job. Many of the systems we rely on for care and work, energy and transport, education and health are under pressure to change. Society faces both deeply entrenched and growing challenges that are outpacing the systems we have. We also have opportunities to create new, alternative systems as new knowledge, values and technologies emerge, from artificial intelligence and bitcoin, to circular and renewable systems of production. Rising to the challenge of fixing an existing system and exploring the possibility of creating a new system are different undertakings. The first is about optimising what exists, the second is about creating something different and better. We want this project to yield practical insights for those who want to respond to the systemic challenges of today by stepping into the possibilities of the future. Acting to change systems depends on new ways of seeing both challenge and opportunity: why systems come under strain and what unlocks the potential for alternatives. It depends on better understanding how new systems form, and what and who is part of initiating and driving the transition to them. In putting together this paper and the ones that will follow from it we want to clarify how to assess the need for, invest in and act on the process of deliberate system change. CY - København K DA - 2020/10// PY - 2020 LA - en-GB PB - The Rockwool Foundation UR - https://www.systeminnovation.org/article-building-better-systems Y2 - 2022/06/17/12:02:20 ER - TY - JOUR TI - Building Bridges to an Uncertain Future Lived Now: Lessons from the Use of Participatory Action Research and Theory of Change Towards A Realistic Community-Based Participatory Monitoring and Evaluation System AU - Avila, Enrique M. AU - Tolentino, Lutgarda L. AU - Binondo, Claudia B. AU - Perez, Maripaz L. AU - Apgar, J. Marina T2 - International Journal of Agriculture System AB - Building on experience from the CGIAR Research Program on Aquatic Agricultural Systems implemented by WorldFish in the Visayas and Mindanao regions of the Philippines, known as the VisMin Hub, we describe the development and evolution of a monitoring and evaluation (M&E) system emerging from the facilitated action-reflection cycles of testing and adopting theories of change carried out with community partners through participatory action research (PAR). The former guides our community partners and us, as members of the potentially emergent PAR groups, towards the realization of the community’s vision; the latter facilitates learning to understand what, how and why change is unfolding. Unlike the conventional M&E system where indicators are pre-set at the beginning of program implementation, these processes result in an organically-evolved, communitybased participatory M&E system that is continuously revised according to contexts to guide communities towards realizing their visions. Its ultimate outcome is enhanced people’s capacity to own the product and process, giving rise to an internally-driven change. Towards the end, the paper offers an iterative discussion of learnings from implementing such an approach. DA - 2016/06/11/ PY - 2016 DO - 10.20956/ijas.v4i1.248 DP - pasca.unhas.ac.id VL - 4 IS - 1 SP - 85 EP - 106 LA - en SN - 2580-6815 ST - Building Bridges to an Uncertain Future Lived Now UR - http://pasca.unhas.ac.id/ojs/index.php/ijas/article/view/248 Y2 - 2018/07/27/09:43:59 KW - Aquatic agricultural systems KW - CGIAR KW - Community-based participatory MEL KW - Learning KW - Participatory action research KW - research in development KW - theory of change ER - TY - JOUR TI - Can’t See the Wood For the Logframe: Integrating Logframes and Theories of Change in Development Evaluation AU - Freer, Gordon AU - Lemire, Sebastian T2 - Canadian Journal of Program Evaluation AB - There are numerous ways in which to model the underlying theory of programs. In the context of international development evaluation, the most ubiquitous are likely “logframes” and to some extent “theories of change,” both of which may serve to guide program development and management, monitoring, and evaluation. While logframes and theories of change are often developed in parallel, they are rarely fully integrated in their practical application. Drawing on lessons from a recent theory-based evaluation, this article argues that fully integrating the program theory of change within the program logframe provides for a stronger and more holistic understanding of program progress. DA - 2019/03/01/ PY - 2019 DO - 10.3138/cjpe.53007 DP - DOI.org (Crossref) VL - 33 IS - 3 J2 - CJPE LA - en SN - 1496-7308, 0834-1516 ST - Can’t See the Wood For the Logframe UR - https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53007 Y2 - 2019/08/12/22:03:02 ER - TY - RPRT TI - Capacity Development in a Participatory Adaptive Programme: the Case of the Clarissa Consortium AU - Widmer, Mireille AU - Apgar, Marina AU - Afroze, Jiniya AU - Malla, Sudhir AU - Healey, Jill AU - Constant, Sendrine T2 - CLARISSA Research and Evidence Paper AB - Doing development differently rests on deliberate efforts to reflect and learn, not just about what programmes are doing and achieving, but about how they are working. This is particularly important for an action research programme like Child Labour: Action- Research-Innovation in South and South-Eastern Asia (CLARISSA), which is implemented by a consortium of organisations from across the research and development spectrum, during a rapidly changing global pandemic. Harnessing the potential of diverse skills and complementary strengths across partners in responding to the complex challenge of the worst forms of child labour, requires capacity to work together in novel ways. This Research and Evidence Paper documents how CLARISSA approached capacity development, and what we learnt from our challenges and successes. From the start, the programme incorporated a capacity development strategy resting on self-assessment of a wide range of behavioural and technical competencies that were deemed important for programme implementation, formal training activities, and periodic review of progress through an after-action review (AAR) process. An inventory of capacity development activities that took place during the first year of implementation reveals a wide range of additional, unplanned activities, enabled by the programme’s flexibility and adaptive management strategy. These are organised into eight modalities, according to the individual or collective nature of the activity, and its sequencing – namely, whether capacity development happens prior to, during, or after (from) implementation. We conclude with some reflections on the emergent nature of capacity development. Planning capacity development in an adaptive programme provides a scaffolding in terms of time, resources, and legitimacy that sustains adaptiveness. We also recognise the gaps that remain to be addressed, particularly on scaling up individual learning to collective capabilities, and widening the focus from implementation teams to individuals working at consortium level. CY - Brighton DA - 2022/04/25/ PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - IDS SN - 1 ST - Capacity Development in a Participatory Adaptive Programme UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17351 Y2 - 2022/07/04/11:51:12 ER - TY - BLOG TI - Case Study AU - Better Evaluation T2 - Better Evaluation AB - A case study focuses on a particular unit - a person, a site, a project. It often uses a combination of quantitative and qualitative data. Case studies can be particularly useful for understanding how different elements fit together and how different elements (implementation, context and other factors) have produced the observed impacts. There are different types of case studies, which can be used for different purposes in evaluation. The GAO (Government Accountability Office) has described six different types of case study: 1. Illustrative: This is descriptive in character and intended to add realism and in-depth examples to other information about a program or policy. (These are often used to complement quantitative data by providing examples of the overall findings). 2. Exploratory: This is also descriptive but is aimed at generating hypotheses for later investigation rather than simply providing illustration. 3. Critical instance: This examines a single instance of unique interest, or serves as a critical test of an assertion about a program, problem or strategy. 4. Program implementation. This investigates operations, often at several sites, and often with reference to a set of norms or standards about implementation processes. 5. Program effects. This examines the causal links between the program and observed effects (outputs, outcomes or impacts, depending on the timing of the evaluation) and usually involves multisite, multimethod evaluations. 6. Cumulative. This brings together findings from many case studies to answer evaluative questions. UR - https://www.betterevaluation.org/en/plan/approach/case_study Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Case Study Evaluations AU - United States General Accounting Office AB - his guide from the US General Accounting Office outlines good practice in case study evaluation and establishes a set of principles for applying case studies to evaluations. The paper outlines new ways of thinking about case studies and examines the methodology necessary to get the best from a case study analysis. DA - 1990/// PY - 1990 SP - 154 PB - GAO UR - https://www.gao.gov/special.pubs/10_1_9.pdf Y2 - 2018/10/13/00:00:00 ER - TY - BOOK TI - Case Study Research and Applications: Design and Methods AU - Yin, Dr Robert K. AB - Winner of the 2019 McGuffey Longevity Award from the Textbook & Academic Authors Association (TAA) Recognized as one of the most cited methodology books in the social sciences, the Sixth Edition of Robert K. Yin′s bestselling text provides a complete portal to the world of case study research. With the integration of 11 applications in this edition, the book gives readers access to exemplary case studies drawn from a wide variety of academic and applied fields. Ultimately, Case Study Research and Applications will guide students in the successful use and application ofthe case study research method. CY - Los Angeles DA - 2018/02/02/ PY - 2018 ET - Sixth edition SP - 352 LA - English PB - SAGE Publications, Inc SN - 978-1-5063-3616-9 ST - Case Study Research and Applications UR - https://uk.sagepub.com/en-gb/eur/case-study-research-and-applications/book250150 ER - TY - BLOG TI - Causal Link Monitoring AU - Better Evaluation T2 - Better Evaluation AB - Causal Link Monitoring (CLM) integrates design and monitoring to support adaptive management of projects. CLM helps project planners and managers identify the processes that are required to achieve desired results, and then to observe whether those processes take place, and how. Result-producing processes specify the causal links between results in a logic model or results framework—in other words, the processes between results. CLM focuses on how specific individuals or organizations use one result to achieve another result. Steps in Causal Link Monitoring In CLM, planners start by creating a logic model to help document predictable, agreed-upon elements of the project. Next, they refine the causal links by describing the processes that will transform results at one point in the causal chain to the next. Planners are often less certain about these result-producing processes. Finally, the CLM logic model is enhanced with information about two important sources of uncertainty, contextual factors that may influence the project and diverse perspectives on the problem and its solution. The process can be broken down into seven steps, three for project design, three for monitoring, and a final step in which monitoring data informs redesign: Build a logic model. Identify assumptions about causal links. Enhance the logic model with diverse perspectives and contextual factors. Prioritize areas of observation. Collect monitoring data. Interpret and use monitoring data for adaptive management. Revise the logic model. UR - https://www.betterevaluation.org/en/approach/causal_link_monitoring Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Causal Link Monitoring AU - Britt, Heather AU - Hummelbrunner, Richard AU - Greene, Jacqueline AB - Causal Link Monitoring (CLM) 1 integrates design and monitoring to support adaptive management of projects. CLM helps project planners and managers identify the processes that are required to achieve desired results, and then to observe whether those processes take place, and how. Result-producing processes specify the causal links between results in a logic model or results framework—in other words, the processes between results.2 CLM focuses on how specific individuals or organizations use results to achieve other results. In CLM, planners start by creating a logic model to help document predictable, agreed-upon elements of the project. Next, they refine the causal links by describing the processes that will transform results at one point in the causal chain to the next. Planners are often less certain about these resultproducing processes. Finally, the CLM logic model is enhanced with information about two important sources of uncertainty, contextual factors that may influence the project and diverse perspectives on the problem and its solution. DA - 2017/04// PY - 2017 SP - 41 UR - https://www.betterevaluation.org/sites/default/files/CLM%20Brief_20170615_1528%20FINAL.pdf Y2 - 2018/09/22/00:00:00 ER - TY - BLOG TI - Causal Loop Diagrams: Little Known Analytical Tool | iSixSigma AU - Rushing, William AB - The causal loop diagram is an analytical tool that is seldom used in Six Sigma but nonetheless is still very valuable. It is a foundational tool used in system dynamics, a method of analysis used to develop an understanding of complex systems. DA - 2010/// PY - 2010 LA - en-US ST - Causal Loop Diagrams UR - https://www.isixsigma.com/tools-templates/cause-effect/causal-loop-diagrams-little-known-analytical-tool/ Y2 - 2019/02/06/09:44:05 ER - TY - JOUR TI - Causal mapping for evaluators AU - Powell, Steve AU - Copestake, James AU - Remnant, Fiona T2 - Evaluation AB - Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally i... DA - 2023/09/14/ PY - 2023 DO - 10.1177/13563890231196601 DP - journals.sagepub.com LA - en UR - https://journals.sagepub.com/eprint/K9VKYVWWVXZJCVCPNK9S/full AN - Sage UK: London, England Y2 - 2023/10/20/12:46:55 ER - TY - ELEC TI - Center for Theory of Change T2 - Theory of Change Community LA - en-US UR - https://www.theoryofchange.org/ Y2 - 2019/08/12/22:31:25 ER - TY - RPRT TI - Challenge-led Innovation Workbook. Organising for Systems Innovation at Scale AU - Burkett, Ingrid AB - Organising for Systems Innovation at Scale Our team at Griffith Centre for Systems Innovation have been experimenting with and evolving a Challenge-led Innovation Approach (based on Mission-oriented approaches developed by Mariana Mazzucato at UCL IIPP and others internationally). We are using this approach to guide the way we work internally and engage with our systems innovation partners. We’ve facilitated intensive Re:Treats, worked with government bodies, businesses and civic organisations, and engaged deeply with others exploring this work. We have a bias for developing and testing HOW such approaches could be applied to respond to both local and global challenges rather than getting too caught up in the what and why of such approaches. We decided to openly share our learnings and thinking to date in this workbook, to spark conversations and innovation in both practice and thinking amongst those exploring how we work, and to learn together to address complex systems and challenges. We see this booklet as a first step in a longer learning journey. In it we share an overview of: the principles and processes that sparked our evolution to a Challenge-led Innovation framework (from Mission-oriented). examples of our learnings from other system innovators who are experimenting. an adaptable process to help guide the learning journey. learning tools and canvases to catalyse thinking, practice, and further adaptations. Part One sets out some foundations we’ve identified as important to Challenge-led Innovation. If you want to jump straight into the mapping process, we suggest you skip to Part Two. The final section, Part Three, focuses on what we have learnt about the conditions needed and how to get started on a Challenge-led initiative. CY - Logan DA - 2023/11// PY - 2023 LA - en PB - Griffith Centre for Systems Innovation UR - https://www.griffith.edu.au/__data/assets/pdf_file/0033/1881573/GCSI-Challenge-Led-Innovation-Workbook.pdf Y2 - 2024/02/29/12:07:29 ER - TY - BOOK TI - Change: How to Make Big Things Happen AU - Centola, Damon AB - How to create the change you want to see in the world using the paradigm-busting ideas in this "utterly fascinating" (Adam Grant) big-idea book.​ Most of what we know about how ideas spread comes from bestselling authors who give us a compelling picture of a world, in which "influencers" are king, "sticky" ideas "go viral," and good behavior is "nudged" forward. The problem is that the world they describe is a world where information spreads, but beliefs and behaviors stay the same. When it comes to lasting change in what we think or the way we live, the dynamics are beliefs and behaviors are not transmitted from person to person in the simple way that a virus is. The real story of social change is more complex. When we are exposed to a new idea, our social networks guide our responses in striking and surprising ways. Drawing on deep-yet-accessible research and fascinating examples from the spread of coronavirus to the success of the Black Lives Matter movement, the failure of Google+, and the rise of political polarization, Change presents groundbreaking and paradigm-shifting new science for understanding what drives change, and how we can change the world around us. CY - London DA - 2021/// PY - 2021 DP - Amazon SP - 352 LA - English PB - Little, Brown Spark SN - 978-0-316-45733-0 ST - Change ER - TY - BLOG TI - Changing theories AU - Aston, Thomas T2 - Medium AB - You may, or may not, be surprised to hear that many theories of change lack what we might generally understand as a theory. DA - 2023/10/15/T20:40:09.796Z PY - 2023 LA - en UR - https://thomasmtaston.medium.com/changing-theories-e857aa8fba05 Y2 - 2023/10/17/22:05:40 ER - TY - RPRT TI - CLA Maturity Tool - Card Deck (Implementing Partners version 1) AU - USAID LEARN AB - USAID’s Bureau for Policy, Planning and Learning (PPL) and its support mechanism, LEARN, have developed a Collaborating, Learning, and Adapting (CLA) Framework and Maturity tool to help USAID missions think more deliberately about how to plan for and implement CLA approaches that fit the mission’s context and assist them in achieving their development objectives. While the tool is intended primarily for USAID audiences to be used in participatory self-assessment workshops, the CLA Framework and maturity spectrum are relevant to a wider audience. USAID’s CLA Framework identifies key components and subcomponents of daily work that may be opportunities for intentional, systematic, and resourced CLA. The framework recognizes the diversity of what CLA can look like in various organizations and programs while also giving CLA structure, clarity, and coherence across two key dimensions: • CLA in the Program Cycle: how CLA is incorporated throughout Program Cycle processes, including strategy, project, and activity design and implementation; and • Enabling Conditions: how an organization’s culture, business processes, and resource allocation support CLA integration. Recognizing that CLA is not binary—it’s not an issue of “doing it or not doing it”—PPL and LEARN have developed a spectrum of practice for each of the 16 subcomponents in the framework. The spectrum offers examples of what integration might look like at different stages: Not Yet Present, Emergent, Expanding, Advanced and Institutionalized. The maturity stage descriptions are only illustrative and are intended to spark reflection on current practice and opportunities for improvements. In this resource, each CLA subcomponent page describes the key concepts for that topic and includes a description of the maturity stages. Although the descriptions were originally developed for USAID, the majority of the concepts easily transfer or have equivalents in the partner community. For example, although organizations outside of USAID may not hold “Portfolio Reviews” (part of the Pause & Reflect subcomponent), the majority hold some type of meeting to review programmatic progress. This is the seventh version of the CLA Framework and maturity spectrum. PPL and LEARN will continue reviewed and periodically update them based on user feedback, so if you have comments about the content, please email learning@usaid.gov. We would also love to hear how you’ve used this content with your team or organization. CY - Washington DC DA - 2022/// PY - 2022 PB - USAID UR - https://usaidlearninglab.org/sites/default/files/2022-08/508c_cla_maturity_tool_card_deck_ip_v1_2022-07-29.pdf Y2 - 2023/01/03/12:59:23 ER - TY - JOUR TI - Clearing the fog: new tools for improving the credibility of impact claims AU - Befani, Barbara AU - D'Errico, Stefano AU - Booker, Francesca AU - Giuliani, Alessandra T2 - iied Briefing Papers AB - Development actors facing pressure to provide more rigorous assessments of their impact on policy and practice need new methods to deliver them. There is now a broad consensus that the traditional counterfactual analysis leading to the assessment of the net effect of an intervention is incapable of capturing the complexity of factors at play in any particular policy change. We suggest that evaluations focus instead on establishing whether a clearly-defined process of change has taken place, and improve the validity and credibility of qualitative impact statements. IIED research in Uganda shows that the methods of process tracing and Bayesian updating facilitate a dialogue between theory and evidence that allows us to assess our degree of confidence in ‘contribution claims’ in a transparent and replicable way. DA - 2016/04// PY - 2016 DP - pubs.iied.org ST - Clearing the fog UR - https://pubs.iied.org/17359IIED/ Y2 - 2019/06/04/17:16:18 ER - TY - BLOG TI - Collaborative Outcome Mapping AU - Dart, J. AU - Roberts, M. T2 - Better Evaluation AB - Collaborative Outcomes Reporting (COR) is a participatory approach to impact evaluation based around a performance story that presents evidence of how a program has contributed to outcomes and impacts, that is then reviewed by both technical experts and program stakeholders, which may include community members. Collaborative Outcomes Reporting (COR) is a participatory approach to impact evaluation based around a performance story that presents evidence of how a program has contributed to outcomes and impacts, that is then reviewed by both technical experts and program stakeholders, which may include community members. Developed by Jess Dart, COR combines contribution analysis andMultiple Lines and Levels of Evidence (MLLE), mapping existing data and additional data against the program logic to produce a performance story. Performance story reports are essentially a short report about how a program contributed to outcomes. Although they may vary in content and format, most are short, mention program context and aims, relate to a plausible results chain, and are backed by empirical evidence (Dart and Mayne, 2005). The aim is to tell the ‘story’ of a program’s performance using multiple-lines of evidence. COR adds processes of review by an expert panel and stakeholders, sometimes including community members, to check for the credibility of the evidence about what impacts have occurred and the extent to which these can be credibly attributed to the intervention. It is these components of expert panel review (outcomes panel) and a collaborative approach to developing outcomes (through summit workshops) that differentiate COR from other approaches to outcome and impact evaluation. DA - 2014/// PY - 2014 UR - https://www.betterevaluation.org/en/plan/approach/cort Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Combining Big Data and Thick Data to Improve Services Delivery AU - Ang, Yuen Yuen DA - 2018/// PY - 2018 PB - IBM Center for The Business of Government UR - http://www.businessofgovernment.org/node/2513 ER - TY - JOUR TI - Communications Strategic Intent With Systemigrams: Application to the Network-Enabled Challenge AU - Blair, Charles AU - Boardman, John AU - Sauser, Brian T2 - Systems Engineering AB - The U.K. Ministry of Defense (MoD) has mandated the development of a network enabled capability (NEC) across all of defense, aimed at producing agile military and nonmilitary effects via a network of networks. This paper provides an overview of NEC, representing it as a complex human activity system of systems (SoS), analysis of which cannot rely on purely traditional reductionist engineering approaches, requiring instead a soft-systems engineering approach. A literature review is then provided, covering nontraditional systems methodologies of the past 25 years, highlighting the more recent trend towards multimethodological practice. The paper introduces the systemic diagram, or systemigram, conceptual model, explaining its evolution from a form of visual language to its use as an appreciative learning system in a soft-systems methodology. Using the written prose of MoD policy makers, a systemigram model is constructed which represents the NEC concept, providing a systemic visualization of its complexity and an elucidation of the key SoS attributes of emergence, hierarchy, and boundary. Finally, the NEC systemigram is used in an example storyboarding technique, demonstrating its utility as a platform for stakeholder dialog leading towards a refined model that reflects a deeper understanding of NEC strategy. © 2007 Wiley Periodicals, Inc. Syst Eng 10:309–322, 2007 DA - 2007/12/01/ PY - 2007 DO - 10.1002/sys.20079 DP - ResearchGate VL - 10 SP - 309 EP - 322 J2 - Systems Engineering ST - Communications Strategic Intent With Systemigrams ER - TY - RPRT TI - Complexity-Aware Monitoring AU - Britt, Heather AU - Patsalides, Melissa T2 - Briefing on USAID's Discussion Note DA - 2013/// PY - 2013 SP - 30 M3 - Presentation PB - USAID's Learning, Evaluation, and Research Office UR - http://623elmp01.blackmesh.com/sites/default/files/resource/files/c-am_discussion-note-brief_slides.pdf Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - Complexity-Aware Monitoring and Evaluation AU - Hertz, Tilman AU - Brattander, Eva AU - Rose, Loretta T2 - Journal of MultiDisciplinary Evaluation AB - Background: Addressing today’s sustainability challenges requires adopting a systemic approach where social and ecological systems are treated as integrated social-ecological systems. Such systems are complex, and the international development sector increasingly recognises the need to account for the complexity of the systems that they seek to transform. Purpose: This paper sketches out the elements of a complexity-aware monitoring and evaluation (M&E) system for international development programmes in the area of sustainable development.  Setting: Not applicable. Data Collection and Analysis: The authors draw on existing literature on complexity and evaluation and on their own experience from working in the field of M&E. Findings: An M&E system should not be seen simply as a tool to track compliance against a pre-determined theory of change. Instead, it is most useful as a real-time approach, constantly defining and re-defining narratives for change that help push systems along trajectories of interest. Dealing with complexity involves embracing uncertainty; and this challenges established notions of accountability—something which funders and implementers must begin to redefine together. Keywords: monitoring; evaluation; complexity; social-ecological systems; international development programmes; narratives for change; theory of change DA - 2021/06/21/ PY - 2021 DO - 10.56645/jmde.v17i41.679 DP - journals.sfu.ca VL - 17 IS - 41 SP - 35 EP - 50 LA - en SN - 1556-8180 UR - https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/679 Y2 - 2023/08/06/19:18:08 ER - TY - SLIDE TI - Complexity-Aware Monitoring Discussion Note (Screencast) T2 - USAID A2 - Patsalides, Melissa A2 - Britt, Heather AB - This screencast covers information included in the Complexity-Aware Monitoring Discussion Note. The topic is intended for those seeking cutting-edge solutions to monitoring complex aspects of strategies and projects. Most of the principles and promising approaches discussed here have a significant body of theory and practice behind them, but many have not been used to monitor USAID strategies and projects. We have much to learn about whether and how they can be applied successfully in the Agency. USAID's Office of Learning, Evaluation, and Research in the Bureau of Policy, Planning, and Learning has partnered with different groups through monitoring trials to build a body of evidence regarding which complexity-aware M&E approaches are most effective in the contexts of USAID programming. DA - 2014/// PY - 2014 M3 - Screencast UR - https://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-screencast ER - TY - RPRT TI - Conservation Enterprises: Using A Theory of Change Approach to Synthesize Lessons from USAID Biodiversity Projects AU - Boshoven, Judy T2 - Measuring Impact AB - In an effort to increase the understanding of conservation enterprises’ activities and outcomes and to improve the effectiveness of biodiversity programming, this brief synthesizes lessons from past USAID-funded efforts to support conservation enterprises CY - Washington DC DA - 2017/04// PY - 2017 DP - Zotero SP - 14 LA - en M3 - Technical Brief PB - USAID UR - https://www.fosonline.org/wordpress/wp-content/uploads/2017/04/pbaaf622.pdf Y2 - 2023/12/22/00:00:00 ER - TY - RPRT TI - Context-monitoring for adaptive management AU - Pickwick, Sarah DA - 2022/// PY - 2022 PB - World Vision UR - https://oxfamapps.org/fp2p/wp-content/uploads/2022/01/World-Vision-Context-monitoring-for-adaptive-management-.docx Y2 - 2022/01/12/00:00:00 ER - TY - BLOG TI - Contribution Analysis AU - Better Evaluation T2 - Better Evaluation AB - Contribution Analysis is an approach for assessing causal questions and inferring causality in real-life program evaluations. It offers a step-by-step approach designed to help managers, researchers, and policymakers arrive at conclusions about the contribution their program has made (or is currently making) to particular outcomes. The essential value of contribution analysis is that it offers an approach designed to reduce uncertainty about the contribution the intervention is making to the observed results through an increased understanding of why the observed results have occurred (or not!) and the roles played by the intervention and other internal and external factors. Contribution analysis is particularly useful in situations where the programme is not experimental, i.e. not in trial projects but in situations where the programme has been funded on the basis of a relatively clearly articulated theory of change and where there is little or no scope for varying how the program is implemented. Contribution analysis helps to confirm or revise a theory of change; it is not intended to be used to surface or uncover and display a hitherto implicit or inexplicit theory of change. The report from a contribution analysis is not definitive proof, but rather provides evidence and a line of reasoning from which we can draw a plausible conclusion that, within some level of confidence, the program has made an important contribution to the documented results. UR - https://www.betterevaluation.org/en/plan/approach/contribution_analysis Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Contribution Analysis: An approach to exploring cause and effect AU - Mayne, J. T2 - ILAC Brief AB - Questions of cause and effect are critical to assessing the performance of programmes and projects. When it is not practical to design an experiment to assess performance, contribution analysis can provide credible assessments of cause and effect. Verifying the theory of change that the programme is based on, and paying attention to other factors that may influence the outcomes, provides reasonable evidence about the contribution being made by the programme. DA - 2008/05// PY - 2008 SP - 4 PB - ILAC SN - 15 UR - http://www.pointk.org/resources/files/Contribution_Analysis.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Contribution analysis for adaptive management AU - Apgar, Marina AU - Hernandez, Kevin AU - Ton, Giel T2 - Briefing Paper AB - This briefing note shares practical learning on the use of contribution analysis for adaptive management (CA4AM). It examines how the approach enables programmes to work with theories of change in a practical, reflexive way, and how, combined with assessing evidence of a programme’s contribution to change, its findings can inform programme adaptation. It also examines both how and to what extent CA enables AM through the experiences of four large complex programmes all working towards systems-level change and employing a structured process of reflection on theories of change. Key messages CA4AM can enable programmes to work with theories of change in a practical, reflexive way. It is particularly useful for programmes operating in conditions of complexity, when it is difficult to discern attribution and when systems-level change is the goal. A range of enabling factors help CA4AM to be used most effectively, including contractual flexibility; embedded monitoring, evaluation and learning (MEL); and supportive leadership. DA - 2020/// PY - 2020 DP - Zotero SP - 14 LA - en PB - IDS/GLAM ER - TY - RPRT TI - Contribution Rubrics: A simple way to assess influence AU - Aston, Tom AB - This document explores how organisations can measure the level of influence that they had over an outcome. It summarises influence in terms of the significance of the outcome, the level of contribution and the strength of evidence. DA - 2020/01// PY - 2020 LA - en-GB PB - Independent Consultant ST - Contribution Rubics UR - http://www.kwantu.net/resources-1/2020/1/27/contribution-rubics-a-simple-way-to-assess-influence Y2 - 2020/02/14/10:33:34 ER - TY - JOUR TI - Cost–Benefit Analysis (CBA) or the Highway? An Alternative Road to Investigating the Value for Money of International Development Research AU - Peterson, Heidi T2 - The European Journal of Development Research AB - Research for development (R4D) funding is increasingly expected to demonstrate value for money (VfM). However, the dominance of positivist approaches to evaluating VfM, such as cost-benefit analysis, do not fully account for the complexity of R4D funds and risk undermining efforts to contribute to transformational development. This paper posits an alternative approach to evaluating VfM, using the UK’s Global Challenges Research Fund and the Newton Fund as case studies. Based on a constructivist approach to valuing outcomes, this approach applies a collaboratively developed rubric-based peer review to a sample of projects. This is more appropriate for the complexity of R4D interventions, particularly when considering uncertain and emergent outcomes over a long timeframe. This approach could be adapted to other complex interventions, demonstrating that our options are not merely “CBA or the highway” and there are indeed alternative routes to evaluating VfM. DA - 2023/04/01/ PY - 2023 DO - 10.1057/s41287-022-00565-7 DP - Springer Link VL - 35 IS - 2 SP - 260 EP - 280 J2 - Eur J Dev Res LA - en SN - 1743-9728 ST - Cost–Benefit Analysis (CBA) or the Highway? UR - https://doi.org/10.1057/s41287-022-00565-7 Y2 - 2023/04/13/12:38:56 ER - TY - RPRT TI - Creating the (organisational) conditions for an OM­ based M&E and learning practice AU - Deprez, Steff DA - 2009/// PY - 2009 DP - Zotero SP - 8 LA - en PB - Outcome Mapping Learning Community UR - https://www.outcomemapping.ca/download/steff.deprezzol.co.zw_en_2009-deprez_steff-org%20conditions.pdf Y2 - 2023/12/22/00:00:00 ER - TY - JOUR TI - Critical friends - real time insights for shaping strategy AU - Johnson, Debbie AU - Nathan, Geeta AU - Rahman, Syahirah Abdul T2 - Chapters AB - This chapter focuses on facilitation given by the Innovation Caucus to provide expert academic critique to inform Innovate UK's strategy for UK business innovation. As Innovate UK's strategy developed, it became evident that several policy domains needed critical insights and evidence. An academic critical friend provided the latest academic insights, evidence and wider perspective of the actors in the UK Innovation system, insights of which policy makers often lack. The chapter gave case studies of Innovation Caucus' work with Innovate UK in facilitating extensive research expertise and helping policy makers to open dialogue. It also describes 'The Innovators' Breakfast Club', a webinar series organised by Innovation Caucus, co-developed with Innovate UK, which created live face-to-face dialogue between academic experts and Innovate UK strategy/policy leads. This example showed how critical friends are important in facilitating the initial dialogues needed to build evidence for policy making. The dialogue builds the foundation for a virtuous cycle of learning. The approach provides an opportunity for academic experts to gain insights into the priorities and dilemmas facing policy makers, to shape future academic research and refine the skill set needed to become critical friends. DA - 2022/// PY - 2022 DP - ideas.repec.org SP - 104 EP - 112 LA - en UR - https://ideas.repec.org//h/elg/eechap/20351_10.html Y2 - 2023/12/14/21:26:46 ER - TY - BLOG TI - Critical System Heuristics : a very, very short introduction AU - van 't Hof, Sjon T2 - CSL4D/SystemicAgency DA - 2021/03/12/ PY - 2021 UR - https://csl4d.wordpress.com/2012/03/12/critical-system-heuristics-a-very-very-short-introduction/ Y2 - 2018/10/13/00:00:00 ER - TY - BLOG TI - Critical Systems Heuristics AU - Better Evaluation T2 - Better Evaluation AB - Critical System Heuristics (CSH) provides a framework of questions about a program including what is (and what ought to be) its purpose and its source of legitimacy and who are (and who ought to be) its intended beneficiaries. CSH, as developed by Werner Ulrich and later elaborated upon in collaboration with Martin Reynolds, is an approach used to surface, elaborate, and critically consider boundary judgments, that is, the ways in which people/groups decide what is relevant to the system of interest (any situation of concern). CSH is concerned not only with purposive evaluation, where the system or project has a predefined goal and the focus lies in evaluating the means of reaching it, but also more broadly with purposeful evaluation, where both the means and the ends become subjects of inquiry. CSH rests on the foundations of systems thinking and practical philosophy, both of which emphasize the 'infinite richness' of the real world. In this view, understandings of any situation are inherently incomplete, and therefore based on the selective application of knowledge. By systematically questioning the sources of motivation, control, expertise, and legitimation in the system of interest, CSH allows users to make their boundary judgments explicit and defensible. The immediate goal of a CSH evaluation is to elaborate multiple perspectives on a given situation, but the broader aim is to share these perspectives and thereby cut down on actors 'talking past' each other by promoting mutual understanding. UR - https://www.betterevaluation.org/en/plan/approach/critical_system_heuristics Y2 - 2018/10/13/00:00:00 ER - TY - CHAP TI - Critical systems heuristics AU - Ulrich, Werner AU - Reynolds, Martin T2 - Systems Approaches to Manging Change: A Practical Guide A2 - Reynolds, Martin A2 - Holwell, Sue AB - This chapter provides a detailed introduction to critical systems heuristics and the use of its central tool boundary critique. What Is CSH? Two Studies in Applying CSH Using CSH as an Intervention Tool: Some Basic Concepts A Core Concept of CSH: Systematic Boundary Critique Boundary Critique Applied to NRUA-Botswana Boundary Critique Applied to ECOSENSUS-Guyana Boundary Critique and Personal Competence Recognising Boundary Judgements – and Keeping Them Fluid Towards a New Ethos of Professional Responsibility CY - London DA - 2010/// PY - 2010 SP - 243 EP - 292 PB - Springer UR - http://oro.open.ac.uk/21299/1/systems-approaches_ch6.pdf Y2 - 2018/10/13/00:00:00 ER - TY - BOOK TI - Cultivating Communities of Practice: A Guide to Managing Knowledge AU - Wenger, Etienne AU - McDermott, Richard Arnold AU - Snyder, William AB - Today's marketplace is fueled by knowledge. Yet organizing systematically to leverage knowledge remains a challenge. Leading companies have discovered that technology is not enough, and that cultivating communities of practice is the keystone of an effective knowledge strategy.Communities of practice come together around common interests and expertise- whether they consist of first-line managers or customer service representatives, neurosurgeons or software programmers, city managers or home-improvement amateurs. They create, share, and apply knowledge within and across the boundaries of teams, business units, and even entire companies-providing a concrete path toward creating a true knowledge organization.In Cultivating Communities of Practice, Etienne Wenger, Richard McDermott, and William M. Snyder argue that while communities form naturally, organizations need to become more proactive and systematic about developing and integrating them into their strategy. This book provides practical models and methods for stewarding these communities to reach their full potential-without squelching the inner drive that makes them so valuable.Through in-depth cases from firms such as DaimlerChrysler, McKinsey & Company, Shell, and the World Bank, the authors demonstrate how communities of practice can be leveraged to drive overall company strategy, generate new business opportunities, tie personal development to corporate goals, transfer best practices, and recruit and retain top talent. They define the unique features of these communities and outline principles for nurturing their essential elements. They provide guidelines to support communities of practice through their major stages of development, address the potential downsides of communities, and discuss the specific challenges of distributed communities. And they show how to recognize the value created by communities of practice and how to build a corporate knowledge strategy around them.Essential reading for any leader in today's knowledge economy, this is the definitive guide to developing communities of practice for the benefit-and long-term success-of organizations and the individuals who work in them.Etienne Wenger is a renowned expert and consultant on knowledge management and communities of practice in San Juan, California. Richard McDermott is a leading expert of organization and community development in Boulder, Colorado. William M. Snyder is a founding partner of Social Capital Group, in Cambridge, Massachusetts. DA - 2002/// PY - 2002 DP - Google Books SP - 306 LA - en PB - Harvard Business Press SN - 978-1-57851-330-7 ST - Cultivating Communities of Practice ER - TY - BOOK TI - DE 201: A practitioner's guide to developmental evaluation AU - Dozois, Elizabeth AU - Langlois, Marc AU - Blanchet-Cohen, Natasha AB - Developmental evaluation has emerged fairly recently as a way to support adaptive learning in complex and emergent initiatives. Combining the rigour of evaluation with the flexibility and imagination required for development, this new form of evaluation brings critical thinking to bear on the creative process in initiatives involving high levels of uncertainty, innovation, emergence, and social complexity. This guidebook from J.W. McConnell Family Foundation and the International Institute for Child Rights and Development explores key aspects of Developmental Evaluation (DE) and offers suggestions for implementing the practice and includes a range of DE resources to help with its facilitation. Contents What is developmental evaluation? When is developmental evaluation appropriate? What competencies are needed to be an effective DE? How is developmental evaluation practiced? Challenges Appendices Assessing Readiness Stakeholder analysis Sample Learning Framework Systems Analysis Framework - Key Components Data Analysis Resources CY - Montréal, Que. DA - 2010/// PY - 2010 DP - Open WorldCat LA - English PB - J.W. McConnell Family Foundation SN - 978-1-55058-424-0 ST - DE 201 UR - https://mcconnellfoundation.ca/wp-content/uploads/2017/07/DE-201-EN.pdf Y2 - 2018/10/13/16:22:56 ER - TY - BLOG TI - Debiasing: a systematic discipline and delight for development professionals AU - Chambers, Robert T2 - Institute of Development Studies AB - Do we really need debiasing, yet another word?  Yes, unless anyone can improve on it, because we need a word to describe a rigorous discipline we development professionals need for grounded realism. This has been coming on me slowly. But now explorations and ‘aha!’ moments in India have accumulated and combined into an epiphany.  For […] DA - 2018/12/18/T15:17:36+00:00 PY - 2018 LA - en-GB ST - Debiasing UR - https://www.ids.ac.uk/opinions/debiasing-a-systematic-discipline-and-delight-for-development-professionals/ Y2 - 2020/10/16/14:06:20 ER - TY - BLOG TI - Democratic Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Democratic Evaluation is an approach where the aim of the evaluation is to serve the whole community. This allows people to be informed of what others are doing and sees the evaluator as someone who brokers the process. It generally focuses on inclusive practices which foster participation and collaboration. However it is also used as a means of ensuring public accountability and transparency. UR - https://www.betterevaluation.org/en/plan/approach/democratic_evaluation Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Democratic governance and evaluation. AU - Hanberger, A. T2 - Sixth EES (European Evaluation Society) conference, Berlin, Germany AB - This paper, presented by Anders Hanberger at the Sixth EES (European Evaluation Society) Conference in Berlin, Germany (September 30-October 2, 2004) argues that governance, democracy and evaluation impact each in different ways. It offers a discussion centred around the evaluation of three general democratic governance models and the implications of leading democratic evaluations. "Since governance and democracy are changing phenomena, and evaluation is embedded in these structures, there is a need to illuminate and discuss the role of evaluation. Evaluation is to most people thought of as a democratic tool, but what do we mean with democratic evaluation? Furthermore, what is the role of evaluation in times when democracy and governance are changing? The many meanings of democracy and the shifting role of evaluation in various democratic governance settings are also a motive for discussing governance and democratic evaluation. Governance issues can be discussed in relation to different political systems. However, in this paper the discussion is confined to democratic governance systems. The premise of this paper is that governance, democracy and evaluation affect one another in different ways. Governance is intertwined with democracy, and democracy and governance can be maintained or strengthened by evaluation, for example. Because different models of governance and democracy presuppose one another, evaluating governance models, or programme processes/outcomes where a specific governance model sets up the context, have implications for the model under scrutiny and subsequently for democracy. Accordingly, the implications of democratic evaluations need to be discussed in various ways. For analytical purpose democracy and governance are sometimes kept apart." (Hanberger, 2004)​ DA - 2004/10// PY - 2004 SP - 24 UR - http://www.edusci.umu.se/digitalAssets/66/66094_hanbergergovernance04.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Designing a Participatory Programme at Scale: Phases 1 and 2 of the CLARISSA Programme on Worst Forms of Child Labour AU - Burns, Danny AU - Apgar, Marina AU - Raw, Anna T2 - CLARISSA Working Paper AB - CLARISSA (Child Labour: Action-Research-Innovation in South and South-Eastern Asia) is a large-scale Participatory Action Research programme which aims to identify, evidence, and promote effective multi-stakeholder action to tackle the drivers of the worst forms of child labour in selected supply chains in Bangladesh, Nepal, and Myanmar. CLARISSA places a particular focus on participants’ own ‘agency’. In other words, participants’ ability to understand the situation they face, and to develop and take actions in response to them. Most of CLARISSA’s participants are children. This document shares the design and overarching methodology of the CLARISSA programme, which was co-developed with all consortium partners during and since the co-generation phase of the programme (September 2018–June 2020). The immediate audience is the CLARISSA programme implementation teams, plus the Foreign, Commonwealth & Development Office (FCDO). This design document is also a useful reference point for other programmes trying to build large-scale participatory processes. It provides a clear overview of the CLARISSA programmatic approach, the design, and how it is being operationalised in context. CY - Brighton DA - 2021/07/08/ PY - 2021 LA - en PB - Institute of Development Studies SN - 7 ST - Designing a Participatory Programme at Scale UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/16730 Y2 - 2023/01/10/14:52:40 ER - TY - RPRT TI - Designing Contribution Analysis of Participatory Programming to Tackle the Worst Forms of Child Labour AU - Apgar, Marina AU - Snijder, Mieke AU - Prieto Martin, Pedro AU - Ton, Giel AU - Macleod, Shona AU - Kakri, Shanta AU - Paul, Sukanta T2 - CLARISSA Research and Evidence Paper AB - This Research and Evidence Paper presents the theory-based and participatory evaluation design of the Child Labour: Action-Research- Innovation in South and South-Eastern Asia (CLARISSA) programme. The evaluation is embedded in emergent Participatory Action Research with children and other stakeholders to address the drivers of the worst forms of child labour (WFCL). The report describes the use of contribution analysis as an overarching approach, with its emphasis on crafting, nesting and iteratively reflecting on causal theories of change. It illustrates how hierarchically-nested impact pathways lead to specific evaluation questions and mixing different evaluation methods in response to these questions, critical assumptions, and agreement on causal mechanisms to be examined in depth. It also illustrates how realist evaluation can be combined with contribution analysis to deeply investigate specific causal links in the theory of change. It reflects on learning from the use of causal hotspots as a vehicle for mixing methods. It offers considerations on how to navigate relationships and operational trade-offs in making methodological choices to build robust and credible evidence on how, for whom, and under what conditions participatory programming can work to address complex problems such as child labour. CY - Brighton DA - 2022/08/18/ PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 2 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17603 Y2 - 2024/02/01/00:00:00 ER - TY - RPRT TI - Developing a performance story report: user guide AU - Roughly, A. AU - Dart, J. AB - This report is a nuts and bolts guide to developing a Collaborative Outcomes Report (COR)/ Performance Story Report (PSR) produced by the Australian Government and Jess Dart. It includes practical tips, step-by-step process guides and definitions of key concepts surrounding these approaches. DA - 2009/// PY - 2009 PB - Commonwealth of Australia UR - http://nrmonline.nrm.gov.au/downloads/mql:2162/content Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Developing a Systems Thinking Lens for Collective Leadership AU - O’Donnell, Joan AB - Systems thinking approaches are gaining traction as an effective way of understanding and working with increasing complexity. They are being put forward by the Organisation for Economic Co-operation and Development as well as the World Health Organisation as a way to tackle the complex and unpredictable environments we now operate in. As the world has become increasingly interconnected, national or local boundaries cannot isolate and control social problems. The climate emergency, war and political instability have become everyday realities that accompany an ever-widening gap between those who have and control resources, and those who do not have enough to meet basic needs. The usual responses are becoming more and more ineffective; all interventions have unseen consequences or emergent properties that cannot be predicted using an inputoutput outcomes model. Designing interventions therefore requires a fresh lens to manage our way through turbulence and uncertainty. Most complex situations benefit from a complementary mix of a systematic (linear) decisive intervention as much as a systemic (whole-view) understanding of a proposed intervention. In this sense, adopting a systems thinking approach requires both framing the situation using a systems lens, as well as a commitment to take action and learn forward in realtime. When utilised in this way, a systems thinking lens complements rather than replaces traditional management tools. It draws on well-tested concepts and tools and relies on the systems leader to develop their practice, which includes the art of knowing what is needed and when. It is about threading traditional planning methods together with an understanding of the interrelationships, multiple perspectives and boundary judgements that influence the framing of a situation. It is becoming increasingly clear that effective leadership is embedded and invested: objective management science has no place in a world bound by interconnectedness and unintended consequences. Leaders are, as we shall see, an inherent part of both the problem and the proposed intervention. The purpose of this paper is to share some concepts informed by systems thinking to support you, as leaders in your organisations, to bring a systems-informed lens to your work. It puts you as a practitioner-leader at the centre of your own practice and encourages you to reflect critically on your positionality, as well as the lens you use to understand and intervene in complex issues. This paper has been developed on foot of training piloted with public service leaders in Scotland in the summer of 2022, and acts as an introduction to developing your systems literacy skills. While we are all born with a systems sensibility, many of us lose that sense of connectedness and interdependence with the world as we grow up. This paper acts as a prompt to bring a complexity informed approach to your work practice. It also contains some pointers for further resources and tools that may support your learning. CY - Edinburgh DA - 2023/// PY - 2023 DP - Zotero LA - en PB - Collective Leadership for Scotland UR - https://workforcescotland.wpcomstaging.com/wp-content/uploads/2023/03/873878_SCT0123759776-001_Collective-Leadership-Brochure_FINAL.pdf Y2 - 2023/05/22/00:00:00 ER - TY - RPRT TI - Developing and Using Useful ToCs AU - Mayne, John T2 - Evergreen Briefing Note AB - Developing useful theories of change (ToCs) for anintervention, especially complex interventions, is not straightforward nor something done in a couple of hourswith limited effort. Agreement is needed on a number of basic concepts and terms, key challenges need to be addressed, and there are numerous issues that need to be considered.Butthe effort is worth the investment of resources and time.Theory-based evaluation approaches need good ToCs.Some of the ideas here are discussed in the context of agricultural research for nutrition and health interventions by Mayne and Johnson (2015) (10) Developing and Using Useful ToCs. Available from: https://www.researchgate.net/publication/323868372_Developing_and_Using_Useful_ToCs [accessed Jun 13 2018]. DA - 2018/03// PY - 2018 LA - en UR - https://www.researchgate.net/publication/323868372_Developing_and_Using_Useful_ToCs Y2 - 2018/06/13/13:33:41 ER - TY - JOUR TI - Development and radical uncertainty AU - Feinstein, Osvaldo T2 - Development in Practice AB - Development strategies, programmes and projects are designed making assumptions concerning several variables such as future prices of outputs and inputs, exchange rates and productivity growth. However, knowledge about the future is limited. Uncertainty prevails. The usual approach to deal with uncertainty is to reduce it to risk. Uncertainty is perceived as a negative factor that should and can be eliminated. This article presents an alternative approach which recognises that radical uncertainty is irreducible to risk, identifying a positive dimension of uncertainty and showing its implications for development practice. DA - 2020/11/16/ PY - 2020 DO - 10.1080/09614524.2020.1763258 DP - Taylor and Francis+NEJM VL - 30 IS - 8 SP - 1105 EP - 1113 SN - 0961-4524 UR - https://doi.org/10.1080/09614524.2020.1763258 Y2 - 2022/07/11/10:14:36 ER - TY - CHAP TI - Development Policy and Impact Evaluation: Learning and Accountability in Private Sector Development AU - Ton, Giel T2 - Handbook of Development Policy AB - There is broad recognition of the challenges in evaluating policy and programmes on their contribution to sustainable development. Impact evaluations of PSD programmes are carried out at the behest of a particular configuration of interest groups with different expectations. Some groups want to know whether a programme has worked, others want to know how to do these programmes better, others fear that PSD programmes might result in sub-optimal or adverse development outcomes in recipient countries, and some want to be sure that the programme benefits the private sector in the donors' domestic economies. This chapter discusses these with reference to private sector development programmes and explores how contribution analysis, which is a structured stepwise process of theory-based evaluation, can address these challenges and generate findings to improve learning and accountability. DA - 2021/09/07/ PY - 2021 DP - opendocs.ids.ac.uk LA - en PB - Edward Elgar Publishing Limited SN - 978-1-83910-086-4 ST - Development Policy and Impact Evaluation UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/16926 Y2 - 2021/12/17/14:51:52 ER - TY - BLOG TI - Developmental Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Developmental Evaluation (DE) is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments. DE originators liken their approach to the role of research & development in the private sector product development process because it facilitates real-time, or close to real-time, feedback to program staff thus facilitating a continuous development loop. Michael Quinn Patton is careful to describe this approach as one choice that is responsive to context. This approach is not intended as the solution to every situation. Development evaluation is particularly suited to innovation, radical program re-design, replication, complex issues, crises In these situations, DE can help by: framing concepts, test quick iterations, tracking developments, surfacing issues. This description is from Patton (2010) Developmental Evaluation. Applying Complexity Concepts to Enhance Innovation and Use... "Developmental Evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments. Innovations can take the form of new projects, programs, products, organizational changes, policy reforms, and system interventions. A complex system is characterized by a large number of interacting and interdependent elements in which there is no central control. Patterns of change emerge from rapid, real time interactions that generate learning, evolution, and development – if one is paying attention and knows how to observe and capture the important and emergent patterns. Complex environments for social interventions and innovations are those in which what to do to solve problems is uncertain and key stakeholders are in conflict about how to proceed." UR - https://www.betterevaluation.org/en/plan/approach/developmental_evaluation Y2 - 2018/10/13/00:00:00 ER - TY - JOUR TI - Developmental Evaluation AU - Patton, Michael Quinn T2 - The Nonprofit Quarterly DA - 2006/// PY - 2006 IS - Spring SP - 6 UR - https://www.scribd.com/document/8233067/Michael-Quinn-Patton-Developmental-Evaluation-2006 Y2 - 2018/10/13/00:00:00 ER - TY - BLOG TI - Developmental Evaluation AU - Tamarack Community AB - This webpage from Tamarack Community provides links to audio files of an interview between Mark Cabaj and Michael Patton in which they discuss the topic of development evaluation. This is an "approach that has proven particularly useful and effective at addressing the unique challenges of evaluating the real nature of community change work. Learning Objectives: To deepen appreciation for the role of evaluation and evaluation thinking To explore the distinctions between various types of evaluation To understand more about developmental evaluation and its use To investigate concepts, tools and resources available to support working with developmental evaluation methods" (Tamarack Community) DA - 2012/// PY - 2012 UR - http://www.tamarackcommunity.ca/library/developmental-evaluation-diagnostic-checklist Y2 - 2018/10/13/00:00:00 ER - TY - BOOK TI - Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use AU - Patton, Michael Quinn AB - Developmental evaluation (DE) offers a powerful approach to monitoring and supporting social innovations by working in partnership with program decision makers. In this book, eminent authority Michael Quinn Patton shows how to conduct evaluations within a DE framework. Patton draws on insights about complex dynamic systems, uncertainty, nonlinearity, and emergence. He illustrates how DE can be used for a range of purposes: ongoing program development, adapting effective principles of practice to local contexts, generating innovations and taking them to scale, and facilitating rapid response in crisis situations. Students and practicing evaluators will appreciate the book's extensive case examples and stories, cartoons, clear writing style, "closer look" sidebars, and summary tables. Provided is essential guidance for making evaluations useful, practical, and credible in support of social change.See also Developmental Evaluation Exemplars, edited by Michael Quinn Patton, Kate McKegg, and Nan Wehipeihana, which presents 12 in-depth case studies. CY - New York DA - 2010/08/13/ PY - 2010 DP - Amazon ET - 1 edition SP - 375 LA - English PB - Guilford Press SN - 978-1-60623-872-1 ST - Developmental Evaluation KW - Evaluation KW - Measurement KW - Organizational effectiveness KW - Project management ER - TY - JOUR TI - Developmental evaluation: Bridging the gaps between proposal, program, and practice , Developmental evaluation: Bridging the gaps between proposal, program, and practice AU - Lawrence, Rachael B. AU - Rallis, Sharon F. AU - Davis, Laura C. AU - Harrington, Karen T2 - Evaluation AB - Developmental evaluation supports grant-funded initiatives seeking innovation and change. Programs born from aspirational grant language and plans often need guidance as they work toward creating workable models for social innovation. This article describes the challenge of designing and implementing complex programs and presents a case that illustrates how a program moves from proposal to practice. The Massachusetts Charter Public School Association Capacity Building Network is a complex intervention, funded by the US federal government, aimed at raising school capacity to serve students with disabilities and English language learners. Developmental evaluation served to bridge the gaps between an aspirational proposal, an ambitious and ambiguous program plan, and emerging practices to serve this population of students. Jointly reviewing our experience in this developmental evaluation, the evaluation team and a program director share important thematic lessons learned about the developmental evaluation approach., Developmental evaluation supports grant-funded initiatives seeking innovation and change. Programs born from aspirational grant language and plans often need guidance as they work toward creating workable models for social innovation. This article describes the challenge of designing and implementing complex programs and presents a case that illustrates how a program moves from proposal to practice. The Massachusetts Charter Public School Association Capacity Building Network is a complex intervention, funded by the US federal government, aimed at raising school capacity to serve students with disabilities and English language learners. Developmental evaluation served to bridge the gaps between an aspirational proposal, an ambitious and ambiguous program plan, and emerging practices to serve this population of students. Jointly reviewing our experience in this developmental evaluation, the evaluation team and a program director share important thematic lessons learned about the developmental evaluation approach. DA - 2018/01/01/ PY - 2018 DO - 10.1177/1356389017749276 DP - SAGE Journals VL - 24 IS - 1 SP - 69 EP - 83 J2 - Evaluation LA - en SN - 1356-3890 ST - Developmental evaluation UR - https://doi.org/10.1177/1356389017749276 Y2 - 2018/07/27/09:42:46 ER - TY - RPRT TI - Developmental Evaluation: How Barriers & Enablers Emerge Over Time AU - Baylor, Rebecca AU - Fatehi, Y. K. AU - Esper, H. DA - 2019/// PY - 2019 DP - Zotero SP - 6 LA - en PB - USAID ER - TY - RPRT TI - Developmental Evaluation in Practice: Tips, Tools, and Templates AU - USAID CY - Washington DC DA - 2017/09// PY - 2017 PB - USAID UR - https://wdi.umich.edu/wp-content/uploads/USAID-DEPA-MERL-Developmental-Evaluation-in-Practice-Tips-Tools-and-Templates.pdf Y2 - 2019/02/15/08:56:40 ER - TY - RPRT TI - DFID’s approach to value for money in programme and portfolio management AU - ICAI AB - ICAI published this review on DFID’s approach to value for money in February 2018, and as value for money is both a process and an outcome and cuts across all aspects of DFID’s operations, did not score this review. We made five recommendations and published a follow-up to this review in July 2019. All UK government departments are required to achieve value for money in their use of public funds. In recent years, DFID has been working to build value for money considerations further into its management processes and its relationships with implementers and multilateral partners, establishing itself as a global champion on value for money. Review This review was published in February. Though it was not rated, ICAI made five recommendations, and found that the Department for International Development’s approach to value for money was helping to make UK aid spending go further, but improvements are still needed. Findings This review found that DFID has strengthened its processes and systems for ensuring it gains maximum value for each pound spent, has taken swift remedial action to tackle under-performing programmes, and has become a strong global champion on value for money. The review also found that DFID has been diligent in its efforts to cut waste, detect fraud, and improve efficiency, and that this work is improving the return on the UK investment in aid. However, the review found that DFID’s approach was not adequately reporting and capturing results and value for money at the country portfolio level, or how programmes work together to deliver lasting impact, including reducing future dependency on aid. It also found that weakness in the annual review process could undermine DFID’s approach to value for money. It found that targets were frequently revised, and that there could be pressures for optimistic scoring of programmes. Recommendations Based on this review, we made the following five recommendations to help DFID improve its approach to value for money still further: DFID country offices should articulate cross-cutting value for money objectives at the country portfolio level, and should report periodically on progress at that level. Drawing on its experience with introducing adaptive programming, DFID should encourage programmes to experiment with different ways of delivering results more cost-effectively, particularly for more complex programming. DFID should ensure that principles of development effectiveness – such as ensuring partner country leadership, building national capacity and empowering beneficiaries – are more explicit in its value for money approach. Programmes should reflect these principles in their value for money frameworks, and where appropriate incorporate qualitative indicators of progress at that level. DFID should be more explicit about the assumptions underlying the economic case in its business cases, and ensure that these are taken into account in programme monitoring. Delivery plans should specify points in the programme cycle when the economic case should be fully reassessed. Senior responsible owners should also determine whether a reassessment is needed following material changes in the programme, results targets or context. Annual review scores should include an assessment of whether programmes are likely to achieve their intended outcomes in a cost-effective way. DFID should consider introducing further quality assurance into the setting and adjustment of logframe targets. CY - London DA - 2018/02// PY - 2018 LA - en-GB PB - Independent Commission for Aid Impact (ICAI) UR - https://icai.independent.gov.uk/report/value-for-money/ Y2 - 2019/03/12/16:53:46 ER - TY - BLOG TI - Different kinds of rubrics AU - King, Julian T2 - Evaluation and Value for Money AB - Generic, analytic, holistic, orbic, asterisc, escalieric, cephalopodic DA - 2023/07/25/ PY - 2023 M3 - Substack newsletter UR - https://juliankingnz.substack.com/p/different-kinds-of-rubrics Y2 - 2023/07/26/08:49:24 ER - TY - RPRT TI - Discussion Note: Complexity-Aware Monitoring AU - USAID T2 - Program Cycle AB - This Discussion Note complements ADS 201 and outlines general principles and promising approaches for monitoring complex aspects of USAID development assistance. Complexity-aware monitoring is a type of complementary monitoring that is useful when results are difficult to predict due to dynamic contexts or unclear cause and-effect relationships. Key principles of the Program Cycle include applying analytic rigor, managing adaptively, and utilizing a range of approaches to achieve results. ADS 201.3.5.5 identifies three types of program monitoring – performance, context, and complementary. All USAID programming incorporates performance monitoring and should include context monitoring. Performance monitoring “is the ongoing and systematic collection of performance indicator data and other quantitative or qualitative information to reveal whether implementation is on track and whether expected results are being achieved.” Context monitoring is “[t]he systematic collection of information about conditions and external factors relevant to the implementation and performance of an operating units (OU’s) strategy, projects, and activities.” As stated in ADS 201.3.5.5, complementary monitoring may be used by missions and Washington OUs to complement performance and context monitoring in situations where results are difficult to predict due to dynamic contexts or unclear cause-andeffect relationships. This Discussion Note provides an explanation of when to use complementary monitoring approaches that are complexity-aware and summarizes the three principles of complexity-aware monitoring. DA - 2018/// PY - 2018 M3 - Discussion Paper PB - USAID SN - Version 1 UR - https://usaidlearninglab.org/sites/default/files/resource/files/cleared_dn_complexity-aware_monitoring.pdf Y2 - 2018/03/09/00:00:00 ER - TY - BLOG TI - Does the whole add more than the SUM of its parts? AU - Guerzovich, Florencia T2 - Medium AB - This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots. DA - 2023/07/25/T13:20:42.641Z PY - 2023 LA - en UR - https://medium.com/@florcig/does-the-whole-add-more-than-the-sum-of-its-parts-8b9eb352bb67 Y2 - 2023/08/10/09:00:51 ER - TY - CHAP TI - Doing Development Differently at Scale AU - Pellini, Arnaldo AU - Karetji, Petrarca C. AU - Soekadis, Ade T2 - Knowledge, Politics and Policymaking in Indonesia A2 - Pellini, Arnaldo A2 - Prasetiamartati, Budiati A2 - Nugroho, Kharisma Priyo A2 - Jackson, Elisabeth A2 - Carden, Fred AB - In recent decades there has been an increasing recognition that politics and political institutions matter for development. There is also a much greater interest in contextually grounded approaches. This has stemmed from an acknowledgement that purely technocratic approaches to development often result in failure because they do not take into account the nature of political institutions. Nor do they take account of the context in a particular developing country and the interests and incentives of powerful national actors. Policy processes are embedded in specific social, political and organisational contexts. Approaches that focus on implementing universal best practices in evidence-informed policymaking are unlikely to be successful. Instead, what is needed is an approach that takes the local context as the starting point for understanding what issues are relevant to policymakers and developing contextually appropriate solutions. The authors of this chapter reflect on the management approaches and systems that may be required to enable and support large-scale development programmes to be flexible and adaptive to the local policy context and circumstances. The chapter argues that such programmes struggle to adopt adaptive management principle and that to do that at scale requires some changes in the way such programmes are managed. CY - Singapore DA - 2018/// PY - 2018 DP - Springer Link SP - 131 EP - 146 LA - en PB - Springer Singapore SN - 9789811301674 UR - https://doi.org/10.1007/978-981-13-0167-4_8 Y2 - 2018/09/21/08:57:49 KW - Adaptive Management KW - Doing development differently KW - Evidence-informed policymaking KW - Indonesia KW - Knowledge sector KW - Thinking and working politically ER - TY - RPRT TI - Doing Development Differently in the Global South - Workshop Report AU - CARE CY - Nairobi DA - 2018/12// PY - 2018 PB - CARE and British Council UR - https://insights.careinternational.org.uk/media/k2/attachments/CARE_British-Council_DDD-workshop-report-Dec-2018.pdf Y2 - 2019/02/01/10:14:20 ER - TY - RPRT TI - Doing things differently: Rethinking monitoring and evaluation to understand change AU - Saferworld AB - Doing things differently: Rethinking monitoring and evaluation to understand change Learning paper Over the past four years, Saferworld has put in place a way of monitoring, evaluating and learning from our work focused on behaviour and relationship change. This paper outlines the process we have gone through to adapt, embed, and embrace an approach inspired by Outcome Mapping and Outcome Harvesting (OH). Key benefits of our monitoring, evaluation and learning approach are: It is simple, but promotes complex discussion and analysis. It allows conflict- and gender-sensitivity to be built into programmes; it promotes analysis of and adaptation to context. Bringing front-line staff and partners into wider conversations with others substantially increases cross-organisational learning. CY - London DA - 2016/// PY - 2016 LA - english M3 - Learning paper PB - Saferworld UR - https://www.saferworld.org.uk/resources/publications/1027-doing-things-differently-rethinking-monitoring-and-evaluation-to-understand-change Y2 - 2019/02/05/09:09:14 ER - TY - RPRT TI - DPPD Handbook. A step-by-step guide for development practitioners to apply the Data Powered Positive Deviance method AU - DPPD AB - The Method Positive Deviance (PD) is based on the observation that in every community or organization, there are a few individuals who achieve significantly better outcomes than their peers, despite having similar challenges and resources. These individuals are referred to as positive deviants, and adopting their solutions is what is referred to as the PD approach¹. The method described in this Handbook follows the same logic as the PD approach but uses pre-existing, non-traditional data sources instead of — or in conjunction with — traditional data sources. Non-traditional data in this context broadly refers to data that is digitally captured (e.g. mobile phone records and financial data), mediated (e.g. social media and online data), or observed (e.g. satellite imagery). The integration of such data to complement traditional data sources generally used in PD is what we refer to as Data Powered Positive Deviance² (DPPD). The digital data opportunity Recent developments in the availability of digital data provide an opportunity to look for positive deviants³ in new ways and in unprecedented geographical and on temporal scales. A number of studies⁴ have described the challenges related to the application of the PD approach in development. Given these challenges, there are obvious opportunities for innovation in PD and our particular interest here is in the innovative opportunities offered by non-traditional data, following the increasing “datafication” of development and the growing availability of big datasets in a variety of development sectors⁵. DPPD builds on this and expands our ability to extract value from non-traditional digital data while providing a systematic process for leveraging local know-how and the collective wisdom of communities. Data Powered Positive Deviance The DPPD method described in this Handbook emerged from a process of research and testing and follows the same stages as the PD approach. The difference is that DPPD integrates pre-existing, non-traditional data across the five stages, requiring a series of new and specific methods and practices that are not required in the PD approach. The first stage is also somewhat different because it not only defines the problem, but it also checks if it is suitable and feasible to use the DPPD method for the proposed project. Table 1 lists the five stages of the DPPD method. This Handbook dedicates a section to each stage. Stage 1 Assess problem-method fit Stage 2 Determine positive deviants Stage 3 Discover underlying factors Stage 4 Design and implement interventions Stage 5 Monitor and evaluate DA - 2021/11// PY - 2021 PB - DPPD Initiative UR - https://static1.squarespace.com/static/614dae085246883818475c39/t/619f7f163ed02a77d13fd1bd/1637842759939/DPPD+Handbook+Nov+2021.pdf Y2 - 2021/11/25/15:51:12 ER - TY - RPRT TI - (Draft) Managing to Adapt - Analysing Adaptive Management for Planning, Monitoring, Evaluation, and Learning (draft) AU - Schlingheider, Annika AU - Pellfolk, Erica AU - Maneo, Gabriele AU - Desai, Harsh T2 - Oxfam Research Reports AB - Adaptive management is at the heart of ‘doing development differently’ (Wild et al., 2016). Whether it is here to stay depends on how much it is mainstreamed into existing development programming by donors and implementers alike, especially in planning, monitoring, evaluation, and learning (PMEL) cycles. In this report, we find that mainstreaming adaptive management in PMEL involves three strategies: 1. planning for flexibility; 2. developing locally owned monitoring and evaluation (M&E); and 3. creating an enabling environment for learning. Adopting these strategies contributes to virtuous cycles of PMEL DA - 2017/04// PY - 2017 PB - Oxfam UR - https://frompoverty.oxfam.org.uk/draft-paper-on-adaptive-management-in-oxfam-all-comments-welcome/ Y2 - 2023/08/18/00:00:00 ER - TY - ICOMM TI - DRG Center practical tips to implement more context-adaptive programming AU - Jacobstein, David T2 - #AdaptDev AB - Hi #adaptdevers, The cross-sectoral programs team in the DRG Center have been working in support of ways that programming can be made more aware of, and responsive to, context considerations that have to do with incentives and relationships between different actors - often called power dynamics or political economy. Following several discussions with field folks, we were asked to come up with simple and practical tips that could help staff to implement more context-adaptive programming, or to better Work Politically to use TWP jargon. With assistance from many of our field staff and partners, we've put together three documents (to start) that have now been cleared to share publicly: • Tips on Making Your Solicitations Invite Context-Driven Adaptive Programming - suggestions spanning most sections of a typical solicitation to better weave in Thinking and Working Politically. I'm particularly excited that this document contains embedded links to excerpts from a set of solicitations that model good practice. You can also find a summary of all the excerpts from real solicitations, and links onward to the public posting of those solicitations on Fed Biz Ops or Grants.gov, here. • Tips on Power Dynamics and Theories of Change - suggestions relevant to those working on project designs and logic models. • Tips for Better Use of Advisory Councils - suggestions to set up better advisory bodies that help you to incorporate political economy more automatically into your programmatic adjustments. I hope that you'll find these practical and helpful! Feedback is always welcome. Best, David DA - 2018/02/27/ PY - 2018 UR - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/adaptdev/TVvijW_iUx8/a5-0xdsIDgAJ ER - TY - RPRT TI - Dynamic gridlock: Adaptive Humanitarian Action in the Democratic Republic of Congo AU - Obrecht, Alice T2 - ALNAP Country Study DA - 2018/// PY - 2018 SP - 51 PB - ODI/ALNAP UR - https://www.alnap.org/system/files/content/resource/files/main/alnap-2018-adaptiveness-DRC-case-study_1.pdf Y2 - 2018/11/10/00:00:00 ER - TY - JOUR TI - Dynamic knowledge management strategy development in international non-governmental organisations AU - Walsh, John N. AU - Lannon, John T2 - Knowledge Management Research & Practice AB - Knowledge management strategies are important for firms’ competitive positioning. This paper examines how knowledge management codification and personalisation strategies are developed in response to environmental and organisational dynamics in an international non-governmental organisation. A longitudinal case study of the organisation’s strategic reformulation of its KM strategy over a 2.5 period is drawn upon. The research examines how pressures in the firm’s operating environment led to the organisation identifying the need to leverage the value of local contextual knowledge. Subsequent reformulation required the organisation to change its strategic mix of codification and personalisation over time. Although efforts were focused on increasing personalisation, developments were supported through codification demonstrating a symbiotic, mutually supporting relationship between the strategies. The strategic reformulation involved processes of reflection, repackaging and support activities. DA - 2020/06/30/ PY - 2020 DO - 10.1080/14778238.2020.1785348 DP - Taylor and Francis+NEJM VL - 0 IS - 0 SP - 1 EP - 12 SN - 1477-8238 UR - https://doi.org/10.1080/14778238.2020.1785348 Y2 - 2021/07/30/11:44:31 KW - Knowledge management strategy KW - case study KW - international development KW - knowledge dynamics KW - non-governmental organisation ER - TY - JOUR TI - Editorial: Responsible well-being — a personal agenda for development AU - Chambers, Robert T2 - World Development AB - If development means good change, questions arise about what is good, and what sorts of change matter. Answers can be personally defined and redefined. The changing words, meanings and concepts of development discourse both reflect and influence what is done. The realities of the powerful tend to dominate. Drawing on experience with participatory approaches and methods which enable poor and marginalized people to express their realities, responsible well-being is proposed as a central concept for a development agenda. This links with capabilities and livelihoods, and is based on equity and sustainability as principles. The primacy of personal actions and non-actions in development points to the need for a pedagogy for the non-oppressed. This includes self-critical awareness, thinking through the effects of actions, and enabling those with power and wealth to experience being better off with less. Others are invited and encouraged to reflect, improve on this analysis, and write their own agenda. DA - 1997/11/01/ PY - 1997 DO - 10.1016/S0305-750X(97)10001-8 DP - ScienceDirect VL - 25 IS - 11 SP - 1743 EP - 1754 J2 - World Development SN - 0305-750X ST - Editorial UR - https://www.sciencedirect.com/science/article/pii/S0305750X97100018 Y2 - 2024/01/26/11:14:08 KW - development vocabulary KW - ethics KW - methods KW - participation KW - poverty KW - well-being ER - TY - RPRT TI - Emerging Opportunities: Monitoring and Evaluation in a Tech-Enabled World AU - Raftree, Linda AU - Bamberger, Michael T2 - Discussion Paper AB - Monitoring and evaluation practice is using information and communication technologies for more timely data, and more inclusive voice and feedback. DA - 2014/09// PY - 2014 PB - ITAD & Rockefeller Foundation ST - Emerging Opportunities UR - https://www.rockefellerfoundation.org/report/emerging-opportunities-monitoring/ Y2 - 2017/02/11/11:11:49 ER - TY - RPRT TI - Emerging Technologies and Approaches in Monitoring, Evaluation, Research, and Learning for International Development Programs AU - Bruce, Kerry AU - Gandhi, Valentine J AU - Vandelanotte, Joris AB - Emerging technology is making monitoring, evaluation, research and learning (MERL) more precise and enriching data. However, this evolution is so rapid that it can be difficult to stay informed about the field overall. This paper presents examples of emerging technology that are most often used in MERL for development programs, describes the pros and cons of their use, and discusses technology and ethics concerns that practitioners should keep in mind. The paper covers new types of data sources (application data, sensor data, and drones), new ways to store data (distributed ledger technology and the cloud), and new ways to analyze data (text analytics and supervised and unsupervised learning). DA - 2020/07// PY - 2020 DP - Zotero SP - 20 LA - en PB - MERL Tech ER - TY - BLOG TI - Empowerment Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Empowerment Evaluation is an approach which provides communities with the tools and knowledge that allows them to monitor and evaluate their own performance. First developed by David Fetterman who describe it as “the use of evaluation concepts, techniques, and findings to foster improvement and self-determination.” UR - https://www.betterevaluation.org/plan/approach/empowerment_evaluation Y2 - 2018/10/18/00:00:00 ER - TY - VIDEO TI - Empowerment evaluation AU - Fettermen, David C5 - Vimeo Video DA - 2014/// PY - 2014 UR - https://vimeo.com/96643564 ER - TY - BLOG TI - Empowerment Evaluation Blog AU - Fetterman, David AB - This blog, by Dr David Fetterman, provides a range of resources on empowerment evaluation theory and practice. The blog includes links to videos, guides and relevant academic literature that provide a detailed analysis and discussion of using empowerment evaluation. DA - 2006/05// PY - 2006 UR - http://eevaluation.blogspot.com/?view=magazine Y2 - 2023/10/17/00:00:00 ER - TY - RPRT TI - Energy Governance in Developing Countries — A New Approach AU - Mcculloch, Neil AB - In 2015, leaders from around the world agreed to 17 Sustainable Development Goals (SDGs) to be achieved by 2030. The seventh goal (SDG7) is: “Ensure access to affordable, reliable, sustainable and modern energy for all.” In the same year, the world’s leaders concluded the Paris Agreement to tackle climate change, which will require a global transition in the energy sector away from the use of fossil fuels. Yet, despite growing investments in clean energy in many developing countries, the transition is happening much more slowly than needed. The central reason for this is poor energy governance. This technical brief shows how poor energy governance damages energy access and efforts to improve the quality and reliability of power. It explains the political reasons why energy governance is so bad in many countries and contrasts this with the current system of procuring technical assistance, which largely ignores the energy governance challenge. It shows that a new approach to tackling energy governance is emerging that is better matched to the nature of the problems faced and provides recommendations on how to implement it. CY - London DA - 2021/06// PY - 2021 DP - Zotero SP - 9 LA - en M3 - Briefing PB - The Policy Practice & Chemonics ER - TY - RPRT TI - Engage. Empower. Enact. - Citizen Engagement & Democratic Innovation Programme White Paper AU - Doyle, Linda AU - Smith, Bethan AB - The Cynefin Centre’s Citizen Engagement & Democratic Innovation programme provides tools for collective sense-making in the areas of community development and youth work; civic engagement and democratic innovation; collaborative service/policy design and evaluation; housing/tenant engagement; futures and planning; shared learning and peer to peer knowledge exchange. CY - Conwy DA - 2021/// PY - 2021 DP - Zotero SP - 26 LA - en PB - Cognitive Edge ER - TY - RPRT TI - Engaging scientists through institutional histories AU - Prasad, Shambu AU - Hall, Andrew AU - Thummuru, Laxmi T2 - ILAC Brief AB - An institutional history is a narrative that records key points about how institutional arrangements – new ways of working – evolve over time creating more effective ways to achieve goals. It can be used to document institutional innovations in projects and to highlight barriers to change. An institutional history draws out and synthesizes lessons for research organizations and partners as well as for others in similar circumstances. DA - 2006/11// PY - 2006 SP - 2 PB - CGIAR SN - 14 UR - https://www.betterevaluation.org/sites/default/files/ILAC_Brief14_institutional.pdf Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Enhancing partner and system-level learning: 8 Tips from MEL Managers AU - Gover, Dun AU - Nasution, Zulka AU - Okutu, David AU - Bolder, Meghan AU - Henao, Lina T2 - MSD in MEL Brief AB - Effective learning is a key driver of market systems change, with the potential to enhance system competitiveness, resilience, and inclusiveness. Shifting the Locus of Learning: Catalyzing Private Sector Learning to Drive Systemic Change recently outlined a rationale for enhancing the scale and quality of learning in a system and identifying 10 strategies programs can contextualize to catalyze learning. These strategies are also backed with robust examples from 13 programs doing this work across 11 countries. To deepen insights on what MSD Monitoring, Evaluation and Learning (MEL) Managers have experienced in putting several of those strategies into practice, the Feed the Future Market Systems and Partnerships (MSP) Activity convened a series of peer discussions as part of a larger initiative (see Figure 1). This brief shares the collective learning and experience on this topic of three senior MEL Managers who were interested in and had experience with this topic. The group represented full-time, program-based MEL Leads working on MSD programs funded by USAID and DFAT, based in Fiji, Albania, and Kosovo working for Adam Smith International, SwissContact, and DT Global, respectively. From those discussions, this paper synthesizes eight tips from MEL Managers for practically enhancing partner and system-level learning: 1. Identify the right decision-maker(s) at potential partners. 2. Use a co-creation process to identify learning opportunities. 3. Use diagnostics and assessments to strengthen partner and system capacity for actionable learning. 4. Use a phased capacity strengthening process tied to behavior change. 5. Measure partners’ continued investment in and use of learning—not the continuation of specific learning activities. 6. Work with sector-level institutions for scale but be aware of risks. 7. Leverage informal communities of practice to share learning. 8. Use the right terminology to talk about partner and system-focused learning. DA - 2023/// PY - 2023 DP - Zotero LA - en PB - USAID SN - 3 UR - https://agrilinks.org/sites/default/files/media/file/MSD%20in%20MEL%20Brief%203_PS%20Learning_508.pdf Y2 - 2023/10/02/09:38:55 ER - TY - BLOG TI - EPE Week: Lee-Anne Molony on Collaborative Outcomes Reporting AU - Molony, L A T2 - AEA365 a Tip-a-day by and for Evaluators AB - This AEA365 blog, written by Lee-Anne Molony a Principal Consultant at Clear Horizon, provides a brief overview on Collaborative Outcomes Reporting (COR). DA - 2013/04/23/ PY - 2013 UR - https://aea365.org/blog/epe-week-lee-anne-molony-on-collaborative-outcomes-reporting/ Y2 - 2018/10/13/00:00:00 ER - TY - BOOK TI - Essentials of Utilization-Focused Evaluation AU - Patton, Michael Quinn AB - Based on Michael Quinn Patton's best-selling Utilization-Focused Evaluation, this briefer book provides an overall framework and essential checklist steps for designing and conducting evaluations that actually get used. The new material and innovative graphics present the utilization-focused evaluation process as a complex adaptive system, incorporating current understandings about systems thinking and complexity concepts. The book integrates theory and practice, is based on both research and professional experience, and offers new case examples and cartoons with Patton's signature humor. DA - 2011/08/22/ PY - 2011 DP - Google Books SP - 489 LA - en PB - SAGE Publications SN - 978-1-4833-0697-1 KW - Reference / Research KW - Research ER - TY - COMP TI - EvalC3 AU - Davies, Rick AB - Tools for developing, exploring and evaluating predictive models of expected outcomes DA - 2019/// PY - 2019 ET - 2.3.39 UR - https://evalc3.net/ Y2 - 2017/05/25/19:31:44 ER - TY - RPRT TI - Evaluability assessment for Impact Evaluation: Guidance, checklists and decision support AU - Peersman, Greet AU - Guijt, Irene AU - Pasanen, Tiina AB - This guidance note focuses on the utility of, and guidance for, evaluability assessment before undertaking an impact evaluation. The primary audience for this guidance note is evaluators conducting an evaluability assessment for impact evaluation. The secondary audience is people commissioning or managing an evaluability assessment for impact evaluation, as well as funders of an impact evaluation. Sections one and two provide an overview of evaluability assessment and how it can be used for impact evaluation. Section three provides guidance for planning to undertake an evaluability assessment for impact evaluation. This is informative for all intended users of the guidance note. Section four includes checklists and decision support for evaluability assessments. The checklist is geared to those conducting the evaluability assessment and can be adapted to suit a particular context or purpose. The decision support provides those conducting an evaluability assessment with evidence-based recommendations for impact evaluation funders and commissioners, about whether, when and how to proceed with the evaluation. Sections five and six provide guidance on what to do after the assessment is concluded, and offer lessons learned from evaluability assessments in practice. DA - 2015/08// PY - 2015 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9802.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - Evaluability Checklist for Post Project Evaluation AU - Zivetz, Laurie AU - Cekan, Jindra AB - Considerations for planning a post project evaluation during the project, at the end, or after it has closed. DA - 2020/// PY - 2020 PB - Valuing Voices UR - https://valuingvoices.com/wp-content/uploads/2017/08/Valuing-Voices-Checklists.pdf Y2 - 2021/02/18/13:20:22 ER - TY - RPRT TI - Evaluating CLARISSA: Innovation Driven by a Participatory Learning Agenda AU - Apgar, Marina AU - Snijder, Mieke AU - Kakri, Shanta AU - Macleod, Shona AU - Paul, Sukanta AU - Sambo, Anna AU - Ton, Giel T2 - CLARISSA working paper AB - Children end up in child labour as a result of many, often unknown or hidden, interactions between multiple actors and multiple factors within households, communities, and labour systems, leading to unpredictable outcomes for children and other sector stakeholders and sometimes resulting in the worst forms of child labour (WFCL). It is a complex problem, and interventions aimed at tackling it are also, inevitably, complex and challenging. The way they influence change is non-linear, causality is uncertain, and unintended consequences may result. Programmes such as the Child Labour: Action-Research-Innovation in South and South-Eastern Asia (CLARISSA) that are engaging with such intractable challenges and aim to reach the most left behind (children in WFCL) are operating in conditions of complexity. This complexity poses significant challenges to the way programmes are designed, planned, implemented, and evaluated, and requires a move away from linear and predetermined models. In this Working Paper, we share our experience and early learning about how to design and implement monitoring, evaluation and learning that intentionally embraces the challenge of complexity. CY - Brighton DA - 2020/06/25/ PY - 2020 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 2 ST - Evaluating CLARISSA UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/15456 Y2 - 2023/01/10/14:37:04 ER - TY - JOUR TI - Evaluating complex interventions: A theory-driven realist-informed approach AU - Douthwaite, Boru AU - Mayne, John AU - McDougall, Cynthia AU - Paz-Ybarnegaray, Rodrigo T2 - Evaluation AB - There is a growing recognition that programs that seek to change people’s lives are intervening in complex systems, which puts a particular set of requirements on program monitoring and evaluation. Developing complexity-aware program monitoring and evaluation systems within existing organizations is difficult because they challenge traditional orthodoxy. Little has been written about the practical experience of doing so. This article describes the development of a complexity-aware evaluation approach in the CGIAR Research Program on Aquatic Agricultural Systems. We outline the design and methods used including trend lines, panel data, after action reviews, building and testing theories of change, outcome evidencing and realist synthesis. We identify and describe a set of design principles for developing complexity-aware program monitoring and evaluation. Finally, we discuss important lessons and recommendations for other programs facing similar challenges. These include developing evaluation designs that meet both learning and accountability requirements; making evaluation a part of a program’s overall approach to achieving impact; and, ensuring evaluation cumulatively builds useful theory as to how different types of program trigger change in different contexts. DA - 2017/07/01/ PY - 2017 DO - 10.1177/1356389017714382 DP - SAGE Journals VL - 23 IS - 3 SP - 294 EP - 311 J2 - Evaluation LA - en SN - 1356-3890 ST - Evaluating complex interventions UR - https://doi.org/10.1177/1356389017714382 Y2 - 2020/10/15/14:36:58 ER - TY - RPRT TI - Evaluating complex interventions in international development AU - Masset, Edoardo AU - Shrestha, S. AU - Juden, M. T2 - CEDIL Methods Working Paper 6 AB - This paper reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. Complex interventions are those that are characterised by multiple components, multiple stakeholders, or multiple target populations. They may also be interventions that incorporate multiple processes of behavioural change. While such interventions are very common and receive a large proportion of development aid budgets, they are rarely subject to rigorous evaluations. The CEDIL Methods Working Paper, ‘Evaluating Complex Interventions in International Development’, reviews promising methods for the evaluation of complex interventions that are new or have been used in a limited way. It offers a taxonomy of complex interventions in international development and draws on literature to discuss several methods that can be used to evaluate these interventions. The paper focuses its attention on methods that address causality and allow us to state conclusively whether an intervention works or not. It shows that several rigorous methods developed in different disciplines can be adapted and used to evaluate complex interventions in international development. CY - London and Oxford DA - 2021/12// PY - 2021 LA - en-GB PB - CEDIL UR - https://cedilprogramme.org/publications/cedil-methods-working-paper-6/ Y2 - 2022/06/17/12:31:00 ER - TY - RPRT TI - Evaluating complex interventions: What are appropriate methods? AU - Masset, Eduardo AB - In the CEDIL Methods brief, ‘Evaluating complex interventions: What are appropriate methods?’ we identify four types of complex development interventions: long causal chain interventions, multicomponent interventions, portfolio interventions, and system-level interventions. These interventions are characterised by multiple activities, multiple outcomes, multiple components, a high level of interconnectedness, and non-linear outcomes. CY - London and Oxford DA - 2022/05/16/T09:22:06+00:00 PY - 2022 LA - en-GB M3 - CEDIL Methods Brief 7 PB - CEDIL UR - https://cedilprogramme.org/publications/cedil-methods-brief/cedil-methods-brief-7/ Y2 - 2022/06/17/12:30:52 ER - TY - RPRT TI - Evaluating Complexity. Propositions for improving practice AU - Preskill, Hallie AU - Gopal, Srik AB - 9 propositions can help evaluators measure progress on complex social problems. CY - Boston DA - 2014/11// PY - 2014 PB - FSG UR - http://www.fsg.org/publications/evaluating-complexity KW - IMPORTANT ER - TY - BLOG TI - Evaluating complexity, simplistically AU - Aston, Thomas T2 - Medium AB - A critical appraisal's of CEDIL papers on Evaluating Complex Interventions... A study was recently published by the Centre of Excellence for Development Impact and Learning (CEDIL) entitled Evaluating complex interventions in international development. This is the sort of title that raises great expectations. Complexity is a hugely popular theme and many of us are keen to know more about how to evaluate efforts that seek to achieve results amid complexity. In April 2021, CEDIL conducted a webinar on the paper, and in July 2021 CEDIL published a blog. In September 2021, I wrote a blog expressing some reservations regarding the focus of the study; its apparent over-emphasis on interventions, under-emphasis on context, as well as its choice of some supposedly under-used methods. These methods were: (1) factorial designs; (2) adaptive trials; (3) Qualitative Comparative Analysis (QCA); (4) synthetic control; (5) agent-based modelling, and system dynamics. DA - 2022/02/16/T14:42:04.608Z PY - 2022 LA - en UR - https://thomasmtaston.medium.com/evaluating-complexity-simplistically-f587778a1b32 Y2 - 2022/06/17/13:00:07 ER - TY - THES TI - Evaluating development projects: exploring a synthesis model of the logical framework approach and outcome mapping AU - Yang, Ting AB - Under the current results-driven development agenda, sound evaluation, and a corresponding evaluation toolkit, need to be in place to examine whether and to what extent development interventions have achieved their targeted objectives and results, and to generate lessons for further development learning and improvement. My review of the literature shows that innovative and appropriate evaluation approaches are needed to address key challenges in evaluation such as the tension between learning and accountability objectives, the need to unpack the mechanisms linking outputs and outcomes or goal, and to add an actor perspective. Irrespective of project type, the Logical Framework Approach (LFA) is often a standard requirement of major official donor agencies on projects they fund, so as to fulfil bureaucratic imperatives. However, it is often considered inadequate in addressing key challenges in development evaluation. Given the dominant status of the LFA with such strong support from donors, it is helpful to seek a ‘middle way’: a combination of the LFA with other approaches in order to address some of its inadequacies, while satisfying donor agencies’ requirements. A synthesis of the LFA and Outcome Mapping (OM) is one such option. This thesis explores the practical value and usefulness of a synthesis model empirically. Applying the model in two case study aid projects, I found that it serves well as a theory-based evaluation tool with a double-stranded (actor strand and results chain) theory of change. The model helps reconcile learning and accountability and add explanatory power and an explicit actor perspective. It also helps establish causation and enable attribution claims at various results levels with its different elements. The model has some limitations but my results suggest it can be usefully adopted. The choice of its application depends on project evaluation context and purpose in specific cases. DA - 2018/// PY - 2018 DP - sussex.primo.exlibrisgroup.com LA - eng ST - Evaluating development projects UR - http://sro.sussex.ac.uk/id/eprint/79800/1/Yang%2C%20Ting.pdf Y2 - 2022/12/05/00:00:00 ER - TY - BOOK TI - Evaluating interventions that prevent or counter violent extremism: A practical guide A3 - Hofman, Joanna A3 - Sutherland, Alex DA - 2018/// PY - 2018 DP - Crossref SP - 173 LA - en PB - RAND Corporation ST - Evaluating interventions that prevent or counter violent extremism UR - https://www.rand.org/pubs/research_reports/RR2094.html Y2 - 2019/09/17/10:53:42 ER - TY - JOUR TI - Evaluating megaprojects: From the ‘iron triangle’ to network mapping AU - Lehtonen, Markku T2 - Evaluation AB - Evaluation literature has paid relatively little attention to the specific needs of evaluating large, complex industrial and infrastructure projects, often called ‘megaprojects’. The abundant megaproject governance literature, in turn, has largely focused on the so-called ‘megaproject pathologies’, i.e. the chronic budget overruns, and failure of such projects to keep to timetables and deliver the expected social and economic benefits. This article draws on these two strands of literature, identifies shortcomings, and suggests potential pathways towards an improved evaluation of megaprojects. To counterbalance the current overemphasis on relatively narrowly defined accountability as the main function of megaproject evaluation, and the narrow definition of project success in megaproject evaluation, the article argues that conceptualizing megaprojects as dynamic and evolving networks would provide a useful basis for the design of an evaluation approach better able to promote learning and to address the socio-economic aspects of megaprojects. A modified version of ‘network mapping’ is suggested as a possible framework for megaproject evaluation, with the exploration of the multiple accountability relationships as a central evaluation task, designed to reconcile learning and accountability as the central evaluation functions. The article highlights the role of evaluation as an ‘emergent’ property of spontaneous megaproject ‘governing’, and explores the challenges that this poses to the role of the evaluator. DA - 2014/07/01/ PY - 2014 DO - 10.1177/1356389014539868 DP - SAGE Journals VL - 20 IS - 3 SP - 278 EP - 295 J2 - Evaluation LA - en SN - 1356-3890 ST - Evaluating megaprojects UR - http://journals.sagepub.com/doi/abs/10.1177/1356389014539868 ER - TY - JOUR TI - Evaluating programming that thinks and works politically: Challenges and emerging practice AU - Jacobstein, David AU - Swift, Sarah T2 - New Directions for Evaluation AB - Issues of power are not new to program evaluation. What is new is a consideration of how programming uses insights into incentives that shape and adapt implementation. How should one evaluate in a way that explicitly assesses the ways in which a program considers power? One of the innovative topics deriving from the democracy and governance space is the approach of thinking and working politically (TWP) which is seeing increased use in development programming. TWP suggests different mental models and practical approaches to achieving development objectives in ways that are more contextually grounded and informed by power dynamics. This article describes several of the core challenges to evaluation of TWP and also a rubric of considerations for more effective evaluation practices in this emerging field. DA - 2022/// PY - 2022 DO - 10.1002/ev.20527 DP - Wiley Online Library VL - 2022 IS - 176 SP - 69 EP - 78 LA - en SN - 1534-875X ST - Evaluating programming that thinks and works politically UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20527 Y2 - 2023/04/13/09:06:47 ER - TY - RPRT TI - Evaluating Systems Change Results: an inquiry framework AU - Cabaj, Mark DA - 2018/// PY - 2018 DP - Zotero SP - 22 LA - en PB - Tamarack Institute UR - https://www.tamarackcommunity.ca/evaluating-impact-evaluating-systems-change ER - TY - RPRT TI - Evaluating the impact of flexible development interventions using a ‘loose’ theory of change: Reflections on the Australia-Mekong NGO Engagement Platform AU - Davies, Rick AB - • For some interventions, tight and testable theories of change are not appropriate – for example, in fast moving humanitarian emergencies or participatory development programmes, a more flexible approach is needed. • However, it is still possible to have a flexible project design and to draw conclusions about causal attribution. This middle path involves ‘loose’ theories of change, where activities and outcomes may be known, but the likely causal links between them are not yet clear. • In this approach, data is collected ‘after the event’ and analysed across and within cases, developing testable models for ‘what works’. More data will likely be needed than for projects with a ‘tight’ theory of change, as there is a wider range of relationships between interventions and outcomes to analyse. The theory of change plays an important role in guiding the selection of data types. • While loose theories of change are useful to identify long term impacts, this approach can also support short cycle learning about the effectiveness of specific activities being implemented within a project’s lifespan. DA - 2016/03// PY - 2016 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10361.pdf Y2 - 2018/11/10/00:00:00 ER - TY - JOUR TI - Evaluating the ‘logical framework approach’ towards learning-oriented development evaluation AU - Gasper, Des T2 - Public Administration and Development AB - The logical framework approach has spread enormously, including increasingly to stages of review and evaluation. Yet it has had little systematic evaluation itself. Survey of available materials indicates several recurrent failings, some less easily countered than others. In particular: focus on achievement of intended effects by intended routes makes logframes a very limiting tool in evaluation; an assumption of consensual project objectives often becomes problematic in public and inter-organizational projects; and automatic choice of an audit form of accountability as the priority in evaluations can be at the expense of evaluation as learning. DA - 2000/// PY - 2000 DO - 10.1002/1099-162X(200002)20:1<17::AID-PAD89>3.0.CO;2-5 DP - Wiley Online Library VL - 20 IS - 1 SP - 17 EP - 28 LA - en SN - 1099-162X UR - http://onlinelibrary.wiley.com/doi/abs/10.1002/1099-162X%28200002%2920%3A1%3C17%3A%3AAID-PAD89%3E3.0.CO%3B2-5 Y2 - 2022/12/05/22:10:23 ER - TY - BLOG TI - Evaluating with the Success Case Method AU - Gram, Tom T2 - Performance X Design AB - The method was developed by Robert Brinkerhoff as an alternative (or supplement) to the Kirkpatrick approach and its derivatives. It is very simple and fast (which is part of it’s appeal) and goes something like this: Step 1. Identify targeted business goals and impact expectations Step 2. Survey a large representative sample of all participants in a program to identify high impact and low impact cases Step 3. Analyze the survey data to identify: a small group of successful participants a small group unsuccessful participants Step 4. Conduct in-depth interviews with the two selected groups to: document the nature and business value of their application of learning identify the performance factors that supported learning application and obstacles that prevented it. Step 5. Document and disseminate the story report impact applaud successes use data to educate managers and organization The process produces two key outputs In-depth stories of documented business effect that can be disseminated to a variety of audiences Knowledge of factors that enhance or impede the effect of training on business results. Factors that are associated with successful application of new skills are compared and contrasted with those that impede training. It answers practical and common questions we have about training and other initiatives: What is really happening? Who’s using what, and how well? Who’s not using things as planned? What’s getting used, and what isn’t? Which people and how many are having success? Which people and how many are not? What results are being achieved? What value, if any, is being realized? What goals are being met? What goals are not? Is the intervention delivering the promised and hoped for results? What unintended results are happening? What is the value of the results? What sort of dollar or other value can be placed on the results? Does the program appear to be worthwhile? Is it producing results worth more than its costs? What is its return on investment? How much more value could it produce if it were working better? How can it be improved? What’s helping? What’s getting in the way? What could be done to get more people to use it? How can everyone be more like those few who are most successful? DA - 2011/02/24/ PY - 2011 UR - https://performancexdesign.wordpress.com/2011/02/24/evaluating-with-the-success-case-method/ Y2 - 2018/10/22/00:00:00 ER - TY - JOUR TI - Evaluation and learning in complex, rapidly changing health systems: China’s management of health sector reform AU - Xiao, Yue AU - Husain, Lewis AU - Bloom, Gerald T2 - Globalization and Health AB - Healthcare systems are increasingly recognised as complex, in which a range of non-linear and emergent behaviours occur. China’s healthcare system is no exception. The hugeness of China, and the variation in conditions in different jurisdictions present very substantial challenges to reformers, and militate against adopting one-size-fits-all policy solutions. As a consequence, approaches to change management in China have frequently emphasised the importance of sub-national experimentation, innovation, and learning. Multiple mechanisms exist within the government structure to allow and encourage flexible implementation of policies, and tailoring of reforms to context. These limit the risk of large-scale policy failures and play a role in exploring new reform directions and potentially systemically-useful practices. They have helped in managing the huge transition that China has undergone from the 1970s onwards. China has historically made use of a number of mechanisms to encourage learning from innovative and emergent policy practices. Policy evaluation is increasingly becoming a tool used to probe emergent practices and inform iterative policy making/refining. This paper examines the case of a central policy research institute whose mandate includes evaluating reforms and providing feedback to the health ministry. Evaluation approaches being used are evolving as Chinese research agencies become increasingly professionalised, and in response to the increasing complexity of reforms. The paper argues that learning from widespread innovation and experimentation is challenging, but necessary for stewardship of large, and rapidly-changing systems. DA - 2018/11/20/ PY - 2018 DO - 10.1186/s12992-018-0429-7 DP - BioMed Central VL - 14 IS - 1 SP - 112 J2 - Globalization and Health SN - 1744-8603 ST - Evaluation and learning in complex, rapidly changing health systems UR - https://doi.org/10.1186/s12992-018-0429-7 Y2 - 2019/03/15/10:31:43 ER - TY - RPRT TI - Evaluation Building Blocks: A Guide AU - McKegg, K. AU - Oakden, J. AU - Wehipeihana, N. AU - King, Julian AB - This guide presents the Kinnect Group’s approach to evaluation. Developed through ten years of collaborative work, it has been well tested. While there are many other guides to evaluation and different ways to do it, our approach emphasises the place of evaluative reasoning in evaluation. The guide draws on the work of many evaluation theorists, as well as our practice-based body of knowledge. We hope that using the building blocks as explained in the guide will help you in your endeavour to do credible and useful evaluation. We find this approach works for us. Our clients tell us it gives them meaningful and insightful findings that they can use to take action. Our view to intellectual property is simple. What you read is yours to use. We just ask that you cite the guide when you draw from it. For those of you who engage with this guide and use it in your practice, we look forward to hearing from you. We are particularly interested in feedback where you find aspects of the guide work well for you. In that way we can all continue to build and evolve our practice. CY - Auckland DA - 2018/// PY - 2018 PB - Kinnect Group UR - https://www.kinnect.co.nz/site_files/32691/upload_files/blog/evaluationbuildingblocks_a-guide_final_v1.pdf?dl=1 Y2 - 2023/09/29/10:47:45 ER - TY - RPRT TI - Evaluation for equitable development results A2 - Segone, M AB - This document is made up of a range of Evaluation Working Papers (EWP) focused on evaluation for equitable development. Put together by evaluation specialists they present strategic evaluation findings, lessons learned and innovative approaches and methodologies. Part 1:Evaluation and equity Evaluation to accelerate progress towards equity, social justice and human rights 2 Human rights and gender equality in evaluation 13 When human rights is the starting point for evaluation 25 Strengthening Equity- focused evaluations through insights from feminist theory and approaches 39 Decolonizing evaluation in a developing world. Implications and cautions for Equity-focused evaluations 59 Part 2: Methodological implications for Equity-focused evaluations Methodological issues to design and implement equity-focused evaluations 86 Developmental evaluation for Equity-focused evaluations 102 Systems thinking and Equity-focused evaluations 115 Methodological challenges in using programme theory to evaluate pro-poor and equity-focused programmes by Patricia Rogers, Royal Melbourne Institute of Technology University and Richard Hummelbrunner, Independent consultant 142 Case Study and equity in Evaluationby Saville Kushner, University of the West of Englan 172 Values-Engaged Evaluationsby Jennifer Greene, University of Illinois at Urbana-Champaign 192 Part 3: Examples of Equity-focused evaluations Evaluating the contribution of UNDP to equity-focused public policies in Brazil and China 210 Using a human rights approach to evaluate ILO’s discrimination strategy 222 CONEVAL experience in evaluating interventions for Indigenous populations in Mexico 244 UNICEF supported evaluations with elements of equity-focused evaluations 258 CY - New York DA - 2011/// PY - 2011 SP - 308 PB - UNICEF UR - http://www.clear-la.cide.edu/sites/default/files/Evaluation_for_equitable%20results_web.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Evaluation for Improvement: A Seven Step Empowerment Evaluation Approach for Violence Prevention Organizations AU - Cox, PJ AU - Keener, D AU - Woodward, T AU - Wandersman, A AB - This guide, written by Pamela J. Cox, Dana Keener, Tifanee L. Woodard, & Abraham H. Wandersman for the Centers for Disease Control and Prevention (CDC) outlines a seven step process for hiring an evaluator to implement an empowerment evaluation. The process begins with preparing for the hiring and concludes with an assessment of an evaluation to ensure its sustainability. Excerpt "...concerns and experiences with independent evaluation led to the development of participatory evaluation approaches as a way to promote an organization’s use of evaluation for the improvement of its strategies. Although there are many participatory evaluation approaches, empowerment evaluation places an explicit emphasis on building the evaluation capacity of individuals and organizations so that evaluation is integrated into the organization’s day-to-day management processes. Through empowerment evaluation, both individual and organizational evaluation capacity are increased through a “learn-by-doing” process, whereby organizations and their staff evaluate their own strategies. Specifically, organizations hire an evaluator to work with them in conducting an evaluation of their strategies. Rather than evaluating an organization’s strategies and presenting an evaluation “report card,” empowerment evaluators coach individuals and organizations through an evaluation of their own strategy(ies) by providing them with the knowledge, skills, and resources they need to conduct just such an evaluation." Contents Empowerment Evaluation: An Overview 11 Principles of Empowerment Evaluation 11 Step 1: Preparing for the Hiring Process 23 Step 2: Writing a Job Announcement 31 Step 3: Finding Potential Empowerment Evaluators 39 Step 4: Assessing the Candidates 45 Step 5: Writing An Evaluation Contract 55 Step 6: Building an Effective Relationship With Your Evaluator 61 Step 7: Assessing and Sustaining the Evaluation 65 Appendix A: Resources for General Evaluation and Empowerment Evaluation 73 Appendix B: Worksheets for Hiring an Empowerment Evaluator 75 Appendix C: Sample Hiring Committee Confidentiality Statement 89 Appendix D: Sample Job 90 Appendix E: Sample Request For Proposals 91 Appendix F: Sample Interview Questions 93 Appendix G: Sample Budget and Narrative for an Evaluation Team 95 DA - 2009/// PY - 2009 UR - https://www.cdc.gov/violenceprevention/pdf/evaluation_improvement-a.pdf Y2 - 2018/10/18/00:00:00 ER - TY - ELEC TI - Evaluation of development programmes - OECD AU - OECD AB - Data & research on evaluation of development programmes inc. paris declaration, budget support, multilateral effectiveness, impact evaluation, joint evaluations, governance, aid for trade DA - 2019/// PY - 2019 UR - https://www.oecd.org/dac/evaluation/ Y2 - 2019/11/27/11:59:43 ER - TY - BOOK TI - Evaluation of International Development Interventions: An Overview of Approaches and Methods AU - Vaessen, Jos AU - Lemire, Sebastian AU - Befani, Barbara AB - This guide provides an introductory overview of a range of methods that have been selected for their actual and potential use in the field of international development evaluation. For each method, a detailed guidance note presents the method’s main features and procedural steps, key advantages and disadvantages, as well as its applica­bility. Each guidance note includes references for relevant background readings (basic and advanced) as well as references to other additional resources of interest. Both the choice of approaches and methods and the associated guidance are by no means definitive. IEG plans to periodically update the guide as evaluation prac­tices evolve. DA - 2020/11// PY - 2020 DP - DOI.org (Crossref) LA - en PB - World Bank ST - Evaluation of International Development Interventions UR - http://elibrary.worldbank.org/doi/book/10.1596/34962 Y2 - 2021/08/05/20:47:06 ER - TY - RPRT TI - Evaluation of the market systems development approach: Lessons for expanded use and adaptive management at Sida Volume I: Evaluation Report AU - Ruffer, Tim AU - Bailey, Helen AU - Dahlgren, Stefan AU - Spaven, Patrick AU - Winters, Mark T2 - Evaluation of the market systems development approach AB - This report presents the findings of an evaluation of Sida’s management of the market systems development (MSD) approach. It aims to inform thinking on how Sida can best manage its growing portfolio of MSD programs. Beyond this, it provides insights relevant to Sida’s wider support to complex and adaptive programs. The evaluation identified several factors that affect Sida’s ability to ensure that conducive conditions are in place for effective MSD programs and good development programming more generally. Sida’s relatively flexible framework of rules, guidelines and systems for project management provide the space needed for staff to innovate and manage adaptively. But for this to happen consistently and effectively, Sida needs to invest more deliberately in building the capacity of its staff in relevant areas. In addition, leadership and incentives are key to shaping a culture of active experimentation and learning. This needs to be supported with clearer guidance for those involved in the design and appraisal of MSD projects; and strengthened oversight of project performance, including through adjustments to Sida’s contracts and funding agreements. CY - Stockholm DA - 2018/12// PY - 2018 PB - Sida UR - https://www.sida.se/contentassets/bfe15e8902fa4dbb864bd478c2f14df1/2018_2a_evaluation_market_systems_dev_approach_vol-1.pdf Y2 - 2019/01/08/12:11:40 ER - TY - RPRT TI - Evaluation of the market systems development approach: Lessons for expanded use and adaptive management at Sida Volume II: Case studies AU - Ruffer, Tim AU - Bailey, Helen AU - Dahlgren, Stefan AU - Spaven, Patrick AU - Winters, Mark T2 - Evaluation of the market systems development approach AB - This report presents the findings of an evaluation of Sida’s management of the market systems development (MSD) approach. It aims to inform thinking on how Sida can best manage its growing portfolio of MSD programs. Beyond this, it provides insights relevant to Sida’s wider support to complex and adaptive programs. The evaluation identified several factors that affect Sida’s ability to ensure that conducive conditions are in place for effective MSD programs and good development programming more generally. Sida’s relatively flexible framework of rules, guidelines and systems for project management provide the space needed for staff to innovate and manage adaptively. But for this to happen consistently and effectively, Sida needs to invest more deliberately in building the capacity of its staff in relevant areas. In addition, leadership and incentives are key to shaping a culture of active experimentation and learning. This needs to be supported with clearer guidance for those involved in the design and appraisal of MSD projects; and strengthened oversight of project performance, including through adjustments to Sida’s contracts and funding agreements. CY - Stockholm DA - 2018/12// PY - 2018 PB - Sida UR - https://www.sida.se/contentassets/bfe15e8902fa4dbb864bd478c2f14df1/2018_2a_evaluation_market_systems_dev_approach_vol-1.pdf Y2 - 2019/01/08/12:11:40 ER - TY - BLOG TI - Evaluative reasoning in complexity AU - King, Julian T2 - Evaluation and Value for Investment DA - 2023/06/11/ PY - 2023 UR - https://juliankingnz.substack.com/p/evaluative-reasoning-in-complexity?utm_source=substack&utm_medium=email Y2 - 2024/01/04/23:23:00 ER - TY - RPRT TI - Everyday patterns for shifting systems AU - GCSI DA - 2022/10/25/ PY - 2022 DP - Zotero LA - en PB - GCSI UR - https://www.griffith.edu.au/__data/assets/pdf_file/0024/1867002/Right-Scaling_Patterns_TSI-and-GCSI.pdf ER - TY - RPRT TI - Everyday patterns for shifting systems - Right scaling AU - GCSI DA - 2022/10/25/ PY - 2022 DP - Zotero LA - en PB - GCSI UR - https://www.griffith.edu.au/__data/assets/pdf_file/0024/1867002/Right-Scaling_Patterns_TSI-and-GCSI.pdf ER - TY - RPRT TI - Evidence-led adaptive programming: Lessons from MUVA AU - Sharp, Samuel AU - Riemenschneider, Nils AU - Selvester, Kerry AB - Calls for more ‘adaptive programming’ have been prominent in international development practice for over a decade. Learning-by-doing is a crucial element of this, but programmes have often found it challenging to become more learning oriented. Establishing some form of reflective practice, against countervailing incentives, is difficult. Incorporating data collection processes that generate useful, timely and practical information to inform these reflections is even more so.This paper explores MUVA - an adaptive female economic empowerment programme in Mozambique. MUVA, we suggest, is atypically evidence-led. It combines systematic, inclusive reflective practice with extensive real-time data collection. We describe the fundamental features of MUVA’s monitoring, evaluation and learning (MEL) approach that supported this. One, how data collection and analysis are synchronised with set cycles for learning and adapting projects. Two, how MEL systems are designed to prioritise actionable learning, with data collection oriented more to the needs of implementing staff than to the reporting requirements of funders.This approach was enabled by building collective ownership over the programme’s objectives and the purpose of MEL from the outset. Implementers are asked about their motivations, and these are related to the programme’s Theory of Change. The evidence culture is supported by the proximity of MEL staff to implementing staff; and through structuring upwards accountability to funders around justifying evidence-based adaptations instead of reporting on more narrow indicators. We conclude by considering the relevance, or not, of MUVA’s approach to programmes in other contexts or issue areas trying to replicate a similarly evidence-informed approach to adaptive management. Key messages Learning-by-doing is essential to adaptive programming, but it can be challenging to establish data collection processes that generate useful, timely and practical information. MUVA – a female economic empowerment programme in Mozambique – has an atypically evidence-led adaptive management approach. This has two fundamental features. One, data collection and analysis are synchronised with set cycles for learning and adapting projects. Two, data collection is oriented more to the needs of implementing staff than to the reporting requirements of funders. Monitoring, evaluation and learning (MEL) systems are designed to prioritise actionable learning. This approach was enabled by building collective ownership over the programme’s objectives and the purpose of MEL from the outset. Implementers are asked about their motivations, and these are related to the programme’s Theory of Change. The evidence culture is supported by the proximity of MEL staff to implementing staff; and through structuring upwards accountability to funders around justifying evidence-based adaptations instead of reporting on more narrow indicators. CY - London DA - 2022/06/29/ PY - 2022 LA - en-gb PB - ODI ST - Evidence-led adaptive programming UR - https://odi.org/en/publications/evidence-led-adaptive-programming-lessons-from-muva/ Y2 - 2022/07/04/10:45:09 ER - TY - THES TI - Examining the results and adaptation ideas in foreign aid AU - Janus, Heiner AB - This thesis applies ideational and institutional theories to analyse how two specific ideas, results and adaptation, have changed the theory and practice of development cooperation. The thesis addresses the question of why the results and adaptation ideas are often treated as binaries and how this debate has evolved historically. In a first theoretical paper, the evolution of results and adaptation is conceptualised as a combination of institutional layering and diffusion within development organisations. The second theoretical paper applies ideational theory, in particular, the coalition magnet framework, to China as a donor country. The empirical papers apply ideational and institutional theories to study aid projects funded by the World Bank and China in the Rwandan agriculture sector. The third paper analyses through which mechanism, results-based principal-agent relationships or problem-driven iterative adaptation, the World Bank’s Program for Results in the agriculture sector in Rwanda has led to increased agricultural productivity. The paper combines causal process tracing and contribution analysis to investigate two underlying theories of change of the Program for Results. The fourth paper applies the same framework and methodology to the Chinese Agricultural Technology Demonstration Center in Rwanda. The fifth paper compares both projects, the World Bank project and the Chinese project. The thesis finds that the ideas of results and adaptation are often presented as mutually exclusive mainly at the general level of public philosophies or paradigms, but show overlap and potential for integration on the level of framing policy problems and policy solutions. The thesis also demonstrates that there is unexplored potential for convergence between China and Development Assistance Committee donors around “coalition magnet” ideas. The empirical part of the thesis reveals how results-based and adaptive causal mechanisms co-exist within given aid interventions by the World Bank and China, how these interact and how they ultimately contribute to achieving development outcomes. The key finding is that the broader political context of the Rwandan agricultural sector is the main factor for determining development outcomes, which neither the World Bank project nor the Chinese projects take into account. The comparison of the World Bank’s and China’s interventions finds that donor organisations need to address how results-based ideas in combination adaptive development ideas can be better tailored to fit into the specific context of the Rwandan agriculture sector. CY - Manchester DA - 2020/// PY - 2020 LA - en M3 - PhD Thesis PB - University of Manchester UR - https://www.research.manchester.ac.uk/portal/en/theses/examining-the-results-and-adaptation-ideas-in-foreign-aid(33eb1913-0918-4147-8080-f36f3f444c18).html Y2 - 2022/03/30/14:26:31 ER - TY - RPRT TI - Exit for Sustainability Checklists AU - Cekan, Jindra AB - These Checklists for Exit along the Sustainability Project Cycle are based on lessons from ex-post evaluations that Valuing Voices has done or researched. Ex-post project completion evaluations are rare and even more so those that consulted partners and participants in the field about sustained impacts. Over $3.5 trillion has been spent on public foreign aid projects in the past 70 years (OECD 2019) yet we have evaluated less than 1% of these projects for sustainability. Our Valuing Voices ex-post research of 39 organizations’ evaluations of sustainability shows that most project results decrease (20-90%) as early as two years ex-post in addition to An Asian Development Bank study of post-completion sustainability found that “some early evidence suggests that as many as 40% of all new activities are not sustained beyond the first few years after disbursement of external funding,” Most project exits are in the last quarter and sustainability handover assumptions are not validated expost. Learning from what was sustained helps us know how to exit for sustainability from the very onset of the project (green slices) as compared to the typical project cycle (orange), above. We encourage those tasked with funding, designing, implementing, monitoring & evaluating projects to use these longer checklists and view the full recording shared with participants. A partial PowerPoint can also be found on „Sustaining All of our Hard Work“ presentation for the Vienna Evaluation Network (10/20). These checklists are aimed at donors/designers and implementers of foreign aid projects outcomes and impacts and can be adapted by local NGOs, national governments, private sector, academics, to create exit plans. Local participation in creating these and feedback on how well exit is going will help them sustain results. DA - 2020/// PY - 2020 PB - Valuing Voices UR - https://valuingvoices.com/wp-content/uploads/2021/03/Exit-For-Sustainability-Checklists-Dec2020-2.pdf Y2 - 2021/02/18/13:20:22 ER - TY - RPRT TI - Exploring new territories for evaluation AU - Sharp, Cathy AB - Drawing on action research, this paper recasts evaluation as ‘action inquiry’, an embedded evaluative learning practice that can help navigate complexity when enacting collective leadership. It is offered as an invitation to inquiry amongst a reasonably well-informed audience of policy makers and practitioners who work in and for public services. It will particularly interest those who provide research, evaluation and facilitation support, and those seeking to develop a more relational approach to research and evaluation. Action inquiry is a model of practising change together in environments where ‘nothing is clear, and everything keeps changing’ that significantly challenges the prevailing discourse on evaluation. Action inquiry can be wrapped around and enmeshed within initiatives and programmes that work with complexity - anywhere where success will depend on the quality of relationships that can be developed. The paper reviews some important interrelated concepts that underpin the ideals of collective leadership and public service reform and which confront deeply embedded traditional notions of leadership, expertise and participation. These offer important challenges to ideas about how change happens and recognise that relationships are at the heart of practising change. This warrants a re-examination of the high expectations of evidence-based or informed practice. Collective leadership makes new demands of evidence as it rests on help to determine ‘wise actions’ in real-life situations. This confronts the practical reality of how to work together in conditions often expressed as ‘dynamic’ or ‘turbulent’ and the added human complexities of power, emotions and relationships; too often these elements are denied or avoided aspects of a change process. Facilitated action inquiry makes these elements part of the conversations, in the midst of ‘work-as-we-are-doing-it’, to increase areas of choice for individuals and a group as a whole. Within public policy, there is an overwhelming amount of evidence about ‘what works’ yet change seems to be stubborn and slow. The paper explores some of the deep-rooted vestiges of a ‘hierarchy of evidence’ and assumptions about standardisation and generalisability that act as a ‘barrier to transformation’. These include the narrow framing of what counts as evidence and consequent relegation of community perspectives, lived experience and practice-based evidence. The paper acknowledges the clear appetite for different approaches to evaluation, especially those that better reflect deeply held values and avoid creating a culture of ‘gaming’, rooted in fear of failure and loss of funding, at the expense of learning. The need for new forms of developmental evaluative thinking, collaborative inquiry and action research to create embedded learning is well overdue. Action inquiry is a desirable and necessary response to the complex situations and challenges of human services and recognises the essentialness of knowledge co-production. It is a model of co-creation at every stage and endorses the idea that people learn from participation in evaluation and by testing theories of change through action. Action inquiry builds on the idea of inquiry, or a moment-to-moment awareness and quality of attention and draws from several elements of action research practice. It sees inquiry as an intervention in itself, one that furthermore, explicitly seeks to enhance the probability of the success of a programme, focus on learning, the collaborative development of practice-based knowledge and positive relationships. The paper highlights the importance of building inquiry into living systems, the role of facilitation, systemic inquiry, and evaluative thinking. It proposes an expansion of ideas of appreciation as a relational and collaborative practice that is a driver of emergence. Social recognition that acknowledges someone’s social value to the community and implies mutual moral obligations to cooperation and participation is particularly crucial in a work context that requires successful coordination and multiple contributions to achieve results across hierarchies of position, professional rank and sectors. Hence, appreciation goes beyond the idea of positivity to include social recognition, valuing more explicit forms of inquiry, building participants’ aspirations to design new social systems and acting in new ways to embed change. In developing this discussion, the paper contributes to emerging dialogues about the need for a model of ‘5th generation evaluation’. Such a model would be based on the idea that appreciative and challenging inquiry that is contextual, relational and open-minded will create better opportunities for change and development. The paper sets out some ‘provocative propositions’ that can help us to navigate this terrain, perhaps of a fledgling ‘5th generation approach’ to inquiry. Facilitated action inquiry can hold the key to developing both new knowledge and an adaptive, collaborative and improvisational skill-set, able to respond in new ways to systemic and complex issues on the ground. It’s common to hear the expression ‘it’s all about relationships’ and it is clearly time to shift our focus to relationships; not relationships as ‘things’, but as co-created and dynamic relational processes in which we are embedded. In this way we can bring new qualities to our talking to each other about our various and shared visions of a better future. CY - Edinburgh DA - 2018/// PY - 2018 DP - Zotero LA - en PB - Collective Leadership for Scotland UR - https://www.humanlearning.systems/uploads/collectiveleadershipreport1.pdf Y2 - 2023/05/22/00:00:00 ER - TY - RPRT TI - Exploring new territories for evaluation - Provocative propositions AU - Sharp, Cathy DA - 2018/// PY - 2018 LA - en UR - https://www.humanlearning.systems/uploads/collectiveleadershipreport1.pdf Y2 - 2023/05/22/00:00:00 ER - TY - JOUR TI - Exploring ways to reconcile accountability and learning in the evaluation of niche experiments AU - Regeer, Barbara J. AU - de Wildt-Liesveld, Renée AU - van Mierlo, Barbara AU - Bunders, Joske F. G. T2 - Evaluation AB - While evaluation is seen as a mechanism for both accountability and learning, it is not self-evident that the evaluation of niche experiments focuses on both accountability and learning at the same time. Tensions exist between the accountability-oriented needs of funders and the learning needs of managers of niche experiments. This article explores the differences in needs and expectations of funders and managers in terms of upwards, downwards and internal accountability. The article shows that as the multi-stakeholder contexts in which niche experiments take place give rise to various requirements, tensions in evaluation are essentially a specific manifestation of tensions between niche experiments and their multiple contexts. Based on our findings, an adjusted accountability framework is proposed, including several strategies that can reconcile a learning approach with accountability needs in niche experiments aiming to change current practices in a more sustainable direction. DA - 2016/01/01/ PY - 2016 DO - 10.1177/1356389015623659 DP - SAGE Journals VL - 22 IS - 1 SP - 6 EP - 28 J2 - Evaluation LA - en SN - 1356-3890 UR - http://journals.sagepub.com/doi/abs/10.1177/1356389015623659 ER - TY - BLOG TI - Extending epistemology for programme evaluation – can After Action Reviews become spaces for critical reflection? AU - Apgar, Marina AU - Snijder, Mieke T2 - Methodspace - Sage AB - The term After Action Review (AAR) is becoming more common in the world of development evaluation, particularly in programmes that focus on evaluation as a formative learning process. As structured and facilitated learning moments, AARs take many shapes always built to support a specific team in a specific moment on their journey. Across such a diversity of practice how do we understand the evolution of the AAR as a method with an action research orientation? Being mindful always of the risk of instrumentalising and co-opting participatory methods, there is danger that AARs become an empty programme ritual, remaining at the surface and failing to achieve critical reflection and so falling short of their intention of pushing for deeper change in our practice. As the evaluation and learning team of two large complex, multi-partner projects using research to address development challenges, we have been applying the AAR method while adapting to virtual COVID working with partners across the world. As action researchers we have been thinking about what critical reflection means within a programme AAR process and whether and how they can open up a second person inquiry space. DA - 2020/10/31/ PY - 2020 LA - en-US UR - https://www.methodspace.com/blog/extending-epistemology-for-programme-evaluation-can-after-action-reviews-become-spaces-for-critical-reflection Y2 - 2023/12/19/10:43:59 ER - TY - RPRT TI - Failing Forward: How CARE is Focusing on What Goes Wrong to Improve Impact AU - Janoch, Emily T2 - CLA Case Competition AB - CARE's Failing Forward initiative is sparking opportunities to showcase the ideas that don't work so we can spend more time implementing the ones that do. It's changing the conversation inside the organization, and leading to changes in the way we design and implement programs. DA - 2019/08/07/ PY - 2019 LA - en M3 - Text PB - CARE ST - Failing Forward UR - https://usaidlearninglab.org/library/failing-forward-how-care-focusing-what-goes-wrong-improve-impact Y2 - 2020/02/14/12:30:01 ER - TY - BLOG TI - Fast Forwarding to Systems Maps of Corruption: Getting to Usable Analysis More Quickly AU - Woodrow, Peter T2 - CJL AB - For AC practitioners, systems mapping is essential but also difficult. To lessen obstacles, Peter Woodrow proposes a scaffolded approach. An Experiment in “Fast Forwarding” Drawing upon our near-decade work on corruption, we recently decided to try an experiment: we would present “common patterns” of corruption as tentative models to adapt and add to—rather than try to teach people to do systems mapping from scratch. In this teaching experiment, each common pattern would function as a kind of “scaffolding” or framework that participants can build on to generate a cogent systems map as the basis for identifying possible points of intervention and subsequent program planning. DA - 2023/07/09/T23:09:19.358Z PY - 2023 LA - en ST - Fast Forwarding to Systems Maps of Corruption UR - https://www.corruptionjusticeandlegitimacy.org/post/fast-forwarding-to-systems-maps-of-corruption-getting-to-usable-analysis-more-quickly Y2 - 2023/10/02/09:57:50 ER - TY - RPRT TI - FCDO Beneficiary Engagement AU - FCDO AB - This guide has been developed to help build confidence and capability, distilling useful tips and considerations that may help teams think through programme delivery issues and interpret elements of the PrOF Rules. This PrOF Guide lays out: - The definition of beneficiary engagement. - The case for beneficiary engagement. - FCDO’s approach to beneficiary engagement. - Practical tips for how to integrate beneficiary engagement throughout the programme cycle, including guiding questions to ask, rules of thumb to apply, tools to use and challenges and special topics to consider during Design, Mobilisation, Delivery and Closure phases. At its core, beneficiary engagement is about processes that recognise the dignity and support the agency of the people whose lives we are trying to improve. It is about beneficiaries and programme constituents having a say over what assistance they receive and how they receive it. It is about engaging beneficiaries and programme constituents as people with valuable insights and capabilities, rather than a compliance exercise. It is about empowering all beneficiaries and programme constituents to improve their lives by engaging them in helping us make better design and delivery decisions for the programmes that affect their lives. It’s about ensuring that a diverse set of voices are heard. Harnessing the power of beneficiary engagement can also improve outcomes and help programmes reach them more efficiently. It helps define and promote Value for Money, improve transparency and ensure that beneficiaries are safe from harm and empowered to speak out wherever harm does occur. Beneficiary engagement is supported by FCDO policy commitments, PrOF Rules, internal guidance and key international commitments. Beneficiary engagement is applicable to, and valuable in, a range of contexts, including humanitarian contexts. Beneficiary engagement requires time and resources, but programmes can help ensure the benefits of engagement outweigh the costs by Doing No Harm, Engaging Early and Closing the Loop. It is ultimately the Programme Responsible Owner’s responsibility to determine what beneficiary engagement is suitable and feasible for a programme. Quality beneficiary engagement is not about applying the one “right” approach but rather thoughtfully considering key questions and applying key principles and proven tools to the programme’s context to achieve a programme that supports the dignity and agency of all beneficiaries as much as possible. CY - London DA - 2021/03// PY - 2021 LA - en-GB PB - FCDO UR - https://www.bond.org.uk/resources/fcdo-beneficiary-engagement/ Y2 - 2023/07/17/11:12:39 ER - TY - BLOG TI - Finding and using causal hotspots: a practice in the making AU - Apgar, Marina AU - Snijder, Mieke T2 - Institute of Development Studies AB - This is the second blog in our series ‘Lessons on using Contribution Analysis for impact evaluation’. In our first blog, we introduced Contribution Analysis (CA) as an overarching approach to theory-based evaluation and the idea of causal hotpots as a way to zoom in, unpack and make the hard choices of where to focus evaluation research. Identifying specific links in the theory of change (ToC) with specific evaluation questions enables you to then choose appropriate methods. We have applied the approach in diverse settings, testing how robust Contribution Analysis can really be. Here we walk you through an example. DA - 2021/09/15/ PY - 2021 LA - en-US UR - https://www.ids.ac.uk/opinions/finding-and-using-causal-hotspots-a-practice-in-the-making/ Y2 - 2024/02/13/00:00:00 ER - TY - JOUR TI - Five Misunderstandings About Case-Study Research AU - Flyvbjerg, Bent T2 - Qualitative Inquiry AB - This article examines five common misunderstandings about case-study research: (a) theoretical knowledge is more valuable than practical knowledge; (b) one cannot generalize from a single case, therefore, the single-case study cannot contribute to scientific development; (c) the case study is most useful for generating hypotheses, whereas other methods are more suitable for hypotheses testing and theory building; (d) the case study contains a bias toward verification; and (e) it is often difficult to summarize specific case studies. This article explains and corrects these misunderstandings one by one and concludes with the Kuhnian insight that a scientific discipline without a large number of thoroughly executed case studies is a discipline without systematic production of exemplars, and a discipline without exemplars is an ineffective one. Social science may be strengthened by the execution of a greater number of good case studies. DA - 2006/04// PY - 2006 DO - 10.1177/1077800405284363 DP - DOI.org (Crossref) VL - 12 IS - 2 SP - 219 EP - 245 J2 - Qualitative Inquiry LA - en SN - 1077-8004, 1552-7565 UR - http://journals.sagepub.com/doi/10.1177/1077800405284363 Y2 - 2023/01/17/09:26:48 ER - TY - NEWS TI - Five tips for evaluating your impact in international development AU - Pasanen, Tiina T2 - The Guardian AB - How can impact evaluations actually be helpful? These top tips will save development professionals time, energy and money. International development can be messy – with uncertain, complex settings, and multiple partners with different interests, goals and capacities. At the same time, we are under increasing pressure to demonstrate impact. We have to show that our projects have made a real change in people’s lives and that donor or taxpayer money hasn’t been wasted. But impact evaluations, which are often seen as a solution to this, aren’t always used. And in some cases they are unhelpful. So how can we improve the quality of impact evaluations, so that they produce results that are useful? This is where evaluability assessments come in. By asking whether we should evaluate a project – and if so, when and how – these assessments can improve the quality of impact evaluations. Here are five other reasons to assess evaluability before starting an impact evaluation. DA - 2015/08/28/ PY - 2015 SE - Working in Development UR - https://www.theguardian.com/global-development-professionals-network/2015/aug/28/impact-evaluations-international-development-how-to Y2 - 2018/11/10/00:00:00 ER - TY - JOUR TI - Foresight and International Development* AU - Bingley, Kate T2 - IDS Bulletin AB - This article provides an overview of the use of foresight-type approaches and techniques in policy-related work in international development. It draws primarily on published and grey literatures, as well as select interviews with foresight practitioners. It begins with a brief introduction to the approaches and tools used in the field of strategic foresight, and then a broad mapping of the foresight landscape as relevant to international development. It provides reflections on the evidence of use and impact of foresight initiatives, and makes suggestions around future directions for foresight in international development. DA - 2016/// PY - 2016 DO - 10.19088/1968-2016.152 VL - 47 IS - 4 UR - http://bulletin.ids.ac.uk/idsbo/article/view/2777/ONLINE%20ARTICLE Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - Fostering Innovation and Entrepreneurialism in Public Sector Reform AU - Cummings, Clare T2 - Public Administration and Development AB - Summary There is growing recognition within the international development sector that there is a need for a new, more effective approach to engaging in public sector reform. This article builds on an emerging body of work that advocates more entrepreneurial and adaptive public sector reform programming. Drawing on knowledge and theory from public sector management, psychology and entrepreneurialism, this article aims to understand what motivates public sector workers to work entrepreneurially and suggest how these ideas can be applied to the way in which international development agencies engage in public sector reform work. This requires a shift in thinking from predesigned, large, externally led programmes promoting international best practice to interventions in which agencies adopt the role of a facilitator, helping to establish the enabling conditions for local partners to work entrepreneurially, developing their own solutions to the problems that they identify in their work. Copyright ? 2015 John Wiley & Sons, Ltd. DA - 2015/12/08/ PY - 2015 DO - 10.1002/pad.1735 VL - 35 IS - 4 SP - 315 EP - 328 J2 - Public Administration and Development SN - 0271-2075 UR - https://doi.org/10.1002/pad.1735 Y2 - 2018/10/12/00:00:00 KW - Development KW - Entrepreneurship KW - Innovation KW - Public sector reform KW - motivation ER - TY - JOUR TI - Fostering Learning in Large Programmes and Portfolios: Emerging Lessons from Climate Change and Sustainable Development AU - Harvey, Blane AU - Pasanen, Tiina AU - Pollard, Alison AU - Raybould, Julia T2 - Sustainability AB - In fields like climate and development, where the challenges being addressed can be described as “wicked”, learning is key to successful programming. Useful practical and theoretical work is being undertaken to better understand the role of reflexive learning in bringing together different knowledge to address complex problems like climate change. Through a review of practical cases and learning theories commonly used in the areas of resilience, climate change adaptation and environmental management, this article: (i) reviews the theories that have shaped approaches to reflexive learning in large, highly-distributed climate change and resilience-building programmes for development; and (ii) conducts a comparative learning review of key challenges and lessons emerging from early efforts to promote and integrate reflexive learning processes in programmes of this nature. Using a case study approach, the authors focus on early efforts made in four large, inter-related (or nested) programmes to establish, integrate and sustain learning processes and systems. Eight themes emerged from the review and are considered from the perspective of learning programmes as emergent communities of practice. By investigating how these themes play out in the nested programming, the paper contributes to the limited existing body of evidence on learning in large climate change programmes and identifies areas where future efforts might focus. DA - 2017/02/21/ PY - 2017 DO - 10.3390/su9020315 DP - Crossref VL - 9 IS - 2 SP - 315 LA - en SN - 2071-1050 ST - Fostering Learning in Large Programmes and Portfolios UR - http://www.mdpi.com/2071-1050/9/2/315 Y2 - 2019/03/12/14:17:01 ER - TY - RPRT TI - Foundations of the After Action Review Process AU - Morrison, John E AU - Meliza, Larry L T2 - Special Report AB - The U.S. Army has adopted the After Action Review AAR as its primary method of providing feedback after unit collective training exercises. The AAR is an interactive discussion in which unit members decide what happened, why it happened, and how to improve or sustain collective performance in future exercises. other services and organizations outside the military are also beginning to employ the AAR as a feedback tool. This report describes the twenty-five year history of AAR research and development and the major behavioral research areas contributing to AAR development and refinement. In addition, this report defines goals for future AAR research. CY - Alexandria, VA DA - 1999/07// PY - 1999 DP - apps.dtic.mil LA - en PB - Army Research Institute SN - 42 UR - https://apps.dtic.mil/sti/citations/ADA368651 Y2 - 2024/01/12/15:35:29 ER - TY - BLOG TI - From mouthset to mindset shifts in co-creating systems change AU - Blomkamp, Emma AU - Snow, Thea AU - Burkett, Ingrid T2 - Griffith Centre for Systems Innovation DA - 2023/08/08/T02:21:53.025Z PY - 2023 LA - en UR - https://medium.com/good-shift/from-mouthset-to-mindset-shifts-in-co-creating-systems-change-69caf9401f7b Y2 - 2023/11/07/11:01:50 ER - TY - RPRT TI - From political economy analysis to doing development differently: a learning experience AU - Booth, David AU - Harris, Daniel AU - Wild, Leni AB - A study of how development efforts can be strengthened by an awareness of political economy, reflecting on the experiences of PoGo policy researchers. CY - London DA - 2016/01// PY - 2016 PB - ODI ST - From political economy analysis to doing development differently UR - http://www.odi.org/publications/10235-political-economy-analysis-doing-development-differently Y2 - 2019/06/10/00:00:00 KW - IMPORTANT ER - TY - JOUR TI - From PRA to PLA and pluralism: practice and theory AU - Chambers, Robert AB - PRA (participatory rural appraisal) and the more inclusive PLA (participatory learning and action) are families of participatory methodologies which have evolved as behaviours and attitudes, methods, and practices of sharing. During the 1990s and 2000s PRA/PLA has spread and been applied in most countries in the world. Among the multifarious domains of application, some of the more common have been natural resource management and agriculture, programmes for equity, empowerment, rights and security, and community-level planning and action. Related participatory methodologies which have co-evolved and spread widely as movements include farmer participatory research, Integrated Pest Management, Reflect, Stepping Stones and Participatory Geographic Information Systems. Ideologically and epistemologically PRA/PLA seeks and embodies participatory ways to empower local and subordinate people, enabling them to express and enhance their knowledge and take action. It can be understood as having three main components: facilitators’ behaviours, attitudes and mindsets linked with precepts for action; methods which combine visuals, tangibles and groups; and sharing without boundaries. The interplay of these resonates with theories of chaos, complexity, emergence and deep simplicity, especially self-organising systems on the edge of chaos. Good practice has moved towards an eclectic pluralism in which branding, labels, ownership and ego give way to sharing, borrowing, improvisation, creativity and diversity, all these complemented by mutual and critical reflective learning and personal responsibility. Keywords: participatory methodologies; networks; pluralism; practice; theory. DA - 2007/// PY - 2007 DP - opendocs.ids.ac.uk LA - en ST - From PRA to PLA and pluralism UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/4079 Y2 - 2023/10/17/13:56:00 ER - TY - JOUR TI - Getting beneath the surface in program planning, monitoring and evaluation: Learning from use of participatory action research and theory of change in the CGIAR Research Program on Aquatic Agricultural Systems AU - Apgar, J Marina AU - Allen, Will AU - Albert, Joelle AU - Douthwaite, Boru AU - Paz Ybarnegaray, Rodrigo AU - Lunda, Jeston T2 - Action Research AB - Many rural poor and marginalized people strive to make a living in social-ecological systems that are characterized by multiple and often inequitable interactions across agents, scale and space. Uncertainty and inequality in such systems require research and development interventions to be adaptive, support learning and to engage with underlying drivers of poverty. Such complexity-aware approaches to planning, monitoring and evaluating development interventions are gaining strength, yet, there is still little empirical evidence of what it takes to implement them in practice. In this paper, we share learning from an agricultural research program that used participatory action research and theory of change to foster learning and support transformative change in aquatic agricultural systems. We reflect on our use of critical reflection within participatory agricultural research interventions, and our use of theory of change to collectively surface and revisit assumptions about how change happens. We share learning on the importance of being strengths-based in engaging stakeholders across scales and building a common goal as a starting point, and then staging a more critical practice as capacity is built and opportunities for digging deeper emerge. DA - 2017/03/01/ PY - 2017 DO - 10.1177/1476750316673879 DP - SAGE Journals VL - 15 IS - 1 SP - 15 EP - 34 J2 - Action Research LA - en SN - 1476-7503 ST - Getting beneath the surface in program planning, monitoring and evaluation UR - https://doi.org/10.1177/1476750316673879 Y2 - 2018/07/27/10:06:48 ER - TY - BLOG TI - Getting intentional about M&E: choosing suitable approaches for adaptive programmes AU - Management, Global Learning for Adaptive T2 - GLAM Blog AB - Does the choice of monitoring and evaluation (M&E) approaches and tools matter for adaptive programmes? In short, yes: monitoring, evaluation and learning (MEL) and adaptive management (AM) are intertwined. While programme monitoring data and evaluation results are not the only sources of evidence that programmes use for learning and iteration, they often are amongst most important ones — or at least they should be. Selecting what type of information to collect and analyse — and how — is critical for any type of programme. However, what AM especially focuses on is intentionally building in opportunities for structured and collective reflection, ongoing and real-time learning, course correction and decision-making in order to improve effectiveness. DA - 2020/02/07/T15:28:36.809Z PY - 2020 LA - en ST - Getting intentional about M&E UR - https://medium.com/glam-blog/getting-intentional-about-m-e-choosing-suitable-approaches-for-adaptive-programmes-f76c6b2790d9 Y2 - 2020/10/14/13:32:21 ER - TY - JOUR TI - Getting Practical With Causal Mechanisms: The application of Process-Tracing Under Real-World Evaluation Constraints AU - Raimondo, Estelle T2 - New Directions for Evaluation AB - Over the past decade, the field of development evaluation has seen a renewed interest in methodological approaches that can answer compelling causal questions about what works, for whom, and why. Development evaluators have notably started to experiment with Bayesian Process Tracing to unpack, test, and enhance their comprehension of causal mechanisms triggered by development interventions. This chapter conveys one such experience of applying Bayesian Process Tracing to the study of citizen engagement interventions within a conditional cash transfer program under real-world evaluation conditions. The chapter builds on this experience to discuss the benefits, challenges, and potential for the applicability of this approach under real-world evaluation conditions of time, money, and political constraints. DA - 2020/// PY - 2020 DO - 10.1002/ev.20430 DP - Wiley Online Library VL - 2020 IS - 167 SP - 45 EP - 58 LA - en SN - 1534-875X ST - Getting Practical With Causal Mechanisms UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20430 Y2 - 2021/02/18/14:26:31 ER - TY - RPRT TI - Getting Real about Unknowns in Complex Policy Work AU - Andrews, Matt AB - As with all public policy work, education policies are demanding. Policy workers need to ‘know’ a lot—about the problems they are addressing, the people who need to be engaged, the promises they can make in response, the context they are working in, and the processes they will follow to implement. Most policy workers answer questions about such issues within the structures of plan and control processes used to devise budgets and projects. These structures limit their knowledge gathering, organization and sense-making activities to up-front planning activities, and even though sophisticated tools like Theories of Change suggest planners ‘know’ all that is needed for policy success, they often do not. Policies are often fraught with ‘unknowns’ that cannot be captured in passive planning processes and thus repeatedly undermine even the best laid plans. Through a novel strategy that asks how much one knows about the answers to 25 essential policy questions, and an application to recent education policy interventions in Mozambique, this paper shows that it is possible to get real about unknowns in policy work. Just recognizing these unknowns exist—and understanding why they do and what kind of challenge they pose to policy workers—can help promote a more modest and realistic approach to doing complex policy work. CY - Oxford DA - 2021/11/24/ PY - 2021 DP - DOI.org (Crossref) LA - en M3 - RISE Working Paper PB - Research on Improving Systems of Education (RISE) SN - 21/083 UR - https://riseprogramme.org/publications/getting-real-about-unknowns-complex-policy-work Y2 - 2021/12/16/15:37:35 ER - TY - RPRT TI - GIZ’s evaluation system AU - GIZ Evaluation Unit AB - This provides a general description of GIZ’s new evaluation system, which is designed to implement our new policy and to achieve the objectives of evaluation reform. This general description is accompanied by two other documents that provide supplementary details of GIZ’s key evaluation instruments: project evaluations for BMZ business and corporate strategic evaluations. They are geared in particular to GIZ staff members who commission, implement or support evaluations and to evaluators who conduct evaluations on behalf of GIZ as well as repre-sentatives of commissioning parties, clients and cooperation partners. The current versions are very much a ‘work in progress’ in many regards. We are well aware that we will only be able to meet our own high expectations of our evaluations if we continue to develop their quality in dialogue with project staff, our commissioning parties, clients and partners and with evaluation practitioners and experts from the academic, scientific and research community. We look forward to taking on this task and to sharing our experience with all interested parties. CY - Bonn DA - 2018/// PY - 2018 PB - GIZ GmbH UR - https://www.giz.de/en/downloads/GIZ_EVAL_EN_general%20description.pdf Y2 - 2023/03/28/09:51:55 ER - TY - RPRT TI - Glossary of Key Terms in Evaluation and Results Based Management AU - OECD CY - Paris DA - 2009/// PY - 2009 PB - OECD UR - https://www.oecd.org/dac/evaluation/2754804.pdf Y2 - 2019/11/27/11:58:30 ER - TY - CHAP TI - Glossary of Knowledge Management AU - Serrat, Olivier T2 - Knowledge Solutions: Tools, Methods, and Approaches to Drive Organizational Performance A2 - Serrat, Olivier AB - The knowledge management discipline can be cryptic. These Knowledge Solutions define its most common concepts in simple terms. CY - Singapore DA - 2017/// PY - 2017 DP - Springer Link SP - 1055 EP - 1061 LA - en PB - Springer Singapore SN - 978-981-10-0983-9 UR - https://doi.org/10.1007/978-981-10-0983-9_120 Y2 - 2019/06/17/13:22:12 KW - Definitions KW - Knowledge management KW - Taxonomy KW - Terminology ER - TY - RPRT TI - Governance Diaries: An Approach to Governance Research from the Ground Up AU - Loureiro, Miguel AU - Joshi, Anuradha AU - Barnes, Katrina AU - Chaimite, Egídio AB - Research on empowerment and accountability tends to focus on collective action and its potential for empowering citizens undertaking the action and on achieving state accountability. In fragile, conflict and violence-affected settings (FCVAS) collective action is rare and risky. So how do citizens, particularly the chronically poor and most marginalised, interact and make claims on the different public authorities that exist in these settings, and how do these interactions contribute to citizens’ sense of empowerment and accountability? Given the current agenda of ‘leave no one behind’, an understanding of how such populations interact with public authorities to meet their governance needs can help identify the constraints to achieving development for all in these challenging settings. We developed ‘governance diaries,’ a cross between a panel survey and multi-sited ethnographies, as an iterative approach to capture their experiences around governance issues over time. We explain here how this approach works, and the challenges and opportunities it offers for research. CY - Brighton DA - 2020/02/14/ PY - 2020 DP - opendocs.ids.ac.uk LA - en PB - IDS ST - Governance Diaries UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/15119 Y2 - 2020/10/01/11:01:01 ER - TY - GEN TI - Guidance on tasks and deliverables for different evaluation phases AU - Methods Lab AB - This tool describes the five key phases of evaluation, from planning and design, to implementation and communication of results. It provides a list of the main tasks and deliverables for each phase, intended for use by anyone managing an impact evaluation. This tool was developed by Irene Guijt, Simon Hearn, Tiina Pasanen and Patricia Rogers for use in Methods Lab projects. It follows to some extent the BetterEvaluation Rainbow Framework. DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10646.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - Guidance on using the revised Logical Framework - How to note AU - DFID AB - The principal changes to the logframe from the earlier (2008) 4 x 4 matrix are: - Objectively Verifiable Indicators (OVIs) have been separated into their component elements (Indicator, Baseline and Target), and Milestones added. - Means of Verification has been renamed ‘Source’. - Inputs are now quantified in terms of funds (expressed in Sterling for DFID and all partners) and use of DFID staff time (expressed as annual Full-Time Equivalents (FTEs). - A DFID Share box now indicates the financial value of DFID’s Inputs as a percentage of the whole. - Assumptions are shown at Purpose and Output level only. - Risks are shown at Activities level only, but also rated at Output level; - At the Output level, the Impact Weighting is now shown in the logframe together with a Risk Rating for individual Outputs. - Activities are now shown separately (so do not normally appear in the logframe sent for approval), although they can be added to the logframe if this is more suitable for your purposes. - A renewed emphasis on the use of disaggregated beneficiary data within indicators, baselines and targets. The Logical Framework (logframe) was significantly re-designed in February 2009. In January 2011 a slightly amended logframe template was introduced at the same time as the launch of the new DFID Business Case. Given the extent of changes that took place in February 2009 it was only necessary to make minor amendments in January 2011. These amendments are as follows: - Results Chain terminology aligned across DFID (the terms Goal and Purpose in the old logframe template have been replaced by Impact and Outcome) - Rows added to allow achieved results to be captured alongside the planned results determined at project design stage - Word version of the logframe template removed – excel is the preferred format for logical frameworks from January 2011 - Indicator numbering introduced within logframe template CY - London DA - 2011/01// PY - 2011 M3 - Practice Paper PB - DFID UR - https://assets.publishing.service.gov.uk/media/5a7c9e2d40f0b6629523ab0f/using-revised-logical-framework-external.pdf Y2 - 2024/01/30/10:49:54 ER - TY - RPRT TI - Guide for Adopting Remote Monitoring Approaches During COVID-19 AU - USAID AB - USAID maintains staff and operations in more than 80 countries around the world, all of which the COVID-19 pandemic will disrupt. USAID remains committed to protecting the health and safety of our staff, while continuing appropriate oversight of our programs and ensuring the accountable and effective use of U.S. taxpayer funds. In the current operating environment, USAID and implementing partners face new challenges in implementing activities, monitoring progress, collecting data, and tracking indicators. As we adapt our approaches, we will work with implementing partners to find innovative, responsible, and safe ways to monitor and evaluate programming. Digital tools can support novel approaches to remote monitoring. Responsible use of digital tools also supports operating unit alignment with the Digital Data Collection mandate in the Agency’s new Digital Strategy. This guide provides information for Agency staff and implementing partners on remote monitoring techniques and when they can be employed. We encourage use of this guide to identify and pursue appropriate remote-monitoring approaches for your needs. CORs/AORs and Activity Managers should work with implementing partners to document updated approaches in each agreement’s plans for monitoring, evaluation, and learning (MEL), and should upload these amended Activity MEL Plans into the Agency Secure Image and Storage Tracking System (ASIST) as soon as possible. This can be done in collaboration with Mission and Washington Bureau M&E Specialists, as appropriate DA - 2020/// PY - 2020 DP - Zotero SP - 9 LA - en PB - USAID ER - TY - JOUR TI - Guidelines for Drawing Causal Loop Diagrams AU - Daniel H., Kim T2 - The Systems Thinker AB - The old adage “if the only tool you have is a hammer, everything begins to look like a nail” can also apply to language. If our language is linear and static, we will tend to view and interact with our world as if it were linear and static. Taking a complex, dynamic, and circular world and linearizing it into a set of snapshots may make things seem simpler, but we may totally misread the very reality we were seeking to understand. Making such inappropriate simplifications “is like putting on your brakes and then looking at your speedometer to see how fast you were going” says Bill Isaacs of the MIT Center for Organizational Learning. Articulating Reality Causal loop diagrams provide a language for articulating our understanding of the dynamic, interconnected nature of our world. We can think of them as sentences which are constructed by linking together key variables and indicating the causal relationships between them. By stringing together several loops, we can create a coherent story about a particular problem or issue. DA - 1992/// PY - 1992 VL - 3 IS - 1 SP - 5 EP - 6 UR - http://www.cs.toronto.edu/~sme/SystemsThinking/GuidelinesforDrawingCausalLoopDiagrams.pdf Y2 - 2018/03/09/00:00:00 ER - TY - GEN TI - Guiding questions to help narrow the scope of an evaluation AU - Methods Lab AB - Time and budget constraints can mean that programmes are not able to assess all possible evaluation questions; this is especially true for multi-component or multi-site programmes operating in challenging environments. This tool identifies areas of enquiry to help programmes prioritise the number of questions and measurement indicators used. This tool was developed by Anne Buffardi for use in in Methods Lab projects. DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10038.pdf Y2 - 2018/11/10/00:00:00 ER - TY - JOUR TI - Hacking by the prompt: Innovative ways to utilize ChatGPT for evaluators AU - Ferretti, Silva T2 - New Directions for Evaluation AB - “Hacking by the prompt”—writing simple yet creative conversational instructions in ChatGPT's message window—revealed many valuable additions to the evaluator's toolbox for all stages of the evaluation process. This includes the production of terms of reference and proposals for the dissemination of final reports. ChatGPT does not come with an instruction book, so evaluators must experiment creatively to understand its potential. The surprising performance of ChatGPT leads to the question: will it eventually substitute for evaluators? By describing ChatGPT through four personality characteristics (pedantic, “I know it all,” meek, and “speech virtuoso”), this article provides case examples of the potential and pitfall of ChatGPT in transforming evaluation practice. Anthropomorphizing ChatGPT is debatable, but the result is clear: tongue-in-cheek personality characteristics helped hack ChatGPT more creatively while remaining aware of its challenges. This article combines practical ideas with deeper reflection on evaluation. It concludes that ChatGPT can substitute for evaluators when evaluations mostly focus on paperwork and conventional approaches “by the book” (an unfortunate trend in the sector). ChatGPT cannot substitute engagement with reality and critical thinking. Will ChatGPT then be a stimulus to rediscover the humanity and the reality we lost in processes? DA - 2023/// PY - 2023 DO - 10.1002/ev.20557 DP - Wiley Online Library VL - 2023 IS - 178-179 SP - 73 EP - 84 LA - en SN - 1534-875X ST - Hacking by the prompt UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20557 Y2 - 2023/12/11/09:47:47 ER - TY - BLOG TI - HERMES (Highly Extensible Resource for Modeling Event-Driven Supply Chains) AU - HERMES T2 - hermes.psc.edu AB - Systems modeling is a tool for policymakers and program managers to capture all the direct and indirect effects of changes to a supply chain, identify sustainable solutions to the root causes of issues and save time, effort and resources in costly trial and error. Without systems modeling, evaluating the supply chain, identifying gaps, and implementing solutions can be insufficient, unsustainable and costly. Stock2 Vaccine supply chains are complex systems, comprising all the equipment, personnel, policies and processes needed to deliver a vaccine from its point of origin to the population. Understanding how the various components of a vaccine supply chain interact with each other is critical to evaluating supply chain function, identifying the root causes of issues and formulating sustainable solutions. HERMES (Highly Extensible Resource for Modeling Event-Driven Supply Chains) is a software program that allows users to generate a detailed computer simulation model of a supply chain DA - 2015/// PY - 2015 UR - http://hermes.psc.edu/ Y2 - 2018/03/09/00:00:00 ER - TY - RPRT TI - HIV, food and drugs: Livelihoods, nutrition and Anti-retroviral Therapy (ART) in Kenya and Zambia AU - Samuels, Fiona AU - Rutenberg, Naomi T2 - Briefing Paper AB - Key points • ART has restored the health of many people living with HIV, but their livelihoods lag behind • Good nutrition is important for people on ART. Food supplementation can help, but is no substitute for sustainable livelihoods • The livelihoods of people on ART can be bolstered through skills, livelihood networks, assets and cash or food transfers DA - 2008/// PY - 2008 SP - 4 PB - ODI SN - 45 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/3355.pdf Y2 - 2018/10/19/00:00:00 ER - TY - BLOG TI - Horizontal Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Horizontal evaluation is an approach that combines self-assessment by local participants and external review by peers. Originally developed to evaluate new methodologies for agricultural research and development, horizontal evaluation has wider potential for application. In its original setting, the focus of horizontal evaluation is the actual R&D methodology itself rather than the project per se or the team or organisation that developed it UR - https://www.betterevaluation.org/en/plan/approach/horizontal_evaluation Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Horizontal Evaluation: Stimulating social learning among peers AU - Thiele, G AU - velasco, C AU - Manrique, K T2 - ILAC Brief AB - Horizontal evaluation is a flexible evaluation method that combines self-assessment and external review by peers. We have developed and applied this method for use within an Andean regional network that develops new methodologies for research and development (R&D). The involvement of peers neutralizes the lopsided power relations that prevail in traditional external evaluations, creating a more favourable atmosphere for learning and improvement. The central element of a horizontal evaluation is a workshop that brings together a group of ‘local participants’ who are developing a new R&D methodology and a group of ‘visitors’ or ‘peers’ who are also interested in the methodology. The workshop combines presentations about the methodology with field visits, small group work and plenary discussions. It elicits and compares the perceptions of the two groups concerning the strengths and weaknesses of the methodology; it provides practical suggestions for improvement, which may often be put to use immediately; it promotes social learning among the different groups involved; and it stimulates further experimentation with and development of the methodology in other settings. DA - 2006/// PY - 2006 SP - 4 SN - 13 UR - http://hdl.handle.net/10568/70133 ER - TY - BLOG TI - How adaptive M&E from the peace sector can help demonstrate the value of aid AU - Kratzer, Sebastian AB - Over the last decade, the peace sector has been developing and adapting Monitoring and Evaluation (M&E) systems and tools to fit their contexts and ways of working. This evolution may hold some insights for the aid community in how to go beyond more traditional, backwards-looking M&E to navigate today’s volatile, interest-based world of politics and aid. DA - 2023/10/13/ PY - 2023 UR - https://frompoverty.oxfam.org.uk/how-adaptive-me-from-the-peace-sector-can-help-demonstrate-the-value-of-aid/ Y2 - 2023/08/15/07:57:02 ER - TY - BLOG TI - How Adaptive Management is challenging the monitoring and evaluation of complex programmes AU - Simister, Nigel T2 - INTRAC AB - By Nigel Simister Adaptive management is a broad approach designed to support development or humanitarian programmes in complex or uncertain … DA - 2018/05/18/T13:10:39+01:00 PY - 2018 LA - en UR - https://www.intrac.org/adaptive-management-challenging-monitoring-evaluation-complex-programmes/ Y2 - 2018/08/10/09:52:41 ER - TY - MGZN TI - How AI Fits into Lean Six Sigma AU - Holweg, Matthias AU - Davenport, Thomas H. AU - Snyder, Ken T2 - Harvard Business Review AB - AI already is being used in some areas of process improvement, and the usage of this technology — including generative AI — promises to grow. That’s because it can perform tasks faster and much less expensively than humans alone. But it will never fully replace people — and that poses management challenges. DA - 2023/11/09/T13:35:41Z PY - 2023 DP - hbr.org SN - 0017-8012 UR - https://hbr.org/2023/11/how-ai-fits-into-lean-six-sigma Y2 - 2023/11/14/22:24:48 ER - TY - JOUR TI - How are Research for Development Programmes Implementing and Evaluating Equitable Partnerships to Address Power Asymmetries? AU - Snijder, Mieke AU - Steege, Rosie AU - Callander, Michelle AU - Wahome, Michel AU - Rahman, M. Feisal AU - Apgar, Marina AU - Theobald, Sally AU - Bracken, Louise J. AU - Dean, Laura AU - Mansaray, Bintu AU - Saligram, Prasanna AU - Garimella, Surekha AU - Arthurs-Hartnett, Sophia AU - Karuga, Robinson AU - Mejía Artieda, Adriana Elizabeth AU - Chengo, Victoria AU - Ateles, Joanes T2 - The European Journal of Development Research AB - The complexity of issues addressed by research for development (R4D) requires collaborations between partners from a range of disciplines and cultural contexts. Power asymmetries within such partnerships may obstruct the fair distribution of resources, responsibilities and benefits across all partners. This paper presents a cross-case analysis of five R4D partnership evaluations, their methods and how they unearthed and addressed power asymmetries. It contributes to the field of R4D partnership evaluations by detailing approaches and methods employed to evaluate these partnerships. Theory-based evaluations deepened understandings of how equitable partnerships contribute to R4D generating impact and centring the relational side of R4D. Participatory approaches that involved all partners in developing and evaluating partnership principles ensured contextually appropriate definitions and a focus on what partners value. DA - 2023/04/01/ PY - 2023 DO - 10.1057/s41287-023-00578-w DP - Springer Link VL - 35 IS - 2 SP - 351 EP - 379 J2 - Eur J Dev Res LA - en SN - 1743-9728 UR - https://doi.org/10.1057/s41287-023-00578-w Y2 - 2023/04/13/11:06:18 ER - TY - RPRT TI - How Decent is Decent Work? Using SenseMaker to understand workers’ experiences AU - Mager, Franziska AU - Smith, Becca AU - Gujit, Irene DA - 2018/05/31/ PY - 2018 DP - DOI.org (Crossref) LA - en PB - Oxfam ST - How Decent is Decent Work? UR - http://hdl.handle.net/10546/620476 Y2 - 2020/02/06/09:09:00 ER - TY - RPRT TI - How do we know we are doing good work? AU - Sharp, Cathy AU - McLaughlin, Dot AU - Whitley, Janet AU - Lawson, Karen AB - Collective leadership feels timely and important in an uncertain, fast changing, and challenging world. This report comes at this heightened moment of urgency and appetite for renewal, bringing potential to do things differently in public services and communities. The opportunities and challenges of true collaboration-in-practice, in the absence of blueprints, mean that it has never been more necessary to change ways of working and foreground learning. As outcomes remain important, and time and resources are scarcer than ever, the approach of collective leadership makes the creation of impact a shared, conscious, and actionable choice. A pathway to ultimate impact at scale is co-created through dialogue about expectations and contributions, and by design, not assuming change will happen because we have good intentions. Collective Leadership for Scotland (CLfS) has a strong vision and enjoys continuing active interest, drawing together participants from a variety of agencies working in public services. People are looking for fresh thinking, space and time for reflection, connection with others, a chance to think about how to tackle difficulties, and to test out what it takes to do, and continue to do, the work of collaborative public service. These motivations are deepened and brought into sharper focus by the pandemic, with an added interest in developing skills in online facilitation. CLfS contributes to building a critical mass for system change, to help to sustain the ambitions of the Christie Commission and the delivery of the National Outcomes for Scotland. There remains further potential to realise wider and deeper impact amongst organisations, communities, and wider systems. The conclusions of this report are likely to have wider resonance beyond interests in the CLfS programmes. This report deepens understanding of some of the challenges of commissioning, convening, and the scope for deeper impact through building reflective and relational leadership practices. It also outlines social and experiential sensemaking and facilitation practices to strengthen the action inquiry approach as a deliberate learning strategy, building cultures that support new forms of collaborative inquiry and systemic action research. CY - Edinburgh DA - 2022/09// PY - 2022 DP - Zotero LA - en PB - Collective Leadership for Scotland UR - https://workforcescotland.wpcomstaging.com/wp-content/uploads/2022/11/Collective-Leadership-for-Scotland_Impact-Report_18-November-2022.pdf Y2 - 2023/05/22/00:00:00 ER - TY - SOUND TI - How do we know what works? Interview with Prof. Michael Woolcock, Lead Social Scientist at the World Bank A2 - Evans, Alice T3 - Rocking our priors AB - Dr Alice Evans and leading experts discuss growth, governance, & gender inequalities. Alice is a Lecturer at King's College London, and Faculty Associate at the Harvard Kennedy School. DA - 2018/03/17/ PY - 2018 LA - en UR - https://soundcloud.com/user-845572280/how-do-we-know-what-works-dr-michael-woolcock Y2 - 2022/01/28/13:00:43 ER - TY - RPRT TI - How Does Participatory Action Research Generate Innovation? Findings from a Rapid Realist Review AU - Snijder, Mieke AU - Apgar, J. Marina T2 - CLARISSA Emerging Evidence Report AB - This Emerging Evidence Report shares evidence of how, for whom, and under what circumstances, Participatory Action Research (PAR) leads to innovative actions. A rapid realist review was undertaken to develop programme theories that explain how PAR generates innovation. The methodology included peer-reviewed and grey literature and moments of engagement with programme staff, such that their input supported the development and refinement of three resulting initial programme theories (IPTs) that we present in this report. Across all three IPTs, safe relational space, group facilitation, and the abilities of facilitators, are essential context and intervention components through which PAR can generate innovation. Implications from the three IPTs for evaluation design of the CLARISSA programme are identified and discussed. The report finishes with opportunities for the CLARISSA programme to start building an evidence base of how PAR works as an intervention modality, such as evidencing group-level conscientisation, the influence of intersecting inequalities, and influence of diverse perspectives coming together in a PAR process. CY - Brighton DA - 2021/07/23/ PY - 2021 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 6 ST - How Does Participatory Action Research Generate Innovation? UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/16754 Y2 - 2024/02/22/11:16:40 ER - TY - BLOG TI - How should donors manage adaptively? Market Systems Development as a case study. AU - Itad T2 - Itad AB - Itad has recently completed a strategic evaluation for Sida to help them work through how they can best manage programmes that are adaptive and apply systems approaches - the conclusions from the evaluation about what funders need to do to manage adaptively are of broader relevance. DA - 2018/12/20/T10:52:27+00:00 PY - 2018 LA - en-US ST - How should donors manage adaptively? UR - https://www.itad.com/how-should-donors-manage-adaptively-market-systems-development-as-a-case-study/ Y2 - 2019/01/08/12:25:21 ER - TY - MGZN TI - How to Build Movements with Cyclical Patterns in Mind - Non Profit News | Nonprofit Quarterly AU - Algoso, Dave AU - Guerzovich, Florencia AU - Gattoni, Soledad T2 - Nonprofit Quarterly AB - The world changes too much for anyone who is invested in social change work to imagine that this work is linear and predictable. Opportunities come and go, whether caused by a pandemic or political shifts. This much most social movement leaders and activists intuitively understand. But what can be done with this realization? How might movement groups better prepare for moments of opportunity? We want to explore how we can create the changes we want to see by responding to the changes that are outside our control. DA - 2023/06/05/ PY - 2023 UR - https://nonprofitquarterly.org/how-to-build-movements-with-cyclical-patterns-in-mind/?utm_source=substack&utm_medium=email Y2 - 2023/10/03/11:30:33 ER - TY - RPRT TI - How to design a monitoring and evaluation framework for a policy research project AU - Pasanen, Tiina AU - Shaxson, Louise AB - Policy research projects face a number of challenges: policy processes are complex, involve multiple actors and often feature a significant time-lag between research and what may or may not happen as a result of it. To complicate matters further, the scope and scale of policy research projects are increasingly moving away from single research studies towards multi-component, multi-site and multi-sector endeavours. These factors mean that developing an overarching monitoring and evaluation framework can be challenging. But it is not impossible. The guidance note aims to support the first steps in designing and structuring the M&E framework (that is, what aspects or areas of policy research projects to monitor and evaluate, why, when and how). It builds on an M&E framework for policy research projects developed and tested by the Research and Policy in Development (RAPID) programme of the Overseas Development Institute, and draws on case studies of how it has been used by RAPID and other research teams. DA - 2016/02// PY - 2016 SP - 40 M3 - Working and Discussion Papers PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10259.pdf Y2 - 2018/11/10/00:00:00 ER - TY - BLOG TI - How to develop capacity in the international development sector AU - Prieto Martín, Pedro T2 - CLARISSA AB - In her acclaimed study of effective cross-organizational teamwork, Harvard Business School Professor Amy C. Edmonson concluded that “trying things that... DA - 2022/08/12/T08:50:14+00:00 PY - 2022 LA - en-US UR - https://clarissa.global/how-to-develop-capacity-in-the-international-development-sector/ Y2 - 2024/01/29/00:00:00 ER - TY - RPRT TI - How to do Process Tracing: A Method for Testing “How Change Happened” in Complex and Dynamic Settings AU - Lynn, Jewlya AU - Stachowiak, Sarah AU - Beyers, Jennifer AB - Process tracing is a causal methodology that can help people understand how a particular large-scale change actually happened within a complex dynamic environment. Much of the existing literature provides important information about the method; we wrote this brief to help more people operationalize the concepts and learn about practical steps for using this method more easily, with quality, and toward a more equitable world. This piece was written based on our experiences implementing process tracing when our experience showed that existing materials on the method had a lot more conceptual than practical information. We’ve approached this as people with some successful (and some unsuccessful) experience with the method itself, alongside deep experience in evaluating initiatives and strategies in complex and dynamic settings. We focus not on the Bayesian side of process tracing but rather on how this can be implemented in a way that’s more participatory and lifts up the experiences and wisdom of those closest to the work and the problems being tackled. We hope this contributes to and helps make more approachable the important work of political scientists and methodologists upon which this work sits. CY - Seattle DA - 2022/10// PY - 2022 PB - ORS Impact UR - https://www.orsimpact.com/directory/how-to-do-process-tracing.htm Y2 - 2024/02/19/09:56:32 ER - TY - VIDEO TI - How to make a multilayer pie chart in Excel AU - Karina Adcock AB - Create a multilevel donut chart in excel -------------------------------------------------------------------------------- #exceltips #exceltutorials #excelcharts SUBSCRIBE: https://goo.gl/c46YPs Microsoft Office 365, Beta Channel, Version 2104 --------------------------------------------------------------------------------- IF YOU LIKED THIS VIDEO YOU MAY ALSO LIKE: - Recreating a pie chart from a newspaper in PowerPoint    • How to make a pie... DA - 2021/04/14/ PY - 2021 DP - YouTube UR - https://www.youtube.com/watch?v=jaAGX3MiU6c Y2 - 2023/04/27/13:59:13 ER - TY - BLOG TI - How to make theories of change more useful? AU - Aston, Thomas T2 - Medium AB - A fair amount has been written recently questioning the value added of theories of change. Have we gone through a hype cycle? Are they… DA - 2021/02/05/T22:42:20.434Z PY - 2021 LA - en UR - https://thomasmtaston.medium.com/how-to-make-theories-of-change-more-useful-fc969076a44d Y2 - 2021/02/18/14:44:33 ER - TY - BLOG TI - How to Monitor and Evaluate an Adaptive Programme: 7 Takeaways AU - Sikustahili, Gloria AU - Adkins, Julie AU - Makongo, Japhet AU - Milligan, Simon T2 - From Poverty to Power AB - Monitoring and Evaluation needs to be different to support the new generation of 'adaptive programmes' in aid. 4 M&E specialists in Tanzania explain how. DA - 2020/09/18/T06:30:00+00:00 PY - 2020 LA - en-GB ST - How to Monitor and Evaluate an Adaptive Programme UR - https://oxfamblogs.org/fp2p/how-to-monitor-and-evaluate-an-adaptive-programme-7-takeaways/ Y2 - 2020/10/15/11:51:38 ER - TY - RPRT TI - How-To Note: Developing a Project Logic Model (and its Associated Theory of Change) AU - USAID T2 - Program Cycle AB - This How-To Note describes considerations for developing a project logic model, as well as steps for thinking through a more complete theory of change (TOC). A logic model is a graphic or visual depiction that summarizes key elements of a TOC, and it is often used as a facilitation tool during the design process. There are many types of logic models, including but not limited to logical frameworks (logframes), results chains, results frameworks, and local actor-oriented models, among others. The project logic model and its associated TOC are included in the Project Appraisal Document (PAD) that approves a project design (see ADS 201.3.3.13). While this How-To Note focuses on logic models at the project level, logic models are also used at the strategy level (specifically, results frameworks – see Box 1), and often at the activity level. The concepts and steps presented here are generally applicable to the process of developing logic models and TOCs throughout the Program Cycle. DA - 2017/07// PY - 2017 SP - 24 M3 - How To Note SN - Version 2 UR - https://usaidlearninglab.org/system/files/resource/files/project_logic_model_how_to_note_final_sep1.pdf Y2 - 2024/01/30/00:00:00 ER - TY - RPRT TI - How useful are RCTs in evaluating transparency and accountability projects? AU - Leavy, Jennifer T2 - MAVC Working Paper CY - Brighton DA - 2014/09// PY - 2014 PB - IDS UR - http://www.makingallvoicescount.org/publication/how-useful-are-rcts-in-evaluating-transparency-accountability/ Y2 - 2016/04/22/10:10:35 ER - TY - JOUR TI - How We Model Matters: A Manifesto for the Next Generation of Program Theorizing AU - Whynot, Jane AU - Lemire, Sebastian AU - Montague, Steve T2 - Canadian Journal of Program Evaluation AB - In this concluding article, grounded on the exemplary contributions contained in the preceding pages, the guest editors scale the proverbial soapbox and present a manifesto to guide the pursuit and advancement of the next generation of program theorizing. Formulating ten declarations for program theory development and examination, the modest hope of the authors is to motivate and inspire reflective evaluation practitioners to broaden their views, approaches, and techniques for future program theorizing. DA - 2019/03/01/ PY - 2019 DO - 10.3138/cjpe.53070 DP - DOI.org (Crossref) VL - 33 IS - 3 J2 - CJPE LA - en SN - 1496-7308, 0834-1516 ST - How We Model Matters UR - https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/53070 Y2 - 2019/08/12/21:34:54 ER - TY - JOUR TI - Illogical Framework: The Importance of Monitoring and Evaluation in International Development Studies AU - Pomerantz, Jessica R. T2 - Cornell Policy Review AB - In my brief experience with monitoring and evaluation, I have become convinced that it is critically important both as an international development project component and as a field of academic study. Throughout my academic career at Cornell University, I have, at times, argued that monitoring and evaluation has actually impeded development efforts, but here I wish to amend my opinion. Bad monitoring and evaluation can sabotage development projects and our meaningful interpretation of development impacts; failures can appear to be successes and vice-versa. As a student and practitioner of monitoring and evaluation, I have drawn the conclusions listed below and I submit them for your consideration. • Monitoring and evaluation is a key element of the international development industry applicable to many areas of public administration, domestic and international. • International development failures could be discovered and averted or corrected given proper monitoring and evaluation activities. • Anecdotal evidence from development activities in Afghanistan provides one example of the international community’s lack of attention to monitoring and evaluation concerning an ongoing development catastrophe. • Higher education ought to be filling the monitoring and evaluation knowledge gap but to date is failing to do so. DA - 2011/// PY - 2011 DP - Zotero VL - 01 IS - 01 LA - en UR - http://www.cornellpolicyreview.com/illogical-framework-the-importance-of-monitoring-and-evaluation-in-international-development-studies/?pdf=73 Y2 - 2023/01/12/00:00:00 ER - TY - RPRT TI - Impact assessment and the quest for the Holy Grail AU - Bymolt, Roger T2 - KIT Working papers CY - Amsterdam DA - 2015/// PY - 2015 PB - KIT Royal Tropical Institute SN - 2015-3 UR - https://www.kit.nl/wp-content/uploads/2018/08/Impact-assessment-and-the-quest-for-the-holy-Grail.pdf Y2 - 2019/03/12/15:04:24 ER - TY - RPRT TI - Impact evaluation for portfolio programmes on policy influence: Reflections on the Indonesian Poverty Reduction Support Facility AU - Mackenzie, Jessica AU - Hearn, Simon AB - • Donors are increasingly using portfolio-based programmes that embrace ‘good failure’ and adaptive, political programming. • However, measuring the impact of these programmes is challenging, especially for those working on policy influence and building country systems; not only do you need to measure the positive and negative impact of the overall portfolio, but also the different pathways tested. • Programmes, therefore, need a light-touch monitoring and evaluation system that allows it to remain flexible. • Good practice examples of portfolio-based programmes present six strategies to evaluate impact: 1. Develop appropriate logic models 2. Collect observational data throughout implementation 3. Develop stories of change or case studies 4. Understand causal relationships without a counterfactual 5. Purposefully select which activities to study 6. Be explicit about how impact will be valued across the portfolio. • These strategies are only useful if monitoring and evaluation is placed at the centre of programme decision-making. DA - 2016/04// PY - 2016 SP - 48 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10463.pdf Y2 - 2018/11/10/00:00:00 ER - TY - BOOK TI - Impact Evaluation of Development Interventions: A Practical Guide AU - White, Howard AU - Raitzer, David AB - This book offers guidance on the principles, methods, and practice of impact evaluation. It contains material for a range of audiences, from those who may use or manage impact evaluations to applied researchers. DA - 2017/12/26/ PY - 2017 DP - www.adb.org LA - en PB - Asian Development Bank SN - 978-92-9261-058-6 978-92-9261-059-3 ST - Impact Evaluation of Development Interventions UR - https://www.adb.org/publications/impact-evaluation-development-interventions-practical-guide Y2 - 2022/01/28/12:12:16 ER - TY - RPRT TI - Impact Evaluation: Taking stock and looking ahead - Conference report AU - Visser, Irene AU - Guijt, Irene AU - Kusters, Cecile AB - This report summarises the presentations and discussions of the Conference ‘Impact evaluation. Taking stock and looking ahead’, which took place in Wageningen on March 25 and 26, 2013. The Conference was organised and funded by the Centre for CY - Wageningen DA - 2013/01/01/ PY - 2013 DP - www.academia.edu LA - en PB - Centre for Development Innovation ST - Impact Evaluation UR - https://www.academia.edu/111110248/Impact_Evaluation_Taking_stock_and_looking_ahead Y2 - 2023/12/11/09:28:43 ER - TY - RPRT TI - Impact Evidence and Beyond: Using Evidence to Drive Adoption of Humanitarian Innovations AU - Dodgson, Kate AU - Crowley, Catie T2 - Scaling series AB - This learning paper provides guidance to humanitarian innovators on how to use evidence to enable and drive adoption of innovation. Innovation literature and practice show time and time again that it is difficult to scale innovations. Even when an innovation is demonstrably impactful, better than the existing solution and good value for money, it does not automatically get adopted or used in mainstream humanitarian programming. Why do evidence-based innovations face difficulties in scaling and how can innovators best position their innovation to scale? This learning paper is for innovators who want to effectively use evidence to support and enable their journey to scale. It explores the underlying social, organisational and behavioural factors that stifle uptake of innovations. It also provides guidance on how to use, prioritise and communicate evidence to overcome these barriers. The paper aims to help innovators generate and present their evidence in more tailored and nuanced ways to improve adoption and scaling of their innovations. CY - London DA - 2021/05/07/ PY - 2021 DP - Zotero LA - en PB - Elrha UR - https://www.elrha.org/researchdatabase/impact-evidence-and-beyond-using-evidence-to-drive-adoption-of-humanitarian-innovations-scaling-series/ Y2 - 2021/10/28/00:00:00 ER - TY - BLOG TI - Impact networks: measuring towards success AU - Small Foundation T2 - Small Foundation AB - How does Small Foundation think about the impact… DA - 2021/06/29/T08:18:53+00:00 PY - 2021 ST - Impact networks UR - https://smallfoundation.ie/impact-networks-measuring-towards-success/ Y2 - 2022/06/17/11:36:45 ER - TY - BLOG TI - In praise of... Logframes AU - Teskey, Graham T2 - From Poverty to Power AB - In all the jobs I have held, the only training that has ever stayed with me was a three-day course on logframes, held in a very pleasant beach hotel on Fiji’s beautiful coral coast. This was a few months after I joined what was then the Overseas Development Administration, DFID’s forerunner. Three days on logframes. Yes really. Our Pacific team were gathered together to learn this new skill. The course was designed not only to help us think rigorously about how change happens, but also to ensure that we had a shared understanding of what constitutes good design. Twenty-five years later I am aware that logframes are out of fashion: references to the ‘tyranny of the logframe’ abound, its vertical determinism, and its lack of flexibility. Donors seem to have replaced logframes with things called ‘Program Logics’, as in the schematic below. You don’t have to be able to read it to get the picture… (and it’s logframes that are supposed to be too vertical…?). DA - 2021/08/05/ PY - 2021 UR - https://oxfamapps.org/fp2p/in-praise-of-logframes/ Y2 - 2021/08/05/21:17:01 ER - TY - BLOG TI - In pursuit of the deep and sustained shifts necessary for adaptive management AU - Simister, Nigel T2 - INTRAC AB - The final blog in Nigel Simister's series on adaptive management and the M&E of complex projects and programmes. DA - 2018/06/01/T14:22:47+01:00 PY - 2018 LA - en UR - https://www.intrac.org/pursuit-deep-sustained-shifts-necessary-adaptive-management/ Y2 - 2018/07/17/10:42:36 ER - TY - BOOK TI - Inclusive participatory approaches: A facilitator’s guide AU - Zaremba, Haley AU - Elias, Marlène AU - Devi, J. Tulasi AU - Priyadarshini, Pratiti AB - In community engagement and participatory processes, facilitators must make intentional efforts and adopt inclusive strategies to include marginalized and frequently overlooked groups. Yet, there is a lack of guidance on how to inclusively facilitate participatory processes. Facilitators are therefore often poorly prepared to engage with the power relations that underlie these processes, including those between the facilitator and participants and among participants themselves. This guide addresses this shortcoming by presenting strategies that have been shown to enhance the meaningful participation of women and marginalized groups in participatory processes. The aim is to equip facilitators with tools to create inclusive participatory spaces and community engagement. Although it refers to processes related to natural resource management, the guide is also applicable to the facilitation of participatory processes focusing on other issues or fields of significance to communities. DA - 2021/12// PY - 2021 DP - cgspace.cgiar.org LA - en SN - 978-92-9255-234-3 ST - Inclusive participatory approaches UR - https://cgspace.cgiar.org/handle/10568/117461 Y2 - 2023/10/17/13:45:26 ER - TY - RPRT TI - Increasing Participation in Evaluation AU - Baker, A AU - Bruner, B T2 - Effectiveness Initiatives AB - The Increasing Participation in Evaluation bulletin was developed by Anita Baker with Beth Bruner to help organizations integrate evaluative thinking into their organizational practice. This three page bulletin discusses how Organization Staff, Evaluators, and Funders are typically involved in participatory evaluation. The guide also defines the term "Ripple", and how to accomplish Ripple as well as examining what it looks like when Executive Leaders and Management Staff use Evaluative Thinking. Contents How are Organization Staff, Evaluators, and Funders typically involved in Participatory Evaluation? Organization Managers/Staff Roles Evaluators/Roles Funder Roles Shared Roles What is Ripple Anyway? How do you accomplish Ripple? What does it look like when Executive Leaders and Management Staff Use Evaluative Thinking? How to use these Bulletins DA - 2014/// PY - 2014 SP - 3 PB - Bruner Foundation UR - http://www.evaluativethinking.org/docs/EvaluativeThinking.bulletin.10.pdf Y2 - 2018/10/19/00:00:00 ER - TY - BLOG TI - Innovating for inclusive rigour in peacebuilding evaluation AU - Apgar, Marina AU - Báez-Silva, Ángela Maria AU - Deng, Ayak Chol AU - Fairey, Tiffany AU - Rohrbach, Livia AU - Alamoussa, Dioma AU - Bradburn, Helene AU - Cubillos, Edwin AU - Gray, Stephen AU - Wingender, Leslie T2 - Institute of Development Studies AB - Inclusive and rigorous peacebuilding evaluation is both vital and complex. In this blog we share examples of how we are innovating our methodologies to move towards participatory and adaptive practice. DA - 2022/04/22/T09:21:37+00:00 PY - 2022 LA - en-GB UR - https://www.ids.ac.uk/opinions/innovating-for-inclusive-rigour-in-peacebuilding-evaluation/ Y2 - 2022/04/22/14:14:00 ER - TY - RPRT TI - Innovation histories: A method from learning from experience AU - Douthwaite, B AU - Ashby, J T2 - ILAC Brief AB - Preparing an ‘innovation history’ is a method for recording and reflecting on an innovation process. People who have been involved in the innovation jointly construct a detailed written account (sometimes referred to as a ‘learning history’) based on their recollections and on available documents. The process of preparing this history stimulates discussion, reflection and learning amongst stakeholders. Subsequent planning can build on the lessons learned, formulate a shared vision and act as a catalyst for change. Based on the initial detailed account of the innovation process, more concise informational products can be prepared that summarize the innovation process for wider dissemination of findings. These may include public awareness materials, policy briefs or articles in professional journals. DA - 2005/// PY - 2005 SP - 4 SN - 5 UR - https://cgspace.cgiar.org/handle/10568/70176 Y2 - 2018/10/18/00:00:00 ER - TY - BLOG TI - Innovation History AU - Better Evaluation T2 - Better Evaluation AB - Preparing an ‘innovation history’ is an option for recording and reflecting on an innovation process. People who have been involved in the innovation jointly construct a detailed written account (sometimes referred to as a ‘learning history’) based on their recollections and on available documents. The process of preparing this history stimulates discussion, reflection and learning among stakeholders. Subsequent planning drawing on the innovation history, can (i) build on the lessons learned, (ii) inform a shared vision, (iii) act as a catalyst for change and (iv) improve future performance. An innovation history is developed in stages. Based on the initial detailed account of the innovation process, more concise informational products can be prepared that summarize the innovation process for internal use. Products designed for wider dissemination of findings can help external parties build upon and expand their knowledge and understanding about how innovations are brought about. Such products may include public awareness materials, policy briefs and articles in professional journals. They may be based on the study of an individual case or on reviews that compare and contrast experiences across several cases. Innovation histories are underpinned by two sets of concepts that guide data gathering and analysis. The first set comes from a model of the innovation process called the ‘learning selection’ model (Douthwaite, 2002). The second set is derived from social network analysis (e.g. Cross and Parker, 2004). These concepts help participants involved in creating the innovation history to appreciate innovation as an evolutionary process driven by experiential learning cycles. The experimentation and learning involved in this evolutionary process leads to the generation of novelty, followed by its selection and promulgation. During this process, technologies become ‘fitter’, in the sense that they perform better. Stakeholder group’s social networks influence how this evolutionary learning selection process evolves hence the importance of identifying, tracking and documenting them. UR - https://www.betterevaluation.org/en/plan/approach/innovation_history Y2 - 2018/10/18/00:00:00 ER - TY - BLOG TI - Innovative M&E from the Sandbox and beyond AU - Vester, Søren AU - Tran, Samuel T2 - Medium AB - In this blog we are sharing a digest of some of the many useful and innovative monitoring, evaluation and learning resources and efforts that have come through the M&E Sandbox in 2022. A lot of these resources have been shared by our community in response to the overwhelmingly positive feedback from the launch of the Sandbox (please keep them coming!). We hope you find it useful. We have grouped these efforts and resources under six broad questions: - How do we measure systems transformation? - How do we know if we are on track? - How do we rethink complexity and independence in evaluation? - Why, how and for whom do we measure? - How do we generate insights and learn? - How do we make decisions and adapt? DA - 2023/01/20/T14:03:18.824Z PY - 2023 LA - en UR - https://medium.com/@undp.innovation/innovative-m-e-from-the-sandbox-and-beyond-9234d0977796 Y2 - 2023/01/24/09:29:34 ER - TY - RPRT TI - Insights for Influence: Understanding Impact Pathways in Crisis Response AU - Clark, Louise AU - Carpenter, Jo AU - Taylor, Joe AB - The Covid-19 Responses for Equity (CORE) programme was a three-year initiative funded by the Canadian International Development Research Centre (IDRC) that brought together 20 projects from across the global South to understand the socioeconomic impacts of the Covid-19 pandemic, improve existing responses, and generate better policy options for recovery. The research covered 42 countries across Africa, Asia, Latin America, and the Middle East to understand the ways in which the pandemic affected the most vulnerable people and regions, and deepened existing vulnerabilities. Research projects covered a broad range of themes, including macroeconomic policies for support and recovery; supporting essential economic activity and protecting informal businesses, small producers, and women workers; and promoting democratic governance to strengthen accountability, social inclusion, and civil engagement. The Institute of Development Studies (IDS) provided knowledge translation (KT) support to CORE research partners to maximise the learning generated across the research portfolio and deepen engagement with governments, civil society, and the scientific community. As part of this support, the IDS KT team worked with CORE project teams to reconstruct and reflect on their impact pathways to facilitate South-South knowledge exchange on effective strategies for research impact, and share learning on how the CORE cohort has influenced policy and delivered change. This report presents an overview of these impact pathways and the lessons learnt from a selection of the projects chosen to represent the diversity of approaches to engage policymakers, civil society, and the media to generate and share evidence of the effect of the pandemic on diverse vulnerable groups. CY - Brighton DA - 2023/11/10/ PY - 2023 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies ST - Insights for Influence UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/18172 Y2 - 2023/11/13/12:39:09 ER - TY - BLOG TI - Institutional Histories AU - Better Evaluation T2 - Better Evaluation AB - An institutional history (IH) is a narrative that records key points about how institutional arrangements – new ways of working – have evolved over time and have created and contributed to more effective ways to achieve project or programme goals. An IH is generated and recorded in a collaborative way by scientists, farmers and other stakeholders. A key intention behind institutional histories is to introduce institutional factors into the legitimate narrative of success and failure in research organizations. Histories can be written by using interviews and ‘writeshops’ to construct a timeline, gain a clear understanding of roles and relationships, enquire into what triggers successful innovations and reflect on failures. Lessons drawn from this analysis can be used to improve performance. The dialogue that is promoted between the actors during the preparation of institutional histories can promote learning and capacity building. UR - https://www.betterevaluation.org/en/plan/approach/institutional_histories Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Integrated Governance: Achieving Governance Results and Contributing to Sector Outcomes AU - Frazer, Sarah AU - Granius, Mark AU - Brinkerhoff, Derick W. AU - McGregor, Lisa T2 - RTI Press Publication AB - Achieving broad-based socio-economic development requires interventions that bridge disciplines, strategies, and stakeholders. Effective sustained progress requires more than simply an accumulation of sector projects, and poverty reduction, individual wellbeing, community development, and societal advancement do not fall neatly into sectoral categories. However, researchers and practitioners recognize key operational challenges to achieving effective integration that stem from the structures and processes associated with the current practice of international development. Integration calls for the intentional linking of intervention designs, implementation, and evaluation across sectors and disciplines to achieve mutually reinforcing outcomes. In this report, we summarize the results of a study we conducted to explore the challenges facing governance programs that integrate with sector interventions to achieve governance outcomes and contribute to sector outcomes. Through a review of policy documents and project reports from recent integrated governance programs and interviews with donor and practitioner staff, we found three integrated governance programming variants, an emphasis on citizen and government collaboration to improve service delivery, interventions that serve as the glue between sectors, and a balancing act for indicators to measure contribution to sectoral outcomes. Our analysis identified four key success factors: contextual readiness, the application of learning and adapting approaches, donor support, and recognition of the limitations of integrated governance. We then discuss recommendations and implications and for answering the challenge of integrating governance and sector programming to achieve development outcomes. CY - Research Triangle Park, NC DA - 2022/05/19/ PY - 2022 DP - DOI.org (Crossref) LA - en PB - RTI Press SN - RR-0046-2205 ST - Integrated Governance UR - https://www.rti.org/rti-press-publication/integrated-governance Y2 - 2022/07/01/08:09:30 ER - TY - RPRT TI - Integrating Big Data and Thick Data to Transform Public Services Delivery AU - Ang, Yuen Yuen AB - This report explains how public organizations can combine big data and thick data to transform public services delivery—a strategy called mixed analytics. Governments can greatly enhance the value of big data by combining it with “thick” data—rich qualitative information about users, such as their values, goals, and consumption behavior, obtained by observing or interacting with them in their daily lives. Big data holds great promise for improving public services delivery and innovation in government, but they are not a panacea . Having lots of data can be overwhelming or have little utility if the data are “thin”—that is, they lack meaning for users or fail to capture issues that matter most . By yielding insights into what citizens really care about and how they consume services, thick data can inform both the collection and analysis of big data. This report introduces the concept of “mixed analytics,” integrating big data and thick data to transform government decision making, public services delivery, and communication. The report presents three case studies of organizations that employ mixed analytics at the international, federal, and city level, respectively. Together, this research offers a set of transferable lessons for agencies at all levels of government: Lesson 1: Big data is a means to an end, rather than an end. Lesson 2: Thick data can identify unexpected problems or previously unexpressed needs. Lesson 3: Thick data can inform the analysis of big data. Lesson 4: Mixed analytics can offer both scale and depth. Lesson 5: Applying technology is a social activity, not an isolated technical task. Lesson 6: The best solutions are not always high-tech. The report concludes with five actionable recommendations for public managers. Make data and technology relevant to the people who use it. Leverage thick data at appropriate stages of the problem-solving process. Build an interdisciplinary team of quantitative and qualitative experts who work closely with stakeholders. Combine big and thick data to improve communication Improve government agencies’ knowledge of mixed research methods. This report builds on multiple past IBM Center reports about how government can leverage data and analytics to improve decisions, including Data-Driven Government: The Role of Chief Data Officers, by Jane Wiseman; Ten Actions to Implement Big Data Initiatives: A Study of 65 Cities, by Alfred Ho and Bo McCall, and Realizing The Promise of Big Data, by Kevin DeSouza. The report also complements several chapters that assess the past and potential use of data across the public sector in our recent book, Government For The Future: Reflection and Vision For Tomorrow’s Leaders. At a time when the US and other governments continue to enhance their use of data as a strategic asset for transformation, we hope that Professor Ang’s report provides useful insights for government managers and stakeholders. DA - 2019/// PY - 2019 DP - Zotero SP - 47 LA - en PB - IBM Center for The Business of Government ER - TY - GEN TI - Interview protocol: Most significant turning points AU - Prieto Martin, Pedro AU - Faith, Becky AB - This interview protocol was used for a research project on adaptiveness in technology for governance initiatives in Kenya. For more information, please read the research report at: Prieto Martin, P.; Hernandez, K.; Faith, B. and Ramalingam, B. (2017) Doing Digital Development Differently: Lessons in adaptive management from technology for governance initiatives in Kenya, MAVC Research Report, Brighton: Institute of Development Studies, ids.ac.uk/project/making-all-voices-count DA - 2017/10// PY - 2017 PB - Institute of Development Studies UR - https://www.academia.edu/34704208 Y2 - 2017/09/28/00:00:00 KW - IMPORTANT ER - TY - BLOG TI - Introducing the CLA Maturity Tool for Implementing Partners AU - Salib, Monalisa AU - Ziegler, Jessica T2 - Social Impact AB - On December 13, 2022, Social Impact hosted a webinar in our Evidence for Impact series, "Strengthen Your Team's CLA Practices: Introducing the CLA Maturity Tool for USAID Implementers." Through the session, we shared more about the origin and history of the USAID tool upon which this version is based, why and how we updated DA - 2023/01/11/T17:58:55+00:00 PY - 2023 LA - en-US UR - https://socialimpact.com/introducing-the-cla-maturity-tool-for-implementing-partners/ Y2 - 2023/03/20/11:34:01 ER - TY - BLOG TI - Introduction to International Futures AU - International Futures at the Pardee Center T2 - International Futures Help System AB - INTERNATIONAL FUTURES HELP SYSTEM Introduction to IFs Purposes Visual Representation of IFs Issues and Modules Quick Survey of IFs Issues and Modules IFs Background IFs Geographic Representation of the World IFs Time Horizon Instructional Use Acknowledgements Feedback Support for IFs Use Development Mode Features DA - 2016/// PY - 2016 UR - https://www.du.edu/ifs/help/intro/index.html Y2 - 2018/03/09/00:00:00 ER - TY - RPRT TI - It's All About MeE: Using Structured Experiential Learning (“e”) to Crawl the Design Space AU - Pritchett, Lant AU - Samji, Salimah AU - Hammer, Jeffrey T2 - Working Paper 322 AB - This paper argues that within-project variations in design can serve as their own counterfactual, reducing the incremental cost of evaluation and increasing the direct usefulness of evaluation to implementing agencies. It suggests combining monitoring (‘M’), structured experiential learning (‘e’), and evaluation (‘E’) so as to facilitate innovation and organisational capability building while also providing accountability … CY - Washington DC DA - 2013/04// PY - 2013 PB - Center for Global Development ST - It's All About MeE UR - http://www.gsdrc.org/document-library/its-all-about-mee-using-structured-experiential-learning-e-to-crawl-the-design-space/ Y2 - 2017/05/17/15:17:34 ER - TY - BLOG TI - Iterative Design and Monitoring for Adaptive Management: How Causal Link Monitoring can help AU - Britt, Heather AU - Hummelbrunner, Richard AU - Green, Jackie T2 - Better Evaluation AB - Development actors are embracing the concept and practice of adaptive management, using evidence to inform ongoing revisions throughout implementation. In this guest blog, Heather Britt, Richard Hummelbrunner and Jackie Greene discuss a practical approach that donors and partners can use to agree on what’s most important to monitor as a project continues to evolve. DA - 2018/09/26/ PY - 2018 UR - https://www.betterevaluation.org/en/blog/iterative-design-and-monitoring-adaptive-management-how-causal-link-monitoring-can-help Y2 - 2018/10/13/00:00:00 ER - TY - JOUR TI - Keeping it Real: Using Mechanisms to Promote Use in the Realist Evaluation of the Building Capacity to Use Research Evidence Program AU - Punton, Melanie AU - Vogel, Isabel T2 - New Directions for Evaluation AB - This chapter explores the use of mechanisms within the realist evaluation of the Building Capacity to Use Research Evidence (BCURE) program, a £15.7 million initiative aiming to improve the use of evidence in decision-making in low and middle-income countries. The evaluation was commissioned to establish not just whether BCURE worked but also how and why capacity building can contribute to increased use of evidence in policymaking in the very different contexts in which the program operated. This chapter argues that using mechanisms helped provide nuanced and robust insights into these questions, while also strengthening the usefulness and policy relevance of the evaluation. Drawing primarily on qualitative data, including interviews with more than 500 stakeholders over 3 years, the evaluation explored the mechanisms that promote capacities to use evidence in decision-making, through developing and testing realist context-intervention-mechanism-outcome configurations (CIMOs). Uncovering the value of mechanisms for policy and program learning was not easy, and the chapter sets out some of the thorny challenges faced and how the BCURE evaluation navigated these. Ultimately, the use of mechanisms in the BCURE evaluation helped to generate practical and nuanced insights that fed directly into the design of a £17 million follow-up program. The seven mechanisms uncovered are continuing to inform our own and others’ work on institutional capacity change in a wide range of fields. DA - 2020/// PY - 2020 DO - 10.1002/ev.20427 DP - Wiley Online Library VL - 2020 IS - 167 SP - 87 EP - 100 LA - en SN - 1534-875X ST - Keeping it Real UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20427 Y2 - 2021/02/18/14:26:34 ER - TY - BOOK TI - Key Concepts in Ethnography AU - O'Reilly, Karen CY - London DA - 2009/// PY - 2009 PB - SAGE Publications ER - TY - JOUR TI - Knowledge Management strategies and processes in Agile Software Development: a systematic literature review AU - Ouriques, Raquel Andrade Barros AU - Wnuk, Krzysztof AU - Gorschek, Tony AU - Svensson, Richard Berntsson T2 - CoRR AB - Knowledge-intensive companies that adopt Agile Software Development (ASD) relay on efficient implementation of Knowledge Management (KM) strategies to promotes different Knowledge Processes (KPs) to gain competitive advantage. This study aims to DA - 2018/// PY - 2018 DP - Zotero VL - abs/1807.04962 LA - en UR - https://arxiv.org/ftp/arxiv/papers/1807/1807.04962.pdf Y2 - 2018/12/19/00:00:00 ER - TY - RPRT TI - Knowledge Retention and Transfer (KRT) - Model and Matsurity Matrix AU - USAID T2 - CLA Toolkit AB - The Maturity Matrix is not a standalone tool. It is one of the three major components of the KRT Model. The Maturity Matrix is meant to be used in conjunction with the KRT Toolkit and Implementation Plan. CY - Washington DC DA - 2019/06// PY - 2019 PB - USAID LEARN UR - https://usaidlearninglab.org/sites/default/files/resource/files/version_2_cla_toolkit_staff_transitions_tool_20190613.pdf Y2 - 2023/10/26/09:40:00 ER - TY - RPRT TI - Knowledge Retention and Transfer (KRT) - Model overview presentation AU - USAID AB - What problem is the KRT model trying to solve? For workforces that experience continuous staff turnover, the lack of systematic knowledge transfer can often lead to: - Loss of programmatic momentum, - Duplication of efforts and frustration, and - Wasted time and resources. The Knowledge Retention and Transfer (KRT) model provides tools, processes, and practices to individuals, teams, offices, and organizations to improve knowledge handover, which in turn improves efficiency and programmatic and operational learning. CY - Washington DC DA - 2023/07// PY - 2023 PB - USAID UR - https://usaidlearninglab.org/sites/default/files/resource/files/version_2_cla_toolkit_staff_transitions_tool_20190613.pdf Y2 - 2023/10/26/09:40:00 ER - TY - JOUR TI - Large language model applications for evaluation: Opportunities and ethical implications AU - Head, Cari Beth AU - Jasper, Paul AU - McConnachie, Matthew AU - Raftree, Linda AU - Higdon, Grace T2 - New Directions for Evaluation AB - Large language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text-based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep-learning AI techniques, where language models are trained on extremely large text data from the internet and then re-used for downstream tasks with limited fine-tuning required. They offer exciting opportunities for evaluators to automate and accelerate time-consuming tasks involving text analytics and text generation. We estimate that over two-thirds of evaluation tasks will be affected by LLMs in the next 5 years. Use-case examples include summarizing text data, extracting key information from text, analyzing and classifying text content, writing text, and translation. Despite the advances, the technologies pose significant challenges and risks. Because LLM technologies are generally trained on text from the internet, they tend to perpetuate biases (racism, sexism, ethnocentrism, and more) and exclusion of non-majority languages. Current tools like ChatGPT have not been specifically developed for monitoring, evaluation, research, and learning (MERL) purposes, possibly limiting their accuracy and usefulness for evaluation. In addition, technical limitations and challenges with bias can lead to real world harm. To overcome these technical challenges and ethical risks, the evaluation community will need to work collaboratively with the data science community to co-develop tools and processes and to ensure the application of quality and ethical standards. DA - 2023/// PY - 2023 DO - 10.1002/ev.20556 DP - Wiley Online Library VL - 2023 IS - 178-179 SP - 33 EP - 46 LA - en SN - 1534-875X ST - Large language model applications for evaluation UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20556 Y2 - 2023/12/11/09:47:35 ER - TY - RPRT TI - Lasting Roots: Naatal Mbay and the Integrated Finance Model in Senegal AU - Fowler, Ben AU - Courbois, Laura T2 - MSP Ex-Post Study AB - This report addresses the well-recognized evidence gap1 on the longer-term impacts created by marketdriven programming; specifically, programming influenced by market systems development (MSD) principles. It does so by presenting the findings of an ex-post study conducted three and a half years after the close of USAID’s Feed the Future Senegal Naatal Mbay Activity (hereafter Naatal Mbay) in 2019. It examines the scale and sustainability of changes resulting from Naatal Mbay’s introduction of an integrated finance model (IFM) – described in Error! Reference source not found. below – in the domestic rice sector. This study is one in a series of ex-post evaluations that are being conducted between 2023-2026 on USAID-funded MSD interventions around the world. This study focused on four questions, noted below in Figure 1. These were addressed using a mix of desk research, 122 key informant interviews with market actors and other stakeholders remotely and in Senegal, focus group discussions with 26 rice producers networks in Senegal, and a validation workshop with USAID/Senegal, implementing partner staff and market actors. Findings were analyzed leveraging the Disrupting System Dynamics (DSD) framework (see Figure 4 in the body of the report) as an analytical tool for understanding systems change. CY - Washington DC DA - 2023/08// PY - 2023 DP - Zotero LA - en PB - USAID UR - https://agrilinks.org/post/lasting-roots-ex-post-study-senegal-naatal-mbay-and-integrated-finance-model Y2 - 2023/10/02/00:00:00 ER - TY - RPRT TI - Laudes Rubrics - Measurement and Learning Approach AU - Laudes Foundation AB - How can we measure and learn when promoting systems change? It’s a challenge that has inspired Laudes Foundation to develop a rubrics-based methodology to help us, our partners and the wider field of philanthropy, understand our contribution to change, while learning and adapting to new and unforeseen circumstances. Change cannot be captured by numbers alone because metrics put the focus on what can be counted, not always what’s most important. Rubrics are a framework that set a standard for what ‘good’ looks like – and create a shared language for describing and assessing it using both quantitative and qualitative evidence. At Laudes Foundation, the rubrics are integrated into our grantmaking processes – from the design phase through to measurement, evaluation and learning. The four Laudes Foundation rubrics categories Laudes Foundation has developed 21 rubrics that work across different levels, from processes to long-term impact. When measuring a specific initiative, a smaller set of relevant rubrics are chosen and assessed on a rating scale from ‘harmful’ to ‘thrivable’. The 21 rubrics are categorised into four groups, with some natural overlap between categories B, C and D. Category A focuses on the process-related aspects of initiatives, including design, implementation, monitoring, communication and learning, and organisation and network effectiveness. Category B focuses on the early and later changes that need to happen to create the right conditions to achieve the 2025 outcomes. Category C captures the 2025 outcomes, focusing on how policymakers, financiers, business leaders, and workers and producers behave. Category D captures the 2030 impacts, describing the new reality created as a result of sustained efforts. DA - 2023/// PY - 2023 LA - en PB - Laudes Foundation UR - https://www.laudesfoundation.org/grants/rubrics Y2 - 2023/06/14/11:03:37 ER - TY - RPRT TI - LEARN End of Contract Report AU - Learning Lab AB - In September of 2014, USAID’s Office of Learning, Evaluation & Research (LER) awarded the Learning and Knowledge Management (LEARN) contract to Dexis Consulting Group and subcontractor RTI International.1 This document—the End of Contract Report—captures five and half years of results and reflections for our stakeholders. Our intention is to share the good and the bad, and while this report would not be considered a “tell all,” we think we have a story worth sharing, particularly to USAID CORs and AORs, activity managers, and other implementers of institutional support contracts.2 LEARN’s primary purpose was to support organizational change at USAID. More specifically, the contract was focused on helping USAID staff integrate collaborating, learning, and adapting (CLA) approaches into the design, implementation, monitoring, and evaluation of programs (what is known at USAID as the Program Cycle). It was clear that most USAID staff, whether they realized it or not, were already integrating CLA into their work to some extent. The focus of our efforts, therefore, was to make those practices more systematic, intentional, resourced, and ultimately more widespread throughout the Agency, which would have a ripple effect on implementing partners and even other stakeholders, such as host country governments. This was based on the theory—later borne out by evidence—that by becoming a better learning organization, USAID could be a more effective development organization. And that theory brought the USAID CLA team within USAID’s Bureau for Policy, Planning & Learning (PPL) and LEARN contractors together, driven by a shared purpose of improving how USAID does business. LEARN was designed based on this belief and, as you might expect from a learning-oriented contract, began with more questions than answers. Primary among those questions was: could an institutional support contract do more than carry out requested services—could it actually accelerate positive organizational change at USAID? And if so, how and under what conditions? DA - 2020/04/05/ PY - 2020 DP - Zotero SP - 92 LA - en PB - USAID ER - TY - RPRT TI - Learning about Theories of Change for the Monitoring and Evaluation of Research Uptake AU - Barnett, Chris AU - Gregorowski, Robbie T2 - IDS Practice Paper In Brief AB - This paper captures lessons from recent experiences on using ‘theories of change’ amongst organisations involved in the research–policy interface. The literature in this area highlights much of the complexity inherent in the policymaking process, as well as the challenges around finding meaningful ways to measure research uptake. As a tool, ‘theories of change’ offers much, but the paper argues that the very complexity and dynamism of the research-to-policy process means that any theory of change will be inadequate in this context. Therefore, rather than overcomplicating a static depiction of change at the start (to be evaluated at the end), incentives need to be in place to regularly collect evidence around the theory, test it periodically, and then reflect and reconsider its relevance and assumptions. CY - Brighton DA - 2013/10/01/ PY - 2013 DP - opendocs.ids.ac.uk LA - en PB - IDS SN - 14 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/2995 Y2 - 2021/12/20/12:47:00 ER - TY - RPRT TI - Learning and Accountability AU - Simister, Nigel AU - Scholz, Vera AB - Most people agree that monitoring and evaluation (M&E) should be used for both learning and accountability. However, there is no consensus about which one is more important. The debate matters as there is sometimes tension between the two purposes. In the past there has often been a disconnect between M&E and learning. Many M&E systems are primarily designed to enable accountability to donors. DA - 2020/// PY - 2020 PB - INTRAC UR - https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Learning-and-Accountability.pdf Y2 - 2023/01/24/10:07:05 ER - TY - BOOK TI - Learning and adapting: the use of monitoring and evaluation in countering violent extremism: a handbook for practitioners AU - Dawson, Laura AU - Edwards, Charlie AU - Jeffray, Calum AU - Royal United Services Institute for Defence and Security Studies AB - IN 2013, the Royal United Services Institute for Defence and Security Studies (RUSI) was awarded a grant under the Kanishka Project to develop a handbook for monitoring and evaluating counter violent extremism (CVE) policies and programmes. The aim of this handbook is to support CVE policy-makers and practitioners (those who design, manage and evaluate CVE programmes), by providing them with key terms regarding violent extremism and radicalisation, describing the purpose of evaluation, and providing examples of key methodologies they can employ to conduct monitoring and evaluation (M&E) in this emerging policy field. The handbook will enable readers to understand why, when and how to conduct an evaluation of a CVE policy, programme or project. . DA - 2014/// PY - 2014 DP - Open WorldCat LA - English PB - Royal United Services Institute for Defence and Security Studies SN - 978-0-85516-124-8 ST - Learning and adapting UR - http://www.publicsafety.gc.ca/lbrr/archives/cnmcs-plcng/cn31896-eng.pdf Y2 - 2019/09/17/10:38:28 ER - TY - JOUR TI - Learning and not learning from experience in rural project planning AU - Hulme, David T2 - Public Administration and Development AB - The theme of learning from experience as a means of improving the effectiveness of rural development projects and programmers has been common in recent years. Considerable effort has been put into refining, monitoring and evaluation systems to enhance organizational learning processes. However, an emphasis on normative approaches to evaluation and learning from experience has led to the neglect of research into the actual processes by which rural development agencies utilize experience. The case study presented here points to the shortcomings of such approaches and illustrates the partisan manner in which individuals and organizations treat the lessons of experience. Actively ‘not learning from experience’ is as much a part of organizational processes as learning from experience. This paper examines the implications of this finding and reviews alternative approaches to improving experiential learning in rural development activities. It is found that such a perspective does not generate the innocuous technical prescriptions characteristic of conventional approaches, but a number of useful directions for further research can be identified. DA - 1989/// PY - 1989 DO - 10.1002/pad.4230090102 DP - Wiley Online Library VL - 9 IS - 1 SP - 1 EP - 16 LA - en SN - 1099-162X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/pad.4230090102 Y2 - 2023/09/29/09:55:14 ER - TY - RPRT TI - Learning Evaluation Theory: Brinkerhoff’s Success Case Method AU - Downes, Andrew AB - Brinkerhoff’s model isn’t restricted to learning. It can be used to analyze any major business change, such as the purchase of new equipment or implementation of a new process. It’s based on the assumption that any initiative, no matter how successful or unsuccessful, will always include some success and some failure. It seeks to uncover the most impactful successes and failures of an initiative and then tell the stories behind them, backed by evidence. Your organization can use these stories to learn how to be more successful in the future. DA - 2016/// PY - 2016 SP - 4 PB - Watershed UR - https://www.watershedlrs.com/hubfs/DOCUMENTS/Learning_Evaluation_Brinkerhoff_.pdf Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Learning from Life Story Collection and Analysis With Children Who Work in the Worst Forms of Child Labour in Nepal AU - Karki, Shanta AU - Apgar, Marina AU - Snijder, Mieke AU - Sharma, Ranjana T2 - CLARISSA Learning Note AB - The CLARISSA Nepal team collected and analysed 400 life stories of children and young people engaged in or affected by the worst forms of child labour (WFCL), particularly in the “Adult Entertainment” sector in Nepal, which includes children working in Dohoris (restaurants playing folk music), dance bars, spa-massage parlours, khaja ghars (tea/snack shop) and guest houses. Stories were also collected from children in CLARISSA’s focus neighbourhoods, children in this category include street connected children and those working in transportation, party palaces, domestic labour and construction sites. Of the 400 stories collected, 350 were collected by adult researchers and 50 were collected by children themselves. CY - Brighton DA - 2022/10// PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 2 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17740 Y2 - 2023/10/16/13:41:44 ER - TY - MGZN TI - Learning in the Thick of It AU - Darling, Marilyn AU - Parry, Charles AU - Moore, Joseph T2 - Harvard Business Review AB - After-action reviews identify past mistakes but rarely enhance future performance. Companies wanting to fully exploit this tool should look to its master: the U.S. Army’s standing enemy brigade, where soldiers learn and improve even in the midst of battle. DA - 2005/07/01/T04:00:00Z PY - 2005 DP - hbr.org SN - 0017-8012 UR - https://hbr.org/2005/07/learning-in-the-thick-of-it Y2 - 2024/01/12/15:30:13 KW - Aerospace sector KW - Corporate governance KW - Decision making and problem solving KW - Defense industry KW - Defense sector KW - Fabrication and manufacturing KW - Meeting management KW - Military KW - North America KW - Organizational culture KW - Presentation skills KW - Project management KW - Psychology KW - Telecom ER - TY - RPRT TI - Learning, monitoring and evaluating: achieving and measuring change in adaptive programmes AU - Callaghan, Sarah AU - Plank, Georgia T2 - Synthesis Paper, 5 DA - 2017/05// PY - 2017 DP - Google Scholar PB - DFID-LASER Programme UR - http://dfidlaser.org/wp-content/uploads/2017/07/Synthesis-paper-6-MEL-1-June-2017-FINAL.pdf Y2 - 2017/09/13/09:22:45 ER - TY - BLOG TI - Learning through and about Contribution Analysis for impact evaluation AU - Apgar, Marina AU - Ton, Giel T2 - Institute of Development Studies AB - Over the past five years, colleagues from the Centre for Development Impact (CDI) – a joint initiative between the Institute of Development Studies, Itad and University of East Anglia – have been innovating with and learning how to use Contribution Analysis as an overarching approach to impact evaluation. In this blog series, we share our learning and insights, some of them in raw emergent form, highlight the complexities, nuances, excitements, and challenges of embracing new ways of doing impact evaluation. We begin this series by sharing ideas about ‘causal hotspots’ that first surfaced as ‘aha’ moments in our collaboration on the CDI short course Contribution Analysis for Impact Evaluation – the moments when we found the ways and words to better articulate our ideas and help people navigate the messy realities of theory-based evaluation. DA - 2021/09/10/ PY - 2021 LA - en-US UR - https://www.ids.ac.uk/opinions/learning-through-and-about-contribution-analysis-for-impact-evaluation/ Y2 - 2024/02/13/00:00:00 ER - TY - RPRT TI - Learning to adapt & adapting to learn - Using elements of outcome mapping in the ‘Resilient Adolescents in the Syria Crisis’ programme AU - van Ongevalle, Jan AU - Kvintradze, Ana AU - Rennesson, Gaël AU - Miller, David AB - This learning paper highlights how elements of outcome mapping were used by Save the Children Sweden in a project (2018-2020) that supports adolescents, affected by the Syria crisis, to become more resilient. The paper first outlines how the spheres of influence framework has been applied to develop an actor focused theory of change. It then describes how progress markers, as an alternative to SMART indicators, were formulated to monitor the programme’s results. The paper also outlines how long lists of progress markers were categorised in a more realistic and practical results framework. The paper then continues to elaborate how outcome journals, qualitative data analysis techniques and regular review meetings and reflection workshops were utilised for data collection, for collective learning among programme stakeholders and for informing planning and programme adjustment. Various practical guidelines and tips on how to implement elements of outcome mapping are provided. The final part of the paper explores to what extent outcome mapping was able to foster several key enablers of adaptive programme management and highlights some of the challenges that programme stakeholders faced. Practical recommendations towards the use of outcome mapping in future programmes are also proposed. DA - 2021/06// PY - 2021 PB - Save the Children UR - https://www.outcomemapping.ca/download/Outcome%20Mapping%20Learning%20Paper_SAP_02062021.pdf Y2 - 2022/09/30/08:38:35 ER - TY - RPRT TI - Learning to ADAPT: monitoring and evaluation approaches in climate change adaptation and disaster risk reduction – challenges, gaps and ways forward AU - Silva Villanueva, Paula T2 - SRC Discussion Paper AB - This working paper is a methodological contribution to the emerging debate on monitoring and evaluation (M&E) in the context of climate change adaptationand disaster risk reduction. Effectively managing disaster risk is critical for adapting to the impacts of climate change, however disasters risk reduction M&E practice may be limited in capturing progress towards adaptation. The unique nature of adaptation to climate change calls for experience-based learning M&E processes for discovering the key insights into adaptive capacity and its links to adaptation processes, and to risk and vulnerability reduction at large. The ADAPT guiding principles and indicators set the foundations towards this end. DA - 2011/// PY - 2011 DP - opendocs.ids.ac.uk LA - en PB - IDS SN - 9 ST - Learning to ADAPT UR - https://opendocs.ids.ac.uk/opendocs/handle/123456789/2509 Y2 - 2018/08/22/08:26:33 ER - TY - CHAP TI - Learning to Learn Collectively AU - Wrigley, Rebecca T2 - Capacity AB - CABUNGO, a Malawian NGO, recently evaluated its own performance using the Most Significant Change approach. Rebecca Wrigley describes how, with the support of stakeholders,CABUNGO learned to improve its services. DA - 2008/04// PY - 2008 VL - 33 SP - 13 EP - 13 UR - http://gsbblogs.uct.ac.za/walterbaets/files/2009/09/CapacityandOL2.pdf Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Learning to make a difference: Christian Aid Ireland’s adaptive programme management in governance, gender, peace building and human rights AU - Booth, David AU - Balfe, Karol AU - Gallagher, Róisín AU - Kilcullen, Gráinne AU - O'Boyle, Sarah AU - Tiernan, Alix AB - This paper assesses the relevance of adaptive or trial-and-error approaches to the field of governance, peace building and human rights. Tackling the problems of poverty, vulnerability and exclusion that persist in parts of the world that continue to be affected by violence or political insecurity is difficult for several reasons. For one, because of the complexity of the prevailing social, economic and political systems, solutions to chronic problems are far from obvious. One response to this aspect of the challenge is adaptive programme design and management. This paper is the product of a multi-year collaboration between ODI and the core team of Christian Aid Ireland to assess the relevance of adaptive or trial-and-error approaches to the field of governance, peace building and human rights. It explains the basis on which Christian Aid Ireland’s current five-year programme has become committed to an adaptive approach. It then describes and seeks to draw lessons from the programme’s first year of experience, considering the possible implications for implementation over the coming years. The authors find that to get full benefits from the move to adaptive management, the new ways of working and their underlying principles will need to become more embedded in organisational practices and cultures. CY - London DA - 2018/09// PY - 2018 LA - en PB - ODI ST - Learning to make a difference UR - https://www.odi.org/publications/11191-learning-make-difference-christian-aid-ireland-s-adaptive-programme-management-governance-gender Y2 - 2019/02/01/09:44:17 ER - TY - BLOG TI - Lessons from a trial of the Success Case Method AU - McGuinness, Liz T2 - Better Evaluation AB - The Success Case Method approach is useful for documenting stories of impact and for understanding the factors that help or hinder impact. It is particularly useful for uncovering the contextual forces that influence impact. Originally designed for evaluating corporate training programs, the Success Case Method is now being applied to other programs including international development interventions. Last year, I provided technical assistance to a pilot of the Success Case Method as part of the USAID-funded Complexity-Aware M&E Trials. DA - 2017/07/28/ PY - 2017 UR - https://www.betterevaluation.org/en/blog/lessons_on_SCM Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Lessons Learned from Applying the Data Powered Positive Deviance AU - Pawelke, Andreas AU - Glücker, Andreas AU - Albanna, Basma AU - Boy, Jeremy AB - This report presents six learnings from four pilot projects conducted by the Data Powered Positive Deviance (DPPD) initiative, a global collaboration between the GIZ Data Lab, the UNDP Accelerator Labs Network, the University of Manchester Center for Digital Development, and UN Global Pulse Lab Jakarta. The pilots seek out grassroots solutions to development challenges that range from the interaction between livestock farming and deforestation to gender-based violence and insecurity in dense urban environments in Ecuador, Mexico, Niger and Somalia. The learnings relate to the early stages of the DPPD method, originally proposed by Albanna & Heeks [1], and focus mainly on the access to, and use of digital data. They are summarized as follows: 1. Remain flexible in the face of data unavailability 2. Leverage existing partnerships for data access 3. Map and fill know-how gaps early 4. Scale with caution 5. Look at deviance over time 6. Look beyond individual or community practices and behavior The report is written for development practitioners, data analysts, domain experts, and more generally anyone interested in using new data sources and technologies to uncover successful local solutions to development challenges. DA - 2021/10// PY - 2021 PB - DPPD ER - TY - JOUR TI - Lessons Learned From the Use of the Most Significant Change Technique for Adaptive Management of Complex Health Interventions AU - Ohkubo, Saori AU - Mwaikambo, Lisa AU - Salem, Ruwaida M. AU - Ajijola, Lekan AU - Nyachae, Paul AU - Sharma, Mukesh Kumar T2 - Global Health: Science and Practice AB - Introduction:The Most Significant Change (MSC) technique is a complex-aware monitoring and evaluation tool, widely recognized for various adaptive management purposes. The documentation of practical examples using the MSC technique for an ongoing monitoring purpose is limited. We aim to fill the current gap by documenting and sharing the experience and lessons learned of The Challenge Initiative (TCI), which is scaling up evidence-based family planning (FP) and adolescent and youth sexual and reproductive health (AYSRH) interventions in 11 countries in Asia and sub-Saharan Africa. Methods:The qualitative assessment took place in early 2021 to document TCI’s use and adaptation of MSC and determine its added value in adaptive management, routine monitoring, and cross-learning efforts. Focus group discussions and key informant interviews were conducted virtually with staff members involved in collecting and selecting MSC stories. Results:TCI has had a positive experience with using MSC to facilitate adaptive management in multiple countries. The use of MSC has created learning opportunities that have helped diffuse evidence-based FP and AYSRH interventions both within and across countries. The responsive feedback step in the MSC process was viewed as indispensable to learning and collaboration. There are several necessary inputs to successful use of the method, including buy-in about the benefits, training on good interviewing techniques and qualitative research, and dedicated staff to manage the process. Conclusion:Our assessment results suggest that the MSC technique is an effective qualitative data collection tool to strengthen routine monitoring and adaptive management efforts that allows for flexibility in how project stakeholders implement the process. The MSC technique could be an important tool for global health practitioners, policy makers, and researchers working on complex interventions because they continually need to understand stakeholders’ needs and priorities, learn from lessons and evidence-based practices, and be agile about addressing potential challenges. DA - 2022/02/28/ PY - 2022 DO - 10.9745/GHSP-D-21-00624 DP - www.ghspjournal.org VL - 10 IS - 1 LA - en SN - 2169-575X UR - https://www.ghspjournal.org/content/10/1/e2100624 Y2 - 2022/05/16/10:09:12 ER - TY - RPRT TI - Let’s Take the Con Out of Randomized Control Trials in Development AU - Pritchett, Lant T2 - CID Faculty Working Paper AB - The enthusiasm for the potential of RCTs in development rests in part on the assumption that the use of the rigorous evidence that emerges from an RCT (or from a small set of studies identified as rigorous in a “systematic” review) leads to the adoption of more effective policies, programs or projects. However, the supposed benefits of using rigorous evidence for “evidence based” policy making depend critically on the extent to which there is external validity. If estimates of causal impact or treatment effects that have internal validity (are unbiased) in one context (where the relevant “context” could be country, region, implementing organization, complementary policies, initial conditions, etc.) cannot be applied to another context then applying evidence that is rigorous in one context may actually reduce predictive accuracy in other contexts relative to simple evidence from that context—even if that evidence is biased (Pritchett and Sandefur 2015). Using empirical estimates from a large number of developing countries of the difference in student learning in public and private schools (just as one potential policy application) I show that commonly made assumptions about external validity are, in the face of the actual observed heterogeneity across contexts, both logically incoherent and empirically unhelpful. Logically incoherent, in that it is impossible to reconcile general claims about external validity of rigorous estimates of causal impact and the heterogeneity of the raw facts about differentials. Empirically unhelpful in that using a single (or small set) of rigorous estimates to apply to all other actually leads to a larger root mean square error of prediction of the “true” causal impact across contexts than just using the estimates from non-experimental data from each country. In the data about private and public schools, under plausible assumptions, an exclusive reliance on the rigorous evidence has RMSE three times worse than using the biased OLS result from each context. In making policy decisions one needs to rely on an understanding of the relevant phenomena that encompasses all of the available evidence. CY - Boston DA - 2021/05// PY - 2021 DP - Zotero SP - 40 LA - en PB - Center for International Development, Harvard University SN - 399 UR - https://bsc.cid.harvard.edu/files/bsc/files/2021-05-cid-wp-399-external-validity.pdf Y2 - 2021/06/25/00:00:00 ER - TY - JOUR TI - Letting Evidence Speak for Itself: Measuring Confidence in Mechanisms AU - Befani, Barbara AU - D'Errico, Stefano T2 - New Directions for Evaluation AB - This chapter argues that the credibility of causal mechanisms can be greatly increased by formulating them as statements that are both empirically falsifiable and empirically confirmable. Whether statements can be so depends on the potential availability of the relevant evidence (e.g., no evidence exists that can prove or disprove the existence of God, but good quality evidence is potentially available in many other cases). The Bayes formula can be used to measure the extent to which a given set of empirical observations supports or weakens the belief that a causal mechanism exists. With this approach, confidence in the existence of a mechanism is increased or decreased through a process that can be open, transparent, and shared with the public or groups of stakeholders, reducing cognitive biases, and improving internal validity and consensus around the existence of given mechanisms. The approach is showcased in the evaluation of a learning partnership whereby a knowledge product released by a research organization influenced policy at the municipal level. DA - 2020/// PY - 2020 DO - 10.1002/ev.20420 DP - Wiley Online Library VL - 2020 IS - 167 SP - 27 EP - 43 LA - en SN - 1534-875X ST - Letting Evidence Speak for Itself UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20420 Y2 - 2021/02/18/14:26:30 ER - TY - JOUR TI - Leveraging Experimental Evaluations for Understanding Causal Mechanisms AU - Peck, Laura R. T2 - New Directions for Evaluation AB - Experimental evaluations—especially when grounded in theory-based impact evaluation—can provide insights into the mechanisms that generate program impacts. This chapter details variants of experimental evaluation designs and also analytic strategies that leverage experimental evaluation data to learn about causal mechanisms. The design variants are poised to illuminate causal mechanisms related to program implementation and the contribution of selected components of multifaceted programs. The analysis strategies lend themselves to illuminating causal mechanisms related to participants’ responses to program components as well as to the contributions of selected program components themselves. The chapter offers an example from one, theory-based impact evaluation, which embedded both design and analytic strategies to examine the extent to which specific program components and participant experiences might be identified as causal mechanisms. The particular value in using this theory-based experimental strategy is that the results are rigorous and potentially highly relevant to policy and practice. DA - 2020/// PY - 2020 DO - 10.1002/ev.20422 DP - Wiley Online Library VL - 2020 IS - 167 SP - 145 EP - 160 LA - en SN - 1534-875X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20422 Y2 - 2021/02/18/14:26:35 ER - TY - RPRT TI - Life Stories From Children Working in Bangladesh’s Leather Sector and its Neighbourhoods: Told and Analysed by Children AU - Sayem, Mashrique AU - Sayed, Sayma AU - Maksud, A. K. M. AU - Reaz Hossain, Khandaker AU - Afroze, Jiniya AU - Burns, Danny AU - Raw, Anna AU - Hacker, Elizabeth T2 - CLARISSA Research and Evidence Paper AB - CLARISSA (Child Labour: Action-Research-Innovation in South and South-Eastern Asia) has a participatory and child-centred approach that supports children to gather evidence, analyse it themselves and generate solutions to the problems they identify. The life story collection and collective analysis processes supported children engaged in the worst forms of child labour in Bangladesh to share and analyse their life stories. Over 400 life stories were collected from children who worked in the leather supply chain, or who lived and worked in leather sector neighbourhoods. Using causal mapping, 53 children who were engaged in or had experience of the worst forms of child labour collectively analysed the data. This resulted in children’s life stories becoming the evidence base for revealing macro‑level system dynamics that drive the worst forms of child labour. This paper is a record of the children’s analysis of the life stories and key themes they identified, which formed the basis of a series of seven child-led Participatory Action Research groups. CY - Brighton DA - 2023/11/06/ PY - 2023 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 5 ST - Life Stories From Children Working in Bangladesh’s Leather Sector and its Neighbourhoods UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/18168 Y2 - 2023/11/13/12:56:32 ER - TY - BOOK TI - Linking Assessment to Instruction: Using Dynamic Indicators of Basic Early Literacy Skills in an Outcomes-Driven Model AU - Kaminski, Ruth AU - Cummings, Kelli AB - As educators increasingly are held responsible for student achievement, school personnel struggle to find ways to effectively document student responsiveness to interventions and track progress toward important outcomes. While many educators focus on high-stakes tests as a means of documenting student achievement of important outcomes, other assessment approaches may be better suited to assessing student progress. Assessment that can be used to adapt teaching to meet student needs is called formative assessment. Because the primary purpose of formative assessment is to support student learning, it may arguably be considered the most important assessment practice in which educators engage. This paper will focus on linking assessment to instruction to improve student outcomes through the use of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) within an Outcomes-Driven Model. DA - 2008/// PY - 2008 SP - 7 PB - Dynamic Measurement Group UR - https://dibels.org/papers/PM_BDA_032708.pdf Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - Livelihoods in the Za'atari Camp: Impact evaluation of Oxfam’s Cash for Work activities in the Za’atari camp (Jordan) AU - Lombardini, Simone AU - Mager, Franziska DA - 2019/10/22/ PY - 2019 DP - DOI.org (Crossref) LA - en PB - http://hdl.handle.net/10546/620883 ST - Livelihoods in the Za'atari Camp UR - http://hdl.handle.net/10546/620883 Y2 - 2020/02/06/09:09:32 ER - TY - BOOK TI - Locating the energy for change: an introduction to appreciative inquiry AU - Elliott, Charles CN - HD58.8 .E44 1999 CY - Winnipeg DA - 1999/// PY - 1999 DP - Library of Congress ISBN SP - 288 PB - International Institute for Sustainable Development SN - 978-1-895536-15-7 ST - Locating the energy for change KW - Appreciative inquiry KW - Organizational change ER - TY - JOUR TI - Lost Causal: Debunking Myths About Causal Analysis in Philanthropy AU - Lynn, Jewlya AU - Stachowiak, Sarah AU - Coffman, Julia T2 - The Foundation Review DA - 2021/09/01/ PY - 2021 DO - 10.9707/1944-5660.1576 DP - DOI.org (Crossref) VL - 13 IS - 3 J2 - The Foundation Review LA - en SN - 1944-5660 ST - Lost Causal UR - https://scholarworks.gvsu.edu/tfr/vol13/iss3/6 Y2 - 2023/01/27/15:23:12 ER - TY - ELEC TI - M&E Universe AU - intrac T2 - INTRAC AB - The M&E Universe is a free, online resource developed by INTRAC to support development practitioners involved in monitoring and evaluation (M&E). It consists of a series of short papers (2-6 pages) on different subjects related to M&E. It can be explored through an online platform (intrac.org/universe) that is compatible with most web browsers. The M&E Universe is designed for M&E practitioners with different levels of experience and expertise in M&E, from those new to M&E who want an entry level into the subject to experienced practitioners wanting to broaden their knowledge. Initial contact with the Map will provide people with a starting point for understanding different aspects of M&E and how they link to each other. More experienced practitioners can browse the Map to identify more advanced subjects of interest, such as sampling methods, different forms of evaluation, organisational M&E systems and complex methodologies for data collection and analysis. DA - 2019/06// PY - 2019 LA - en UR - https://www.intrac.org/resources/me-universe/ Y2 - 2019/08/12/22:38:49 ER - TY - RPRT TI - Making adaptive rigour work - Principles and practices for strenghening monitoring, evaluation and learning for adaptive management AU - Ramalingam, Ben AU - Wild, Leni AU - Buffardi, Anne L AB - Adaptive programmes can be accountable, rigorous and high quality in how they use evidence by taking an ’adaptive rigour’ approach. Core development and humanitarian challenges are complex, and require processes of testing, learning and iteration to find solutions – adaptive management offers one approach for this. Yet large bureaucracies and development organisations can have low tolerance for experimentation and learning, and adaptive management can be viewed as an excuse for ‘making things up as you go along’. This briefing from the Global Learning for Adaptive Management (GLAM) initiative argues that adaptive programmes can be accountable, rigorous and high quality in how they use evidence – but this requires rethinking some key assumptions about how they are practised. The paper sets out three key elements of an ‘adaptive rigour’ approach: - Strengthening the quality of monitoring, evaluation and learning data and systems. - Ensuring appropriate investment in monitoring, evaluation and learning across the programme cycle. - Strengthening capacities and incentives to ensure the effective use of evidence and learning as part of decision-making, leading ultimately to improved effectiveness. CY - London DA - 2019/04// PY - 2019 DP - Zotero LA - en PB - ODI/GLAM ER - TY - RPRT TI - Making humanitarian response more flexible: Challenges and questions AU - Obrecht, Alice AU - Bourne, S. CY - London DA - 2018/08// PY - 2018 M3 - ALNAP Background Paper PB - ALNAP/ODI UR - https://www.alnap.org/system/files/content/resource/files/main/ALNAPpaper%20Making%20humanitarian%20response%20more%20flexible_1.pdf Y2 - 2018/09/18/12:04:15 ER - TY - BOOK TI - Making Outcome Mapping Work - Evolving experiences from around the World A3 - Jones, Harry CY - Ottawa DA - 2007/03// PY - 2007 DP - Zotero LA - en PB - Outcome Mapping Learning Community UR - https://www.odi.org/publications/3639-making-outcome-mapping-work-evolving-experiences-around-world Y2 - 2020/10/09/00:00:00 ER - TY - BOOK TI - Making Outcome Mapping Work - Volume 2 - Innovations in Participatory Planning, Monitoring and Evaluation A3 - Hearn, Simon A3 - Schaeffer, Heidi A3 - van Ongevalle, JAn DA - 2009/09// PY - 2009 DP - Zotero LA - en PB - Outcome Mapping Learning Community ER - TY - JOUR TI - Management not models: adaptability, responsiveness, and a few lessons from football AU - Maclay, Christopher T2 - Development in Practice AB - Despite a swathe of critiques of logframes and other blueprint approaches to development over the last 30 years, most aid infrastructure continues to concentrate on the design and subsequent implementation of closed models. This article does not propose an alternative to blueprints, but challenges the inflexibility of their implementation, which is inadequate given the complex nature of social change. It proposes a supplementary management and learning approach which enables implementers to be dynamic, adaptive, and responsive to problems and opportunities. Emphasising the role of donors, the paper presents a case study of one donor-led programme in Bangladesh doing just this. DA - 2015/01/02/ PY - 2015 DO - 10.1080/09614524.2015.983460 DP - Taylor and Francis+NEJM VL - 25 IS - 1 SP - 42 EP - 57 SN - 0961-4524 ST - Management not models UR - https://doi.org/10.1080/09614524.2015.983460 Y2 - 2022/12/05/21:47:44 ER - TY - RPRT TI - Managing staff transistions through CLA: preserving institutional memory as staff come and go AU - USAID T2 - CLA Toolkit AB - This document is relevant for any position or hiring mechanism. While this document does not explicitly address what happens when someone arrives into a newly-created position, which has its own set of challenges, many of the principles, actions, and resources can be applied in that context. Section 1 offers guidance for how to set up and implement systems at Mission/OU or office to ensure that all staff help preserve institutional memory and enable continuity of relationships. This section is for any staff in a Mission/OU working to build systems that improve handovers and knowledge capture and sharing across the Mission or OU. If a Mission or OU is just beginning to develop a system to manage staff transitions, you may want to start by developing a system within an office or team and then expand it out to other offices or to the Mission/OU writ large. Section 2 includes key principles and actions individuals could consider following when arriving or departing a position, regardless of hiring mechanism and position. This includes staff new to the Mission/OU and staff who are currently in the Mission/OU and are moving into a new position. Section 3 provides a select number of resources to help you take a systematic and comprehensive approach to manage staff transitions as effectively and efficiently as possible at the individual or organizational level. CY - Washington DC DA - 2019/06// PY - 2019 PB - USAID LEARN UR - https://usaidlearninglab.org/sites/default/files/resource/files/version_2_cla_toolkit_staff_transitions_tool_20190613.pdf Y2 - 2023/10/26/09:40:00 ER - TY - RPRT TI - Market systems change rubric AU - Loveridge, Donna AB - This systems change rubric describes different performance levels according to various systems elements, such as policy (formal rules), practices and relationships and connections. Programmes can use the rubric to assess the performance of systems to help decide where and how to intervene, or during and post-implementation to conduct progress assessments, and assess the effectiveness of interventions and type, breadth and depth of systems change. Each performance level description highlights the type of data and information that needs to be collected. One analysis is completed, users can compare this to the performance descriptions to see which level best matches the analysis. This helps programmes draw conclusions about systems changes. The rubric was developed in 2020 and tested in 2021 and builds on systems change thinking and frameworks from two previous FSG publications. It can be used as: pre-intervention to conduct an assessment during an intervention to conduct progress assessments and reflect on the effectiveness of interventions to change systems and inform decision making post intervention to make judgements about whether interventions were valuable given the resources, time and effort spent Useful for: Implementation managers to determine the effectiveness of interventions, as well as by Monitoring, Evaluation and Learning (MEL) teams to track progress against expected outcomes. CY - Oxford DA - 2022/01// PY - 2022 DP - Zotero LA - en PB - Oxford Policy Management ER - TY - RPRT TI - Measuring and Monitoring Adaptive Learning: A Landscape Review AU - LaFond, Anne AU - Adrian, Haley AB - This landscape review on measuring and monitoring adaptive learning highlights the learning from five adaptive programming guidelines and toolkits and one implementation science framework to inform the monitoring and evaluation of adaptive learning. The introduction of adaptive learning processes and skillsets in global health programming is part of an emerging strategy to advance a learning culture within projects and teams to improve health program performance. The monitoring and evaluation of adaptive learning is an emergent field aiming to monitor how adaptive learning processes have been introduced, how they are used, and whether they are having the intended results. Although there is a growing body of literature on adaptive programming more generally, there is a limited knowledge base on the monitoring and evaluation of adaptive learning interventions and their impacts. Unlike other implementation strategies or program management approaches, there are no standard metrics or a monitoring and evaluation framework to track the integration, implementation, and effectiveness of adaptive learning in health programming. CY - Washington DC DA - 2022/05// PY - 2022 DP - Zotero LA - en PB - USAID MOMENTUM Knowledge Accelerator UR - https://usaidmomentum.org/resource/adaptive-learning-measures-landscape-review/ ER - TY - SLIDE TI - Measuring Impact: A Learning Approach to Strengthening USAID Biodiversity Programs T2 - Updates from the field: perspectives on Learning from USAID-funded Activites A2 - Lauck, Liz CY - USAID DA - 2014/// PY - 2014 UR - https://usaidlearninglab.org/sites/default/files/resource/files/kmrg_061114_ppt.pdf Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - Measuring Impact. Making Use of the Portfolio: Organizational Learning at USAID AU - Peabody, Shawn T2 - Measuring Impact AB - This technical analysis explores previous and ongoing social learning efforts, best practices, challenges, and lessons in USAID as a foundation for improving the implementation and design of the Agency’s forestry and biodiversity programs. This analysis is particularly relevant as the Bureau of Economic Growth, Education and the Environment’s Office of Forestry and Biodiversity (E3/FAB) begins to develop a Cross-Mission Learning Program under the Measuring Impact initiative, a five-year activity to promote the adoption of best practices in the USAID program cycle among Missions using biodiversity funds. The findings reported in this document can help inform the overall design and structure of the Learning Program and provide insight into possible challenges and best practices. CY - Washington DC DA - 2015/05// PY - 2015 DP - Zotero SP - 40 LA - en PB - USAID ER - TY - RPRT TI - MERL Tech State of the Field - The evolution of MERL Tech AU - Raftree, Linda DA - 2020/04// PY - 2020 DP - Zotero SP - 30 LA - en PB - MERL Tech UR - https://merltech.org/resources/merl-tech-state-of-the-field-the-evolution-of-merl-tech/ ER - TY - ELEC TI - MIRADI: Adaptive Management Software for Conservation Projects DA - 2018/// PY - 2018 UR - https://www.miradi.org/ ER - TY - VIDEO TI - Miradi Measures Dashboard Demo AU - Salzer, D AB - Live demonstration (view in HD) of a new online measures dashboard site (http://miradi.sitkatech.com) that allows users of the Miradi Adaptive Management software (http://miradi.org) to share the progress they are making towards achieving their desired results DA - 2013/// PY - 2013 UR - https://www.youtube.com/watch?v=i2nrZDvVBxE ER - TY - RPRT TI - Mission-Based Monitoring, Evaluation and Learning (MEL) Platforms Assessment Report AU - Arntson, Laura AU - Giannoni, Tonya AU - Peek, Nancy AU - Saarlas, Kristin AB - In 2016, the Bureau of Policy, Planning and Learning (PPL) commissioned an internal stocktaking of USAID's mission-based MEL (Monitoring, Evaluation, and Learning) Platforms. CY - Washington DC DA - 2017/12/04/ PY - 2017 LA - en PB - USAID UR - https://usaidlearninglab.org/library/mission-based-monitoring%2C-evaluation-and-learning-platforms-assessment-report Y2 - 2019/03/12/14:30:19 ER - TY - BLOG TI - MLE or MEL in adaptive programming? AU - Pellini, Arnaldo T2 - Knowledge counts DA - 2018/06/21/ PY - 2018 UR - https://arnaldopellini.org/2018/06/21/mle-or-mel-in-adaptive-programming/ Y2 - 2018/07/17/10:57:17 ER - TY - JOUR TI - Models for Foresight Use in International Development AU - Rhydderch, Alun T2 - IDS Bulletin AB - This article sets out the components of the foresight approach that has been adopted by many governments in the developed world, and identifies elements of this 'dominant' approach that may hinder its uptake in developing countries. Instead, it suggests that a less rigid, more exploratory and normative approach may be better suited to many developing country contexts. With reference to the writings and practice of the creator of 'la prospective', Gaston Berger, it argues for an attitude that combines bold and inclusive thinking about how to create better futures with the pragmatic engagement with political and administrative systems that can help bring these about. DA - 2016/// PY - 2016 DO - 10.19088/1968-2016.153 VL - 47 IS - 4 UR - http://bulletin.ids.ac.uk/idsbo/article/view/2778/ONLINE%20ARTICLE Y2 - 2018/03/09/00:00:00 ER - TY - BLOG TI - Monitoring and evaluation: five reality checks for adaptive management AU - Pasanen, Tiina T2 - ODI Insight DA - 2017/12/14/ PY - 2017 UR - https://www.odi.org/comment/10588-monitoring-and-evaluation-five-reality-checks-adaptive-management Y2 - 2018/11/10/00:00:00 ER - TY - BLOG TI - Monitoring and evaluation for adaptive programming AU - Roche, Chris AU - Kelly, Linda T2 - Devpolicy Blog from the Development Policy Centre AB - Chris Roche and Linda Kelly with six take-aways on what is being tried and learnt in setting up monitoring and evaluation frameworks for adaptive programs. DA - 2018/09/18/T06:00:54+10:00 PY - 2018 LA - en-US UR - http://www.devpolicy.org/monitoring-and-evaluation-for-adaptive-programming-20180918/ Y2 - 2018/09/24/08:15:51 ER - TY - JOUR TI - Monitoring and evaluation for thinking and working politically AU - Aston, Thomas AU - Roche, Chris AU - Schaaf, Marta AU - Cant, Sue T2 - Evaluation AB - This article explores the challenges of monitoring and evaluating politically informed and adaptive programmes in the international development field. We assess the strengths and weaknesses of some specific evaluation methodologies which have been suggested as particularly appropriate for these kinds of programmes based on scholarly literature and the practical experience of the authors in using them. We suggest that those methods which assume generative causality are particularly well suited to the task. We also conclude that factoring in the politics of uncertainty and evidence generation and use is particularly important in order to recognize and value diverse experiential knowledge, integrate understandings of the local context, accommodate adaptation and realistically grapple with the power relations which are inherent in evaluation processes. DA - 2022/01/01/ PY - 2022 DO - 10.1177/13563890211053028 DP - SAGE Journals VL - 28 IS - 1 SP - 36 EP - 57 J2 - Evaluation LA - en SN - 1356-3890 UR - https://doi.org/10.1177/13563890211053028 Y2 - 2022/03/21/11:55:27 ER - TY - RPRT TI - Monitoring and Evaluation in the Development Sector AU - KPMG CY - Zurich DA - 2014/// PY - 2014 DP - Zotero LA - en PB - KPMG International UR - https://assets.kpmg.com/content/dam/kpmg/pdf/2014/09/2014-survey-monitoring-evaluation-v4.pdf ER - TY - RPRT TI - Monitoring and evaluation of policy influence and advocacy AU - Tsui, Josephine AU - Hearn, Simon AU - Young, John T2 - Working Paper AB - Policy influence and advocacy are increasingly regarded as a means of creating sustainable policy change in international development. It is often also seen as a difficult area to monitor and evaluate. Yet there is an increasingly rich strand of innovation in options to monitor, evaluate and learn from both the successes and failures of policy influence and advocacy interventions. This paper explores current trends in monitoring and evaluating policy influence and advocacy; discusses different theories of how policy influence happens; and presents a number of options to monitor and evaluate different aspects of advocacy interventions. Case studies describe how some organisations have used these options in practice to understand their impact and improve their advocacy strategies. DA - 2014/03// PY - 2014 SP - 90 PB - ODI SN - 395 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8928.pdf Y2 - 2018/12/10/00:00:00 ER - TY - RPRT TI - Monitoring And Evaluation Tools and Approaches to Support Adaptive Management AU - Pasanen, Tiina DA - 2019/// PY - 2019 PB - GLAM ER - TY - RPRT TI - Monitoring and learning for country-level portfolio decision-making and adaptation AU - Buffardi, Anne L AU - Mason, Paige AU - Hutchings, Claire AU - Sharp, Samuel T2 - Briefing Note AB - Most measurement and adaptive management approaches were developed for and from individual projects. This briefing aims to guide measurement and management of country-level portfolios of work. It identifies potential purposes portfolio-level analyses can fulfil, types of adaptation, and the relative role of monitoring, learning and evaluation (MEL). Drawing on reviews of practice from the United States Agency for International Development (USAID) and the United Kingdom Department for International Development (DFID), guidance notes, and experiences of members of the Global Learning for Adaptive Management (GLAM) initiative, it offers recommendations and considerations that are particularly relevant for this level of analysis and management. Key messages - Portfolio-level analyses can serve eight potential purposes, each of which answers different questions, involves adaptation at different times and levels, and requires different types of evidence. Identifying the purpose(s) and how the component parts relate to each other should guide the development of monitoring and learning systems. - Portfolios are oriented more towards breadth than depth, involve more people with different perspectives, and draw on multiple sources of evidence with potentially greater variation in quality. - In practice, four activities appear to be applied most frequently at a portfolio level: alignment of indicators and aggregation of monitoring data; synthesis of multiple sources and types of information to provide a summary of outputs, outcomes, common observations and trends; periodic review and reflection sessions; and strategic planning, design or refresh of the portfolio strategy. - The extent to which evidence-informed portfolio management is facilitating learning and adaptation has not been well documented to date, and we suggest potential indicators to do so. CY - London DA - 2019/05// PY - 2019 DP - Zotero LA - en PB - ODI/GLAM UR - https://www.odi.org/publications/11351-monitoring-and-learning-country-level-portfolio-decision-making-and-adaptation Y2 - 2019/05/31/00:00:00 ER - TY - RPRT TI - Monitoring and Result Measurement for Adaptive Programming - How to Use Data to Manage a MSD Program: Lessons from PRISMA AU - Khan, Khaled AU - Seely, Kevin AU - Ridwan, Mustika AU - Mulya, Bodhiya AB - Using monitoring data to improve interventions is harder than it seems. Decision-makers are often busy implementing activities, unclear about their roles in data collection and analysis, and uncertain what data matters most or when. PRISMA, an AUD77 million agricultural Market Systems Development (MSD) programme funded by DFAT Australia, has encountered these challenges. With the programme completing its first five year phase, this case study shares ten key lessons divided into three sections: shaping the culture, developing systems, and top management decisions. These lessons aim to help program, sector and intervention managers make better use of monitoring data to improve interventions. DA - 2019/// PY - 2019 PB - PRISMA UR - https://aip-prisma.or.id/data/public/uploaded_file/05_Monitoring%20and%20Result%20Measurement%20for%20Adaptive%20Programming%20-%20How%20to%20Use%20Data%20to%20Manage%20a%20MSD%20Program%20-%20Lessons%20from%20PRISMA.pdf Y2 - 2020/02/14/12:40:55 ER - TY - RPRT TI - Monitoring, Evaluation, Accountability and Learning in Emergencies AU - Dominique Morel AU - Hagens, Clara CY - Baltimore, USA DA - 2012/// PY - 2012 PB - Catholic Relief Services UR - https://www.crs.org/sites/default/files/tools-research/monitoring-evaluation-accountability-and-learning-in-emergencies.pdf Y2 - 2022/02/24/15:44:19 ER - TY - RPRT TI - Monitoring, Evaluation and Learning approaches in an adaptive management context AU - Tsui, Josephine T2 - Helpdesk Request DA - 2016/09// PY - 2016 SP - 15 PB - EPS PEAKS, ODI ER - TY - RPRT TI - Monitoring, Evaluation and Learning in NGO Advocacy - Findings from Comparative Policy Advocacy MEL Review Project AU - Coe, Jim AU - Majot, Juliette AB - For organizations committed to social change, advocacy often figures as a crucial strategic element. How to assess effectiveness in advocacy is, therefore, important. The usefulness of Monitoring, Evaluation and Learning (MEL) in advocacy are subject to much current debate. Advocacy staff, MEL professionals, senior managers, the funding community, and stakeholders of all kinds are searching for ways to improve practices – and thus their odds of success – in complex and contested advocacy environments. This study considers what a selection of leading advocacy organizations are doing in practice. We set out to identify existing practice and emergent trends in advocacy-related MEL practice, to explore current challenges and innovations. The study presents perceptions of how MEL contributes to advocacy effectiveness, and reviews the resources and structures dedicated to MEL. CY - London DA - 2013/02// PY - 2013 PB - ODI and Oxfam UR - https://www.alnap.org/help-library/monitoring-evaluation-and-learning-in-ngo-advocacy-findings-from-comparative-policy Y2 - 2021/03/26/10:53:04 ER - TY - RPRT TI - Monitoring for problem-solving, adaptive management, reporting and learning AU - Dominique Morel AU - Dzino-Silajdzic, Velida AU - Hagens, Clara AB - Internal and external stakeholders have different information needs over a project’s life, for purposes that include adaptive management, accountability, compliance, reporting and learning. A project’s monitoring, evaluation, accountability and learning, or MEAL, system should provide the information needed by these stakeholders at the level of statistical reliability, detail and timing appropriate to inform data use. In emergency contexts where the situation is still fluid, ‘informal monitoring’ has proved helpful to staff’s ongoing assessment of the broader environment in order to identify changes in the situation, in other actors’ responses, and in priority unmet needs that would require corresponding changes in the response.2 The same distinction between informal monitoring of possible changes in the project’s operating context—whether identified as project assumptions and risk factors or not—and formal monitoring of the activities included in the response and project indicators, is relevant for development contexts too. Informal monitoring: Ongoing assessment of changes in operating context Formal monitoring: Tracking progress against project activities and indicators Within formal monitoring, it is useful to further differentiate between light monitoring and rigorous monitoring: - Light monitoring aims to provide timely feedback on new activities (or new locations or target groups) or aspects of the project’s theory of change (activity-to-output or output-to-IR change) logic that staff are less confident about, to check for early signs that progress is being made and that assumptions are holding true while there is still ample time to make adjustments if necessary.3 - Rigorous monitoring aims to collect representative data for evidence-based project management, reporting and learning, not just at midterm but throughout project implementation. CY - Baltimore, USA DA - 2020/04// PY - 2020 PB - Catholic Relief Services UR - https://www.crs.org/sites/default/files/tools-research/monitoring_for_problem_solving_adaptive_mgt_reporting_and_learning_2020.pdf Y2 - 2022/02/24/15:44:19 ER - TY - BLOG TI - Most Significant Change AU - Better Evaluation T2 - Better Evaluation AB - The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why. The are three basic steps in using MSC: Deciding the types of stories that should be collected (stories about what - for example, about practice change or health outcomes or empowerment) Collecting the stories and determining which stories are the most significant Sharing the stories and discussion of values with stakeholders and contributors so that learning happens about what is valued. MSC is not just about collecting and reporting stories but about having processes to learn from these stories – in particular, to learn about the similarities and differences in what different groups and individuals value. It provides some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders. By itself it is not sufficient for impact evaluation as it does not provide information about the usual experience but about the extremes. If you imagine a normal distribution of outcomes for individuals then the stories often come from the extremity of positive change. It can be useful to explicitly add a process to generate and collect stories from the extremity of little or negative change. MSC can be very helpful in explaining HOW change comes about (processes and causal mechanisms) and WHEN (in what situations and contexts). It can therefore be useful to support the development of programme theory (theory of change, logic models). UR - https://www.betterevaluation.org/en/plan/approach/most_significant_change Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - MOTION HANDBOOK - Developing a transformative Theory of Change AU - Palavicino, Carla Alvial AU - Matti, Cristian AU - Witte, Jenny AB - A step-by-step guide on how to develop a Transformative Theory of Change, for innovation projects, programmes and organisations working on systems transformation. The MOTION project was initiated with one key question in mind: how can we help projects and organisations be more transformative, using the framework and concept provided by the multi-level perspective? And what kind of tools, methods and frameworks can we co-design that translate scientific concepts into practises relevant for policy practitioners? This led us into a co-creation journey during which researchers from the Transformative Innovation Policy Consortium (TIPC) and EIT Climate-KIC project partners experimented, reflected and learned from each other in building the approach that we share in this handbook. Through this journey, we gained a deeper understanding of what the portfolio approach means in a transformative system change context and which skills and competences are needed to facilitate processes of co-creation in the science-policy-practice interface. We had the opportunity to configure the key building blocks of our theoretical approach, the Transformative Outcomes, into practical insights and actions that can easily be applied by innovation organisations at many levels. This handbook is the culmination of the journey as it translates key learnings from the MOTION project into practical insights that are relevant to practitioners working on systems transformation. CY - Utrecht DA - 2022/01// PY - 2022 PB - TIPC, Utrecht University UR - https://transitionshub.climate-kic.org/publications/motion-handbook-developing-a-transformative-theory-of-change/ Y2 - 2023/01/24/09:46:49 ER - TY - RPRT TI - Multi-project programmes functions, forms and implications for evaluation and learning AU - Buffardi, Anne AU - Hearn, Simon AB - Multi-project programmes can serve different purposes. For instance, they may coordinate multiple implementing entities; standardise management and technical support; compare intervention approaches across different contexts; enhance leverage through joint action; or foster sustainability by building relationships among organisations. • At the same time, multi-project programmes are costly, potentially duplicate other mechanisms that fulfil similar functions, and can dilute focus and create confusion. • To guide decisions on what coordination, evaluation and learning mechanisms are needed, it is helpful to articulate the intended purpose of using a programme model. Identifying the purpose(s) can help staff determine what types of knowledge sharing strategies may be most useful to the programme, and how project and programme theories of change can be sequenced DA - 2015/12// PY - 2015 SP - 20 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10353.pdf Y2 - 2018/11/10/00:00:00 ER - TY - BOOK TI - Multiple case study analysis AU - Stake, Robert E. AB - Examining situational complexity is a vital part of social and behavioral science research. This engaging text provides an effective process for studying multiple cases--such as sets of teachers, staff development sessions, or clinics operating in different locations--within one complex program. The process also can be used to investigate broadly occurring phenomena without programmatic links, such as leadership or sibling rivalry. Readers learn to design, analyze, and report studies that balance common issues across the group of cases with the unique features and context of each case. Three actual case reports from a transnational early childhood program illustrate the author's approach, and helpful reproducible worksheets facilitate multicase recording and analysis. CN - LB1028 .S733 2005 CY - New York DA - 2006/// PY - 2006 DP - Library of Congress ISBN SP - 342 PB - The Guilford Press SN - 978-1-59385-248-1 978-1-59385-249-8 UR - https://www.amazon.com/Multiple-Study-Analysis-Robert-Stake/dp/1593852487 KW - Case method KW - Cross-cultural studies KW - Education KW - Europe, Eastern KW - Research Methodology KW - Step by Step (Program) ER - TY - BOOK TI - Navigating Complexity in International Development: Facilitating Sustainable Change at Scale AU - Burns, Danny AU - Worsley, Stuart AB - Governments and organizations invest huge sums of money in development interventions to explicitly address poverty and its root causes. However, a high proportion of these do not work. This is because interventions are grounded in flawed assumptions about how change happens -- change is rarely linear, yet development interventions are almost entirely based on linear planning models. Change is also characterized by unintended consequences, which are not predictable by planners and by power dynamics, which drive outcomes towards vested interests rather than real need. Development processes need to engage effectively with these sorts of complex system dynamics. This book provides a conceptual framework for this thinking, offers detail case studies of interventions which have been built on this philosophy and which demonstrate key facets of it. It articulates some clear methodological underpinnings for this work, and draws out the implications both for development, practice and practitioners. CY - Rugby DA - 2015/10/15/ PY - 2015 SP - 198 LA - English PB - Practical Action Publishing SN - 978-1-85339-852-0 ST - Navigating Complexity in International Development KW - IMPORTANT ER - TY - VIDEO TI - Navigating with Action Inquiry (Collective Leadership Animations) AU - Keira Oliver AB - "Have you ever wondered if there might be a better way to work together? Perhaps you would like to find ways to avoid habits, or patterns of behaviour, that whilst you say you don’t like them, you seem unable to escape." Written & narrated by Cathy Sharp (Research for Real) and animated by Keira Oliver (Collective Leadership for Scotland) and her son (10yr) in an animation-sprint style (created quickly, but with care), this film summarises the ideas and practices of action inquiry. It introduces a set of statements designed to stretch, challenge and encourage us to talk about these ideas to find better ways to work together. DA - 2020/05/19/ PY - 2020 DP - YouTube UR - https://www.youtube.com/watch?v=UX2MF6bW6GQ Y2 - 2023/05/22/13:42:41 ER - TY - RPRT TI - NGO-IDEAs Impact Toolbox - Participatory Monitoring of Outcome and Impact AU - Gohl, Eberhard AU - Causemann, Bernward AB - NGO-IDEAs (NGO – Impact on Development, Empowerment and Actions) NGO-IDEAs is a cooperation of about 40 non-govermental organisations (NGOs) from South Asia, East Africa and the Philippines and 14 German NGOs working in the field of development cooperation. It identifies and develops jointly with all partners, concepts and tools for NGOs in the areas of Outcome and Impact Assessment and Monitoring & Evaluation (M&E). NGO-IDEAs is further being supported by VENRO, the umbrella organisation of development non-governmental organisations in Germany as well as PARITÄT, the legal holder of the project. The Federal Ministry for Economic Cooperation and Development (BMZ) has cofinanced the project. NGO-IDEAs is not just another study evaluating the impact of NGOs’ work – it combines research & development, knowledge management, learning & training as well as advice & coaching to initiate a collective learning process for all partners involved. Additionally, NGOIDEAs intends to create a valuable resource base for use by NGOs. NGO-IDEAs aims at: • Empowering community based organisations or groups and the poor among the rural communities to use and practice impact monitoring for project management • Empowering NGOs to further improve the effectiveness, impact and sustainability of their efforts • Making social changes more visible for implementing and funding NGOs, thus improving development practice • Improving public recognition of NGOs and CBOs and their contribution to development. The NGO-IDEAs “Impact Toolbox” is to enable NGOs and grassroots organisations to monitor projects together with the so called target groups involved, in a manner that will enhance positive outcomes and impacts, and reduce negative ones. It focuses on joint setting of goals, on monitoring them and finally on taking joint decisions about the further design and direction of interventions. The instruments of the NGO-IDEAs “Impact Toolbox” are simple and participatory. Simple means: setting out from people’s knowledge and know-how, therefore, easy to understand and apply. Application can easily be fitted into the “normal” activities of the NGOs or grassroots organisations. The participatory character emerges through democratic elements promoting a “Culture of Learning” that the people can assimilate CY - Stuttgart DA - 2011/// PY - 2011 DP - Zotero LA - en PB - Impact+ UR - http://www.ngo-ideas.net/tiny_tools/index.html Y2 - 2023/07/14/00:00:00 ER - TY - JOUR TI - Nimble adaptation: Tailoring monitoring, evaluation, and learning methods to provide actionable data in complex environments AU - Serpe, Lauren AU - Ingram, Mason AU - Byom, Kate T2 - New Directions for Evaluation AB - This chapter examines good practices in implementing effective Monitoring, Evaluation, and Learning (MEL) systems within complex international development Democracy, Human Rights, and Governance (DRG) programs, which are characterized by challenges of non-linearity, limited evidence of theories of change, and contextual and politically contingent nature of outcomes. The chapter presents three cases of MEL systems in complex projects implemented by Pact across distinct and diverse operating contexts – Zimbabwe, Cambodia, and Somalia – to illustrate those projects’ MEL approaches that enabled continuous adaptation. The authors analyze the cases to respond to two questions: (1) What are the key elements of effective adaptive management-focused MEL systems in complex environments? (2) What is practical guidance for designing and enabling complexity-responsive and effective adaptive management-focused MEL systems? The case studies illustrate three key elements: (1) Information gathering that closely links context, research, and performance data; (2) Systems for reflection that offer scheduled learning moments of varying frequency and intensity, as well as multiple feedback mechanisms; and (3) Enabling structures that promote adaptive mindsets and attitudes within project teams. DA - 2022/// PY - 2022 DO - 10.1002/ev.20523 DP - Wiley Online Library VL - 2022 IS - 176 SP - 97 EP - 106 LA - en SN - 1534-875X ST - Nimble adaptation UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20523 Y2 - 2023/04/13/09:10:18 ER - TY - JOUR TI - No Mechanism Without Context: Strengthening the Analysis of Context in Realist Evaluations Using Causal Loop Diagramming AU - Renmans, Dimitri AU - Holvoet, Nathalie AU - Criel, Bart T2 - New Directions for Evaluation AB - Realist evaluation is an approach with a strong emphasis on causal mechanisms and the context in which they are triggered. However, recent reviews of published realist evaluations show that context is often understudied. This is problematic, as a thorough understanding of the relationship between context and causal mechanisms is crucial in assisting policymakers to make appropriate and targeted decisions that improve the intervention. Therefore, we set out to test whether combining realist evaluation with the “systems thinking” approach and, more specifically, causal loop diagramming, could help strengthen the analysis of context. We did this through a study of a performance-based financing (PBF) intervention in the Ugandan health care sector by the Belgian development agency, Enabel. PBF allocates funds to health workers and/or health facilities based on their performance, and introduces additional management support tools, provides extra monitoring and supervision, and promotes community participation in management issues, among other activities. In this case, we found that the proposed combined methodological approach indeed adds value to the analysis, as it leads to insights into the role played by the underlying system that otherwise may have been overlooked. Moreover, such information may provide clear directions to policymakers on how to improve the intervention in a sustainable way. Finally, causal loop diagrams help to visualize complex causal interactions and to communicate them to policymakers. DA - 2020/// PY - 2020 DO - 10.1002/ev.20424 DP - Wiley Online Library VL - 2020 IS - 167 SP - 101 EP - 114 LA - en SN - 1534-875X ST - No Mechanism Without Context UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20424 Y2 - 2021/02/18/14:26:34 ER - TY - BLOG TI - Old Wine in New Bottles? 6 ways to tell if a programme is really ‘doing development differently’ AU - Green, Duncan T2 - From Poverty to Power AB - Some top adaptive management exponents give their top tips on how to distinguish spin from reality, when looking at an avowedly AM programme DA - 2018/11/13/T07:30:53+00:00 PY - 2018 LA - en-GB ST - Old Wine in New Bottles? UR - https://oxfamblogs.org/fp2p/old-wine-in-new-bottles-6-ways-to-tell-if-a-programme-is-really-doing-development-differently/ Y2 - 2018/11/13/11:49:11 ER - TY - RPRT TI - OPM’s approach to assessing Value for Money AU - King, Julian AU - OPM AB - This document offers practical guidance for assessing value for money (VfM) in international development programmes. Since 2016, evaluation expert Julian King has been working with OPM to develop and deploy a robust approach to VfM assessment. The approach combines cutting-edge evaluation practice with concepts from economic evaluation to respond to donor requirements for accountability and good resource allocation, as well as to support reflection, learning, and adaptive management. CY - Oxford DA - 2018/02/05/ PY - 2018 LA - en PB - Oxford Policy Management UR - http://www.opml.co.uk/publications/opm%E2%80%99s-approach-assessing-value-money Y2 - 2018/03/16/09:58:46 ER - TY - ELEC TI - opm-vfm-approach-2.pdf UR - https://www.julianking.co.nz/wp-content/uploads/2023/12/opm-vfm-approach-2.pdf?utm_source=substack&utm_medium=email Y2 - 2024/02/12/14:06:34 ER - TY - RPRT TI - Opportunities for Using Complexity-Aware Approaches to Theory of Change AU - Goodier, Sarah AU - Apgar, Marina T2 - SDC-IDS Briefing Note 8 AB - The purpose of this briefing note is to review opportunities for using complexity aware approaches to Theory of Change (ToC) to inform the SDC approach. It provides an overview of complexity-aware approaches and then focuses on demonstrating how complexity thinking can support programming by building on the frameworks currently being used in the project/programme cycle management (PCM) processes. It is aimed at SDC staff, in particular Programme Officers and staff of partner organisations involved in the management of SDC interventions. CY - Brighton DA - 2018/08// PY - 2018 DP - opendocs.ids.ac.uk LA - en PB - IDS UR - https://opendocs.ids.ac.uk/opendocs/handle/123456789/14040 Y2 - 2019/03/06/11:07:33 ER - TY - ELEC TI - Organisational Performance Index AU - The Kenya Institute of Management AB - The OPI Business Excellence Model exists to help organizations improve their performance and succeed in the competitive global marketplace. We are dedicated to improving African organizations regardless of sector, size or structure. The OPI was developed in response to feedback from organization participating in Company of the Year Award (COYA) and SME of the Year Award (SMOYA), who called on the Kenya Institute of Management (KIM) to refine the assessment methodology in order to emphasize the relationship between instilling smart business processes and generating strong business results. Whilst there are numerous management tools and techniques commonly used, the OPI Excellence Model is a practical, non-prescriptive framework which provides a holistic view of the organisation and it can be used to determine how these different methods fit together and complement each other. The Model can therefore be used in conjunction with any number of these tools, based on the needs and function of the organisation, as an overarching framework for developing sustainable excellence. We also administer the Annual Company of the Year Awards (COYA) and the SME of the Year Awards (SMOYA). Our key services are to identify and recognize role-model organizations, share best management practices, and help organizations achieve best-in-class performance levels. The OPI is an excellence model that generates a rating between 1 and 10, which sets a minimum score that an organisation must attain to remain competitive. The attainment of a minimum Organisation Performance Index ( OPI) demonstrates that organisations have applied an organisational strategy that drives effectiveness and competitiveness; failure to attain the minimum OPI demonstrates the organisation has not engaged the processes required to enable world class organisational performance. DA - 2014/// PY - 2014 UR - https://www.kim.ac.ke/index.php/opi/ ER - TY - BOOK TI - Organizational Performance Index (OPI) Handbook: A practical guide to the OPI tool for practitioners and development professionals AU - Pact AB - This handbook is intended to help development practitioners understand how to use and what can be achieved by Pact’s Organizational Performance Index (OPI) tool. This document will provide practitioners and managers with information on how the Pact OPI works and how best to incorporate it in their country strategy or program. CY - Washington DC, United States DA - 2015/// PY - 2015 PB - Pact UR - http://hkdepo.am/up/docs/OPIhandbook_pact.pdf Y2 - 2018/09/22/00:00:00 ER - TY - BLOG TI - Outcome Harvesting AU - Wilson-Grau, R. T2 - Better Evaluation AB - Outcome Harvesting collects (“harvests”) evidence of what has changed (“outcomes”) and, then, working backwards, determines whether and how an intervention has contributed to these changes. Outcome Harvesting has proven to be especially useful in complex situations when it is not possible to define concretely most of what an intervention aims to achieve, or even, what specific actions will be taken over a multi-year period. DA - 2015/// PY - 2015 UR - https://www.betterevaluation.org/en/plan/approach/outcome_harvesting Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Outcome Harvesting AU - Wilson-Grau, Ricardo AU - Britt, Heather AB - Outcome Harvesting was developed by Ricardo Wilson-Grau with colleagues Barbara Klugman, Claudia Fontes, David Wilson-Sánchez, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz, and Wolfgang Richert. Over the past 8 years, Outcome Harvesting and has been used to monitor and evaluate the achievements of hundreds of networks, non-governmental organizations, research centers, think tanks, and communitybased organizations around the world. This brief is intended to introduce the concepts and approach used in Outcome Harvesting to grant makers, managers, and evaluators, with the hope that it may inspire them to learn more about the method and apply it to appropriate contexts. Thus, it is not a comprehensive guide to or explanation of the method, but an introduction to allow evaluators and decision makers to determine if the method is appropriate for their evaluation needs. Where possible, we have included examples to illustrate how Outcome Harvesting is applied to real situations. For each case story, organizations were asked to provide a description of the outcome and a summary of the role played by the organization. Sometimes they added other information such as the outcome’s significance. Some details and identifiers were redacted for confidentiality purposes. A draft of this brief was graciously commented on by Bob Williams, Fred Carden, Sarah Earl, Richard Hummelbrunner and Terry Smutylo. The final text is, of course, the sole responsibility of the authors and editor. DA - 2012/// PY - 2012 PB - Ford Foundation UR - https://usaidlearninglab.org/sites/default/files/resource/files/outome_harvesting_brief_final_2012-05-2-1.pdf Y2 - 2018/09/22/00:00:00 ER - TY - ELEC TI - Outcome Harvesting. Understanding social change results in complex circumstances AU - outcomeharvesting.net AB - This website is a source of applications, events and resources to support the development of a community of practitioners of Outcome Harvesting. BetterEvaluation is one of three sister sites of OutcomeHarvesting.net DA - 2018/// PY - 2018 UR - http://outcomeharvesting.net/ ER - TY - BLOG TI - Outcome Mapping AU - Better Evaluation T2 - Better Evaluation AB - Outcome mapping (OM) is a methodology for planning, monitoring and evaluating development initiatives in order to bring about sustainable social change. As the name suggests, its niche is understanding outcomes; the so-called ‘missing-middle’ or ‘black box’ of results that emerge downstream from the initiative’s activities but upstream from longer-term economic, environmental, political or demographic changes. At the planning stage, the process of outcome mapping helps a project team or program be specific about the actors it intends to target, the changes it hopes to see and the strategies appropriate to achieve these. For ongoing monitoring, OM provides a set of tools to design and gather information on the results of the change process, measured in terms of the changes in behaviour, actions or relationships that can be influenced by the team or program. DA - 2013/10// PY - 2013 UR - https://www.betterevaluation.org/en/plan/approach/outcome_mapping Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Outcome mapping: A method for tracking behavioural changes in development programs AU - Smutylo, Terry T2 - ILAC Brief AB - This guide published by the Institutional Learning and Change (ILAC) Initiative provides a detailed overview of using outcome mapping as an evaluation tool. Contents Expressing results as changes in behaviour Outcome mapping terms The three stages of outcome mapping Figure 1. The three stages and twelve steps of outcome mapping Stage 1. Intentional design Figure 2. The four basic questions of the intentional design stage Stage 2. Outcome and performance monitoring Stage 3. Evaluation planning Using outcome mapping The Ceja Andina Project The Agro-industry and Market Development project for Arracacha DA - 2005/// PY - 2005 SP - 4 SN - 7 UR - https://www.outcomemapping.ca/download/csette_en_ILAC_Brief07_mapping.pdf Y2 - 2018/10/18/00:00:00 ER - TY - BOOK TI - Outcome Mapping: Building Learning and Reflection into Development Programs AU - Earl, Sarah AU - Carden, Fred AU - Smutylo, Terry AU - Patten, Michael Quinn AB - "Outcome Mapping provides not only a guide to essential evaluation map-making, but also a guide to learning and increased effectiveness, and affirmation that being attentive along the journey is as important as, and critical to, arriving at a destination." - Michael Quinn PattonMore and more, development organizations are under pressure to demonstrate that their programs result in significant and lasting changes in the well-being of their intended beneficiaries. However, such "impacts" are often the product of a confluence of events for which no single agency or group of agencies can realistically claim full credit. As a result, assessing development impacts is problematic, yet many organizations continue to struggle to measure results far beyond the reach of their programs. "Outcome Mapping" recognizes that development is essentially about people relating to each other and their environment. The originality of this approach lies in its shift away from assessing the products of a program to focus on changes in behaviour, relationships, actions, and activities in the people, groups, and organizations it works with directly. In doing so, "Outcome Mapping" debunks many of the myths about measuring impact. It will help a program be specific about the actors it targets, the changes it expects to see, and the strategies it employs and, as a result, be more effective in terms of the results it achieves. This publication explains the various steps in the outcome mapping approach and provides detailed information on workshop design and facilitation. It includes numerous worksheets and examples. CY - Ottawa DA - 2001/// PY - 2001 DP - Amazon LA - English PB - IDRC Books SN - 978-0-88936-959-7 ST - Outcome Mapping ER - TY - BLOG TI - Outcome Mapping FAQs AU - Outcome Mapping Learning Community T2 - Outcome Mapping AB - FAQ resource produced by IDRC's Evaluation Unit, with contributions from Kaia Ambrose, Sarah, Earl, Jan Van Ongevalle and Julius Nyangaga. We've added it here to answer some of your OM related questions. DA - 2018/10/18/ PY - 2018 UR - https://www.outcomemapping.ca/about/faqs.php ER - TY - JOUR TI - Outcome mapping for health system integration AU - Tsasis, Peter AU - Evans, Jenna AU - Forrest, David AU - Jones, Richard Keith T2 - Journal of Multidisciplinary Healthcare AB - Health systems around the world are implementing integrated care strategies to improve quality, reduce or maintain costs, and improve the patient experience. Yet few practical tools exist to aid leaders and managers in building the prerequisites to integrated care, namely a shared vision, clear roles and responsibilities, and a common understanding of how the vision will be realized. Outcome mapping may facilitate stakeholder alignment on the vision, roles, and processes of integrated care delivery via participative and focused dialogue among diverse stakeholders on desired outcomes and enabling actions. In this paper, we describe an outcome-mapping exercise we conducted at a Local Health Integration Network in Ontario, Canada, using consensus development conferences. Our preliminary findings suggest that outcome mapping may help stakeholders make sense of a complex system and foster collaborative capital, a resource that can support information sharing, trust, and coordinated change toward integration across organizational and professional boundaries. Drawing from the theoretical perspectives of complex adaptive systems and collaborative capital, we also outline recommendations for future outcome-mapping exercises. In particular, we emphasize the potential for outcome mapping to be used as a tool not only for identifying and linking strategic outcomes and actions, but also for studying the boundaries, gaps, and ties that characterize social networks across the continuum of care. DA - 2013/03// PY - 2013 DO - 10.2147/JMDH.S41575 VL - 6 SP - 99 EP - 107 LA - en SN - 1178-2390 UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3603332/ Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - Outcome mapping: learning brief AU - Buell, Stephanie AU - Malallah, Haneen AU - Mason, Paige T2 - Briefing Paper AB - Adaptive programmes recognise that certain changes, particularly in behaviours, are complex, non-linear and difficult to measure. This briefing note explores the use of outcome mapping (OM) as a monitoring, evaluation and learning (MEL) approach to track behavioural change and inform adaptation for two programmes: the Pathways to Resilience in Semi-arid Economies (PRISE) research consortium and the Accountability in Tanzania programme (AcT I and AcT II). It discusses the implementation of OM, the ways in which it has enabled adaptation and enabling contexts in order to identify key considerations for MEL specialists and programme managers as they determine whether OM may be the right fit, and how best to use the approach. Key messages OM has a number of different benefits as a MEL approach, including unpacking different uses of information at different levels of programme implementation; helping to develop a common language around progress markers; and going beyond monitoring to inform adaptation throughout implementation. These benefits are important aspects of monitoring, evaluation and learning for adaptive management (MEL4AM), as they provide richer evidence for decision-making at a frequency that could mean real-time learning and change. OM works best when it is embedded throughout the organisation, and as part of programme and organisational culture, rather than tasked to a MEL unit or individual. DA - 2020/// PY - 2020 LA - en PB - ODI/GLAM ST - Outcome mapping UR - https://www.odi.org/publications/17354-outcome-mapping-learning-brief Y2 - 2020/10/14/13:32:30 ER - TY - RPRT TI - Outcome Measurement in Local Governance Programmes: a Power Dimension AU - McGee, Rosemary AU - Pettit, Jethro T2 - WIP Paper CY - Brighton DA - 2013/05// PY - 2013 DP - Google Scholar PB - IDS UR - http://www.ids.ac.uk/publication/outcome-measurement-in-local-governance-programmes-a-power-dimension Y2 - 2017/06/28/10:55:37 ER - TY - RPRT TI - Overview for Social Sector Leaders: Assessing Innovation Potential for Social Impact AU - Global Knowledge Initiative AB - We aim to provide decision makers with greater insight and confidence into the process of assessing innovation impact potential. Rather than considering the role innovation plays after an investment is made, or based on historic evaluations of how innovation has or has not delivered solutions to a problem, this approach is forward-looking. This customizable toolset assesses the future impact that innovation can deliver in a system to tackle particularly complex problems DA - 2016/04// PY - 2016 SP - 30 UR - http://globalknowledgeinitiative.org/wp-content/uploads/2016/10/AIIP-Toolset-Overview-With-Tools_GKI-FINAL.pdf Y2 - 2018/03/09/00:00:00 ER - TY - ELEC TI - Pando AU - Root Change T2 - Root Change AB - Pando is a platform that gives organizations a way to visualize, learn from, and engage with the social systems in which they work. We have designed Pando to help users build trust, strengthen relationships, and work together to achieve greater social impact. Grounded in social network analysis, Pando allows users to collect and visualize organizations and their relationships in real-time on simple and easy-to-use relationship maps. Relationship maps are managed by map administrators. These are actors interested in learning about a particular system and making the tool available to those working in the system. Pando is integrated with Keystone Accountability’s Feedback Commons, an online tool that allows map administrators to collect and analyze qualitative feedback about levels of trust and relationship quality among map participants. Key features of Pando include relationship mapping, feedback surveys, and dashboards. DA - 2020/// PY - 2020 UR - https://mypando.org/index Y2 - 2020/10/15/11:19:03 ER - TY - JOUR TI - Paradoxes and partnerships: a study of knowledge exploration and exploitation in international development programmes AU - Lannon, John AU - Walsh, John N. T2 - Journal of Knowledge Management AB - Purpose – This paper aims to look at how organisational partnerships balance knowledge exploration and exploitation in contexts that are rife with paradoxes. It draws on paradox theory to examine the partnership’s response to the explore-exploit relationship. DA - 2019/08/28/ PY - 2019 DO - 10.1108/JKM-09-2018-0605 DP - DOI.org (Crossref) VL - ahead-of-print IS - ahead-of-print J2 - JKM LA - en SN - 1367-3270, 1367-3270 ST - Paradoxes and partnerships UR - https://www.emerald.com/insight/content/doi/10.1108/JKM-09-2018-0605/full/html Y2 - 2019/09/17/09:41:04 ER - TY - ELEC TI - ParEvo - Web-assisted participatory evolution of scenarios AU - Davies, Rick AB - ParEvo is a method of exploring alternative futures or histories, using a participatory evolutionary process (hence ParEvo). The process is designed to be used by multiple people, to produce a branching structure of storylines about what did, or could, happen. Participants are anonymous and can choose the extent to which they collaborate with others. Participants are also able to tag, comment on, and evaluate the storylines that are generated. Completed exercises can be analysed using downloaded data describing the content of storylines and the structure of people’s participation. Outcomes can be both cognitive (e.g. how we think about the future) and behavioural (e.g. how we respond to the future). DA - 2020/// PY - 2020 UR - https://parevo.org/ Y2 - 2022/07/01/08:46:16 ER - TY - RPRT TI - Participation by All - The keys to sustainability of a CRS food security project in Niger AU - Cekan, Jindra AU - Kagendo, Rutere AU - Towns, Alexandra AB - This project that ran from 2006-2012 in Niger and was implemented by three NGOs: CRS, Cooperative for Assistance and Relief Everywhere (CARE), and Helen Keller International (HKI) under the direction of United States Agency for International Development (USAID) Office of Food for Peace (FFP) as a multi-year assistance program (MYAP) to support food security activities in the Dosso, Tahoua, and Zinder regions. PROSAN focused on increasing agricultural production and agro-enterprise, improving household health and nutrition status, reinforcing the capacities of health agents, and enhancing community resiliency. Here are the highlights from the report which itself is an excerpt from a longer analysis we did. Also please note one Annex highlights the similarlties/ differences we found to USAID/ FFP’s 4 elements of sustainability: AIM, METHODS, AND RESEARCH QUESTIONS The aim of this sustainability evaluation was to explore perceptions of sustainability from Nigeriens involved in PROSAN, former CRS staff and donors. It focused on evaluating participants’ adherence to project outcomes and their creation of new innovations. It also evaluated partners’ involvement in sustaining project outcomes. This evaluation used qualitative and quantitative methods including community mapping, focus group discussions, beneficiary interviews, and key stakeholder interviews. The evaluation was carried out in six communities in the Dosso region, with more than 500 interviewees, focusing on the following research questions: Sustainability of activities and groups: Are the communities sustaining the activities three to five years after the end of the project? What can we learn from the communities and their post-project implementation partners? Spread and unexpected outcomes: If the project was considered a success in the eyes of the community, how well did it spread? Fostering Sustainability: What are the long-term prospects for continued sustainability? FINDINGS Three years after PROSAN’s conclusion, the project was considered a success by community members, national partners, the implementer (CRS), and donor (USAID) staff. The main findings include: 1. SUSTAINABILITY OF ACTIVITIES AND GROUPS Eighty percent (80%)[*] of all activities were reported to have become self-sustained and community innovations have emerged: On average, households reported moving from being food secure for 3-6 months per year during PROSAN to 8-12 months at the time of this evaluation, which is a remarkable impact [1]. CRS_Niger_PROSAN_Sustainability_Evaluation_pdf [1] Women reported greater income through the increase in sales of food that was produced and processed due to the grain mills [1]. Respondents also reported improved household health, hygiene, and nutrition, with 91% of survey respondents indicating that their health and sense of well being had improved, especially through the efforts of the health posts and clinics that CRS helped build and the government of Niger’s efforts in sustaining them with resources and staff [1]. Community groups/committees have continued and are well-supported by NGO partners: 81% of the committees set up by PROSAN were functioning at the time of this evaluation, with many participants discussing ways to sustain best practices within their communities, and members still receiving regular trainings or updates [1]. Several new and refresher trainings come through national partners, NGOs, and new channels such as radio programs [1]. Some new NGOs and international organizations have built upon PROSAN’s success, for instance, by using land previously managed by PROSAN for a new vegetable gardening training program, building hygiene programs on past health awareness efforts, or extending agricultural credit for further inputs [1]. Twenty percent (20%) of implemented activities were not sustained or have stagnated: While hygiene practices were sustained by households and there was widespread latrine construction, sanitation was poor in the villages, and most latrines had fallen into disrepair [1]. Fewer than 50% of women reported practicing exclusive breastfeeding for children less than six months of age [1]. While almost half of all health committees no longer exist, new health clinics staff have replaced some of the work of the committees with health and agricultural promotion messages now being sent via radio, television, and cell phones [1]. Literacy training and theater groups have completely ceased [1]. With the exception of the Système Communautaire d’Alerte Précoce-Réponses aux Urgences’ (SCAP-RU) SCAP-RU early warning system which has expanded, other resilience activities such as roadwork and caring for the environment are a lesser priority due in part to the lack of food and cash-incentives to continue doing them [1]. 2. SPREAD AND UNEXPECTED OUTCOMES New innovations and ceased activities reflected the project’s legacy: Community innovations have emerged such as collective funds paying for cleaners of the new health center, community-imposed sanctions for births occurring outside of the health centers, and the monitoring of savings from well water sales. National partners have praised the project, with many lamenting its withdrawal. One non-PROSAN village told an Agriculture Ministry staff and potential NGO partner that “No one should bring a program here unless it is like PROSAN.” PROSAN-trained masons, well repair technicians, and village youth have learned land recuperation techniques (zai holes, bunds and demi-lunes) that helped generate income beyond project communities. Project activities that received free inputs have largely stopped being implemented once the incentives were withdrawn such as Food for Training (FFT), Food for Work (FFW), or Cash for Work (CFW) (e.g. literacy, seedlings, latrines, theater etc.); nonetheless the inputs were highly valued and have continued to support agriculture and health (carts, bicycles). 3. FOSTERING SUSTAINABILITY The following areas were identified as potential barriers to sustainability that could be systematically explored in other projects: Although most committees are still functioning, there are no processes in place to engage and train youth and new inhabitants of the villages. While village communities have been maintained, there is an increasing lack of ministry resources (e.g., staff, transportation, and communications) to take the place of NGOs like CRS after a program ends. There is little management of knowledge around project data, which is further exacerbated by staff changes in NGOs, government ministries, and donors. Project data (proposal content, monitoring data, evaluation results, participant lists, partner names, and exit agreements) must be managed ethically, locally and be held online, accessible for future projects to use and for villages to conduct self-evaluations. CY - Baltimore, USA DA - 2016/// PY - 2016 PB - Valuing Voices UR - https://www.crs.org/sites/default/files/tools-research/participation-by-all.pdf Y2 - 2023/08/14/07:14:53 ER - TY - JOUR TI - Participatory action research AU - Cornish, Flora AU - Breton, Nancy AU - Moreno-Tabarez, Ulises AU - Delgado, Jenna AU - Rua, Mohi AU - de-Graft Aikins, Ama AU - Hodgetts, Darrin T2 - Nature Reviews Methods Primers AB - Participatory action research (PAR) is an approach to research that prioritizes the value of experiential knowledge for tackling problems caused by unequal and harmful social systems, and for envisioning and implementing alternatives. PAR involves the participation and leadership of those people experiencing issues, who take action to produce emancipatory social change, through conducting systematic research to generate new knowledge. This Primer sets out key considerations for the design of a PAR project. The core of the Primer introduces six building blocks for PAR project design: building relationships; establishing working practices; establishing a common understanding of the issue; observing, gathering and generating materials; collaborative analysis; and planning and taking action. We discuss key challenges faced by PAR projects, namely, mismatches with institutional research infrastructure; risks of co-option; power inequalities; and the decentralizing of control. To counter such challenges, PAR researchers may build PAR-friendly networks of people and infrastructures; cultivate a critical community to hold them to account; use critical reflexivity; redistribute powers; and learn to trust the process. PAR’s societal contribution and methodological development, we argue, can best be advanced by engaging with contemporary social movements that demand the redressingl of inequities and the recognition of situated expertise. DA - 2023/04/27/ PY - 2023 DO - 10.1038/s43586-023-00214-1 DP - www.nature.com VL - 3 IS - 1 SP - 1 EP - 14 J2 - Nat Rev Methods Primers LA - en SN - 2662-8449 UR - https://www.nature.com/articles/s43586-023-00214-1 Y2 - 2023/10/06/13:02:25 ER - TY - RPRT TI - Participatory Approaches AU - Guijt, Irene T2 - Methodological Briefs: Impact Evaluation AB - This guide, written by Irene Guijt for UNICEF, looks at the use of participatory approaches in impact evaluation. Using participatory approaches means involving stakeholders, particularly those affected by intervention, in the evaluation process. This includes involvement in the design, data collection, analysis, reporting, and management of the study. Excerpt "By asking the question, ‘Who should be involved, why and how?’ for each step of an impact evaluation, an appropriate and context-specific participatory approach can be developed. Managers of UNICEF evaluations must recognize that being clear about the purpose of participatory approaches in an impact evaluation is an essential first step towards managing expectations and guiding implementation. Is the purpose to ensure that the voices of those whose lives should have been improved by the programme or policy are central to the findings? Is it to ensure a relevant evaluation focus? Is it to hear people’s own versions of change rather than obtain an external evaluator’s set of indicators? Is it to build ownership of the UNICEF programme? These, and other considerations, would lead to different forms of participation by different combinations of stakeholders in the impact evaluation." CY - Florence DA - 2014/// PY - 2014 SP - 23 PB - UNICEF SN - 5 UR - https://www.unicef-irc.org/publications/pdf/brief_5_participatoryapproaches_eng.pdf Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Participatory approaches: a facilitator's guide AU - VSO AB - This book provides a set of guidelines for people who will be involved in participatory processes and projects with specific design focus on VSO (Voluntary Services Overseas) volunteers. It looks at appropriate levels of participation; pitfalls of participatory approaches (PA); best practice in facilitation; and tools for participation. The guide is organised into three parts (I) Principles, (II) Methods and (III) Toolkit. Part I gives a background to PA with a comparative analysis of PA in relation to top-down approaches, and within the range of PAs; looks at the role PA in VSO; discusses how to facilitate participatory processes with multiple stakeholders; presents a framework for PA on different levels of participation; and examines key facilitation skills needed to support participatory activities. Part II collates a range of participatory methods that have been used successfully in the field by VSO volunteers. Methods are categorised according to this suitability for use at different stages of a project process. Examples are also given of methods that can be used for specific purposes, such as participatory organisational appraisal and gender/diversity analysis. Part III gives tips on how to choose the most appropriate tool and how to organise participatory workshops and small group activities. It also systematically records a range of tools used by development workers all over the world with reference to what tool is appropriate in what situation. A profile of each tool includes guidelines on its purpose, potential applications and variations, as well as possible pitfalls. Illustrative case studies taken form real experiences of development workers in the field are also included. CY - London DA - 2004/// PY - 2004 PB - VSO UR - https://www.participatorymethods.org/resource/participatory-approaches-facilitators-guide Y2 - 2023/10/17/13:38:26 ER - TY - BLOG TI - Participatory Evaluation AU - Better Evaluation T2 - Better Wvaluation AB - Participatory evaluation is an approach that involves the stakeholders of a programme or policy in the evaluation process. This involvement can occur at any stage of the evaluation process, from the evaluation design to the data collection and analysis and the reporting of the study. A participatory approach can be taken with any impact evaluation design, and with quantitative and qualitative data. However, the type and level of stakeholder involvement will necessarily vary between different types, for example between a local level impact evaluation and an evaluation of policy changes (Gujit 2014, p.1). It is important to consider the purpose of involving stakeholders, and which stakeholders should be involved how, in order to maximise the effectiveness of the approach. UR - https://www.betterevaluation.org/en/plan/approach/participatory_evaluation Y2 - 2018/10/18/00:00:00 ER - TY - CONF TI - Participatory evaluation design and sense-making - making meaning through co-constructed lenses AU - King, Julian AU - Elliott, Kerry AU - Hollingsworth, Hilary T2 - Australian Evaluation Society International Evaluation Conference AB - Based on a presentation at Australian Evaluation Society International Evaluation Conference, Brisbane, 28 September 2023. The importance of participatory approaches to evaluation is well-understood. Involving the right people and balancing voices at the stakeholder table can help to make evaluations more valid, credible, useful, and actually used. Not all evaluations, however, are done in participatory ways. This presentation shares the experience of three evaluators in two recent projects with clients and stakeholders who were unaccustomed to participatory evaluation. In both cases, new ways of working were introduced, requiring participants to come into the evaluation tent, collaborate and learn with us. Engagement processes deliberately focused on incrementally building deep stakeholder input into evaluation design - including key evaluation questions, context analysis, program theory, rubrics, methods of data gathering, and reporting formats. The resulting evaluation frameworks represent an agreed, shared set of lenses for making sense of the evidence. Through this process, relationships and trust were cemented, stakeholders understood the evaluation process and endorsed the resulting frameworks. The client and sector have become more interested, more engaged stakeholders and more savvy evaluation participants and consumers. As the evaluations progress, the evaluation design builds in ongoing stakeholder involvement in reviewing emergent footprints of system change and making evaluative judgements. Participatory approaches are many, varied, and flexible. We will share our reflections and generalisable learning from the participatory approaches we applied in these evaluations. We will invite the audience to reflect on their own practices and experiences, and provoke them to consider whether participatory evaluation is 'just' a methodological option or a foundational element of good evaluation. C1 - Brisbane C3 - Evaluation and Value for Investment DA - 2023/09/28/ PY - 2023 UR - https://juliankingnz.substack.com/p/making-meaning-through-co-constructed?publication_id=1205622&utm_campaign=email-post-title&r=qn2r Y2 - 2023/09/28/10:10:21 ER - TY - JOUR TI - Participatory evaluation. What is it? Why do it? What are the challenges? AU - Zukoski, A. AU - Luluquisen, Mia T2 - Community-based public health policy & practice AB - Yogi Berra was right when he remarked, “If you don’t know where you’re going, you’ll wind up somewhere else.” When we collectively apply our hopes and energies to improving our communities, how do we know if we’re making the right choices along the way? How will we notice when we are spinning our wheels and what changes to make? How will we know that we are making a difference? And how will we know what success looks like to everyone in the group? By taking a community-based public health approach to our work, we create an opportunity to engage in a particular type of evaluation—participatory evaluation – that can help answer those questions. For those groups that are interested in this approach, this policy brief discusses the key concepts of participatory evaluation and some tips for applying it. We also present some real-life examples from two evaluators who work with grantees of the Partnership for the Public’s Health Initiative. DA - 2002/// PY - 2002 DP - Semantic Scholar ST - Participatory evaluation. What is it? ER - TY - RPRT TI - Participatory Monitoring & Evaluation: Learning from change AU - Guijt, Irene AU - Gaventa, John T2 - IDS Policy Briefing AB - Development organisations need to know how effective their efforts have been. But who should make these judgements, and on what basis? Usually it is outside experts who take charge. Participatory monitoring and evaluation (PM&E) is a different approach which involves local people, development agencies, and policy makers deciding together how progress should be measured, and results acted upon. It can reveal valuable lessons and improve accountability. However, it is a challenging process for all concerned since it encourages people to examine their assumptions about what constitutes progress, and to face up to the contradictions and conflicts that can emerge. DA - 1998/// PY - 1998 SP - 6 PB - IDS SN - 12 UR - http://www.ids.ac.uk/files/dmfile/PB12.pdf Y2 - 2018/10/19/00:00:00 ER - TY - BLOG TI - Participatory Rural Appraisal AU - Better Evaluation T2 - Better Evaluation AB - "Participatory Rural Appraisal (PRA) recently renamed Participatory Learning for Action (PLA), is a methodological approach that is used to enable farmers to analyse their own situation and to develop a common perspective on natural resource management and agriculture at village level. PRA is an assessment and learning process that empowers farmers to create the information base they need for participatory planning and action. Outsiders contribute facilitation skills and external information and opinions. Many different tools have been developed for use in PRA. There are four main classes: tools used in group and team dynamics; tools for sampling; options for interviews and dialogue; and options for visualisation and preparing diagrams. Most countries have had some experience with PRA and local publications are available. IIED regularly reports on new developments in its PLA notes (Pretty et al 1995)." (Bie, 1998) UR - https://www.betterevaluation.org/en/approach/PRA Y2 - 2018/10/19/00:00:00 ER - TY - JOUR TI - Participatory Systemic Inquiry AU - Burns, Danny AB - This article explores Participatory Systemic Inquiry processes through two examples of practise. The first is about embedding public engagement in UK higher education, the second is about water infrastructure development and local capacity development in small towns situated around Lake Victoria. These examples illustrate why it is necessary to understand the wider systemic dynamics within which issues are situated, and how this helps to identify workable and sustainable solutions to problems. It describes the learning architectures which were constructed to hold the local and thematic inquiries and then to extend them. It also demonstrates the methods which operationalised these processes and explores some of the methodological differences between this approach and other approaches to qualitative and participatory research. DA - 2012/05// PY - 2012 DO - 10.1111/j.1759-5436.2012.00325.x DP - opendocs.ids.ac.uk LA - en SN - 1759-5436 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/7488 Y2 - 2023/11/13/14:57:26 ER - TY - RPRT TI - Peacebuilding design, monitoring, and evaluation: A Training Package for participants and trainers at intermediate to advanced levels AU - Ernstorfer, Anita AU - Barnard-Webster, Kiely AB - This training package includes 7 Training Modules and a set of Annexes (Annexes A-O). The Training Modules build on each other and should ideally be used in a sequenced way in a training setting. However, for groups with specific training needs around particular areas, modules can also be used individually, but need to be tailored by the trainers and facilitators to meet the needs of specific audiences. The annexes provide worksheets and hand-outs that can be used as resources during the training for specific modules and exercises. CY - New York DA - 2019/02// PY - 2019 PB - Carnegie Corporation UR - https://www.cdacollaborative.org/wp-content/uploads/2019/02/PB-DME-Training-Package-final.pdf Y2 - 2022/06/17/13:12:01 ER - TY - RPRT TI - Peer reviews - Guidance for facilitators and participants AU - Wadley, Ian T2 - Mediation Practice Series AB - Essential points for practitioners and donors • Mediation offers a cost-effective and proven method for resolving armed conflict. Between 1985 and 2015, 75 per cent of armed conflicts in the world were resolved through agreement rather than by force. In most cases these processes will have involved third party facilitation or support. • Professional mediators understand the high stakes involved in their work to prevent, mitigate and resolve armed conflict. In addition, they and their financial supporters are increasingly required to demonstrate ‘value-for-money’ to ensure continued funding. • However, traditional monitoring and evaluation (M&E) methods are not well suited to this task, typically imposing artificially linear project models on a dynamic conflict situation, as well as compliance reporting that moves attention away from real value. • Traditional M&E methods tend to focus on documenting the past and generating vast amounts of data, rather than enabling timely adaptation of the project in the present. • Traditional M&E approaches rely heavily on external evaluation consultants. Even in the best of cases this may interfere with the mediation process and impose a heavy time burden on the project team, leading to low acceptance of traditional M&E approaches by mediation practitioners. • In contrast, an ideal M&E approach for mediation should deliver useful insights in even the most dynamic and sensitive mediation environments, impose a light reporting burden, and be readily accepted by mediation teams. It should protect discretion and trust, enable rapid adaptation, and also provide some assurance that donor funds are being well spent. CY - Geneve DA - 2021/03// PY - 2021 PB - Centre for Humanitarian Dialogue UR - https://www.hdcentre.org/wp-content/uploads/2017/12/HDC_MPS7_EN-REV2-WEB.pdf Y2 - 2023/08/15/07:40:00 ER - TY - ENCYC TI - Performance Story AU - Dart, J. AU - Mayne, J. T2 - Encyclopedia of Evaluation AB - This is a concise description of ‘Performance Story’ written by Jess Dart and John Mayne for the Sage ‘Encyclopaedia of Evaluation’. DA - 2005/// PY - 2005 SP - 307 EP - 309 PB - Sage Publications UR - http://methods.sagepub.com/reference/encyclopedia-of-evaluation/n410.xml Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - PLA Notes 32: Participation, Literacy and Empowerment AU - PLA T2 - Participatory Learning and Action Notes CY - London DA - 1998/06// PY - 1998 DP - pubs.iied.org PB - IIED ST - PLA Notes 32 UR - http://pubs.iied.org/6137IIED/ Y2 - 2017/07/11/12:33:22 ER - TY - RPRT TI - Planning & Navigating Social Change - Tools for Pacific voyagers AU - Orr, Douglas Epeli AU - Cavatore, Maria AU - Aston, Tom AB - Those working to advance social change in the Pacific must understand, work with and respond to the complex and changing relationships and dynamics of power that exist within such networks, and situate their work in the context of decolonisation and self-determination. But the approaches typically used by NGOs to plan our projects and programs have failed to give due attention to these dynamics. The result is that our projects fall short of achieving their promise, despite being technically sound and logical. In this guide, we aim to provide a process and tools that prioritise and draw out the rich, often implicit, knowledge that Pacific Islanders have about our contexts to plan and manage social change initiatives in a manner that values and responds to this ocean of relationships. Drawing from these relationship-based and voyaging traditions, we emphasise the need to: • frame change in terms of change in people; e.g. change in the situation of groups of people and change in terms of the actions of people; • identify a course towards a particular ‘change destination’; • identify the signs to read that will help you determine where you are on that course; • be constantly attentive to the context as you make the journey, and • be ready to adjust your course in response to changes in your context. To support this way of working, we have selected a number of tools from approaches that we believe fit. In particular, we have drawn from the following approaches: • Thinking and Working Politically and other ‘politically aware’ approaches that recognise that the success of a development initiative will be determined by the interplay of relationships of power within a particular context, not just by having the right technical solution; • Outcome mapping because of its focus on people and relationships, and on identifying and supporting needed changes in the behaviour of specific actors in relation to others. This is key, because it’s individuals who deliver services, individuals who decide whether to allocate budget and resources to address a problem, and individuals who make the choice to believe whether a service is relevant to solving their issues; • Adaptive management because of its recognition that development initiatives take place within complex and changing systems and need to adjust to these and that this requires ongoing learning and adaptation. CY - Fiji DA - 2019/// PY - 2019 PB - Oxfam UR - https://www.pasifikarising.org/wp-content/uploads/2019/09/Navigating-Social-Change_FINAL_web_250919.pdf Y2 - 2020/02/06/09:07:46 ER - TY - RPRT TI - Political Economy Analysis – Guidance for legal technical assistance AU - Denney, Lisa AU - Domingo, Pilar AB - Political economy analysis is about understanding how change in relation to rule of law and international development is embedded within and shaped by political and economic relations that interact and are particular to each context. These political economy dynamics determine the distribution of power and resources within any given society and must be taken into account when attempting to achieve change. Political economy analysis is important in the justice sector because rule of law and dispute resolution processes reflect the rules of the game about who wins and who loses in the allocation of power and resources. Using political economy analysis in legal technical assistance can help to ensure more relevant and effective assistance that sustainably addresses concrete problems. It can also help to avoid the risk that poorly targeted assistance might inadvertently exacerbate existing power structures in a way that was not intended. DA - 2017/01// PY - 2017 SP - 24 M3 - Guidance note PB - ROLE UK UR - http://www.roleuk.org.uk/sites/default/files/files/PEA%20-%20Guidance%20for%20legal%20technical%20assistance.pdf Y2 - 2018/10/11/00:00:00 ER - TY - JOUR TI - Portfolios of Agile Projects: A Complex Adaptive Systems’ Agent Perspective AU - Sweetman, Roger AU - Conboy, Kieran T2 - Project Management Journal AB - While agile approaches can be extremely effective at a project level, they can impose significant complexity and a need for adaptiveness at the project portfolio level. While this has proven to be highly problematic, there is little research on how to manage a set of agile projects at the project portfolio level. What limited research that does exist often assumes that portfolio-level agility can be achieved by simply scaling project level agile approaches such as Scrum. This study uses a complex adaptive systems lens, focusing specifically on the properties of projects as agents in a complex adaptive portfolio to critically appraise current thinking on portfolio management in an agile context. We then draw on a set of 30 expert interviews to develop 16 complex adaptive systems (CAS)-based propositions as to how portfolios of agile projects can be managed effectively. We also outline an agenda for future research and discuss the differences between a CAS-based approach to portfolio management and traditional approaches. DA - 2018/10/11/ PY - 2018 DO - 10.1177/8756972818802712 VL - 49 IS - 4 ER - TY - BLOG TI - Positive Deviance AU - Better Evaluation T2 - Better Evaluation AB - Positive Deviance (PD) refers to a behavioral and social change approach which is premised on the observation that in any context, certain individuals confronting similar challenges, constraints, and resource deprivations to their peers, will nonetheless employ uncommon but successful behaviors or strategies which enable them to find better solutions. Through the study of these individuals– subjects referred to as “positive deviants” - the PD approach suggests that innovative solutions to such challenges may be identified and refined from their outlying behavior. UR - https://www.betterevaluation.org/en/plan/approach/positive_deviance Y2 - 2018/10/19/00:00:00 ER - TY - JOUR TI - Positive deviance, big data, and development: A systematic literature review AU - Albanna, Basma AU - Heeks, Richard T2 - The Electronic Journal of Information Systems in Developing Countries AB - Positive deviance is a growing approach in international development that identifies those within a population who are outperforming their peers in some way, eg, children in low-income families who are well nourished when those around them are not. Analysing and then disseminating the behaviours and other factors underpinning positive deviance are demonstrably effective in delivering development results. However, positive deviance faces a number of challenges that are restricting its diffusion. In this paper, using a systematic literature review, we analyse the current state of positive deviance and the potential for big data to address the challenges facing positive deviance. From this, we evaluate the promise of “big data-based positive deviance”: This would analyse typical sources of big data in developing countries—mobile phone records, social media, remote sensing data, etc—to identify both positive deviants and the factors underpinning their superior performance. While big data cannot solve all the challenges facing positive deviance as a development tool, they could reduce time, cost, and effort; identify positive deviants in new or better ways; and enable positive deviance to break out of its current preoccupation with public health into domains such as agriculture, education, and urban planning. In turn, positive deviance could provide a new and systematic basis for extracting real-world development impacts from big data. DA - 2019/// PY - 2019 DO - 10.1002/isd2.12063 DP - Wiley Online Library VL - 85 IS - 1 SP - e12063 LA - en SN - 1681-4835 ST - Positive deviance, big data, and development UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/isd2.12063 Y2 - 2021/11/25/14:49:21 KW - big data KW - developing countries KW - machine learning KW - mobile data KW - positive deviance KW - systematic literature review ER - TY - BLOG TI - Positive Deviance Initiative AU - Positive Deviance Initiative DA - 2018/// PY - 2018 UR - https://positivedeviance.org/ Y2 - 2018/10/19/00:00:00 ER - TY - ELEC TI - Poverty Stoplight AU - povertystoplight.org AB - What is the Poverty Stoplight? The Stoplight is a tool that seeks to activate the potential of families and communities to lift themselves out of poverty. Using a technology platform, it offers a self-assessment survey and intervention model that enables people to develop practical solutions to overcome their specific needs. We work with organizations from different sectors around the world and bring together a network of powerful actors with one clear mission: eliminate poverty worldwide. The Stoplight Community is comprised of government agencies, private companies, small and medium-sized enterprises, non-profits, cooperatives, microfinance organizations, academic institutions, and sports clubs, among others. DA - 2018/// PY - 2018 UR - https://www.povertystoplight.org/ Y2 - 2023/04/27/13:54:23 ER - TY - JOUR TI - PRA five years later - where are we now? / DRP - Cinco años después, ¿dónde nos encontramos? AU - Chambers, Robert AU - Guijt, Irene T2 - Forest, Trees and People Newsletter AB - PRA five years later: where are we now? DA - 1995/// PY - 1995 DP - www.academia.edu IS - 26/27 LA - es ST - PRA five years later UR - https://www.academia.edu/111110280/PRA_five_years_later_where_are_we_now Y2 - 2023/12/11/09:36:31 ER - TY - BOOK TI - Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective AU - Chen, Huey AB - The Second Edition of Practical Program Evaluation shows readers how to systematically identify stakeholders’ needs in order to select the evaluation options best suited to meet those needs. Within his discussion of the various evaluation types, Huey T. Chen details a range of evaluation approaches suitable for use across a program’s life cycle. At the core of program evaluation is its body of concepts, theories, and methods. This revised edition provides an overview of these, and includes expanded coverage of both introductory and more cutting-edge techniques within six new chapters. Illustrated throughout with real-world examples that bring the material to life, the Second Edition provides many new tools to enrich the evaluator’s toolbox. CY - Los Angeles DA - 2014/11/13/ PY - 2014 DP - Amazon ET - Second edition SP - 464 LA - English PB - SAGE Publications, Inc SN - 978-1-4129-9230-5 ST - Practical Program Evaluation ER - TY - RPRT TI - Practioners Guide to Assessing Systems Change AU - Gover, Dun AU - Nasution, Zulka AU - Okutu, David AU - Bolder, Meghan AU - Henao, Lina T2 - MSD in MEL Brief DA - 2023/// PY - 2023 PB - USAID SN - 1 UR - https://agrilinks.org/sites/default/files/media/file/MSD%20in%20MEL%20Brief%201_Practioners%20Guide%20to%20Assessing%20Systems%20Change_06.14.pdf Y2 - 2023/10/02/09:38:55 ER - TY - JOUR TI - Practising change together – where nothing is clear, and everything keeps changing AU - Sharp, Cathy T2 - Action Learning: Research and Practice AB - This paper explores the thinking and practice of ‘action inquiry’ an embedded learning practice that can help navigate complexity when practising change together. The paper uses examples from social contexts where there are concerns about community wellbeing and health care. These are drawn from collaborative or collective leadership development programmes within public services that seek to bring new attention to the qualities of how people think, converse and interact, as part of their collective professional practice. This treats social action as a relational and dialogical practice, something that we do together as professionals by engaging in reflective inquiry and action. The paper suggests that action inquiry offers a prospect of rekindling the links between ‘action learning’ and collaborative leadership by developing a co-mission and a mutual commitment to a new type of learning partnership. Action inquiry can be wrapped around and enmeshed within initiatives and programmes that work with complexity, anywhere where effective social action will depend on the quality of relationships that can be developed. This research was funded by two separate Scottish Government commissions, where the author was a learning partner. The paper also draws on the further reflections of some of the practitioners most centrally involved. DA - 2020/01/02/ PY - 2020 DO - 10.1080/14767333.2020.1712838 DP - Taylor and Francis+NEJM VL - 17 IS - 1 SP - 10 EP - 23 SN - 1476-7333 UR - https://doi.org/10.1080/14767333.2020.1712838 Y2 - 2023/05/22/14:08:48 ER - TY - VIDEO TI - Practitioners Guidance to Assessing Systems Change: Co-Authors Preview AU - Gover, Dun AU - Nasution, Zulka AU - Okutu, David AU - Bolder, Meghan AU - Henao, Lina AB - Check out this video to see what’s inside our new resource: Practitioners' Guidance to Assessing Systems Change, developed by MEL Managers for MEL Managers. (Check out the Guidance here https://bit.ly/MSPMELClinics.) Hear from the authors about which parts they love the most and how this guide challenges MEL managers to assess systems change as an ongoing aspect of implementation, generating feedback that teams need to better understand and catalyze change, for more impact. DA - 2023/07// PY - 2023 PB - USAID UR - https://www.youtube.com/watch?v=X_szw6nIwbA Y2 - 2023/10/02/09:38:55 ER - TY - BOOK TI - Principles-Focused Evaluation: The GUIDE AU - Patton, Michael Quinn AB - How can programs and organizations ensure they are adhering to core principles--and assess whether doing so is yielding desired results? From evaluation pioneer Michael Quinn Patton, this book introduces the principles-focused evaluation (P-FE) approach and demonstrates its relevance and application in a range of settings. Patton explains why principles matter for program development and evaluation and how they can serve as a rudder to navigate the uncertainties, turbulence, and emergent challenges of complex dynamic environments. In-depth exemplars illustrate how the unique GUIDE framework is used to determine whether principles provide meaningful guidance (G) and are useful (U), inspiring (I), developmentally adaptable (D), and evaluable (E). User-friendly features include rubrics, a P-FE checklist, firsthand reflections and examples from experienced P-FE practitioners, sidebars and summary tables, and end-of-chapter application exercises. CY - New York DA - 2017/10/16/ PY - 2017 ET - 1st edition SP - 435 LA - English PB - Guilford Press SN - 978-1-4625-3182-0 ST - Principles-Focused Evaluation UR - https://www.guilford.com/books/Principles-Focused-Evaluation/Michael-Quinn-Patton/9781462531820 ER - TY - RPRT TI - Process Monitoring of Impacts: Towards a new approach to monitor the implementation of Structural Fund Programmes AU - Hummelbrunner, Richard AU - Huber, Wolf AU - Arbter, Roland DA - 2005/// PY - 2005 SP - 22 PB - ÖAR Regionalberatung UR - http://archiv.bundeskanzleramt.at/DocView.axd?CobId=14624 Y2 - 2018/09/22/00:00:00 ER - TY - MANSCPT TI - Process Tracing as a Practical Evaluation Method: Comparative Learning from Six Evaluations AU - Wadeson, Alix AU - Monzani, Bernardo AU - Aston, Tom DA - 2020/03// PY - 2020 DP - Zotero LA - en UR - https://mande.co.uk/wp-content/uploads/2020/03/Process-Tracing-as-a-Practical-Evaluation-Method_23March-Final-1.pdf ER - TY - RPRT TI - Process Tracing Innovations in Practice: Finding the Middle Path AU - Aston, Thomas AU - Wadeson, Alix T2 - CDI Practice Paper AB - Evaluation practitioners in the international development sector have given considerable attention in recent years to process tracing as a method for evaluating impact, including discussion of how to assess the relative importance of causal factors. Despite the increasing interest, there is a relative dearth of examples of practical learning and evidence of applying process tracing in practice. This CDI Practice Paper draws on comparative learning from applying three different types of process tracing in international development initiatives. It argues in favour of a ‘middle path’ of applying evidence tests and rubrics to structure evaluative judgements rather than formal Bayesian updating or looser forms of process tracing. It also calls attention to the potential added value of taking a participatory approach, offering practical recommendations for how to do this effectively. CY - Brighton DA - 2023/03/27/ PY - 2023 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 25 ST - Process Tracing Innovations in Practice UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17913 Y2 - 2023/03/28/09:32:09 ER - TY - RPRT TI - Producer-led value chain analysis: The missing link in value chain development AU - Ghore, Yogesh AB - Introduction and rationale The concept of a value chain is increasingly being applied in the design and implementation of development programs aimed at poverty reduction. As an analytical tool, it provides a useful framework for understanding key activities, relationships, and mechanisms that allow producers, processors, buyers, sellers, and consumers—separated by time and space—to gradually add value to products and services as they pass from one link of the chain to another, making it a “value chain” (UNIDO, 2009). While it has been popular in the private sector ever since it was conceptualized by Michael Porter in the 1980s, more recently various donors and governments have shown interest in its use and have applied it to a range of development interventions, particularly in the area of sector development, livelihoods promotion, small and medium enterprise (SME) development, and rural and economic development. Academics and development organizations have designed numerous instruments for value chain analysis (VCA) and implementation. Beginning in the early 2000s, international organizations and donor agencies have sponsored the development of these tools, including the United States Agency for International Development (USAID) Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), the World Bank, the International Finance Corporation (IFC), and the International Labour Organization (ILO). These guides and tools have been used in many development programs to address organizational, donor and local and regional priorities for development. While VCA guides and tools are important in understanding markets and relationships among key stakeholders, a recent comparative review of popularly used guides by Donovan et al. (2013) suggests that most of these guides are designed to be implemented independently of the local context and do not sufficiently focus on mutual learning, whether related to tool design or to the outcomes and impacts of the designed project activities. VCA is often conducted by external experts and the knowledge generated in the process is often confined to reports. Without the capacity building and effective participation of women and men producers—smallholder farmers who hold critical knowledge about the local context—an important link is missing in the entire process of VCA. The insufficient attention to the human, social, and other contextual factors undermines the full potential of the value chain systems approach, not only for economic outcomes, but for the long term sustainability of the intended benefits. So what are the ways to effectively engage producers in the community in the VCA? The producer-led process described here tries to address this basic question. It introduces simplified tools and an approach that ensures farmer participation in data collection, analysis, and identification of opportunities and constraints, and design of value chain interventions. This participation requires: a) an environment in which they feel comfortable to share their knowledge and insights, which is often not the case when extractive surveys and questionnaires are administered to collect information; and b) the use of simple and participatory tools that will allow them to provide inputs into the process of VCA and also help them to understand complex value chain systems and use this understanding for making livelihood decisions. The key steps and tools described are aimed at empowering the women and men farmers to make informed decisions about their own enterprises and how they relate to the value chain, thereby directly contributing to, and influencing, the overall process of value chain development. The process of VCA involving these steps and tools was developed at Coady Institute and first tested with Oxfam Canada and its local partners in Ethiopia in August 2012. DA - 2015/// PY - 2015 DP - Zotero LA - en PB - Coady ER - TY - RPRT TI - Program Cycle - How-To Note: Strategy-Level Portfolio Review AU - USAID CY - Washington D.C. DA - 2017/07// PY - 2017 PB - USAID PPL UR - https://usaidlearninglab.org/sites/default/files/resource/files/how_to_note_portfolio_review_final_compliant_1_r.pdf Y2 - 2019/02/25/13:48:41 ER - TY - ELEC TI - Program Cycle Road Map AU - USAID DA - 2016/// PY - 2016 UR - http://usaidprojectstarter.org/content/program-cycle-road-map Y2 - 2018/02/06/10:07:46 ER - TY - JOUR TI - Program Logic Foundations: Putting the Logic Back into Program Logic AU - Hawkins, Andrew J. T2 - Journal of MultiDisciplinary Evaluation AB - Abstract Background: Program logic is one of the most used tools by the public policy evaluator. There is, however, little explanation in the evaluation literature about the logical foundations of program logic or discussion of how it may be determined if a program is logical. This paper was born on a long journey that started with program logic and ended with the logic of evaluation. Consistent throughout was the idea that the discipline of program evaluation is a pragmatic one, concerned with applied social science and effective action in complex, adaptive systems. It gradually became the central claim of this paper that evidence-based policy requires sound reasoning more urgently than further development and testing of scientific theory. This was difficult to reconcile with the observation that much evaluation was conducted within a scientific paradigm, concerned with the development and testing of various types of theory. Purpose: This paper demonstrates the benefits of considering the core essence of a program to be a proposition about the value of a course of action. This contrasts with a research-based paradigm in which programs are considered to be a type of theory, and in which experimental and theory-driven evaluations are conducted. Experimental approaches focus on internal validity of knowledge claims about programs and on discovering stable cause and effect relationships—or, colloquially, ‘what works?’. Theory-driven approaches tend to focus on external validity and in the case of the realist approach, the search for transfactual causal mechanisms—extending the ‘what works’ mantra to include ‘for whom and in what circumstances’. On both approaches, evaluation aspires to be a scientific pursuit for obtaining knowledge of general laws of phenomena, or in the case of realists, replicable context-mechanism-outcome configurations. This paper presents and seeks to justify an approach rooted in logic, and that supports anyone to engage in a reasonable and democratic deliberation about the value of a course of action. It is consistent with systems thinking, complexity and the associated limits to certainty for determining the value of a proposed, or actual, course of action in the social world. It suggests that evaluation should learn from the past and have an eye toward the future, but that it would be most beneficial if concerned with evaluating in the present, in addressing the question ‘is this a good idea here and now? Findings: In seeking foundations of program logic, this paper exposes roots that extend far deeper than the post-enlightenment, positivist and post-positivist social science search for stable cause and effect relationships. These roots lie in the 4th century BCE with Aristotle’s ‘enthymeme’. The exploration leads to conclusions about the need for a greater focus on logic and reasoning in the design and evaluation of programs and interventions for the public good. Science and research are shown to play a crucial role in providing reasons or warrants to support a claim about the value of a course of action; however, one subordinate to the alpha-discipline of logical evaluation and decision making that must consider what is feasible given the context, capability and capacity available, not to mention values and ethics. Program Design Logic (PDL) is presented as an accessible and incremental innovation that may be used to determine if a program makes sense ‘on paper’ in the design stage as well as ‘in reality’ during delivery. It is based on a configurationalist theory of causality and the concepts of ‘necessary’ and ‘sufficient’ conditions. It is intended to guide deliberation and decision making across the life cycle of any intervention intended for the public good. DA - 2020/11/19/ PY - 2020 DP - journals.sfu.ca VL - 16 IS - 37 SP - 38 EP - 57 LA - en SN - 1556-8180 ST - Program Logic Foundations UR - https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/657 Y2 - 2021/05/18/14:45:11 ER - TY - RPRT TI - Project Evaluation and the Project Appraisal Reporting System AU - Rosenberg, L.J. AU - Posner, L.D. AU - Hanley, E.J. DA - 1970/07/24/ PY - 1970 PB - Fry Consultants Inc. UR - https://pdf.usaid.gov/pdf_docs/PNADW881.pdf Y2 - 2019/08/08/22:38:36 ER - TY - BOOK TI - Purposeful Program Theory: Effective Use of Theories of Change and Logic Models AU - Funnell, Sue C. AU - Rogers, Patricia J. AB - Program Theory in Evaluation Practice is a ground-breaking reference that teaches how to develop an explicit causal model that links an intervention (project, program or policy) with its intended or observed impacts and using this to guide monitoring and evaluation. Peerless in its explanation of why and how to use and develop program theory, the book is rich with examples and alternative approaches. The book is an invaluable resource to faculty and students as well as professionals in professional development programs, education, social work, and counseling. DA - 2011/02// PY - 2011 LA - en-au PB - Wiley ST - Purposeful Program Theory UR - https://www.wiley.com/en-au/Purposeful+Program+Theory%3A+Effective+Use+of+Theories+of+Change+and+Logic+Models-p-9780470478578 Y2 - 2023/01/12/11:17:09 ER - TY - BOOK TI - Purposeful Program Theory: Effective Use of Theories of Change and Logic Models AU - Funnell, Sue C. AU - Rogers, Patricia J. AB - Between good intentions and great results lies a program theory—not just a list of tasks but a vision of what needs to happen, and how. Now widely used in government and not-for-profit organizations, program theory provides a coherent picture of how change occurs and how to improve performance. Purposeful Program Theory shows how to develop, represent, and use program theory thoughtfully and strategically to suit your particular situation, drawing on the fifty-year history of program theory and the authors' experiences over more than twenty-five years."From needs assessment to intervention design, from implementation to outcomes evaluation, from policy formulation to policy execution and evaluation, program theory is paramount. But until now no book has examined these multiple uses of program theory in a comprehensive, understandable, and integrated way. This promises to be a breakthrough book, valuable to practitioners, program designers, evaluators, policy analysts, funders, and scholars who care about understanding why an intervention works or doesn't work." —Michael Quinn Patton, author, Utilization-Focused Evaluation"Finally, the definitive guide to evaluation using program theory! Far from the narrow 'one true way' approaches to program theory, this book provides numerous practical options for applying program theory to fulfill different purposes and constraints, and guides the reader through the sound critical thinking required to select from among the options. The tour de force of the history and use of program theory is a truly global view, with examples from around the world and across the full range of content domains. A must-have for any serious evaluator." —E. Jane Davidson, PhD, Real Evaluation Ltd.Companion Web site: josseybass.com/go/funnellrogers DA - 2011/02/09/ PY - 2011 SP - 582 LA - Inglés ST - Purposeful Program Theory ER - TY - RPRT TI - Putting learning at the centre: Adaptive development programming in practice AU - Valters, Craig AU - Cummings, Clare AU - Nixon, Hamish AB - Adaptive programming suggests, at a minimum, that development actors react and respond to changes in the political and socio-economic operating environment. It emphasises learning and the development practitioner is encouraged to adjust their actions to find workable solutions to problems that they may face. Being prepared to react to change may seem like common sense – and indeed it is. However much development thinking and practice remains stuck in a linear planning model which discourages learning and adaptation, in part because projects are seen as ‘closed, controllable and unchanging systems’ (Mosse, 1998: 5). This paper critically engages with this problem and makes clear why and how learning needs to be at the centre of adaptive development programming. It begins by clarifying why and what kind of learning matters for adaptive programming. The paper then turns its focus to how strategies and approaches applied throughout a programme’s conception, design, management and M&E can enable it to continually learn and adapt. CY - London DA - 2016/03// PY - 2016 PB - ODI ST - Adapting development UR - https://www.odi.org/publications/10367-putting-learning-centre-adaptive-development-programming-practice Y2 - 2016/05/11/12:37:16 KW - IMPORTANT ER - TY - BLOG TI - Q&A: Complexity-Aware Planning for Stabilization Programming AU - Martins, Christy T2 - DAI - Checkpoint DA - 2018/08/13/ PY - 2018 UR - http://dai-global-checkpoint.com/q-and-a-complexity-aware-planning-for-stabilization-programming.html Y2 - 2018/08/20/11:11:20 ER - TY - RPRT TI - Qualitative Comparative Analysis - A Rigorous Qualitative Method for Assessing Impact AU - Baptist, Carrie AU - Befani, Barbara DA - 2015/01// PY - 2015 PB - Coffey UR - http://www.coffey.com/assets/Ingenuity/Qualitative-Comparative-Analysis-June-2015.pdf Y2 - 2016/05/12/10:50:17 ER - TY - RPRT TI - Qualitative comparative analysis: A valuable approach to add to the evaluator’s toolbox? Lessons from recent applications AU - Schatz, Florian AU - Welle, Katharina T2 - CDI Practice Paper DA - 2016/01// PY - 2016 PB - IDS SN - 13 UR - http://cdimpact.org/blog/qualitative-comparative-analysis-%E2%80%93-addition-evaluator%E2%80%99s-toolbox Y2 - 2016/05/12/10:41:57 ER - TY - BLOG TI - Qualitative Impact Assessment Protocol (QUIP) AU - Better Evaluation T2 - Better Evaluation AB - The QUIP sets out to generate differentiated evidence of impact based on narrative causal statements elicited directly from intended project beneficiaries without use of a control group. Evidence of attribution is sought through respondents’ own accounts of causal mechanisms linking X to Y alongside Z rather than by relying on statistical inference based on variable exposure to X. This narrative data is intended to complement quantitative evidence on changes in X, Y and Z obtained through routine project monitoring. UR - https://www.betterevaluation.org/en/plan/approach/QUIP Y2 - 2018/10/19/00:00:00 ER - TY - BOOK TI - Quality standards for development evaluation AU - OECD AU - Development Assistance Committee AB - Based on a broad international consultative process, the DAC Quality Standards for Development Evaluation are a reference guide to good practice in development evaluation. With a view to improving the quality of evaluation processes and products, and facilitating collaboration, this reference guide lays out standards for each phase of a typical evaluation process: from defining purpose, to planning, designing, implementing, reporting, and learning from and using evaluation results. CY - Paris, France DA - 2010/// PY - 2010 DP - Open WorldCat LA - en PB - OECD SN - 978-92-64-08390-5 UR - https://doi.org/10.1787/9789264167100-ar Y2 - 2019/11/27/12:01:45 ER - TY - RPRT TI - QUIP: Understanding clients through in-depth interviews AU - IMP-ACT T2 - IMP-ACT PRACTICE NOTES AB - This Practice Note by Imp-Act gives a step-by-step guide to developing and conducting in-depth interviews using the QUIP approach, and analysing the information and making conclusions based on what you have learned. DA - 2004/// PY - 2004 SP - 6 PB - Institute of Development Studies SN - 2 UR - https://sptf.info/images/pn2_quip.pdf Y2 - 2018/10/19/00:00:00 ER - TY - BOOK TI - RAND Program Evaluation Toolkit for Countering Violent Extremism AU - Helmus, Todd AU - Matthews, Miriam AU - Ramchand, Rajeev AU - Beaghley, Sina AU - Stebbins, David AU - Kadlec, Amanda AU - Brown, Michael AU - Kofner, Aaron AU - Acosta, Joie DA - 2017/// PY - 2017 DP - Crossref LA - en PB - RAND Corporation SN - 978-0-8330-9724-8 UR - http://www.rand.org/pubs/tools/TL243.html Y2 - 2019/09/17/11:20:09 ER - TY - BLOG TI - Randomised Controlled Trial AU - Better Evaluation T2 - Better Evaluation AB - Randomised controlled trials (RCTs), or randomised impact evaluations, are a type of impact evaluation which uses randomised access to social programmes as a means of limiting bias and generating an internally valid impact estimate. UR - https://www.betterevaluation.org/en/plan/approach/rct Y2 - 2018/10/19/00:00:00 ER - TY - BLOG TI - Rapid Action Learning and COVID-19 AU - Chambers, Robert T2 - Sanitation Learning Hub AB - A-ha! A moment etched in my memory: 20 or so researchers were... DA - 2020/10/06/T09:46:33+00:00 PY - 2020 LA - en-GB UR - https://sanitationlearninghub.org/2020/10/06/rapid-action-learning-for-covid-19/ Y2 - 2020/10/16/14:03:59 ER - TY - RPRT TI - Rapid Action Learning for Sanitation and Hygiene Programming AU - Chambers, Robert AU - Myers, Jamie AU - Vernon, Naomi T2 - Frontiers of Sanitation AB - There is a glaring gap and compelling need for approaches and methods that realign to new rigour through timeliness, cost-effectiveness, relevance and being actionable. Over the past few years, the Sanitation Learning Hub, in collaboration with the Government of India, Praxis, WSSCC and WaterAid India, have been developing Rapid Action Learning approaches. Multiple approaches have been trialled, with flexible formats, but the essential criteria is that learning is timely, relevant and actionable. These learning approaches are the focus of the latest edition of the Frontiers of Sanitation series. This Frontiers explains the advantages and disadvantages of the approaches trialled and sets out a challenge to those working in the water, sanitation and hygiene sector to: Reflect on what, for you, constitutes rigour. Adopt and adapt approaches to fit your context and needs. Develop your own approaches. Record your experiences and lessons learnt. Take the time to share your experiences with us CY - Brighton DA - 2020/09// PY - 2020 DP - Zotero SP - 21 LA - en PB - IDS SN - 15 ER - TY - RPRT TI - Rapid outcome mapping approach: a guide to policy engagement and influence AU - Young, John AU - Shaxson, Louise AU - Jones, harry AU - Hearn, Simon AU - Datta, Ajoy AU - Cassidy, Caroline DA - 2014/// PY - 2014 SP - 84 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9011.pdf Y2 - 2018/12/10/00:00:00 ER - TY - BLOG TI - “Real”​ process tracing: part 1 — context AU - Aston, Thomas T2 - Medium AB - When asserting the value of theory-based methods, you often here words like “black boxes” and “causal mechanisms.” These are commonly… DA - 2020/12/26/T09:59:27.959Z PY - 2020 LA - en ST - “Real”​ process tracing UR - https://thomasmtaston.medium.com/real-process-tracing-part-1-context-6a52777a6a98 Y2 - 2021/02/18/14:47:07 ER - TY - BLOG TI - Realist Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings. Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?. The complete realist question is: “What works, for whom, in what respects, to what extent, in what contexts, and how?”. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context. UR - https://www.betterevaluation.org/en/approach/realist_evaluation Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - Realist Evaluation AU - Pawson, Ray AU - Tilley, Nick AB - Contents: Introduction 1. The nature of programmes and how they work 2. Basic concepts in the explanation and understanding of programmes 3. Strategies and methods of realist evaluation 4. Realism’s place in the policy cycle: formative, summative and synthetic approaches 5. The nature, presentation and use of findings from realist evaluation 6. Conclusion: strengths, limitations and relationships with other approaches Appendix I – ‘Thinking it through’: an exercise in realist hypothesis making. Appendix II – ‘Varieties of realist evaluation’: pocket illustrations of quantitative, qualitative, formative and synthetic applications. Appendix III – ‘Would it work here?’: a grid to help decide on the feasibility of mounting a programme ‘on your patch’. DA - 2004/// PY - 2004 SP - 36 UR - http://www.communitymatters.com.au/RE_chapter.pdf Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - Realist Impact Evaluation: An introduction AU - Westhorp, Gill AB - • Realist evaluation is a member of a family of theory-based evaluation approaches which begin by clarifying the ‘programme theory’: the mechanisms that are likely to operate, the contexts in which they might operate and the outcomes that will be observed if they operate as expected. • Realist approaches assume that nothing works everywhere for everyone: context makes a big difference to programme outcomes. A realist evaluation asks not ‘what works?’ but ‘how or why does this work, for whom, in what circumstances?’ • Realist impact evaluation is most appropriate for evaluating new initiatives or programmes that seem to work but where ‘how and for whom’ is not yet understood; programmes that have previously demonstrated mixed patterns of outcomes; and those that will be scaled up, to understand how to adapt the intervention to new contexts. DA - 2014/09// PY - 2014 SP - 12 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9138.pdf Y2 - 2018/11/10/00:00:00 ER - TY - BLOG TI - Redefining Rigor: Describing Quality Evaluation in Complex, Adaptive Settings AU - Preskill, Hallie AU - Lynn, Jewlya T2 - FSG AB - Traditionally, evaluation has focused on understanding whether a program is making progress against pre-determined indicators. In this context, the quality of the evaluation is often measured in part by the “rigor” of the methods and scientific inquiry. DA - 2016/02/01/T10:11:54-05:00 PY - 2016 LA - en ST - Redefining Rigor UR - https://www.fsg.org/blog/redefining-rigor-describing-quality-evaluation-complex-adaptive-settings Y2 - 2021/11/09/12:25:49 ER - TY - RPRT TI - Refining advocacy assessment: reflections from practice AU - Buffardi, Anne AU - Hearn, Simon AU - Tilley, Helen T2 - Working Paper AB - • This paper revisits how existing advocacy evaluation frameworks classify people and activities, and define and assess outcomes. We discuss how assessments could be more specific and propose bounding the scope of inquiry in one of four ways: strategy specific, outcome oriented, actor-centric or system-wide. • In classifying activities, the same action or event may be situated at different phases of the change pathway – in some cases used as a tactic to influence a policy outcome, and in others an intended outcome itself. • Because advocacy is more relational than other types of more technical development interventions, there will be fewer sources of directly observable data, and the direction of potential bias may be unknown. • In terms of learning, advocacy initiatives are contextually dependent, therefore lessons may be less directly transferable to subsequent phases of an initiative or to other settings. Organisations have bounded repertoires and the transferability of skill sets is limited, so advocates adapt how and with whom they engage more than what they do DA - 2017/03// PY - 2017 PB - ODI SN - 500 UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/11335.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - Reflections from a Realist Evaluation in Progress: Scaling Ladders and Stitching Theory AU - Punton, Melanie AU - Vogel, Isabel AU - Lloyd, Rob T2 - CDI Practice Paper 18 AB - Realist evaluation provides valuable insights into how and why programmes lead to change, and can generate transferable lessons to help practitioners roll out or scale up an intervention. However, as ... CY - Brighton DA - 2016//04/ PY - 2016 LA - en PB - IDS ST - Reflections from a Realist Evaluation in Progress UR - http://www.ids.ac.uk/publication/reflections-from-a-realist-evaluation-in-progress-scaling-ladders-and-stitching-theory Y2 - 2018/07/27/10:31:22 ER - TY - GEN TI - Reflections on the Utilization-Focused Evaluation (UFE) Process AU - Lim, Yvonne AU - Mizumoto, Ann AB - This presentation from the Strengthening ICTD Research Capacity in Asia (SIRCA) provides an overview of how UFE was used in their SIRCA programme. It was presented at the Evaluation Conclave 2010, New Delhi, India The key objectives of the program are to: Enhance research capacity in Asia through rigorous academic research Create a space for dialogue on ICT4D social science research issues in Asia Create linkages through a mentorship program Disseminate findings in publications and conferences Contents SIRCA Programme SIRCA Key Objectives SIRCA Evaluation UFE Learnings UFE Challenges Evaluation is over…but there’s a lasting outcome... DA - 2010/10/28/ PY - 2010 PB - Strengthening ICTD Research Capacity in Asia (SIRCA) Programme UR - https://evaluationinpractice.files.wordpress.com/2010/12/sirca_conclave-2010-presentation-3_yl.pdf Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Reforming donors in fragile states: using public management theory more strategically AU - Gulrajani, Nilima AU - Honig, Dan AB - This paper identifies ways in which donors can be more effective in fragile and conflict-affected states by exploiting theories and concepts drawn from public management. CY - London DA - 2016/04// PY - 2016 LA - en M3 - Research Report PB - Overseas Development Institute ST - Reforming donors in fragile states UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10479.pdf Y2 - 2018/02/16/08:21:21 ER - TY - RPRT TI - Reforming solid waste management in Phnom Penh AU - Denney, Lisa T2 - Working Politically in Practice AB - This paper tracks the efforts of an Asia Foundation team and local stakeholders as they worked to support improvements in the solid waste management sector in Phnom Penh, Cambodia. The team worked in a flexible way with a range of partners, and with particular focus on understanding the incentives and politics affecting service delivery. While reform of the sector remains in progress, steps have been taken to introduce more competition and better public sector management of solid waste collection in the city. This case study lays out the real-time decisions and processes which drove the strategy and implementation of this project, providing useful insights into how politically astute and flexible programs can be successfully implemented. This case has emerged from an action research process, which was led by a researcher from the Overseas Development Institute (ODI) and conducted over the course of almost two years. By capturing and analyzing the experiences of the program team in Phnom Penh, the paper intends to provide practical insights for others in the development community aiming to implement similar kinds of programming. DA - 2016/05// PY - 2016 SN - 8 UR - http://asiafoundation.org/publication/reforming-solid-waste-management-phnom-penh Y2 - 2016/03/23/16:47:09 ER - TY - JOUR TI - Reframing, refining, and reconceptualising the worst forms of child labour through participatory adaptive programming AU - Apgar, Marina AU - Burns, Danny T2 - Journal of modern slavery AB - This article explores the potential of using participatory action research as an adaptive programming modality to drive learning and innovation to tackle the drivers of (and seek to eliminate) the Worst Forms of Child Labour. We draw on our experience from early phases of implementation of a large-scale action research programme, which despite the constraints covid-19 posed in moving to full implementation and participatory engagement with children and other stakeholders on the ground, is already generating rich learning about the opportunities and challenges of designing programmes that respond to the complex reality of WFCL. We share early learning about what it takes to be fully open to using the lived experience of programme development, and early findings from scoping and mapping of the dynamics of social norms, business practices and urban neighbourhoods and supply chains influencing WFCL in Bangladesh and Nepal, to frame and reframe the questions and response strategies and operationalise a participatory adaptive intent to work with hidden and complex dynamics that characterise the WFCL. DA - 2021/// PY - 2021 DP - Zotero VL - 6 IS - 4 LA - en ER - TY - BLOG TI - Regional Healthcare Ecosystem Analyst for Healthcare Associated Infections AU - Lee, Bruce Y. T2 - Grantome AB - Understanding and preventing the spread of both endemic and emerging healthcare-associated infectious diseases throughout hospitals and nursing homes is a national priority. Our work has shown that the many disparate inpatient healthcare facilities in a region can form a complex healthcare ecosystem connected by both direct and indirect patient sharing allowing pathogens in one health care facility to readily spread to other facilities. Our goal is to further develop RHEA (Regional Healthcare Ecosystem Analyst) into a user-friendly software tool that decision makers (e.g., policy makers, funders, product developers and manufacturers, healthcare administrators, infection prevention specialists, researchers, and educators) can readily use to help healthcare ecosystems prevent and control the spread of an endemic pathogen, methicillin- resistant Staphylococcus aureus (MRSA), and an emerging pathogen, carbapenem-resistant Enterobacteriaceae (CRE). This next generation of RHEA will bring multiple innovations by: 1) further elucidating the interconnectedness of the ecosystem and showcasing the value of cooperation among facilities versus the current individual facility approach to infectious disease control, 2) incorporating new healthcare ecosystem responses for endemic MRSA and emerging CRE, 3) continuing our work integrating economic and operational models into a framework of infectious disease epidemiological models, 4) imbuing virtual patients with characteristics linked to MRSA and CRE transmission and outcome risk, 5) building a comprehensive """"""""virtual laboratory"""""""" to help address many existing and future healthcare ecosystem infectious disease questions, and 6) building a user-friendly modeling tool that decision makers can use. The project will continue our team's longstanding modeling work and developing computational tools for decision makers via three specific aims. First, completing Specific Aim 1 will expand RHEA by adding the following capabilities: i) integrated clinical outcome and economic models, ii) more extensive and detailed patient characteristics linked to changes in infection risk and costs, iii) expanded HAI control measures, and iv) stochastic and adaptive disease parameters to represent emerging and evolving diseases. Next, Specific Aim 2 will use our newly expanded RHEA framework to model a sample endemic pathogen, MRSA, and a sample emerging pathogen, CRE to identify optimal control strategies even when pathogen characteristics are evolving. Finally, Specific Aim 3 will involve developing a user-friendly interface for RHEA and deploying it as a healthcare ecosystem computational modeling tool that various stakeholders (e.g., policy makers, healthcare administrators, infection control specialists, funders, and product manufacturers) can readily use to make decisions regarding the control of healthcare-associated infections. Public Health Relevance Understanding and preventing the spread of healthcare-associated infectious diseases throughout hospitals and nursing homes is a national priority. Simulation models can serve as virtual laboratories to help identify best practice solutions for containing common [such as methicillin-resistant Staphylococcus aureus (MRSA)] and emerging [e.g., carbapenem-resistant Enterobacteriaceae (CRE)] causes of healthcare associated infections (HAIs). Our goal is to develop our software RHEA (Regional Healthcare Ecosystem Analyst) into a computational modeling tool that can be used directly by decision makers to identify, develop, and evaluate strategies and interventions to reduce HAIs across a large geographic region. DA - 2018/// PY - 2018 UR - http://grantome.com/grant/NIH/R01-HS023317-01 Y2 - 2018/03/09/00:00:00 ER - TY - RPRT TI - Report on outcomes and get everyone involved: The Participatory Performance Story Reporting Technique AU - Dart, J. T2 - Presented at the AES conference in Perth, September 2008 AB - This paper outlines the background and philosophy of Collaborative Outcomes Reporting (COR) and Performance Story Reporting (PSR), providing an overview of these emerging techniques and showing how they can be used as a framework for reporting on contribution to long-term outcomes (or targets) using mixed methods and participatory process. The report serves as an introduction to these approaches in evaluation and discusses their bias, limitations and where they might best be applied. DA - 2008/// PY - 2008 SP - 7 UR - https://www.clearhorizon.com.au/f.ashx/report-on-outcomes-and-get-everyone-involved_the-participatory-performance.pdf ER - TY - GEN TI - Report template on integrating impact into an existing monitoring and evaluation system AU - Methods Lab AB - Many development programme staff will commission an impact evaluation towards the end of a project or programme, only to find that the monitoring system did not provide adequate data about implementation, context, baselines or interim results. This tool provides a template outline for a report making recommendations on how to integrate a focus on impact into a programme’s existing monitoring and evaluation system, as the programme moves into a new phase. This template was developed by Anne Buffardi and Tiina Pasanen for use in Methods Lab projects. DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10037.pdf Y2 - 2018/11/10/00:00:00 ER - TY - GEN TI - Report templates for evaluability assessment AU - Methods Lab AB - An evaluability assessment aims to assess the extent to which, and how best, a project can be evaluated in a reliable and credible fashion. These templates are intended to help anyone conducting an evaluability assessment to structure the final report. This tool was developed by Anne Buffardi and Bronwen McDonald for use in Methods Lab projects. It accompanies The Methods Lab publication ‘Evaluability assessment for impact evaluation: guidance, checklists and decision support’. DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10036.pdf Y2 - 2018/11/10/00:00:00 ER - TY - JOUR TI - Representing theories of change: technical challenges with evaluation consequences AU - Davies, Rick T2 - Journal of Development Effectiveness AB - This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation. The paper then outlines a range of different ways of addressing these problems that could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper. DA - 2018/10/02/ PY - 2018 DO - 10.1080/19439342.2018.1526202 DP - Taylor and Francis+NEJM VL - 10 IS - 4 SP - 438 EP - 461 SN - 1943-9342 ST - Representing theories of change UR - https://doi.org/10.1080/19439342.2018.1526202 Y2 - 2021/04/16/08:24:00 KW - Evaluation KW - evaluability KW - representation KW - theory of change ER - TY - RPRT TI - Responding to complexity: A Case Study on the Use of “Developmental Evaluation for Managing Adaptively” AU - Boisvert, Kayla DA - 2017/06// PY - 2017 PB - University of Massachusetts ST - Evaluating CatComm’s Approach UR - http://catcomm.org/evaluation2017/ Y2 - 2017/11/09/11:16:22 ER - TY - RPRT TI - Rethinking systemic change: economic evolution and institutions. Discussion paper AU - Cunningham, Shawn AU - Jenal, Marcus AB - The primary aim of this research project was to find a conceptually sound definition of systemic change. To do so, it was essential to gain a better understanding of how economies change. The central part of the research work, therefore, was an extended literature review on three bodies of knowledge: evolutionary economics new institutional economics complexity theory There is a growing interest in these bodies of knowledge, combined often called New Economic Thinking, and how they affect economic development. Hence, while rethinking systemic change, this work also contributes to answering the broader question of how market systems approaches can contribute to inclusive economic development. The answer, in short, is to shift the focus away from improving transactions at the micro level towards enabling actors to continuously shape an institutional landscape that supports inclusive economic evolution. This discussion paper briefly presents the key messages from the literature review and seven principles drawing from this literature. The principles can be used by market development practitioners, including technical advisers in donor organisations, programme designers and team leaders, to shape programmes and become more in line with how change happens in the economy. A list of selected references is presented at the end. A technical paper contains a much more detailed discussion of the findings and the principles and an extensive list of references. A case study offers a practitioner perspective through the lens of one market systems development programme: DFID-funded Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD). DA - 2016/12// PY - 2016 PB - BEAM Exchange UR - https://beamexchange.org/uploads/filer_public/ac/39/ac395b26-2a17-4195-a485-412cff275929/systemic_change_discussion_paper.pdf Y2 - 2019/02/01/11:18:18 ER - TY - RPRT TI - Rethinking systemic change: economic evolution and institutions. Technical Paper AU - Cunningham, Shawn AU - Jenal, Marcus AB - The primary aim of this research project was to find a conceptually sound definition of systemic change. To do so, it was essential to gain a better understanding of how economies change. The central part of the research work, therefore, was an extended literature review on three bodies of knowledge: evolutionary economics, new institutional economics and complexity theory. There is a growing interest in these bodies of knowledge, combined often called New Economic Thinking, and how they affect economic development. Hence, while rethinking systemic change, this work also contributes to answering the broader question of how market systems approaches can contribute to inclusive economic development. The answer, in short, is to shift the focus away from improving transactions at the micro level towards enabling actors to continuously shape an institutional landscape that supports inclusive economic evolution. This technical paper provides an in-depth review of the fields of evolutionary economics, new institutional economics and complexity and social change. It is argued that economic development is a complex, non-linear and continuous evolutionary process. Both market and non-market institutions matter greatly in shaping economic performance. The paper then explores the consequences of this understanding for market development practice. It discusses how market development practitioners can engage in and shape an intentional change processes. To translate the theory into practice, seven principles are suggested that can be applied to market systems development practice. The paper concludes with a reframed look at systemic change in market systems development. A discussion paper presents the key messages from the literature review and seven principles drawing from this literature, and a case study offers a practitioner perspective through the lens of one market systems development programme: DFID-funded Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD). DA - 2016/12// PY - 2016 PB - BEAM Exchange UR - https://beamexchange.org/uploads/filer_public/ac/39/ac395b26-2a17-4195-a485-412cff275929/systemic_change_discussion_paper.pdf Y2 - 2019/02/01/11:18:18 ER - TY - RPRT TI - Rethinking systemic change in practice - perspectives from NUTEC-MD in northern Uganda AU - Koleros, Andrew AB - Document description This document is currently being finalised and will be published shortly. Please try again soon. Recent research by The BEAM Exchange seeks to understand theoretical perspectives on how market systems approaches can contribute to inclusive economic development through systemic change. It produced three main insights. Economies are evolving systems, building on the mechanisms of variety creation, selection and amplification. Current economic performance, including aspects like the inclusiveness of growth and economic evolution, are shaped by the ability of a society to explore different options for institutional arrangements and adjust them over time. This process of evolution is complex. While some aspects can be designed and managed, others need to be explored through a process of learning and adjustment. This paper responds to these insights by providing a practitioner perspective through the lens of one programme: “Northern Uganda: Transforming the Economy through Climate Smart Agribusiness – Market Development (NUTEC-MD)”, implemented by Palladium Group. It begins with a brief introduction to the programme and key aspects of its design phase. It then provides a summary of the three main research insights, with reflections on their applicability for programme design based on the NUTEC-MD experience. Finally, it provides some conclusions for the wider practitioner community based on this case. CY - London DA - 2016/12// PY - 2016 M3 - Case Study PB - BEAM Exchange UR - https://beamexchange.org/resources/860/ Y2 - 2017/05/17/16:33:28 ER - TY - RPRT TI - Rethinking value for money for adaptive, politically smart programmes - Lessons from Institutions for Inclusive Development in Tanzania AU - Laws, Ed AB - - This short paper draws out lessons for working effectively with and through partners, based on the experience of the Institutions for Inclusive Development (I4ID) programme – an adaptive, politically smart governance programme in Tanzania. • Cultivating effective partnerships can be a key part of delivering locally legitimate projects that have the potential to create sustainable change. Adaptive and politically informed ways of working create specific opportunities and challenges for doing this well. • Flexible and adaptive programmes are deliberately designed to experiment and to make small investments in different areas, to see what will work. While this is often important for making headway on complex challenges, it can also leave partners exposed and can undermine trust. • It can also be challenging to balance the need to meet accountability commitments to donors while allowing local partners to take the lead in pursuing their own objectives. • Co-creating plans, priorities and activities with partners has the potential to resolve some of these tensions. But the time and patience required to do this successfully should not be underestimated, and can be difficult to maintain in the face of pressure from donors to see results within a confined timeframe. CY - London DA - 2020/03// PY - 2020 M3 - Briefing note PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/202002_odi_i4id_briefing_note_gender_web.pdf Y2 - 2021/02/18/13:28:43 ER - TY - RPRT TI - Retrospective ‘Outcome Harvesting’: Generating robust insights about a global voluntary environmental network AU - Rassmann, Kornelia AU - Smith, Richard AU - Mauremootoo, John AU - Wilson-Grau, Ricardo DA - 2013/04// PY - 2013 SP - 16 PB - Better Evaluation UR - https://www.betterevaluation.org/sites/default/files/Retrospective%20outcome%20harvesting.pdf Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - Revealing the Relational Mechanisms of Research for Development Through Social Network Analysis AU - Apgar, Marina AU - Fournie, Guillaume AU - Haesler, Barbara AU - Higdon, Grace Lyn AU - Kenny, Leah AU - Oppel, Annalena AU - Pauls, Evelyn AU - Smith, Matthew AU - Snijder, Mieke AU - Vink, Daan AU - Hossain, Mazeda T2 - The European Journal of Development Research AB - Achieving impact through research for development programmes (R4D) requires engagement with diverse stakeholders across the research, development and policy divides. Understanding how such programmes support the emergence of outcomes, therefore, requires a focus on the relational aspects of engagement and collaboration. Increasingly, evaluation of large research collaborations is employing social network analysis (SNA), making use of its relational view of causation. In this paper, we use three applications of SNA within similar large R4D programmes, through our work within evaluation of three Interidsiplinary Hubs of the Global Challenges Research Fund, to explore its potential as an evaluation method. Our comparative analysis shows that SNA can uncover the structural dimensions of interactions within R4D programmes and enable learning about how networks evolve through time. We reflect on common challenges across the cases including navigating different forms of bias that result from incomplete network data, multiple interpretations across scales, and the challenges of making causal inference and related ethical dilemmas. We conclude with lessons on the methodological and operational dimensions of using SNA within monitoring, evaluation and learning (MEL) systems that aim to support both learning and accountability. DA - 2023/04/01/ PY - 2023 DO - 10.1057/s41287-023-00576-y DP - Springer Link VL - 35 IS - 2 SP - 323 EP - 350 J2 - Eur J Dev Res LA - en SN - 1743-9728 UR - https://doi.org/10.1057/s41287-023-00576-y Y2 - 2023/04/13/11:11:00 ER - TY - RPRT TI - Review of the use of ‘Theory of Change’ in International development AU - Vogel, Isabel AB - A new report on Theory of Change (ToC) and its use in International development has just been produced. DA - 2012/04// PY - 2012 PB - DFID ST - DFID research UR - https://www.gov.uk/government/news/dfid-research-review-of-the-use-of-theory-of-change-in-international-development Y2 - 2016/03/24/16:58:26 ER - TY - RPRT TI - Rigorous Impact Evaluation in GIZ Governance Programmes AU - von Schiller, Armin T2 - How-To-Note AB - Why should I integrate an impact assessment in my programme? How will the programme benefit from it? Are the benefits worth the effort and expenses? How do I design and implement it in detail? Who is addressable for support? What should I pay attention to in order to get the most out of it? This note is meant to answer these questions. It addresses leaders and project staff of governance programmes who are interested in using this tool within their specific governance programme or project. This note provides a guideline and good practice recommendations on how to design and conduct an impact assessment and on how to fully utilise the benefits of the results for the programme and for communication with commissioners, partners and other donors. Additionally, this note will point to indirect benefits that can arise and that should not be ignored. Results of impact assessments are highly relevant for the GIZ as an institution. However, in this note we stress the benefits for the programme or project itself. In particular this note addresses the following aspects: • What are rigorous impact assessments and why should GIZ Governance programmes conduct them more often within their programmes? • Which phases does an impact assessment include? How do I set one up and which aspects deserve special attention in each phase to maximise the benefits for my programme? • What are the benefits I can expect from implementing and impact assessment? • Whom to ask at headquarters in case I need support? This note complements the policy brief “Strategic use of Rigorous Impact Evaluation” and the corporate strategic review on “Rigorous Impact Evaluation” written by the GIZ evaluation unit which focuses on the strategic use of rigorous impact evaluations (RIE) at GIZ. Based on the review findings, the policy brief presents recommendations for strategic planning and implementation of purpose-sensitive RIE using a number of key levers. By adopting central coordination and needs-based support mechanisms, the evaluation unit intends to promote the strategic use of RIE for evidence-based learning and decision-making within the organisation. CY - GIZ DA - 2020/// PY - 2020 PB - Bonn UR - https://www.idos-research.de/uploads/media/giz2021-0019en-rigorous-impact-evaluation-giz-governance-programmes_01.pdf Y2 - 2023/03/28/09:48:27 ER - TY - BLOG TI - Rubrics as a harness for complexity AU - Aston, Thomas T2 - Medium AB - In this final blog in the series, I want to look at the potential value of rubrics. While evaluability assessments can help us to… DA - 2020/12/24/T12:08:01.064Z PY - 2020 LA - en UR - https://thomasmtaston.medium.com/rubrics-as-a-harness-for-complexity-6507b36f312e Y2 - 2021/02/18/14:53:43 ER - TY - BLOG TI - Rules of Thumb – good idea or double-edged sword? AU - Green, Duncan T2 - From Poverty to Power AB - A recent conversation with a good governance programme in Myanmar tried to identify its underlying rules of thumb. Was that a good idea? DA - 2020/10/01/T06:30:11+00:00 PY - 2020 LA - en-GB UR - https://oxfamblogs.org/fp2p/rules-of-thumb-good-idea-or-double-edged-sword/ Y2 - 2020/10/01/10:20:41 ER - TY - GEN TI - Sample agendas for an evaluability assessment stakeholder workshop AU - Methods Lab AB - An evaluability assessment aims to assess the extent to which, and how best, an intervention can be evaluated in a reliable and credible fashion. These sample agendas are intended for people convening key stakeholders (such as project implementation staff and managers, donors and government officials) to discuss the purpose and scope of an impact evaluation and to identify key evaluation questions. This tool was developed by Bronwen McDonald, Anne Buffardi and Irene Guijt for use in Methods Lab projects. It accompanies the Methods Lab publication ‘Evaluability assessment for impact evaluation: guidance, checklists and decision support’. DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10042.pdf Y2 - 2018/10/11/00:00:00 ER - TY - GEN TI - Sample criteria to select case studies for evaluation AU - Methods Lab AB - Time and budget constraints often mean that organisations are unable to evaluate all of their programmes, and large programmes, operating in multiple locations, are unable to evaluate all project sites. This tool introduces two sets of criteria to support evaluators and programme managers to select case studies or programmes for evaluation: i) information about how relevant or feasible evaluation is for individual programmes, and ii) across the overall portfolio, strategic thinking around what types of cases are most important to understand. This tool was developed by Anne Buffardi, Irene Guijt, Simon Hearn and Tiina Pasanen for use in The Methods Lab projects. DA - 2018/11// PY - 2018 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10043.pdf Y2 - 2018/11/10/00:00:00 ER - TY - GEN TI - Sample interview questions for evaluability assessment AU - Methods Lab DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10032.pdf Y2 - 2018/11/10/00:00:00 ER - TY - VIDEO TI - Sara Earl Outcome Mapping pt 1, 2, and 3 AU - Earl, Sara AB - These three videos from Sarah Earl provide an introduction to the concepts of Outcome Mapping C5 - YouTube Video DA - 2008/// PY - 2008 NV - 3 UR - https://www.youtube.com/watch?v=fPL_KEUawnc ER - TY - RPRT TI - SAVI Approach Paper 13: Measuring Value for Money and using Value for Money Analysis AU - SAVI AB - SAVI has established its own framework for assessing Value for Money in annual performance – in relation to expenditure, economy, efficiency, effectiveness and equity. Routine tracking and analysis of expenditure and economy ensure that inputs are supplied and services delivered to partners in line with SAVI’s core values, whilst also meeting DFID requirements and competing... CY - London DA - 2016/12// PY - 2016 LA - en-US PB - DFID ST - SAVI Approach Paper 13 UR - http://savi-nigeria.org/approach/savi-approach-paper-13-measuring-value-money-using-value-money-analysis/ Y2 - 2018/02/28/09:03:27 ER - TY - RPRT TI - Scaling up Social Accountability in Complex Governance Systems: A Relational Approach for Evidencing Sustainability AU - Guerzovich, Florencia AU - Wadeson, Alix AB - When social accountability interventions scale up and their sustainability depends on the interactions of many agents and system components, related results are rarely observable at the end of an intervention. The 2019 OECD Development Assistance Committee’s (OECD DAC) revamped evaluations criteria for assessing sustainability acknowledges that such results are often emergent, and should be monitored and evaluated with this in mind. It therefore emphasizes a turn towards assessing complex processes prospectively. It also asks evaluations to consider how likely it is that these results are evident at the time they are monitored or evaluated. However,the social accountability field continues to have gaps regarding doing this effectively in practice. This paper presents and provides evidence from testing an innovative operational approach that has promising potential to support this aim - a sequential, relational rubric. This approach can support practitioners to monitor, evaluate and learn about the causal processes of scale up of social accountability interventions with an eye towards sustainability i.e., considering prospective sustainability. It is grounded in systems thinking, co-production and social learning theory, as well as links with collective governance and social contract theory for development. Evidence yielded from the authors’ testing of this approach on a sample of diverse projects from the Global Partnership for Social Accountability (GPSA) program revealed that the alleged ‘absence of evidence’ dilemma of social accountability scale up is due to ill-fitting concepts and methods for assessment. It challenges existing assumptions and findings that claim that social accountabilityprocesses do not scale and are unsustainable. The authors propose that by using fit-for-purpose concepts and methods with a focus on social learning and compromise – also called a ‘resonance pathway to scale’ which this paper discusses in detail – it is possible to observe loosely coordinated scale up processes at work in many (but not all) social accountability interventions and identify tangible evidence of prospective sustainability. An important caveat is that these processes, the outcomes they generate, and the corresponding evidence often look qualitatively different than the original intervention design and predictions for scale-up at that point in time. This is because the process of deliberation and compromise inherent to social accountability work in dynamic local systems introduces changes and new conditions for uptake by diverse actors in the public sector, civil society, and donor institutions. The paper concludes that even relatively small-scale localized projects of three to five years with budgets of less than one million USD, across different contexts and sectors can produce processes and outcomes which contribute to many forms of sustainability, including via scaleup.Furthermore, the cross-fertilization of learning and aggregation of results for scale-up across projects within and beyond the GPSA (and other programs) can help monitoring evaluation and learning (MEL) and social accountability practitioners alike to deliver on a program’s mandate. Doing so can also create new knowledge for the wider social accountability field that siloed interventions, lacking suitable concepts and methods for assessing scale-up and prospective sustainability, often fail to produce. The paper ends with recommendations for taking forward this approach and the associated benefits, implications and required investments. CY - Washington DC DA - 2024/01// PY - 2024 PB - World Bank UR - http://documents.worldbank.org/curated/en/099248202082451403/IDU143be23531a0f714f561b91515c596de86102 Y2 - 2024/02/13/13:08:03 ER - TY - BOOK TI - Scenarios: An Explorer's Guide AU - Shell International AB - Exploring the Future The future is ‘terra incognita’: although we may be able to guess the outcome of events that lie close to us, as we project beyond this we enter an unmapped zone full of uncertainty. Paradoxically, the range of options this reveals can seem paralysing. No one can definitively map the future, but we can explore the possibilities in ways that are specifically intended to support decision-making. At Shell we use scenario building to help us wrestle with the developments and behaviours that shape what the future may hold and prepare ourselves more effectively. We also believe it can inspire individuals and organisations to play a more active role in shaping a better future - for themselves, or even on a global scale. In this book, we use a metaphor of exploration and map-making to describe how we think about building scenarios. Like a set of maps describing different aspects of a landscape, scenarios provide us with a range of perspectives on what might happen, helping us to navigate more successfully. Exploration - of a territory or the future - involves both analytical thinking rooted in whatever facts are clear, and also informed intuition. This book describes the approach used to develop a set of global scenarios, ‘People and Connections’ several years ago under the guidance of Ged Davis. Since then, scenarios guided by Albert Bressand have been published, and more recently Shell has published a summary of its Energy Scenarios, ‘Scramble’ and ‘Blueprints’, developed under the guidance of the current leadership. These have built on, and extended, our approach. Indeed, Shell has been working with scenarios for almost 40 years, and we are still learning. Since the environment we live and work in is constantly changing, building scenarios demands continual innovation and creativity. CY - The Hague The Netherlands DA - 2008/// PY - 2008 SP - 98 PB - Shell International BV UR - https://www.shell.com/energy-and-innovation/the-energy-future/scenarios/new-lenses-on-the-future/earlier-scenarios/_jcr_content/par/expandablelist/expandablesection_842430368.stream/1447230877395/5ab112e96191fa79e1d30c31dc6e5cd2ce19ed518a4c1445ab32aa4c4b5c7ec5/shell-scenarios-explorersguide.pdf Y2 - 2018/03/09/00:00:00 ER - TY - BOOK TI - Science, Technology, and Innovation in Uganda: Recommendations for Policy and Action AU - Brar, Sukdeep AU - Farley, Sara E. AU - Hawkins, Robert AU - Wagner, Caroline S. T2 - A World Bank Study DA - 2011/// PY - 2011 SP - 132 PB - World Bank UR - http://documents.worldbank.org/curated/en/188271468115452838/pdf/588440PUB0Scie101public10BOX353816B.pdf ER - TY - RPRT TI - Seeing the Combined Effects of Aid Programmes AU - Burge, Richard AU - Nadelman, Rachel AU - McGee, Rosie AU - Fox, Jonathan AU - Anderson, Colin T2 - IDS Policy Briefing AB - Multiple aid agencies often try to support change in the same places, at the same time, and with similar actors. Surprisingly, their interactions and combined effects are rarely explored. This Policy Briefing describes findings from research conducted on recent aid programmes that overlapped in Mozambique, Nigeria, and Pakistan, and from a webinar with UK Foreign, Commonwealth & Development Office (FCDO) advisors and practitioners. The research found three distinct categories of ‘interaction effects’: synergy, parallel play, and disconnect. We explore how using an ‘interaction effects’ lens in practice could inform aid agency strategies and programming. CY - Brighton DA - 2022/05/10/ PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - IDS SN - 196 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17391 Y2 - 2022/07/01/09:03:59 ER - TY - BLOG TI - Sense-making with stakeholders and rubrics AU - King, Julian T2 - Evaluation and Value for Investment AB - Principles and processes DA - 2023/09/10/ PY - 2023 M3 - Substack newsletter UR - https://juliankingnz.substack.com/p/sense-making-with-stakeholders-and?publication_id=1205622&utm_campaign=email-post-title&r=qn2r Y2 - 2023/09/29/10:31:41 ER - TY - RPRT TI - Sensemaking Workshop Preparation Guide and Facilitator Guide and Sensemaking Training AU - UNDP AB - Based on experience from running Sensemaking workshops for UNDP offices and government partners, the Asia-Pacific Regional Innovation Centre developed the Sensemaking Preparation Guide and Facilitator Guide to share its knowledge with teams and organization that are interested in using the Sensemaking process. DA - 2022/01// PY - 2022 LA - en PB - UNDP UR - https://www.undp.org/publications/sensemaking-workshop-preparation-guide-and-facilitator-guide-and-sensemaking-training Y2 - 2023/01/24/10:41:38 ER - TY - BOOK TI - Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis AU - Schneider, Carsten Q. AU - Wagemann, Claudius T2 - Strategies for social inquiry AB - "Qualitative Comparative Analysis (QCA) and other set-theoretic methods distinguish themselves from other approaches to the study of social phenomena by using sets and the search for set relations. In virtually all social science fields, statements about social phenomena can be framed in terms of set relations, and using set-theoretic methods to investigate these statements is therefore highly valuable. This book guides readers through the basic principles of set theory and then on to the applied practices of QCA. It provides a thorough understanding of basic and advanced issues in set-theoretic methods together with tricks of the trade, software handling and exercises. Most arguments are introduced using examples from existing research. The use of QCA is increasing rapidly and the application of set-theory is both fruitful and still widely misunderstood in current empirical comparative social research. This book provides an invaluable guide to these methods for researchers across the social sciences"-- CY - Cambrigde DA - 2012/// PY - 2012 DP - Gemeinsamer Bibliotheksverbund ISBN SP - 350 LA - eng PB - Cambridge Univ. Press SN - 978-1-107-60113-0 978-1-107-01352-0 ST - Set-theoretic methods for the social sciences KW - Comparative method KW - Mathematical models KW - Qualitative Methode KW - Set theory KW - Social science KW - Vergleichende Forschung ER - TY - RPRT TI - Setting new standards for better MEL. Lessons for grantees & funders AU - Colnar, Megan AU - Azevedo, Andrea AU - Tolmie, Courtney AU - Caddick, Hannah AB - How can donors and grantees work together to create effective monitoring, evaluation, and learning (MEL) practices that drive field-wide transformation? The Open Society Foundation’s Fiscal Governance Program found success by focusing on six key approaches, including empowering grantees and relinquishing power. In 2021, an external close-out evaluation by Intention to Impact of the program (which ran for 7 years and gave over $150 million in grants) revealed something pretty remarkable—the program’s deliberate focus on strengthening field-wide monitoring, evaluation, and learning practices was a success. Substantial capacity increases were observed across key institutions and grantees, new complexity-sensitive practices and methods were being actively championed and deployed, and a growing community of better-connected practitioners were exchanging tips and tricks on how to apply smart, context-specific MEL across fiscal governance issues. What’s more, in this evaluation, most grantees gave high praise to these efforts. So, how did this come about? We detail the six different approaches we used in our new publication Setting new standards for better MEL: Lessons for funders and grantees. The approaches range from checking power dynamics to growing skills for evaluative thinking and seeding peer learning and field-wide research. The publication is paired with a toolkit and showcases resources we used and iterated on across the various approaches. DA - 2023/// PY - 2023 DP - Zotero LA - en PB - Better Evaluation UR - https://www.betterevaluation.org/sites/default/files/2023-04/Setting%20new%20standards%20for%20better%20MEL.pdf Y2 - 2023/07/04/00:00:00 ER - TY - BLOG TI - Simplicity, Accountability and Relationships: Three ways to ensure MEL supports Adaptive Management AU - Roche, Chris T2 - From Poverty to Power AB - Based on a recent discussion in Manila, Chris Roche reflects on how Monitoring, Evaluation and Learning can better support 'adaptive programming'. DA - 2018/07/03/T06:30:33+00:00 PY - 2018 LA - en-GB ST - Simplicity, Accountability and Relationships UR - https://oxfamblogs.org/fp2p/simplicity-accountability-and-relationships-three-ways-to-ensure-mel-supports-adaptive-management/ Y2 - 2018/07/17/10:47:20 ER - TY - SLIDE TI - Small Foundation Network Partner Evaluation Toolkit: Network Evaluation for Network Coordinators A2 - Small Foundation AB - This Toolkit This Toolkit presents an approach to network evaluation that is designed for network coordinators. This Toolkit provides guidance for network coordinators on how to: • More effectively use information they are already collecting as part of their routine coordination duties; • Collect other useful data that would support their network’s health and development; • Integrate network evaluation tasks into network activities, like convenings, to streamline the process; • Use data to effectively coordinate, grow, and sustain their networks. The Toolkit also provides examples of surveys, questionnaires, and dashboards, as well as tips for easy implementation. DA - 2021/// PY - 2021 UR - https://smallfoundation.ie/wp-content/uploads/2021/06/SF-Eval-Toolkit_June-2021.pdf Y2 - 2022/06/17/11:34:01 ER - TY - RPRT TI - Small Foundation Network Partner Evaluation Toolkit Network Evaluation for Network Coordinators Frequently Asked Questions AU - Small Foundation AB - This Frequently Asked Questions (FAQ) document is written for Small Foundation network partners. It accompanies Network Partner Evaluation Toolkit, which outlines four evaluation tools that Small Foundation is encouraging, and supporting, its network partners to use to improve their operations and, ultimately, increase their impact. This FAQ document answers typical questions about these evaluation tools and provides additional detailed recommendations for how network coordinators can effectively implement them. This document has five sections: 1) A brief introduction to network theory and the value of evaluation for network coordinators. 2) An overview of the four evaluation tools and answers general questions about incorporating evaluation into network coordination duties. 3) Recommendations regarding the Network Coordinator Administrative Information (tool #1). 4) Information about Social Network Analysis (tool #2). 5) Recommendations regarding the Network Participant Survey (tool #3). 6) Information about the Collaborative Activity Dashboard (tool #4). DA - 2021/// PY - 2021 PB - Small Foundation UR - https://smallfoundation.ie/wp-content/uploads/2021/06/SF-Eval-Toolkit-FAQ_June-2021.pdf Y2 - 2022/06/17/11:37:41 ER - TY - RPRT TI - Social network analysis handbook: connecting the dots in humanitarian programs AU - IRC AB - The following handbook is designed to provide a step by step guide to the application of Social Network Analysis for the IRC. CY - New York USA DA - 2016/07// PY - 2016 PB - International Rescue Committee ST - Social network analysis handbook UR - https://www.rescue.org/resource/social-network-analysis-handbook-connecting-dots-humanitarian-programs Y2 - 2017/01/29/10:54:13 KW - IMPORTANT ER - TY - BLOG TI - Social Return on Investment AU - Better evaluation T2 - Better Evaluation AB - Social Return on Investment (SROI) is a systematic way of incorporating social, environmental, economic and other values into decision-making processes. By helping reveal the economic value of social and environmental outcomes it creates a holistic perspective on whether a development project or social business or enterprise is beneficial and profitable. This perspective opens up new opportunities and forms the basis for innovative initiatives that genuinely contribute to positive social change and poverty reduction for all. SROI balances proving and improving or addresses the paradox between accountability and learning by placing the perspectives of the different stakeholders at the center of the valuation process. ​SROI originated in the USA from social enterprises interested in new ways to value the contributions they were making to society. It later arrived in Europe, where there is an increasing interest in the methodology as noted by recent publications by Context international cooperation in the Netherlands, the New Economics Foundation in the UK and the SROI Network head-quartered in the UK. SROI is used for planning purposes in terms of designing a Theory of Change, or Business Plan, and for assessing to what extent impact is realised or changes need to occur in the Business Plan. Although the SROI approach supports the thinking along the lines of a result chain, it does not support the idea of the components being connected in a linear fashion. The SROI approach is embedded in the acceptance of development taking place in situations of complexities. Here is a simple, illustrative example: A project aims to uplift the standard of living of people in a certain area and a beekeeping initiative is set up. As a result a beekeeper now enjoys regular meals whereas before this was not the case. In traditional Cost Benefit analyses, the value of the lunch would be measured in market prices. However after interviewing the beneficiaries, and applying some valuation tools, it turns out that the ‘real’ value is much higher than the market price; social value has been created above the market / economic value which is now being accounted for. Like traditional cost-benefit analysis, SROI includes a ratio; in this case a Social Return on Investment ratio. Where in traditional cost benefit analyses the ratios would be used to compare different projects, the SROI ratio is much more seen as one element in explaining and communicating general progress of certain developments. The number itself is not seen as the end goal. It can be interpreted as aiding the narrative of this particular initiative. The aspect of stakeholder perspectives is essential in the SROI approach. It is precisely the value perspectives of the stakeholders (and most importantly the key beneficiaries), assessed, not by assuming these values, but by thoughtfully and intellectually engaging the stakeholders themselves, which is at the heart of this innovative (e)valuation approach. DA - 2016/// PY - 2016 UR - https://www.betterevaluation.org/en/approach/SROI Y2 - 2018/10/19/00:00:00 ER - TY - BLOG TI - Sounding clever or being smart? – How to do more with less in evaluating governance programmes AU - Aston, Tom T2 - Care Insights AB - About a decade ago, the development sector fell into the same trap the financial services industry did in the mid-1990s. We were all seduced by clever... DA - 2018/09/25/ PY - 2018 LA - en-gb ST - Sounding clever or being smart? UR - https://insights.careinternational.org.uk/development-blog/sounding-clever-or-being-smart-how-to-do-more-with-less-in-evaluating-governance-programmes Y2 - 2018/09/26/10:29:54 ER - TY - RPRT TI - SPACES MERL: Systems and Complexity White Paper AU - USAID AB - The Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES MERL) project is an activity funded by USAID’s Global Development Lab and the Bureau for Policy, Planning and Learning (PPL). This three-year activity aims to bring a variety of tools and methodologies that decision-makers can use (alone or in combination) to provide comprehensive systems analysis. The activity is being implemented from 2015 to 2018 by a consortium of organizations expert in systems and complexity, including the Global Obesity Prevention Center (GOPC) at Johns Hopkins University (Prime), Global Knowledge Initiative (GKI), LINC and ResilientAfrica Network (RAN). This Systems and Complexity White Paper is a collaborative effort of the SPACES MERL team, designed to frame the international development landscape, with particular reference to USAID-funded activities, for application of systems and complexity approaches to design, monitoring and evaluation. Customized to the systems and complexity layperson with in-depth knowledge of international development practice, the objectives of this white paper are three-fold:  Provide an overview of systems and complexity practice, its current state of application and relevance to international development practice;  Establish a taxonomy of systems and complexity tools, highlighting the fit of those offered by SPACES MERL within the wider landscape; and  Review and provide information on application of SPACES MERL tools, their purpose and construction, required data, and their applicability to specific contexts. DA - 2016/03// PY - 2016 SP - 95 UR - https://pdf.usaid.gov/pdf_docs/PA00M7QZ.pdf Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - Special Issue: Causal Mechanisms in Program Evaluation AU - Schmidt, Johannes T2 - New Directions for Evaluation DA - 2020/// PY - 2020 DO - 10.1002/ev.20357 DP - Wiley Online Library VL - 2020 IS - 167 SP - 1 EP - 6 LA - en SN - 1534-875X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20357 Y2 - 2022/01/28/11:40:35 ER - TY - ELEC TI - Spherit - Contextual reasoning maps AU - ContextSpheres.com T2 - ContextSpheres.com AB - Spherit is a whole view technology system that streamlines meaning making, solutioning, and action taking in business, healthcare, relationships, and life in general, by displaying complex situation-specific topics in context on a customizable conversation map. When the Spherit system is activated it generates and displays a snapshot of any complex situation in a way that makes it digestible, while engaging our intellect and our intuition in the reflecting and meaning-making process. It's a unifying visual learning system that communicates information in a non-linear graphical format. It works for individuals, teams and organizations. It has a "share and compare" feature that fast-tracks conversations between individuals or groups. And it offers powerful metrics. DA - 2004/// PY - 2004 LA - en-US UR - https://spherit.com/ Y2 - 2023/04/27/13:22:03 ER - TY - BLOG TI - SROI Self-assessment tool AU - SROI Network AB - The online self assessment tool was created by the SROI Network in collaboration with Hall Aitken to help users judge how well their evaluation practices adhere to principles of best practice. The tool is comprised of a seven stage questionnaire, showing results as a spider chart illustrating areas of strength and areas for improvement. The questionnaire itself is structured around the seven principles of SROI. These principles include ‘involve stakeholders’, ‘be transparent’ and ‘do not overclaim’, and together make up the core framework around which SROI is based. Users receive a score for each principle, and an overall average, giving them an idea of to what extent they have successfully applied the principles. High scoring users can then use their score to guide them towards applying for assurance and accreditation. The tool provides guidance, support and more advice to people who want to improve the way in which their organisation measures their social value and assesses their service delivery but don’t know where to start. DA - 2016/// PY - 2016 UR - http://www.socialvalueuk.org/what-is-social-value/sroi-self-assessment-tool/ Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - Stakeholder and Social Network Analysis - Guidance Note AU - IRC AB - Stakeholder Analysis is used to identify the actors and relationships that influence project outcomes. This guidance, from IRC, can help you determine how to work and who to partner, coordinate or engage with in order to best achieve the outcome. It builds on existing stakeholder information and typically will include a participatory internal meeting or workshop. DA - 2019/10/04/T10:36:29-04:00 PY - 2019 LA - en M3 - Text PB - International Rescue Committee UR - https://usaidlearninglab.org/library/stakeholder-and-social-network-analysis-guidance-note Y2 - 2020/10/15/10:24:36 ER - TY - RPRT TI - Starting out on Social Return on Investment AU - SROI Network AU - Aitken, Hall AB - This guide, created by the SROI Network in collaboration with Hall Aitken, is aimed at providing guidance to those who have never used SROI on where to begin. The guide provides practical steps to beginning the process of SROI and includes links to tools aimed at supporting its implementation. Excerpt "Most public, private and third sector organisations have a pretty good idea of the costs of what they do. Annual accounts, management accounts, budget reports and a whole accountancy profession add up to a great deal of effort to make sure this is the case. Some organisations are quite good at counting what they do with these resources. They can track the number of users or contacts, or customers. Many can provide some evidence that these activities lead to some sort of change. But very few can explain clearly why all this matters. What would happen if they did not exist? What is the real value of what they do? Social Return on Investment sets out to redress the balance by looking at value not just cost." Contents Part one – why, what and how? Why does social value matter? What is SROI? How do I do SROI? Next steps Part two – progress tool Part three – practical next steps Steps to involve stakeholders Steps to understand what changes Steps to value the things that matter Steps to only include what is material Steps to avoid overclaiming Steps to be transparent Steps to verify the result DA - 2014/// PY - 2014 SP - 8 UR - http://www.socialvalueuk.org/app/uploads/2016/03/Starting%20Out%20Guide.pdf Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - State of the Art on Use of Theory of Change in the Development Sector AU - Goodier, Sarah AU - Apgar, Marina AU - Clark, Louise T2 - SDC-IDS Briefing Note 7 AB - The purpose of this briefing note is to add to SDC’s understanding of Theory of Change (ToC), drawing on the literature and practice to sketch out the current state of the art approach. This involves expanding on ToC beyond SDC’s current practice of using Impact Hypotheses (IH) to bridge it to operational practice and use ToC more explicitly in the project/programme cycle management (PCM) processes. Sharing the state of the art on use of ToC in the development sector, this briefing note outlines what a ToC is, what it is used for and why it is needed in the development sector. It discusses ToC as both a process and a product, providing step by step guidance on how to facilitate a ToC process. The differences between a ToC and a logframe are highlighted. Some key criteria for recognising when you have a ‘good’ ToC are also included. This brief is aimed at SDC staff, in particular Programme Officers, and staff of partner organisations involved in the management of SDC interventions. CY - Brighton DA - 2018/08// PY - 2018 DP - opendocs.ids.ac.uk LA - en PB - IDS UR - https://opendocs.ids.ac.uk/opendocs/handle/123456789/14039 Y2 - 2019/03/06/12:10:50 ER - TY - BLOG TI - Storytelling and Evaluation: Can Pathways be the Theory Missing in Explanatory Narratives? AU - Guerzovich, Florencia T2 - Medium AB - Florencia Guerzovich DA - 2023/10/18/T10:30:01.863Z PY - 2023 LA - en ST - Storytelling and Evaluation UR - https://medium.com/@florcig/storytelling-and-evaluation-can-pathways-be-the-theory-missing-in-explanatory-narratives-0625a63adfc6 Y2 - 2023/11/14/23:11:00 ER - TY - RPRT TI - Storytelling for Systems Change: insights from the field AU - Thea Snow AU - David Murikumthara AU - Teya Dusseldorp AU - Rachel Fyfe AU - Lila Wolff AU - Jane McCracken AB - Capturing the impact of community-led work The Centre for Public Impact, Dusseldorp Forum, and Hands Up Mallee have been exploring how stories can be used to more effectively communicate the impact of community-led systems change work. Community-led place based initiatives are modelling new ways of working - shifting away from top down, program-focussed approaches towards an approach grounded in systems thinking and community-led innovations. However, while these stories of change are sitting in communities, they’re often not being told or celebrated. We wanted to understand why this is, and what might be done to better enable these stories to be shared and heard. The story of storytelling We talked to a range of people to uncover the story of storytelling - including collective impact backbone team members, community members, storytelling experts, and those working in and around community-led systems change initiatives across Australia. We explored the roles stories play in different communities; what good storytelling looks like; what barriers to storytelling might be; and what role stories can play in supporting systems change. Our findings We have learned through this project that stories can be used both to change the system and to evaluate, understand and showcase the change that is occurring in communities. We have heard that different stories require different approaches – stories that are seeking to enable change look different to those that are seeking to celebrate change. DA - 2022/// PY - 2022 PB - Centre for Public Impact UR - https://www.centreforpublicimpact.org/assets/documents/storytelling-for-systems-change-report.pdf Y2 - 2022/07/26/11:32:15 ER - TY - MGZN TI - Strategy as Simple Rules AU - Eisenhardt, Kathleen M. AU - Sull, Donald T2 - Harvard Business Review AB - When the business landscape was simple, companies could afford to have complex strategies. But now that business is so complex, they need to simplify. Smart companies have done just that with a new approach: a few straightforward, hard-and-fast rules that define direction without confining it. DA - 2001/01/01/T05:00:00Z PY - 2001 DP - hbr.org IS - January 2001 SN - 0017-8012 UR - https://hbr.org/2001/01/strategy-as-simple-rules Y2 - 2020/10/01/10:35:20 KW - Entrepreneurial management KW - Internet KW - Strategy KW - Strategy execution ER - TY - RPRT TI - Strategy Development: Most Signficant Change (MSC) Toolkit AU - ODI AB - The Most Significant Change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the programme cycle and provides information to help people manage it. MSC contributes to evaluation because it provides data on impact and outcomes which can be used to help assess the performance of the programme as a whole. Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most important of these by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of the reported changes. When the technique is successfully implemented, whole teams of people begin to focus their attention on programme impact. MSC has had several names since it was conceived, each emphasising a different aspect. Examples are: ‘Monitoring-without-indicators’ – MSC does not make use of predefined indicators, especially ones which have to be counted and measured; or the ‘story approach’ – the answers to the central question about change are often in the form of stories of who did what, when and why, and the reasons the DA - 2009/01// PY - 2009 SP - 3 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6383.pdf Y2 - 2018/10/18/00:00:00 ER - TY - RPRT TI - Strategy Testing: An innovative approach to monitoring highly flexible aid programs AU - Ladner, Debra T2 - Working Politically in Practice AB - The international development community has increasingly embraced the idea that finding durable solutions to complex development problems requires new ways of working that move beyond industry norms. This paper makes an important contribution to the current debate by outlining an innovative monitoring system called Strategy Testing (ST). This is the third paper in the Working Politically in Practice paper series, launched together with the Australian Department of Foreign Affairs and Trade. DA - 2015/09// PY - 2015 PB - The Asia Foundation SN - 3 UR - http://asiafoundation.org/publication/strategy-testing-an-innovative-approach-to-monitoring-highly-flexible-aid-programs/ Y2 - 2019/05/16/00:00:00 ER - TY - DATA TI - Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? AU - Punton, Melanie AU - Welle, Katharina AB - This CDI Practice Paper by Melanie Punton and Katharina Welle explains the methodological and theoretical foundations of process tracing, and discusses its potential application in international development impact evaluations. It draws on two early applications of process tracing for assessing impact in international development interventions: Oxfam Great Britain (GB)’s contribution to advancing universal health care in Ghana, and the impact of the Hunger and Nutrition Commitment Index (HANCI) on policy change in Tanzania. In a companion to this paper, Practice Paper 10 Annex describes the main steps in applying process tracing and provides some examples of how these steps might be applied in practice. DA - 2015/04// PY - 2015 DP - DOI.org (Crossref) LA - en NV - 10 PB - Centre for Development Impact ST - Straws-in-the-wind, Hoops and Smoking Guns UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/5997 Y2 - 2019/06/21/18:48:54 ER - TY - VIDEO TI - Strengthen Your Team’s CLA Practices: Introducing the CLA Maturity Tool for USAID Implementers AU - Social Impact , Inc AB - The Collaborating, Learning & Adapting (CLA) Maturity Tool has been used by USAID operating units since 2015. The tool enables staff to self-assess their current CLA practice and plan ways to improve their internal and external collaboration, organizational learning efforts, and adaptive management practices. Recently, SI built on this resource to create the CLA Maturity Tool for implementing partners (IPs) to provide a version that speaks more directly to the experience of international and local organizations. Learn more here. DA - 2023/01/10/ PY - 2023 DP - YouTube ST - Strengthen Your Team’s CLA Practices UR - https://www.youtube.com/watch?v=zwdO9FZq_PM Y2 - 2023/03/20/11:30:53 ER - TY - BLOG TI - Success Case Method AU - Better Evaluation T2 - Better Evaluation AB - The Success Case Method (SCM) involves identifying the most and least successful cases in a program and examining them in detail. This approach was developed by Robert Brinkerhoff to assess the impact of organizational interventions, such as training and coaching, though the use of SCM is not limited to this context. It is a useful approach to document stories of impact and to develop an understanding of the factors that enhance or impede impact. The Success Case Method deliberately looks at the most, and least, successful participants of a program. The purpose is not to examine the average performance - rather, by identifying and examining the extreme cases, it asks: 'When the program works, how well does it work? What is working, and what is not?'. UR - https://www.betterevaluation.org/en/plan/approach/success_case_method Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Summarising portfolio change: results frameworks at organisational level AU - Simister, Nigel T2 - M&E Paper 10 CY - Oxford DA - 2016/01// PY - 2016 DP - Zotero SP - 22 LA - en PB - Intrac ER - TY - RPRT TI - Supporting adaptive management. Monitoring and evaluation tools and approaches AU - Pasanen, Tiina AU - Barnett, Inka AB - Key messages • This working paper introduces a set of monitoring and evaluation (M&E) tools and approaches, discussing their potential usefulness in supporting adaptive management in development and humanitarian programmes. It emphasises adaptive programmes characterised by complex aspects, such as: (1) they are innovative; (2) they have uncertain pathways for change; or (3) they operate in uncertain or unstable environments. • The majority of these tools have been used in international development for many years. However, adaptive management brings additional challenges for monitoring and evaluating programmes, as they require intentional M&E design from the start that is oriented towards both learning and accountability. • All of the tools and approaches introduced in this paper have potential to support learning and adaptation, although in various ways and at different stages of a programme. Some tools can support strategic planning and diagnosing throughout a programme – especially during design and inception – while others can help analyse causal relationships at specific points in a programme. It is important to tailor the approach used for its intended purpose. However, whether learning and adaptation happens depends also on factors other than the choice of M&E methods. • For some of these approaches a considerable body of evidence already exists but, for many, more practical examples and systematic analysis is needed. In addition to building the evidence base concerned with which approaches are suitable for different types of adaptive programmes and why, it is also important to improve understanding of the enabling environmental conditions necessary for the tools and approaches outlined here to facilitate and strengthen evaluative thinking, evidence-informed decision-making and ongoing programme iteration. CY - London DA - 2019/12// PY - 2019 DP - Zotero LA - en M3 - Working Paper PB - ODI SN - 569 ER - TY - ELEC TI - Supporting users with the OutNav approash AU - Matter of Focus AB - This report has been generated in OutNav, using the theory-based approach to evaluation pioneered by Matter of Focus. Example of a report from the tool, explaining OutNav approach DA - 2019/// PY - 2019 LA - en UR - https://www.outnav.net/view-live-report/g/nlzyexVgPgXqtvgKvfBf2wJvIjw5syiM#findings-pathway-406-stone-4583 Y2 - 2021/02/18/14:20:25 ER - TY - RPRT TI - System Change: A Guidebook for Adopting Portfolio Approaches AU - Wellsch, Brent AB - This guidebook codifies the principles and methods of applying systems change and portfolio approaches to complex development challenges with practical tools and examples. It is based on the empirical learning generated from the collaborative initiatives in UNDP Country Offices in Bhutan, Pakistan, the Philippines, and Viet Nam with support from Regional Innovation Centre for Asia and the Pacific. CY - Bangkok DA - 2022/03/28/ PY - 2022 LA - en PB - UNDP ST - System Change UR - https://www.undp.org/publications/system-change-guidebook-adopting-portfolio-approaches Y2 - 2023/11/07/10:27:13 ER - TY - RPRT TI - System Innovation on Purpose AU - Leadbeater, Charles AU - Winhall, Jennie AB - In Building Better Systems, we introduced four keys to unlock system innovation: purpose and power, relationships and resource flows. These four keys make up a set. Systems are often hard to change because power, relationships, and resource flows are locked together in a reinforcing pattern to serve the system’s current purpose. Systems start to change fundamentally when this pattern is disrupted and opened up. Then a new configuration can emerge, serving a new purpose. In this essay series we delve deeper into these four keys and provide practical advice on how they can be put to use. This essay is about the role that purpose plays in orchestrating complex systems and how system innovators can create a new system around a new sense of purpose. CY - København K DA - 2021/11// PY - 2021 LA - en-GB PB - The Rockwool Foundation UR - https://www.systeminnovation.org/article-the-patterns-of-possibility Y2 - 2022/06/17/12:02:20 ER - TY - RPRT TI - SystemCraft - a primer: How to Tackle our Toughest Problems AU - Simpson, Kate AU - Randall, Ian AB - Systemcraft is our applied framework to help leaders and organisations get started and keep going when faced with complex problems. It is built on our practical experience. It draws on a broad body of research, action and theory from the worlds of complexity thinking, systems theory, adaptive management, leadership development, social movements, development theory and beyond. Systemcraft has been designed to make systems thinking something any leader can apply when they find themselves faced with a complex problem and asking, ‘So what do I do next?’ CY - Nairobi DA - 2020/09// PY - 2020 PB - Wasafiri UR - https://www.wasafirihub.com/wp-content/uploads/2020/10/Wasafiri-SystemCraft-2020-Small.pdf Y2 - 2021/11/09/12:10:22 ER - TY - JOUR TI - Systemic action research: Changing system dynamics to support sustainable change AU - Burns, Danny T2 - Action Research AB - This article explores the characteristics of systemic action research. It looks at the conceptual underpinnings of systemic action research and explores some of the ways in which it differs from (builds on) other forms of action research. It then explores some of the issues and dilemmas faced by systemic action researchers. DA - 2014/03// PY - 2014 DO - 10.1177/1476750313513910 DP - DOI.org (Crossref) VL - 12 IS - 1 SP - 3 EP - 18 J2 - Action Research LA - en SN - 1476-7503, 1741-2617 ST - Systemic action research UR - http://journals.sagepub.com/doi/10.1177/1476750313513910 Y2 - 2020/10/15/14:22:29 ER - TY - RPRT TI - Systems Thinking and Practice: A Guide to Concepts, Principles and Tools for FCDO and Partners AU - Woodhill, Jim AU - Millican, Juliet T2 - K4D AB - This guide is a basic reference on systems thinking and practice tailored to the context and needs of the UK Government’s Foreign, Commonwealth & Development Office (FCDO). It is an output of the FCDO Knowledge for Development Programme (K4D), which facilitated a Learning Journey on Systems Thinking and Practice with FCDO staff during 2021 and 2022. The guide offers a common language and shared framing of systems thinking for FCDO and its partners. It explores what this implies for working practices, business processes and leadership. It also offers links to additional resources and tools on systems thinking. We hope it can support systems thinking to become more commonplace within the culture and practices of FCDO and working relations with partner organisations. CY - Brighton DA - 2022/02/03/ PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies ST - Systems Thinking and Practice UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17862 Y2 - 2023/02/08/15:50:53 ER - TY - BOOK TI - Systems Thinking. Applied. A primer AU - Edson, Robert DA - 2008/10/08/ PY - 2008 VL - 1.1 SP - 66 PB - ASysT Institute UR - http://www.anser.org/docs/systems_thinking_applied.pdf Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - Taking Stock of Empowerment Evaluation: An Empirical Review AU - Miller, Robin Lin AU - Campbell, Rebecca T2 - American Journal of Evaluation AB - Empowerment evaluation entered the evaluation lexicon in 1993. Since that time, it has attracted many adherents, as well as vocal detractors. A prominent issue in the debates on empowerment evaluation concerns the extent to which empowerment evaluation can be readily distinguished from other approaches to evaluation that share with it an emphasis on participatory and collaborative processes, capacity development, and evaluation use. A second issue concerns the extent to which empowerment evaluation actually leads to empowered outcomes for those who have participated in the evaluation process and those who are the intended beneficiaries of the social programs that were the objects of evaluation. The authors systematically examined 47 case examples of empowerment evaluation published from 1994 through June 2005. The results suggest wide variation among practitioners in adherence to empowerment evaluation principles and weak emphasis on the attainment of empowered outcomes for program beneficiaries. Implications for theory and practice are discussed. DA - 2006/09/01/ PY - 2006 DO - 10.1177/109821400602700303 VL - 27 IS - 3 SP - 296 EP - 319 J2 - American Journal of Evaluation SN - 1098-2140 UR - https://pdfs.semanticscholar.org/236a/2854a6b1a611f835efc2b6fb59d9e773d2c1.pdf Y2 - 2018/10/18/00:00:00 ER - TY - BLOG TI - Tale of the converted: how complex social problems have made me question the use of data in driving impact AU - Burgoyne, John T2 - Impact of Social Sciences AB - 15 Shares In practice the way in which research impacts and influences policy and society is often thought to be a rational, ordered and linear process. Whilst this might represent a ‘common sense’ understanding of research impact, in this cross-post John Burgoyne reflects on how upending the primacy of data and embracing complexity can lead to a more nuanced and effective understanding of research impact. DA - 2019/09/23/T11:01:49+00:00 PY - 2019 LA - "en-US" ST - Tale of the converted UR - https://blogs.lse.ac.uk/impactofsocialsciences/2019/09/23/tale-of-the-converted-how-complex-social-problems-have-made-me-question-the-use-of-data-in-driving-impact/ Y2 - 2021/07/30/13:00:18 ER - TY - GEN TI - Template concept note for an impact evaluation AU - Methods Lab DA - 2015/11// PY - 2015 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10040.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - Ten years of Outcome Mapping adaptations and support AU - Smith, Richard AU - Mauremootoo, John AU - Rassmann, Kornelia T2 - OM Resources: Key Community Documents AB - An analysis of how and where Outcome Mapping has been applied, how users have experienced OM and the support options available and required for its use. This research was commissioned by the OMLC Stewards. DA - 2012/// PY - 2012 PB - Outcome Mapping Learning Community UR - https://www.outcomemapping.ca/resource/ten-years-of-outcome-mapping-adaptations-and-support Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - Testing the Waters: A Qualitative Comparative Analysis of the Factors Affecting Success in Rendering Water Services Sustainable Based on ICT Reporting AU - Welle, Katharina AU - Williams, Jennifer AU - Pearce, Joseph AU - Befani, Barbara AB - This research conducted by WaterAid, Itad and IRC aims to understand the factors that facilitate and inhibit the success of ICT-based reporting to improve rural water supply sustainability. DA - 2015//16/ PY - 2015 PB - MAVC ST - Testing the Waters UR - http://itad.com/reports/testing-the-waters-a-qualitative-comparative-analysis-of-the-factors-affecting-success-in-rendering-water-services-sustainable-based-on-ict-reporting/ Y2 - 2016/05/12/10:24:43 ER - TY - JOUR TI - The Actor-Based Change Framework: A Pragmatic Approach to Developing Program Theory for Interventions in Complex Systems AU - Koleros, Andrew AU - Mulkerne, Sean AU - Oldenbeuving, Mark AU - Stein, Danielle T2 - American Journal of Evaluation AB - Despite a wide body of literature on the importance of program theory and the need to tackle complexity to improve international development programming, the use of program theory to underpin interventions aimed at facilitating change in complex systems remains a challenge for many program practitioners. The actor-based change framework offers a pragmatic approach to address these challenges, integrating concepts and frameworks drawn from complexity science and behavioral change literature to develop robust program theory for complex interventions. This article presents the conceptual framework for the approach and describes how it has been applied in practice on an evaluation of a security and justice program in Nepal. It concludes with a discussion of the strengths and weaknesses of the approach in practice and how it can be applied more widely to improve program theory for interventions in complex systems. DA - 2020/03// PY - 2020 DO - 10.1177/1098214018786462 DP - DOI.org (Crossref) VL - 41 IS - 1 SP - 34 EP - 53 J2 - American Journal of Evaluation LA - en SN - 1098-2140, 1557-0878 ST - The Actor-Based Change Framework UR - http://journals.sagepub.com/doi/10.1177/1098214018786462 Y2 - 2021/05/06/13:44:47 ER - TY - RPRT TI - The adoption of innovation in international development organisations: lessons for development co-operation AU - Kumpf, Benjamin AU - Jhunjhunwala, Parnika T2 - OECD Development Co-operation Working Papers AB - Addressing 21st century development challenges requires investments in innovation, including the use of new approaches and technologies. Currently, many development organisations prioritise investments in isolated innovation pilots that leverage a specific approach or technology rather than pursuing a strategic approach to expand the organisation’s toolbox with innovations that have proven their comparative advantage over what is currently used. This Working Paper addresses this challenge of adopting innovations. How can development organisations institutionalise a new way of working, bringing what was once novel to the core of how business is done? Analysing successful adoption efforts across five DAC agencies, the paper lays out a proposed process for the adoption of innovations. The paper features five case-studies and concludes with a set of lessons and recommendations for policy makers on innovation management generally, and adoption of innovation in particular. CY - Paris DA - 2023/05// PY - 2023 DP - Zotero LA - en PB - OECD Publishing SN - 112 UR - https://doi.org/10.1787/21f63c69-en Y2 - 2023/09/14/00:00:00 ER - TY - RPRT TI - The Advocacy Strategy Framework. A tool for articulating an advocacy theory of change AU - Coffman, Julia AU - Beer, Tanya CY - Washington D.C. DA - 2015/03// PY - 2015 PB - Center for Evaluation Innovation UR - http://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework.pdf Y2 - 2016/11/12/17:57:59 ER - TY - RPRT TI - The Alignment, Interest and Influence Matrix (AIIM) guidance note AU - Mendizabal, Enrique AB - In 2007, on the eve of a workshop to introduce a new version of the RAPID approach to DFID policy teams, Enrique Mendizabal and Ben Ramalingam created the Alignment, Interest and Influence Matrix (AIIM), a stakeholder analysis tool that not only helps to identify key stakeholders, but also suggests a possible course of action towards them. CY - London DA - 2010/11// PY - 2010 LA - en PB - ODI/RAPID UR - https://www.odi.org/publications/5288-alignment-interest-and-influence-matrix-aiim-guidance-note Y2 - 2019/04/07/19:22:17 ER - TY - BLOG TI - The Answer is 42. On Data, Information and Knowledge AU - Bours, Dennis T2 - Earth-Eval AB - A recent discussion with some colleagues on the differences between data, knowledge and information made me realize that there still is a lot of confusion when it comes to the use of terms; confusion that goes well beyond my earlier blog post on indicators, measures and metrics. DA - 2015/05/25/T15:43:29-04:00 PY - 2015 LA - en UR - https://www.climate-eval.org/blog/answer-42-data-information-and-knowledge Y2 - 2019/06/20/15:52:13 ER - TY - RPRT TI - The Art and Craft of Bricolage in Evaluation AU - Aston, Thomas AU - Apgar, Marina T2 - CDI Practice Paper AB - This CDI Practice Paper by Tom Aston and Marina Apgar makes the case for ‘bricolage’ in complexity-aware and qualitative evaluation methods. It provides a framework based on a review of 33 methods to support evaluators to be more intentional about bricolage and to combine the component parts of relevant methods more effectively. It discusses two cases from practice to illustrate the value added of taking a more intentional approach. It further argues that navigating different forms of power is a critical skill for bricolage, and that doing so can help to ensure rigour. CY - Brighton DA - 2022/10/14/ PY - 2022 DP - opendocs.ids.ac.uk LA - en PB - Institute for Development Studies SN - 24 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/17709 Y2 - 2023/01/10/14:54:22 ER - TY - JOUR TI - The behaviour change wheel: A new method for characterising and designing behaviour change interventions AU - Michie, Susan AU - van Stralen, Maartje M AU - West, Robert T2 - Implementation Science : IS AB - Background Improving the design and implementation of evidence-based practice depends on successful behaviour change interventions. This requires an appropriate method for characterising interventions and linking them to an analysis of the targeted behaviour. There exists a plethora of frameworks of behaviour change interventions, but it is not clear how well they serve this purpose. This paper evaluates these frameworks, and develops and evaluates a new framework aimed at overcoming their limitations. Methods A systematic search of electronic databases and consultation with behaviour change experts were used to identify frameworks of behaviour change interventions. These were evaluated according to three criteria: comprehensiveness, coherence, and a clear link to an overarching model of behaviour. A new framework was developed to meet these criteria. The reliability with which it could be applied was examined in two domains of behaviour change: tobacco control and obesity. Results Nineteen frameworks were identified covering nine intervention functions and seven policy categories that could enable those interventions. None of the frameworks reviewed covered the full range of intervention functions or policies, and only a minority met the criteria of coherence or linkage to a model of behaviour. At the centre of a proposed new framework is a 'behaviour system' involving three essential conditions: capability, opportunity, and motivation (what we term the 'COM-B system'). This forms the hub of a 'behaviour change wheel' (BCW) around which are positioned the nine intervention functions aimed at addressing deficits in one or more of these conditions; around this are placed seven categories of policy that could enable those interventions to occur. The BCW was used reliably to characterise interventions within the English Department of Health's 2010 tobacco control strategy and the National Institute of Health and Clinical Excellence's guidance on reducing obesity. Conclusions Interventions and policies to change behaviour can be usefully characterised by means of a BCW comprising: a 'behaviour system' at the hub, encircled by intervention functions and then by policy categories. Research is needed to establish how far the BCW can lead to more efficient design of effective interventions. DA - 2011/04/23/ PY - 2011 DO - 10.1186/1748-5908-6-42 DP - PubMed Central VL - 6 SP - 42 J2 - Implement Sci SN - 1748-5908 ST - The behaviour change wheel UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3096582/ Y2 - 2019/08/12/22:18:18 ER - TY - JOUR TI - The Causal Mechanism Claim in Evaluation: Does the Prophecy Fulfill? AU - Schmitt, Johannes T2 - New Directions for Evaluation AB - Despite increased discussions in the community and a common understanding about the virtue of mechanism-based explanation, little is known about the true benefits and challenges of applying causal mechanism analysis in practice. This chapter aims to introduce the reader to the topic of causal mechanisms and synthesize significant findings on this special issue. It begins by laying out definitions and concepts of causal mechanisms in evaluation literature and proposes a two-way classification of causal mechanisms along which the chapters to this issue are structured. The chapter continues by introducing the Causal Mechanism Claim and elaborates on how analyzing causal mechanisms is expected to increase policy relevance and causal capacity in evaluations. Drawing on this issue's rich corpus of firsthand practical experience, this introduction synthesizes key lessons that support or contradict the Causal Mechanism Claim. We find that both parts of the claim—increased policy relevance and strengthened causal capacity—are supported by the authors' experiences as bundled in this issue. However, we also identify challenges related to cross-cutting issues such as communication and practical applicability and point to the importance of method integration. DA - 2020/// PY - 2020 DO - 10.1002/ev.20421 DP - Wiley Online Library VL - 2020 IS - 167 SP - 11 EP - 26 LA - en SN - 1534-875X ST - The Causal Mechanism Claim in Evaluation UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20421 Y2 - 2021/02/18/14:26:28 ER - TY - VIDEO TI - The collaborative exploration of alternative futures: A different approach to Theories of Change AU - CEDIL programme AB - ParEvo is a web application that enables the collaborative construction and exploration of a range of alternative futures: likely and unlikely, desirable and undesirable. These are described in the form of a branching narrative structure, developed over a series of iterations involving the interactions of a group of participants. These detailed storylines about the future contrast with optimistic, skeletal and largely singular views of the future found in diagrammatic ToCs often encountered by evaluators. This webinar will describe a recent ParEvo exercise implemented by the Centre for the Study of Existential Risk (CSER) in Cambridge. In this exercise 11 international experts explored ideas about how global risks associated with biotechnology research could be managed, and mismanaged, in the coming four years. Including how these alternative futures were evaluated by participants and the CSER facilitators. Issues to be discussed by the panel, and others, include comparisons with other representations of Theories of Change and other approaches to the exploration of alternative futures, described variously as scenario planning, futures or foresight work. Speakers: Rick Davies, Lara Mani Tom Hobson An overview of the app can be found here: https://mscinnovations.wordpress.com/ The app website is here: https://parevo.org/ DA - 2022/04/13/ PY - 2022 DP - YouTube ST - The collaborative exploration of alternative futures UR - https://www.youtube.com/watch?v=Hlbpqfw_ve4 Y2 - 2022/07/01/08:44:57 ER - TY - BLOG TI - The Collaborative, Participatory & Empowerment Evaluation Topical Interest Group AU - American Evaluation Association AB - designed to foster collaboration, share knowledge, and promote scholarship, this TIG is a division of the American Evaluation Association (AEA). UR - http://comm.eval.org/cpetig/home Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - The COM-B Theory of Change Mode - v.5 AU - Mayne, John AB - ResearchGate is a network dedicated to science and research. Connect, collaborate and discover scientific publications, jobs and conferences. All for free. DA - 2019/08// PY - 2019 LA - en M3 - Working Paper PB - Palladium UR - https://www.researchgate.net/publication/335404381_The_COMB_ToC_ModelV5 Y2 - 2020/01/15/15:41:50 ER - TY - JOUR TI - The curious case of the realist trial: Methodological oxymoron or unicorn? AU - Nielsen, Steffen Bohni AU - Jaspers, Sofie Østergaard AU - Lemire, Sebastian T2 - Evaluation AB - Realist evaluation and experimental designs are both well-established approaches to evaluation. Over the past 10 years, realist trials—evaluations purposefully combining realist evaluation and experimental designs—have emerged. Informed by a comprehensive review of published realist trials, this article examines to what extent and how realist trials align with quality standards for realist evaluations and randomized controlled trials and to what extent and how the realist and trial aspects of realist trials are integrated. We identified only few examples that met high-quality standards for both experimental and realist studies and that merged the two designs. DA - 2023/09/23/ PY - 2023 DO - 10.1177/13563890231200291 DP - SAGE Journals SP - 13563890231200291 LA - en SN - 1356-3890 ST - The curious case of the realist trial UR - https://doi.org/10.1177/13563890231200291 Y2 - 2023/10/23/10:45:35 ER - TY - SLIDE TI - The Debate about RCTs in Development is over. We won. They lost. T2 - DRi Seminar A2 - Pritchett, Lant AB - There has been a debate in development economics over the last 20 years as some claimed the use of RCTs as a tool for independent impact evaluation would significantly improve development practice and hence development. While right about the methodological claims about the superiority of randomization to produce cleaner estimates of the LATE (local average treatment effect) of projects and programs, this, in and of itself, does not change development practice. All of the five claims needed to sustain a positive model in which RCT/IIE has a major positive impact are demonstrably false. The proponents of RCTs have responded to losing the first round decisively by changing significantly both their claims and their practice. CY - New York DA - 2018/02/21/ PY - 2018 LA - en-US UR - http://www.nyudri.org/events-index/2018/2/22/lant-pritchett-talk-the-debate-about-rcts-in-development-is-over-we-won-they-lost Y2 - 2019/03/15/11:55:02 ER - TY - RPRT TI - The Difference Learning Makes - Factors that enable and inhibit adaptive programming AU - Gray, Stephen AU - Carl, Andy AB - Executive Summary When Christian Aid (CA) Ireland devised its multi-country and multi-year Irish Aid funded Programme Grant II (2017-2022), they opted to move away from a linear programme management approach and to explore an adaptive one. Across seven countries: Angola, Colombia, El Salvador, Guatemala, Israel and the occupied Palestinian territory, Sierra Leone, and Zimbabwe, CA and partner organisations support marginalised communities to realise their rights, reduce violence and address gender inequality. Since 2019, Adapt Peacebuilding has accompanied CA Ireland, CA country teams and partner organisations as they experimented with using a deliberate adaptive approach. The authors were also asked to follow up on an initial study by CA Ireland and Overseas Development Institute in 2018, which described the rationale for adopting this new approach and included early lessons from its first year of implementation. The aim of this study is to help deepen CA Ireland, CA country teams’ and partners’ understanding of (a) whether their application of adaptive programming has resulted in better development outcomes, and (b) how they can better understand the factors that enabled or inhibited the effectiveness of using this approach. Over the past three years, this study has found evidence and multiple examples that show adaptive programming contributed to better development outcomes. The main reasons cited were that these were made possible both from improvements to programming strategies based on proactive reflection and learning, as well as those that stem from the reactive capacity of adaptive programmes to change course in response to unanticipated changes in operating conditions. This study found that adaptive programming has enabled better development practice where organisations are enhancing their skills to better respond and be flexible to contextual challenges. 72% of partners surveyed described adaptive programming as the most useful approach to programme management that they have used. The programme approach has meant that CA and partner staff were better able to explore the significance of change in the context and their contributions to them. It also enabled spaces for meaningful engagement with communities in learning and programme planning processes and encouraged opportunities for experimentation in programming. The study also found that adaptive programming has supported flexible delivery. This led to better outcomes that would not have been possible were the programme not able to make flexible adjustments. The main focus has been the analysis of nine factors that can determine the effectiveness and impact (or otherwise) of using an adaptive approach, flagging important issues for understanding. These factors are identified as: 1) Leadership; 2) Organisational culture; 3) Conceptual understanding; 4) Staff capacities; 5) Partnership approaches; 6) Participation; 7) Methods and tools; 8) Administrative procedures; and 9) The operating context. Together these can provide an analytical framework for assessing an organisation’s ‘adaptive scope’, which can be used as a tool for better understanding an organisation’s potential to generate improved development outcomes via adaptive programming and how to strengthen them. The study concludes with several recommendations for CA Ireland, all of which have relevance for a broader community of donors and implementing organisations interested in the potential of adaptive programming. CY - London DA - 2022/02// PY - 2022 PB - Christian Aid UR - https://www.christianaid.ie/sites/default/files/2022-12/the-difference-learning-makes-factors-that-enable-and-inhibit-adaptive-programming.pdf Y2 - 2024/01/29/00:00:00 ER - TY - JOUR TI - The Environment and Disease: Association or Causation? AU - Hill, Austin Bradford T2 - Proceedings of the Royal Society of Medicine DA - 1965/05// PY - 1965 DP - PubMed Central VL - 58 IS - 5 SP - 295 EP - 300 J2 - Proc R Soc Med SN - 0035-9157 ST - The Environment and Disease UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1898525/ Y2 - 2023/09/29/10:40:25 ER - TY - RPRT TI - The Future of Development: “Make Happen” with Portfolios of Options AU - CHÔRA Foundation AB - This Green Paper intends to review key elements of the problem that Development actors will confront as a new decade opens up ahead of us. It will articulate a solution that we believe should become an inherent feature of Development programs and initiatives. This is the outcome of an intense period of experiences and reflections in the Development space across different geographies and institutional mandates and activities, during which the Foundation has collaborated with institutions such as the UNDP and Grand Challenges Canada. At the heart of our collaborations is a Strategic Innovation and System Transformation Framework, with its associated concepts, working definitions, processes, tools and people. Sourced from CHÔRA Foundation’s knowledge and practice assets, this is a capability we intend to make relevant, customise, scale up and distribute to our partners and stakeholders. We are looking to create with them a space that offers the world a transformational capability. Central to this capability we see a distinctive and robust practice: the design and dynamic management of Portfolios of Strategic Innovation and System Transformation Options. These Portfolios are unique, context relevant, embedded mechanisms for learning, sensemaking and problem solving that social systems leverage to have an impact on themselves and their problems, and to induce the transformations that are necessary to them. It is our view that Portfolios of Options are the most effective means by which human social systems can supply themselves with budgets of possibility that ensure choice and create opportunity. They will also support pragmatic evolutionary outcomes and enable resilience. CY - Haarlem (Netherlands) DA - 2021/03// PY - 2021 PB - CHÔRA Foundation UR - https://static1.squarespace.com/static/600eb85f87ba7b33ef93a72a/t/604b93ead71c9b5f9e5f382a/1615565806067/Portfolios+of+Options+Green+Paper+upload.pdf Y2 - 2021/11/09/12:22:53 ER - TY - RPRT TI - The Futures Toolkit: Tools for Futures Thinking and Foresight Across UK Government AU - HM Government DA - 2017/11// PY - 2017 UR - https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/674209/futures-toolkit-edition-1.pdf Y2 - 2018/03/09/00:00:00 ER - TY - JOUR TI - The Generalizability Puzzle AU - Bates, Mary Ann AU - Glennerster, Rachel T2 - Stanford Social Innovation Review AB - The practice of using rigorous scientific evaluations to study solutions to global poverty is relatively young. Although researchers continue to advance our knowledge of the mechanisms at work, confusion about their role and value persists. Having evidence from specific studies is fine and good, but for policy makers, the point is not simply to understand poverty, but to eliminate it. Do decisions always need to be informed by evidence from the local context? What potential and limits do randomized controlled trials have for improving our knowledge and finding effective answers? Two leaders in anti-poverty research—J-PAL and IPA—dispel some of the myths about their field. In this article, authors from J-PAL argue that rigorous impact evaluations tell us a lot about the world, not just the particular contexts in which they are conducted. Access to this article made possible by MIT. DA - 2017/// PY - 2017 IS - Summer 2017 UR - https://ssir.org/articles/entry/the_generalizability_puzzle Y2 - 2017/10/11/15:11:53 ER - TY - BLOG TI - The good and the ‘not so good’ of our experiences with SenseMaker AU - Mager, Franziska T2 - Oxfam Views & Voices AB - When we purchased a license for the SenseMaker research method in early 2017 (a proprietary data collection and analysis software), the excitement in our more “geeky” teams was palpable. SenseMaker ... Read More DA - 2019/10/29/ PY - 2019 LA - en-GB UR - https://views-voices.oxfam.org.uk/2019/10/the-good-and-the-not-so-good-of-our-experiences-with-sensemaker/ Y2 - 2020/02/06/09:09:35 ER - TY - JOUR TI - The guiding logics and principles for designing emergent transdisciplinary research processes: learning experiences and reflections from a transdisciplinary urban case study in Enkanini informal settlement, South Africa AU - van Breda, John AU - Swilling, Mark T2 - Sustainability Science AB - Transdisciplinarity is not a new science per se, but a new methodology for doing science with society. A particular challenge in doing science with society is the engagement with non-academic actors to enable joint problem formulation, analysis and transformation. How this is achieved differs between contexts. The premise of this paper is that transdisciplinary research (TDR) methodologies designed for developed world contexts cannot merely be replicated and transferred to developing world contexts. Thus a new approach is needed for conducting TDR in contexts characterised by high levels of complexity, conflict and social fluidity. To that end, this paper introduces a new approach to TDR titled emergent transdisciplinary design research (ETDR). A core element of this approach is that the research process is designed as it unfolds, that is, it transforms as it emerges from and within the fluid context. The ETDR outlined in this paper emerged through a case study in the informal settlement (slum) of Enkanini in Stellenbosch, South Africa. This case study demonstrates the context from and within which the ETDR approach and identifies a set of guiding logics that can be used to guide ETDR approaches in other contexts. The study demonstrates that the new logics and guiding principles were not simply derived from the TDR literature, but rather emerged from constant interacting dynamics between theory and practice. Learning how to co-design the research process through co-producing transformative knowledge and then implementing strategic interventions to bring about incremental social change is key to theory development in ways that are informed by local contextual dynamics. There are, however, risks when undertaking such TDR processes such as under-valuing disciplinary knowledge, transferring risks onto a society, and suppressing ‘truth-to-power’. DA - 2019/05/01/ PY - 2019 DO - 10.1007/s11625-018-0606-x DP - Springer Link VL - 14 IS - 3 SP - 823 EP - 841 J2 - Sustain Sci LA - en SN - 1862-4057 ST - The guiding logics and principles for designing emergent transdisciplinary research processes UR - https://doi.org/10.1007/s11625-018-0606-x Y2 - 2022/08/08/08:58:25 ER - TY - BLOG TI - The Hidden Life of Theories of Change AU - Green, Duncan T2 - From Poverty to Power AB - A smart new report explores how a good idea - theories of change - is distorted by the way it is implemented in the aid sector DA - 2020/09/17/T06:30:22+00:00 PY - 2020 LA - en-GB UR - https://oxfamblogs.org/fp2p/the-hidden-life-of-theories-of-change/ Y2 - 2020/10/15/12:04:05 ER - TY - RPRT TI - The Hidden Life of Theories of Change AU - Ho, Wenny AU - Tamas, Peter AU - van Wessel, Margit AB - Theory of Change is thought to be very useful for learning and adaptive management of complex interventions such as advocacy. Nevertheless, the use of Theory of Change is also under critique. One common criticism is that Theory of Change is often used as a framework that fixes agreements rather than as a living, guiding tool that helps reflection and adaptation. However, while such criticism stresses forms of control, little research has looked at the way Theory of Change and advocacy practice relate. This is a pertinent issue considering that formally agreed Theories of Change and realities on the ground can be very different. This raises questions: Do advocates work in ways different from what Theory of Change states, and if so, how, and why? How does the way they strategize relate to formal Theories of Change? With what implications? In this brief, we explore these more hidden aspects of the life of Theories of Change. DA - 2020/// PY - 2020 PB - Hivos UR - https://www.hivos.org/assets/2020/09/The-Hidden-Life-of-Theories-of-Change.pdf Y2 - 2020/10/15/12:05:21 ER - TY - JOUR TI - The journey from rhetoric to reality: participatory evaluation in a development context AU - Chouinard, Jill Anne AU - Cousins, J. Bradley T2 - Educational Assessment, Evaluation and Accountability AB - In this paper, we focus on participatory evaluation in the context of international development and specifically on the emerging empirical knowledge base. In a prior review and critique of research on participatory evaluation (Cousins and Chouinard 2012), we examined 121 studies, with only 21 (17 %) situated in development contexts. However, the circumstances and challenges for international development and for development evaluation are distinct from those found in developed countries and therefore warrant separate consideration. To provide a more focused and detailed understanding of participatory evaluation in international development contexts, in this paper we augment our initial sample to a total of 40 studies on participatory evaluation in development published over the past 16.5 years. Based on an analysis of this research and related theoretical and conceptual contributions we identify and discuss eight emergent themes: multiplicity of relationships, consequences of stakeholder selection, characterization of participation, contextual complexity, methodological requirements, cultural influence, politics and power and learning and capacity building. We conclude with some reflections on an agenda for research. DA - 2015/02/01/ PY - 2015 DO - 10.1007/s11092-013-9184-8 DP - Springer Link VL - 27 IS - 1 SP - 5 EP - 39 J2 - Educ Asse Eval Acc LA - en SN - 1874-8600 ST - The journey from rhetoric to reality UR - https://doi.org/10.1007/s11092-013-9184-8 Y2 - 2020/10/15/14:31:11 ER - TY - BOOK TI - The Learning Power of Listening AU - Guijt, Irene AU - Gottret, Maria Veronica AU - Hanchar, Anna AU - Deprez, Steff AU - Muckenhirn, Rita AB - Steff had the pleasure to co-author the first SenseMaker Practitioner Guide with a group of friends and colleagues supported and published by Oxfam and CRS. This practical guide is for those who wish to use SenseMaker to conduct assessments, monitor progress, and undertake evaluations or research. Drawing on more than a decade of experience, the authors share dozens of examples from international development, providing practical tips and ideas for context-specific adaptations. They show how the method can be used to for difficult-to-measure outcomes related to poverty reduction, social justice, peacebuilding, resilience, gender norms, behavior change, governance and environmental management. ​ SenseMaker is a unique participatory method of inquiry that encourages and enables novel insights not obtained from conventional quantitative and quantitative and qualitative methods. It is action-oriented and, therefore, well-suited for people needing data- informed insights for adaptive management. "Writing this guide together with Irene, Veronica, Anna and Rita was an enormous learning process in itself and has further shaped our thinking and practice. We hope it will support first-time and experienced users to enhance their practice and that it will inspire people to explore and innovate further with the method." DA - 2022/06/07/ PY - 2022 DP - practicalactionpublishing.com PB - PRACTICAL ACTION PUBLISHING SN - 978-1-78853-200-6 UR - https://practicalactionpublishing.com/book/2622/the-learning-power-of-listening Y2 - 2022/07/26/11:39:48 ER - TY - RPRT TI - The Logical Framework: a manager's guide to a scientific approach to design and evaluation AU - PCI AB - Developed by Practical Concepts Inc. in 1979. Key document in the development of the logical model/framework DA - 1979/11// PY - 1979 LA - en PB - Practical Concepts Inc. ST - The Logical Framework UR - https://www.scribd.com/document/50064505/The-Logical-Framework-a-manager-s-guide-to-a-scientific-approach-to-design-and-evaluation Y2 - 2019/08/07/22:32:43 ER - TY - ELEC TI - The M&E (Monitoring & Evaluation) Universe AU - INTRAC T2 - INTRAC AB - The M&E Universe is a free, online resource developed by INTRAC to support development practitioners involved in monitoring and evaluation (M&E). It consists of a series of short papers (2-6 pages) on different subjects related to M&E. To begin exploring the Universe right away, use the button below. To find out more, read on. DA - 2019/09// PY - 2019 LA - en UR - https://www.intrac.org/projects/the-me-universe/ Y2 - 2022/01/28/00:00:00 ER - TY - BOOK TI - The 'Most Significant Change' (MSC) Technique AU - Davies, Rick AU - Dart, Jess AB - This publication is aimed at organisations, community groups, students and academics who wish to use MSC to help monitor and evaluate their social change programs and projects, or to learn more about how it can be used. The technique is applicable in many different sectors, including agriculture, education and health, and especially in development programs. It is also applicable to many different cultural contexts. MSC has been used in a wide variety of countries by a range of organisations. By 2004, MSC had been used both by NGOs and governments in Africa, Asia, Latin America, Europe and Australasia. DA - 2005/// PY - 2005 SP - 104 PB - CARE International UR - http://www.mande.co.uk/wp-content/uploads/2005/MSCGuide.pdf Y2 - 2018/03/09/00:00:00 ER - TY - RPRT TI - The New Development Economics: We Shall Experiment, but How Shall We Learn? AU - Rodrik, Dani T2 - HKS Working Paper No. RWP08-055 AB - Development economics is split between macro-development economists - who focus on economic growth, international trade, and fiscal/macro policies - and micro-development economists - who study microfinance, education, health, and other social programs. Recently there has been substantial convergence in the policy mindset exhibited by micro evaluation enthusiasts, on the one hand, and growth diagnosticians, on the other. At the same time, the randomized evaluation revolution has led to an accentuation of the methodological divergence between the two camps. Overcoming the split requires changes on both sides. Macrodevelopment economists need to recognize the distinct advantages of the experimental approach and adopt the policy mindset of the randomized evaluation enthusiasts. Micro-development economists, for their part, have to recognize that the utility of randomized evaluations is restricted by the narrow and limited scope of their application. As the Chinese example illustrates, extending the experimental mindset to the domain of economy-wide reforms is not just possible, it has already been practiced with resounding success in the most important development experience of our generation. DA - 2008/10/24/ PY - 2008 DP - papers.ssrn.com LA - en PB - SSRN ST - The New Development Economics UR - https://papers.ssrn.com/abstract=1296115 Y2 - 2019/03/15/11:45:38 KW - Economics KW - Finance KW - Health Care KW - International Development KW - International Economics KW - International Trade KW - Macroeconomics KW - Microeconomics KW - Social Policy KW - Welfare ER - TY - JOUR TI - The origins, development, and application of Qualitative Comparative Analysis: the first 25 years AU - Marx, Axel AU - Rihoux, Benoît AU - Ragin, Charles T2 - European Political Science Review AB - A quarter century ago, in 1987, Charles C. Ragin published The Comparative Method, introducing a new method to the social sciences called Qualitative Comparative Analysis (QCA). QCA is a comparative case-oriented research approach and collection of techniques based on set theory and Boolean algebra, which aims to combine some of the strengths of qualitative and quantitative research methods. Since its launch in 1987, QCA has been applied extensively in the social sciences. This review essay first sketches the origins of the ideas behind QCA. Next, the main features of the method, as presented in The Comparative Method, are introduced. A third part focuses on the early applications. A fourth part presents early criticisms and subsequent innovations. A fifth part then focuses on an era of further expansion in political science and presents some of the main applications in the discipline. In doing so, this paper seeks to provide insights and references into the origin and development of QCA, a non-technical introduction to its main features, the path travelled so far, and the diversification of applications. DA - 2014/02// PY - 2014 DO - 10.1017/S1755773912000318 DP - Cambridge Journals Online VL - 6 IS - 01 SP - 115 EP - 142 SN - 1755-7747 ST - The origins, development, and application of Qualitative Comparative Analysis UR - http://journals.cambridge.org/article_S1755773912000318 Y2 - 2016/06/05/11:20:14 ER - TY - RPRT TI - The Patterns of Possibility - How to Recast Relationships to Create Healthier Systems and Better Outcomes AU - Winhall, Jennie AU - Leadbeater, Charles AB - In Building Better Systems, we introduced four keys to unlock system innovation: purpose and power, relationships and resource flows. These four keys make up a set. Systems are often hard to change because power, relationships, and resource flows are locked together in a reinforcing pattern to serve the system’s current purpose. Systems start to change fundamentally when this pattern is disrupted and opened up so that a new configuration can emerge, serving a new purpose. In this article series we delve deeper into these four keys and provide practical advice on how they can be put to use. This article is about relationships. Systems are defined by the patterns of interactions between their parts: their relationships. Those interactions generate the outcomes of the system as a whole. Transforming the outcomes of a system requires remaking its relationships and then unlocking the other keys to system innovation: purpose, power and resources. This shift in relationships allows all those in the system to learn faster, to be more creative. System innovators redesign the relationships in the system to allow dramatically enhanced learning across the system, and thereby generate far better outcomes. CY - København K DA - 2022/02// PY - 2022 LA - en-GB PB - The Rockwool Foundation UR - https://www.systeminnovation.org/article-the-patterns-of-possibility Y2 - 2022/06/17/12:02:20 ER - TY - BOOK TI - The politics of public sector performance: pockets of effectiveness in developing countries T2 - Routledge research in comparative politics A3 - Roll, Michael CY - London DA - 2014/// PY - 2014 DP - Gemeinsamer Bibliotheksverbund ISBN SP - 276 LA - eng M1 - 55 PB - Routledge SN - 978-0-415-64361-0 978-1-138-95639-1 978-1-315-85771-8 ST - The politics of public sector performance ER - TY - RPRT TI - The Qualitative Impact Assessment Protocol (QUIP) AU - Copestake, James AU - Remnant, Fiona T2 - CDS Briefing Paper AB - This brief presents an overview of the QUIP in three steps: the background to the QUIP and its main aims; the data collection and analysis methodology; and QUIP in the context of other approaches to evaluation. Each section can be read independently, depending on time available. DA - 2015/04// PY - 2015 SP - 9 UR - http://www.bath.ac.uk/cds/documents/quip-briefing-paper-march-2015.pdf Y2 - 2018/10/19/00:00:00 ER - TY - RPRT TI - The Rapid Evaluation, Action, and Learning (REAL) Approach: A toolkit to measure and refine changes and interventions in health campaigns AU - PATH AB - Evaluative Tools for Improving Campaign Effectiveness Public health campaigns aim to control disease or deliver essential health services and products through time-limited and periodic channels. Many countries use campaigns to augment or replace routine service delivery, to target certain populations, or to accelerate progress towards coverage targets. The Root Cause Analysis and Rapid Evaluation, Action, and Learning toolkits present a systematic but flexible approach to identifying the root causes of campaign bottlenecks and then designing, testing, and refining solutions to optimize potential impact. These toolkits respond to a growing need for “fit-for-purpose” rapid-testing, adaptive learning approaches to evaluation and the need for a culture shift toward iterative adaptation and improvement that integrates measurement and evidence-informed decision-making into daily practice. The comprehensive package of toolkits, in addition to the individual Root Cause Analysis toolkit and Rapid Evaluation, Action, and Learning toolkit are available below. DA - 2019/// PY - 2019 PB - PATH UR - https://media.path.org/documents/RapidTestingProtocol_Toolkit02.pdf Y2 - 2023/11/15/10:49:10 ER - TY - JOUR TI - The rise of the randomistas: on the experimental turn in international aid AU - Donovan, Kevin P. T2 - Economy and Society AB - In recent years, the use of experimental methodologies has emerged as a central means of evaluating international aid interventions. Today, proponents of randomized control trials (so-called randomistas) are among the most influential of development experts. This paper examines the growth of this thought collective, analysing how uncertainty has become a central concern of development institutions. It demonstrates that transformations within the aid industry – including the influence of evidence-based policy, the economization of development and the retreat from macro-planning – created the conditions of possibility for experimentation. Within this field, the randomistas adeptly pursued a variety of rhetorical, affective, methodological and organizational strategies that emphasized the lack of credible knowledge within aid and the ability of experiments to rectify the situation. Importantly, they have insisted on the moral worth of experimentation; indeed, the experimental ethic has been proposed as the way to change the spirit of development. Through causal certitude, they propose to reduce human suffering. The rise of experimentation has not, however, eliminated accusations of uncertainty; rather, it has redistributed the means through which knowledge about development is considered credible. DA - 2018/01/02/ PY - 2018 DO - 10.1080/03085147.2018.1432153 DP - Taylor and Francis+NEJM VL - 47 IS - 1 SP - 27 EP - 58 SN - 0308-5147 ST - The rise of the randomistas UR - https://doi.org/10.1080/03085147.2018.1432153 Y2 - 2019/03/15/11:57:35 KW - Development KW - Evidence KW - Experimentation KW - aid KW - uncertainty ER - TY - RPRT TI - The road to results: designing and conducting effective development evaluations AU - Imas, Morra AU - G, Linda AU - Rist, Ray C. AB - The analytical, conceptual, and political framework of development is changing dramatically. The new development agenda calls for broader understandings of sectors, countries, development strategies, and policies. It emphasizes learning and continuous feedback at all phases of the development cycle. As the development agenda grows in scope and complexity, development evaluation follows suit. Development evaluator are moving away from traditional implementation and output-focused evaluation models toward results-based evaluation models, as the development community calls for results and embraces the millennium development goals. As the development community shifts its focus away from projects in order to comprehensively address country challenges, development evaluators are seeking methods with which to assess results at the country, sector, theme, policy, and even global levels. As the development community recognizes the importance of not only a comprehensive but also a coordinated approach to developing country challenges and emphasizes partnerships, development evaluators are increasingly engaged in joint evaluations. These joint evaluations, while advantageous in many respects, add to the complexity of development evaluation (OECD 2006). Additionally, development evaluators increasingly face the measurement challenge of determining the performance of an individual development organization in this broader context and of identifying its contribution. This text is intended as a tool for use in building development evaluation capacity. It aims to help development evaluators think about and explore the new evaluation architecture and especially to design and conduct evaluations that focus on results in meeting the challenges of development. DA - 2009/06/01/ PY - 2009 DP - documents.worldbank.org SP - 1 EP - 611 LA - en PB - The World Bank SN - 52678 ST - The road to results UR - http://documents.worldbank.org/curated/en/400101468169742262/The-road-to-results-designing-and-conducting-effective-development-evaluations Y2 - 2018/02/06/09:53:36 ER - TY - RPRT TI - The Role of Outcome Mapping in Developing a Rural Telemedicine System AU - Tucker, William AU - Blake, Edwin AB - We describe the use of Outcome Mapping to guide the design of a rural telemedicine consultation system in South Africa. While Outcome Mapping was not primarily intended to guide design, we show that it tied in well with a cyclical participatory design method for an Information and Communication Technology for Development project DA - 2008/// PY - 2008 PB - University of the Western Cape UR - https://core.ac.uk/download/pdf/62633469.pdf Y2 - 2018/09/22/00:00:00 ER - TY - BLOG TI - The role of research and learning in adaptive programming AU - Babon, Andrea AU - Denney, Lisa T2 - Devpolicy Blog from the Development Policy Centre AB - Andrea Babon and Lisa Denney explore how learning partners - a common feature of aid programs - can operate and feed into programs. DA - 2018/09/20/T06:00:19+10:00 PY - 2018 LA - en-US UR - http://www.devpolicy.org/the-role-of-research-and-learning-in-adaptive-programming-20180920/ Y2 - 2018/09/24/08:15:49 ER - TY - CONF TI - The Success Case Method: A Simple Evaluation Tool to Identify Critical Success Factors and Program Impa AU - Bell, Catherine AU - McDonalds, Diane T2 - 2006 Australian Evaluation Society International Conference T3 - Final Papers AB - Identification and reporting of critical success factors and program impact in an efficient, yet comprehensive manner is an inherent difficulty facing many evaluators of large-scale evaluations. This paper details how two evaluators encountered such problems in the initial review of a large-scale initiative and then successfully addressed these issues through the application of the Success Case Method in a subsequent evaluation of the same program. The Success Case Method is a down-to-earth evaluation tool that can be used for: finding out what is working and what is not, which also provides accurate and trustworthy information that can be used to make timely decisions (Brinkerhoff, 2003, p.3). This paper reports on the advantages of applying this approach to the evaluation of the Primary Welfare Officer Initiative (PWOI), a program that aims to improve the capacity of schools to support students at risk of disengagement and who are not achieving their educational potential. C1 - Darwin, Australia DA - 2006/// PY - 2006 PB - Australian Evaluation Society Inc. UR - https://www.aes.asn.au/images/stories/files/conferences/2006/papers/063%20Catherine%20Bell.pdf Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - The Truth of the Work: Theories of Change in a changing world AU - Reeler, Doug AU - van Blerk, Rubert DA - 2017/// PY - 2017 PB - The Community Development Resource Association UR - http://www.cdra.org.za/uploads/1/1/1/6/111664/the_truth_of_the_work_-_theories_of_change_in_a_changing_world_-_by_doug_reeler_and_rubert_van_blerk_%E2%80%93_cdra_2017_-_final.pdf Y2 - 2021/05/18/14:49:26 ER - TY - RPRT TI - The Use and Abuse of the Logical Framework Approach: A Review of International Development NGOs' Experiences AU - Bakewell, Oliver AU - Garbutt, Anne AB - The logical framework approach (LFA) has come to play a central role in the planning and management of development interventions over the last twenty years. Although the logical framework has become universally known, it is far from universally liked. It has been the subject of much criticism over the years, concerning both the theoretical basis of the approach, and the way it is applied in practice. In this review, we have attempted to take stock of the current views of international development NGOs on the LFA and the ways in which they use it. CY - Stockholm DA - 2005/11// PY - 2005 DP - www.research.manchester.ac.uk LA - English PB - Sida ST - The Use and Abuse of the Logical Framework Approach UR - https://www.alnap.org/system/files/content/resource/files/main/the-use-and-abuse-of-the-logical-framework-approach.pdf Y2 - 2022/12/05/21:43:34 ER - TY - RPRT TI - The Voice of the Farmer In Agricultural Extension: A Review of Beneficiary Assessments of Agricultural Extension and An Inquiry into their Potential as a Management Tool AU - Salmen, Lawrence F T2 - AKIS Discussion Paper DA - 1999/11/18/ PY - 1999 SP - 30 PB - World Bank UR - http://documents.worldbank.org/curated/en/776431468322742990/pdf/multi0page.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - The Whistler principles to accelerate innovation for development impact AU - G7 AB - The Whistler principles to accelerate innovation for development impact : 2018 G7 Summit – Canada 2018 G7 Presidency – Charlevoix, Quebec CY - G7 Summit 2018 DA - 2018/02// PY - 2018 LA - en-US PB - G7 UR - http://www.g7.utoronto.ca/dev/180602-innovation.html Y2 - 2018/11/13/12:09:29 ER - TY - RPRT TI - Theories of Change for Promoting Empowerment and Accountability in Fragile and Conflict-Affected Settings AU - Green, Duncan T2 - IDS Working Paper AB - This paper explores the current state of thinking among a range of aid actors (multilaterals, bilateral, applied scholars and international non-governmental organisations) on how to promote empowerment and accountability in fragile, conflict and violence affected settings. It seeks to identify trends, gaps and weaknesses in that thinking, and propose research questions and hypotheses to test. Three underlying sources of confusion are identified that are hindering progress on both understanding empowerment and accountability in fragile, conflict and violence affected settings, and taking helpful action to promote it. They are: Theory of endogenous change (e.g. on how empowerment and accountability arise in situ) versus the theory of action of an external intervention; Fragility versus conflict: there is no clear justification for combining these different aspects into a single category; and Empowerment versus accountability: donor analysis and practice has been overwhelmingly weighted towards accountability, exhibiting limited understanding or interest in the nature of power. CY - Brighton DA - 2017/10// PY - 2017 LA - en PB - IDS SN - 499 UR - http://www.ids.ac.uk/publication/theories-of-change-for-promoting-empowerment-and-accountability-in-fragile-and-conflict-affected-settings Y2 - 2018/08/02/09:26:28 KW - A4EA KW - External actors KW - Participation ER - TY - RPRT TI - Theories of Change: time for a radical approach to learning in development AU - Valters, Craig AB - The Theory of Change approach demands a radical shift towards more and better learning in development thinking and practice, creating a productive and much-needed space for critical reflection. CY - London DA - 2015/09// PY - 2015 PB - ODI ST - Theories of Change UR - https://www.odi.org/publications/9883-theories-change-time-radical-approach-learning-development Y2 - 2016/08/05/16:42:21 ER - TY - JOUR TI - Theory amidst complexity – using process tracing in ex-post evaluations AU - Krueger, Kate AU - Wright, Molly T2 - New Directions for Evaluation AB - Evaluators who take a complexity-aware approach must consider tradeoffs related to theoretical parsimony, falsifiability, and measurement validity. These tradeoffs may be particularly pronounced with ex-post evaluation designs in which program theory development and monitoring frameworks are often completed before the evaluator is engaged. In this chapter, we argue that theory-based evaluation (TBE) approaches can address unique ex-post evaluation challenges that complexity-aware evaluation (CAE) alone cannot, and that these two sets of approaches are complimentary. We will outline strategies that evaluators may use to conduct rigorous ex-post evaluations of democracy, human rights, and governance (DRG) interventions that merge CAE's inductive approaches with a theory-testing structure. It will illustrate these strategies with two case studies of ex-post evaluation using process tracing (PT). DA - 2022/// PY - 2022 DO - 10.1002/ev.20524 DP - Wiley Online Library VL - 2022 IS - 176 SP - 119 EP - 128 LA - en SN - 1534-875X UR - http://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20524 Y2 - 2023/04/13/09:21:48 ER - TY - RPRT TI - Theory-Based Approaches to Evaluation: Concepts and Practices AU - Treasury Board of Canada Secretariat AB - This document introduces some of the key concepts of theory-based approaches to evaluation. It is hoped that readers will be encouraged by the information and advice provided in this document and will explore the use (e.g., through pilot evaluations) of theory-based approaches to evaluation in a federal setting. To support this, Sections 1.0 to 8.0 of the document describe the general application of theory-based approaches to evaluation, and Sections 9.0 and 10.0 discuss the potential application of theory-based approaches to a range of federal programs. This document is neither an exhaustive training program in theory-based evaluation nor a step-by-step guide to undertaking a theory-based evaluation. Evaluators who wish to integrate theory-based approaches into their practices are encouraged to pursue additional readings (including those referenced in this document) and, as appropriate, to seek additional support in undertaking a theory-based evaluation. DA - 2021/03/22/ PY - 2021 LA - eng PB - Government of Canada ST - Theory-Based Approaches to Evaluation UR - https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/theory-based-approaches-evaluation-concepts-practices.html Y2 - 2022/01/27/22:06:47 ER - TY - RPRT TI - Theory of Change AU - Rogers, Patricia T2 - Methodological Briefs: Impact Evaluation DA - 2014/// PY - 2014 SP - 16 PB - UNICEF SN - 2 UR - https://www.betterevaluation.org/sites/default/files/Theory_of_Change_ENG.pdf ER - TY - JOUR TI - Theory of Change Analysis: Building Robust Theories of Change AU - Mayne, John T2 - Canadian Journal of Program Evaluation AB - Models for theories of change vary widely as do how they are used. What constitutes a good or robust theory of change has not been discussed much. This article sets out and discusses criteria for robust theories of change. As well, it discusses how these criteria can be used to undertake a vigorous assessment of a theory of change. A solid analysis of a theory of change can be extremely useful, both for designing or assessing the designs of an intervention as well as for the design of monitoring regimes and evaluations. The article concludes with a discussion about carrying out a theory of change analysis and an example. DA - 2017/12/04/ PY - 2017 DO - 10.3138/cjpe.31122 DP - DOI.org (Crossref) VL - 32 IS - 2 J2 - CJPE LA - en SN - 1496-7308, 0834-1516 ST - Theory of Change Analysis UR - https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/31122 Y2 - 2020/02/14/12:39:20 ER - TY - ELEC TI - Theory of Change: From novice to master changemaker AU - Changeroo AB - Theory of Change is more than just a tool or instrument. It is a mindset! A mindset to optimise impact management. As the cornerstone of impact management it is indispensable to any purpose-driven organisation targeting social or environmental value creation. It offers you the cornerstone of a learning approach toward change and impact. A way to build a mindset and organisational culture of critical reflection, co-creation with stakeholders and constant analysis of what works, why, for whom and under what circumstances. Are you ready to take impact management to the next level with the help of Theory of Change? We offer you the premier online course to learn everything about this approach. DA - 2020/// PY - 2020 UR - https://theoryofchangecourse.com/ Y2 - 2020/10/15/10:34:44 ER - TY - JOUR TI - Theory of Change in complex Research for Development programmes: challenges and solutions from the Global Challenges Research Fund AU - Chapman, Sarah AU - Boodhoo, Adiilah AU - Duffy, Carren AU - Goodman, Suki AU - Michalopoulou, Maria T2 - The European Journal of Development Research AB - The United Kingdom Research and Innovation (UKRI) Global Challenges Research Fund (GCRF) aimed to address global challenges to achieve the United Nations (UN) Sustainable Development Goals through 12 interdisciplinary research hubs. This research documents key lessons learned around working with Theory of Change (ToC) to guide Monitoring, Evaluation and Learning (MEL) within these complex research for development hubs. Interviews and document reviews were conducted in ten of the research hubs. The results revealed that only one hub invested in an explicit visual system mapping approach, and that funder timelines, budget constraints and issues with capacity and expertise limited the application of these approaches across all hubs. In contrast, many hubs attempted to deal with visual complexity by means of ether constructing multiple, nested ToCs, or a conscious simplification of complexity through reducing their ToC towards a straightforward and uncomplicated chain model or spherical model. While the former approach had some value, most hubs struggled to find capacity to support the full articulation of nested ToCs. In contrast, the latter approach resulted in ToCs which lacked detail or mechanism articulation, but which nevertheless were often ‘fit for purpose’ in ensuring effective communication and coherence across diverse stakeholders and sub-projects. We conclude that in instances where the reporting, funding and management cycles of complex research for development programmes cannot be adapted to properly support learning-based approaches to ToC development, imposing simplicity in the ToC might be fit for purpose. This might also be preferable to more complex visual approaches that are only partially realised. DA - 2023/04/01/ PY - 2023 DO - 10.1057/s41287-023-00574-0 DP - Springer Link VL - 35 IS - 2 SP - 298 EP - 322 J2 - Eur J Dev Res LA - en SN - 1743-9728 ST - Theory of Change in Complex Research for Development Programmes UR - https://doi.org/10.1057/s41287-023-00574-0 Y2 - 2023/04/13/11:21:03 ER - TY - RPRT TI - Theory of change in ten steps AU - Noble, James CY - London DA - 2019/10// PY - 2019 DP - Zotero SP - 28 LA - en PB - NPC ER - TY - BLOG TI - Theory of change software AU - Better Evaluation AB - There are a number of options when it comes to using software to help create a logic model. These range from generic word processing tools (Word, Powerpoint, or their Google Doc or Mac equivalents), to software that has been specifically tailored for visualising Theories of Change, like TOCO or Miradi. You should consider what resources you have to invest in software, both in terms of cost and in time to learn and use the features. If you only have a short timeframe and have simple needs, then a basic tool may suit you better than some of the more complex software available. It's important to investigate a few options and see what is going to be best for you. DA - 2023/07/03/ PY - 2023 LA - en UR - https://www.betterevaluation.org/tools-resources/theory-change-software Y2 - 2023/07/04/10:00:41 ER - TY - RPRT TI - Theory of Change Thinking in Practice AU - van Es, Marjan AU - Guijt, Irene AU - Vogel, Isabel AB - Want to know better how your interventions can contribute to change? A Theory of Change (ToC) approach helps in deepening your understanding - and that of your partners - of how you collectively think change happens and what the effect will be of your intervention. Not only does it show what political, social, economic, and/or cultural factors are in play, it also clarifies your assumptions. Once a ToC has been developed, it can be used to continually reflect on it in ways that allow for adaptation and checking of assumptions of your intervention. This user friendly guideline helps you to use a Theory of Change approach. Theories of change are the ideas and beliefs people have – consciously or not – about why and how the world and people change. How people perceive and understand change and the world around them is infused by their underlying beliefs about life, human nature and society. They are deep drivers of people’s behaviour and of the choices they make. Social change processes are complex and characterised by non-linear feedback loops: our own actions interact with those of others and a myriad of influencing factors. This triggers reactions that cannot be foreseen and makes outcomes of change interventions unpredictable. Given these uncertainties, how can we plan strategically and sensibly? How can social change initiatives move forward in emerging change processes in a flexible way, while remaining focused on the goal? In this context of complexity, Hivos has found a theory of change (ToC) approach useful in guiding its strategic thinking and action, as well as its collaborative efforts with others. As it fosters critical questioning of all aspects of change interventions and supports adaptive planning and management in response to diverse and quickly changing contexts. It contributes to the quality and transparency of strategic thinking, and therefore to personal, organisational and social learning. This guide builds on the experiences of Hivos working with a ToC approach. It is a practical guide for Hivos staff in applying a Theory of Change approach, but is also very useful for others working on social change such as social entrepreneurs and innovators. This guide builds on the work of a ToC Learning Group initiated by Hivos and comprising of staff of the Centre for Development Innovation (CDI) of Wageningen University and Research Centre and of experts Iñigo Retolaza Eguren, Isabel Vogel and Irene Guijt. For current thinking and work on the use of Theory of Change thinking in complex change processes, see http://www.theoryofchange.nl DA - 2015/11// PY - 2015 LA - English UR - https://www.hivos.org/theory-change-thinking-practice Y2 - 2016/04/20/14:29:25 ER - TY - RPRT TI - Theory of Change Workbook: A Step-by-Step Process for Developing or Strengthening Theories of Change AU - Salib, Monalisa AB - While over time theories of change have become synonymous with simple if/then statements, a strong theory of change should actually be a much more detailed, context-specific articulation of how we *theorize* change will happen under a program. DA - 2022/02/15/ PY - 2022 LA - en M3 - Text PB - USAID ST - Theory of Change Workbook UR - https://usaidlearninglab.org/library/theory-change-workbook-step-step-process-developing-or-strengthening-theories-change Y2 - 2022/03/17/13:58:01 ER - TY - BLOG TI - Thinking and Working Politically in Economic Development Programmes – Some Sprints and Stumbles from a DFID Programme in Kyrgyzstan AU - Koleros, Andrew AU - Rinnert, David T2 - From Poverty to Power AB - A DFID programme in Kyrgyzstan offers useful insights into how the Thinking and Working Politically approach can escape from its governance ghetto DA - 2019/02/06/T07:00:19+00:00 PY - 2019 LA - en-GB UR - https://oxfamblogs.org/fp2p/thinking-and-working-politically-in-economic-development-programmes-some-sprints-and-stumbles-from-a-dfid-programme-in-kyrgyzstan/ Y2 - 2019/02/06/09:37:26 KW - Case Study ER - TY - RPRT TI - Thinking and Working politically to support developmental leadership and coalitions: The Pacific Leadership Program AU - Denney, Lisa AU - McLaren, Rebecca T2 - Research Paper CY - Birmingham DA - 2016/10// PY - 2016 PB - DLP Program SN - 41 UR - http://publications.dlprog.org/Denney_McLaren.pdf Y2 - 2018/09/24/08:45:56 ER - TY - RPRT TI - Thinking and working with political settlements AU - Kelsall, Tim AB - • In recent years, Political Settlements Analysis (PSA) has become increasingly influential in academic and policy circles • Despite its intuitive appeal, it seems difficult to use in practice • PSA has a natural affinity with Adaptive Development, Thinking and Working Politically, and Doing Development Differently approaches • By answering the simple diagnostic questions supplied here, development partners can identify the types of political settlements in which they work, and draw some broad operational implications DA - 2016/01// PY - 2016 SP - 8 M3 - Briefing PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10185.pdf Y2 - 2018/12/10/00:00:00 ER - TY - RPRT TI - Thinking big: How to use theory of change for systems change AU - Abercrombie, Rob AU - Boswell, Katie AU - Thomasoo, Rosanna AB - We think that, applied well, theory of change can support charities and funders to take a systemic approach to their work. This report identifies five common pitfalls that organisations fall into when using theory of change, and walks through five rules of thumb that will help organisations to use the approach to tackle complex problems. We think that, applied well, theory of change can support charities and funders to take a systemic approach to their work. This report identifies five common pitfalls that organisations fall into when using theory of change, and walks through five rules of thumb that will help organisations to use the approach to tackle complex problems. CY - London DA - 2018/03// PY - 2018 PB - NPC UR - https://www.thinknpc.org/publications/thinking-big/ Y2 - 2018/08/16/08:40:01 ER - TY - BOOK TI - Thinking in Systems: A Primer AU - Meadows, Donella H. A3 - Wright, Diana AB - In the years following her role as the lead author of the international bestseller, Limits to Growth—the first book to show the consequences of unchecked growth on a finite planet— Donella Meadows remained a pioneer of environmental and social analysis until her untimely death in 2001.Thinking in Systems, is a concise and crucial book offering insight for problem solving on scales ranging from the personal to the global. Edited by the Sustainability Institute’s Diana Wright, this essential primer brings systems thinking out of the realm of computers and equations and into the tangible world, showing readers how to develop the systems-thinking skills that thought leaders across the globe consider critical for 21st-century life.Some of the biggest problems facing the world—war, hunger, poverty, and environmental degradation—are essentially system failures. They cannot be solved by fixing one piece in isolation from the others, because even seemingly minor details have enormous power to undermine the best efforts of too-narrow thinking.While readers will learn the conceptual tools and methods of systems thinking, the heart of the book is grander than methodology. Donella Meadows was known as much for nurturing positive outcomes as she was for delving into the science behind global dilemmas. She reminds readers to pay attention to what is important, not just what is quantifiable, to stay humble, and to stay a learner.In a world growing ever more complicated, crowded, and interdependent, Thinking in Systems helps readers avoid confusion and helplessness, the first step toward finding proactive and effective solutions. DA - 2008/12/03/ PY - 2008 DP - Amazon SP - 240 LA - English PB - Chelsea Green Publishing ST - Thinking in Systems ER - TY - BLOG TI - Thoughts on the use of hypotheses in adaptive management AU - Sparkman, Timothy T2 - BEAM Exchange AB - New insights, opinions and perspectives on market systems development, from experts and practitioners. DA - 2015/01/06/ PY - 2015 ST - BEAM Exchange UR - https://beamexchange.org/community/blogs/2015/1/6/timsparkman/ Y2 - 2016/10/25/19:31:37 ER - TY - JOUR TI - Three Approaches to Monitoring: Feedback Systems, Participatory Monitoring and Evaluation and Logical Frameworks AU - Jacobs, Alex AU - Barnett, Chris AU - Ponsford, Richard T2 - IDS Bulletin AB - This article compares key attributes, strengths and weaknesses of three different approaches to monitoring development interventions: the logical framework approach, participatory monitoring and evaluation (PM&E) and feedback systems. Academic and practitioner literature describes how logframes meet the needs of senior decision-makers to summarise, organise and compare projects. PM&E meets the needs of field staff to work sensitively with intended beneficiaries and support their learning and empowerment. Feedback systems appear to link the two, providing performance data for managers and creating incentives for implementing staff to focus on their intended beneficiaries. Feedback systems build on the rich heritage of PM&E and are compatible with logframes. They may help provide a manageable and effective approach to accountability that links the means and the ends of development interventions. Feedback systems are at an early stage of development. There is a lot to learn about how and where they work best. DA - 2010/// PY - 2010 DO - 10.1111/j.1759-5436.2010.00180.x DP - Zotero VL - 41 IS - 6 SP - 9 LA - en ER - TY - BLOG TI - Three out of Four Practical Lessons about right fitting MEL for Additive Effects AU - Guerzovich, Florencia T2 - Medium AB - This is a series about Monitoring, Evaluating and Learning (MEL) whether sets of interventions/portfolios are adding more together than each one would produce on their own. In post 1, I pointed to coherence, the new OECD-DAC evaluation criteria as a way to bridge the ambition of bringing bigger change with the MEL world. In post 2, I shared 3 of 4 practical lessons I’ve learned in experimenting with MEL systems and exercises that focus explicitly on interactions of interventions/portfolios. In the third post, I bring Paul Pierson’s groundbreaking argument for social science to MEL. Paraphrasing, most contemporary MEL takes a “snapshot” view of interventions and portfolios, distorting their effects and meaning by ripping them from their temporal context. Instead, we should place in time interventions/portfolios with the ambition to add more than the sum of the part by constructing MEL systems looking at “moving pictures” rather than taking snapshots. DA - 2023/08/01/T10:15:56.957Z PY - 2023 LA - en UR - https://medium.com/@florcig/three-out-of-four-practical-lessons-about-right-fitting-mel-for-additive-effects-2e31de67fc71 Y2 - 2023/08/10/09:00:53 ER - TY - JOUR TI - Timely evaluation in international development AU - Webster, Jayne AU - Exley, Josephine AU - Copestake, James AU - Davies, Rick AU - Hargreaves, James T2 - Journal of Development Effectiveness AB - Impact and process evaluations are increasingly used in international development; however they are generally retrospective in outlook. A more timely approach to evaluation aims to identify necessary, feasible and effective changes during a programme or intervention’s lifetime. This paper aims to identify, categorise, describe and critically appraise methods to support more timely evaluation in international development. Potential methods were identified through scoping seminar, public symposium, targeted review of the literature, and the authors’ own experiences and opinions. Findings from the different data sources were reviewed collectively by the author group and triangulated to develop an analytical framework. We identified four purposes of timely evaluation for international development, and critiqued the use of approaches against four dimensions of timeliness and flexibility. Whilst we found significant interest in more timely approaches to evaluation in international development, there was a dearth of published empirical evidence upon which to base strong recommendations. There is significant potential for timely evaluation to improve international development outcomes. New approaches to mixing and adapting existing methods, together with new technologies offer increased potential. Research is needed to provide an empirical evidence base upon which to further develop the application, across sectors and contexts, of timely evaluation in international development. DA - 2018/10/02/ PY - 2018 DO - 10.1080/19439342.2018.1543345 DP - Taylor and Francis+NEJM VL - 10 IS - 4 SP - 482 EP - 508 SN - 1943-9342 UR - https://doi.org/10.1080/19439342.2018.1543345 Y2 - 2019/03/12/16:55:49 KW - Adaptive learning KW - Outcome evaluation KW - Programme improvement KW - impact evaluation ER - TY - RPRT TI - Tiny Tools - Measuring Change in Communities and Groups AU - Causemann, Bernward AU - Gohl, Eberhard AU - Brenner, Verena AB - Introduction to the Overview: Tiny Tools Why “Tiny Tools” for assessing change? Currently, change is mostly assessed by NGO staff or external experts. The vision of this paper is that communities assess and reflect change themselves and make use of that reflection with appropriate tools. All the tools presented here are relatively quick and easy to learn (therefore “tiny”). With Tiny Tools we can assess change in one session. They can therefore be used where there are not baselines. They are structured and systematic, and they are all widely tested: Experience shows that these tools lead to new insights, mobilise enthusiasm and increase the capacity of communities to bring about further change. The Tiny Tools are in line with what Participatory Rural Appraisal (PRA) wanted to achieve. Many are slight variations of PRA tools. For a detailed description of concepts, see the NGO-IDEAs Impact Toolbox (www.ngo-ideas.net/impact_toolbox) and the NGO-IDEAs Manual Self-Effectiveness (www.ngo-ideas.net/monitoring_self_effectiveness). The tools are designed to visualise change, but also enable communities to reflect on the reasons of change or verify assessments. They may be implemented once or continuously over time. We know that the time of community members is precious, and limited. Therefore all Tiny Tools can be performed in a relatively short session, provided facilitators (it could be field staff or project officers) are experienced – and the community knows and trusts them. The amount of time spent on the application of the tools may however be prolonged according to the needs of a community or NGO. All of these tools are easy to learn for a facilitator experienced in participatory processes. Which tool should be introduced to which community? It is typically the decision of a development organisation (or external experts) which tools they want to introduce into a community. The staff need to assess which tool will lead to learning and action. It could also be that the staff realise aspects of change that they do not understand well enough. These tools are good for exploring change that we have not planned for and not anticipated. They are also good for exploring change in a context where we have no prior information. Communities are the best experts for their situation, but we emphasise that the tools should be used in ways that benefit and empower the communities or individuals participating. The tool implementations should lead to consequences on the grassroots as well as the NGO level. All Tiny Tools can be integrated into PIAR, the Analysis tool in the NGO-IDEAs Impact Toolbox. Also, the Tiny Tools can help to prepare for the application of the Toolbox tools. They help to make people aware of changes that can be observed. The following box gives some hints how Tiny Tools relate to the Impact Toolbox tools, and to what extent they help to attribute change to development interventions. CY - Stuttgart DA - 2012/01// PY - 2012 DP - Zotero LA - en PB - Impact+ UR - http://www.ngo-ideas.net/tiny_tools/index.html Y2 - 2023/07/14/00:00:00 ER - TY - ELEC TI - Tips for Finding Unusual Allies and Building Innovative Alliances AU - Jacobstein, David T2 - USAID Learning Lab AB - One of the truisms of thinking and working politically (TWP) is that it requires looking differently at how stakeholders relate to a given development challenge or process, and building coalitions of the “right actors at the right time” who can drive forward progress while working with the grain of the context. While it’s easy to call for coalitions for change and diverse stakeholder engagement, identifying allies beyond those already seized with an issue, and finding ways to collaborate with stakeholders who may have different reasons for joining a coalition, is a challenging task. These are some suggested tips to improve working in that fashion. DA - 2018/// PY - 2018 UR - https://usaidlearninglab.org/library/tips-finding-unusual-allies-and-building-innovative-alliances ER - TY - RPRT TI - Tips on Learning from Context: Formal and Informal Approaches to Understanding the Local Political Economy AU - Swift, Sarah AB - Successfully adapting your programmatic efforts and policy engagement to the context depends on maintaining your understanding of that context. Given this, continuous learning and feedback loops--not only about your programming, but also about the environment you are working within--are important components of USAID’s work. Since our environments are always changing, in ways big and small, this can be intimidating. One can imagine a scenario where learning consumes the limited time and budgets available, especially for small programs. The question becomes: How can we learn enough, at the right times, to limit missed opportunities and warning signs, while avoiding being consumed by data collection and analysis that never influences our work? This document capitalizes on the experiences of USAID staff as well as emerging learning from other donors and researchers to highlight a range of approaches--formal and informal, time-bound and continuous--that can support ongoing learning within the operational tempo of a busy Mission team or Operating Unit. DA - 2018/// PY - 2018 SP - 9 PB - USAID UR - https://usaidlearninglab.org/sites/default/files/resource/files/tips_on_context_monitoring_-_formal_to_informal_.pdf Y2 - 2018/09/22/00:00:00 ER - TY - BOOK TI - Tools for supporting sustainable natural resource management and livelihoods AU - Ashley, Holly AU - Kenton, Nicole AU - Milligan, Angela DA - 2013/// PY - 2013 DP - Open WorldCat LA - English SN - 978-1-84369-949-1 ER - TY - RPRT TI - Toward a Listening Bank: A review of best practices and the efficacy of Beneficiary Assessment AU - Salmen, Lawrence T2 - Social Development Papers AB - "This paper will first provide a descriptive overview of the beneficiary assessment (BA) work done on World Bank projects (by region, sector, phase of cycle, and so forth); it will then discuss impact, both qualitatively (with case studies) and quantitatively; and finally it will propose a course that, if taken, could lead to a Bank that truly listens and is attuned as much to the perspectives of the governments and peoples it serves as to the financial markets it helps sustain." "BAs use the qualitative techniques of conversational interviewing, focus groups, and participant observation with representative samples of key actors, such as the intended—usually poor—beneficiaries, service providers, non-governmental organizations (NGOs) and other local public- and private-sector leaders; analysis and the presentation of results are done as quantitatively as possible." Contents Beneficiary Assessment and the World Bank: An Overview Beneficiary Assessments in Africa Impact Case Studies Impact on Project Design Impact on Direct Cost Savings Summary of Quantifiable Impacts on Project Design and Objectives A Note on Methodology Complete the Paradigm Shift DA - 1998/09// PY - 1998 SP - 34 SN - 23 UR - http://siteresources.worldbank.org/INTPCENG/1143333-1116505682469/20509261/sdp-023-toward-a-listening-bank-ba-larry.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Towards an Alternative Development Management Paradigm? AU - Shutt, Cathy T2 - Rapport 2016:07 AB - Demonstrating results has been a concern in international development cooperation ever since it was started and in recent years there has been an increased focus on achieving and reporting on “results”. Despite the fact that everyone involved in development cooperation wants to make a difference there has been a growing criticism from practitioners about the “results agenda” based on a concern that the approaches used are not fit for purpose. In the EBA-report, Cathy Shutt, at the University of Sussex, scrutinizes the recent critical debate about results based management, and the main arguments and motives behind the criticism. She shows that the debate is not only a matter of obsessive measurement and reporting of meaningless numbers for political accountability, but also a matter of problematic assumptions and how we think about development, evidence and learning. In the report, Shutt also explores what could be learned from those who are not just critiquing results based management approaches but also proposing alternatives. Are these new alternatives an answer to the criticism? CY - Stockholm DA - 2016/// PY - 2016 PB - Expertgruppen för biståndsanalys (EBA) UR - https://eba.se/wp-content/uploads/2016/08/Rapport2016_07_webb.pdf Y2 - 2019/12/02/00:00:00 KW - IMPORTANT ER - TY - RPRT TI - Towards evidence-informed adaptive management AU - Hernandez, Kevin AU - Ramalingam, Ben AU - Wild, Leni T2 - Working Paper AB - Development and humanitarian organisations seeking to be adaptive have emphasised the need to be transparent about complexity and uncertainty; to be honest about their inability to control what happens; and to design programmes that change over time to become more appropriate and relevant. At their heart, adaptive management approaches emphasise the ability to lean, 'unlearn' and adapt programming accordingly. The cornerstone of effective learning is the creation, gathering, accumulation, interpretation and use of data and evidence. This working paper provides development professionals with tools, strategies and ideas to help them use evidence for adaptive management in practical and evidence-informed ways. Key messages - Evidence is central to effective and rigorous adaptive management. However, despite this central importance, exactly how evidence has been used to inform decisions for adapting development and humanitarian programmes in the past remains unobservable to many. - There is a need to strengthen and document evidence-informed adaptive management. This working paper proposes a roadmap to do this. - Those seeking to use evidence for adaptive management will need to manage trade-offs between ensuring a rigorous, documented (and auditable) trail of evidence-informed actions, being pragmatic about the time and resources allocated to documentation and recognising that it may be necessary to proceed without rigorous evidence when it is unavailable. DA - 2019/11// PY - 2019 DP - Zotero SP - 24 LA - en PB - ODI/GLAM SN - 565 ER - TY - BLOG TI - Towards ‘Targeted Systems Change’ AU - Rye, Sam T2 - Fieldnotes by Sam Rye AB - Modelling and communicating how to shift systems DA - 2019/02/09/T10:40:27.009Z PY - 2019 LA - en UR - https://medium.com/fieldnotes-by-sam-rye/towards-targeted-systems-change-7f4db6febb51 Y2 - 2024/03/27/10:15:11 ER - TY - SOUND TI - Transforming M&E for Uncertain and Complex Contexts: The UNDP’s Innovation Sandbox Approach AU - Fraser, Dugan AU - Haldrup, Søren T2 - Powered by Evidence Podcast AB - How can we transform monitoring and evaluation (M&E) into a more adaptive, emergent process to address uncertainty and complexity in todays’ world? How do we move from compliance and accountability to learning – to support better, more timely, decisions? Join GEI Program Manager, Dugan Fraser, as he discusses these questions and others with Special Guest, Søren Haldrup, from UNDP's Strategic Innovation Unit where he manages UNDP's innovation facility and leads a new initiative called the M&E Sandbox. DA - 2022/11/03/ PY - 2022 LA - en ST - Transforming M&E for Uncertain and Complex Contexts UR - https://www.globalevaluationinitiative.org/podcast/transforming-me-uncertain-and-complex-contexts-undps-innovation-sandbox-approach Y2 - 2023/01/03/12:40:47 ER - TY - JOUR TI - Triangulation, Respondent Validation, and Democratic Participation in Mixed Methods Research AU - Torrance, Harry T2 - Journal of Mixed Methods Research AB - Over the past 10 years or so the “Field” of “Mixed Methods Research” (MMR) has increasingly been exerting itself as something separate, novel, and significant, with some advocates claiming paradigmatic status. Triangulation is an important component of mixed methods designs. Triangulation has its origins in attempts to validate research findings by generating and comparing different sorts of data, and different respondents’ perspectives, on the topic under investigation. Respondent validation has sometimes been included in such processes, but it is an element that has not attracted significant attention from the MMR community. The article argues that attention to respondent validation is a significant issue for methodological debate and that it should be an important aspect of the development of democratic participation in MMR. DA - 2012/04/01/ PY - 2012 DO - 10.1177/1558689812437185 DP - SAGE Journals VL - 6 IS - 2 SP - 111 EP - 123 J2 - Journal of Mixed Methods Research LA - en SN - 1558-6898 UR - http://dx.doi.org/10.1177/1558689812437185 Y2 - 2017/02/23/12:43:15 ER - TY - JOUR TI - Twinning “Practices of Change” With “Theory of Change”: Room for Emergence in Advocacy Evaluation AU - Arensman, Bodille AU - van Waegeningh, Cornelie AU - van Wessel, Margit T2 - American Journal of Evaluation AB - Theory of change (ToC) is currently the approach for the evaluation and planning of international development programs. This approach is considered especially suitable for complex interventions. We question this assumption and argue that ToC’s focus on cause–effect logic and intended outcomes does not do justice to the recursive nature of complex interventions such as advocacy. Supported by our work as evaluators, and specifically our case study of an advocacy program on child rights, we illustrate how advocacy evolves through recursive interactions, with outcomes that are emergent rather than predictable. We propose putting “practices of change” at the center by emphasizing human interactions, using the analytical lenses of strategies as practice and recursiveness. This provides room for emergent outcomes and implies a different use of ToC. In this article, we make a clear distinction between theoretical reality models and the real world of practices. DA - 2018/06/01/ PY - 2018 DO - 10.1177/1098214017727364 DP - SAGE Journals VL - 39 IS - 2 SP - 221 EP - 236 J2 - American Journal of Evaluation LA - en SN - 1098-2140 ST - Twinning “Practices of Change” With “Theory of Change” UR - https://doi.org/10.1177/1098214017727364 Y2 - 2020/10/15/14:41:48 ER - TY - RPRT TI - Uncertainty and COVID-19: A turning point for Monitoring Evaluation, Research and Learning? - A discussion note for aid actors, policymakers and practitioners AU - Tyrrel, Lavinia AU - Roche, Chris AU - Jackson, Elisabeth AB - The COVID-19 pandemic has significantly shifted the context in which aid and development is being delivered. The global scale of the pandemic and the speed at which it is spreading mean that the ‘normal’ economic, ideological and organisational influences which shape (if not determine) aid delivery are in flux. This means that – for a relatively short-period – there is scope for aid actors to work collectively to embed more locally-led, politically-informed and adaptive forms of MERL in aid and development practice. These forms of Monitoring Evaluation Research and Learning (MERL) are not only well-suited to the current global pandemic. They also offer ways for aid program decision makers and practitioners to make sense of the complex and uncertain contexts in which much development work takes place. Applying locally-led, politically-informed and adaptive forms of MERL in the COVID-19 context and beyond requires a shift in mindset and approaches. Situations of complexity, in which it is difficult to predict the relationships between cause and effect, do not lend themselves to linear approaches and fixed indicators. Instead, they require ‘navigation by judgement’, ongoing learning and adaptation and greater privileging of local knowledge, and of the perspectives of those who are often excluded. Rather than being focused on upwards accountability, simple numbers and good news stories, the core function of MERL in this context is to support a better understanding – in real-time – of the changing operating context, to generate learning about the immediate impact of policy and program responses and their longer-term effects, and to inform decision making by front line staff. Whether the opportunities afforded by this ‘critical juncture’ are realised will depend on the degree to which those in the aid and development sector use this opportunity to promote a shift in the deep incentive structures within which development agencies are embedded. On the one hand, the pandemic underscores the limits of the linear understandings of change which underpin many orthodox approaches to planning, design and associated MERL. On the other hand, there is a vested interest in the status quo amongst many organisations, consultants, researchers and MERL practitioners. This is because approaches which promote locally-led development inevitably require those in power to relinquish control. While a range of factors make this shift difficult, there is more scope to change internal ways of working in development agencies than is commonly acknowledged. There is no time like the present to advocate for a ‘new normal’ for MERL. DA - 2020/05// PY - 2020 PB - Abt Associates UR - https://abtassocgovernancesoapbox.files.wordpress.com/2020/05/200514-uncertainty-and-covid19-a-turning-point-for-merl-final.pdf Y2 - 2020/10/15/10:04:43 ER - TY - RPRT TI - Understanding ‘theory of change’ in international development: A review of existing knowledge AU - Stein, Danielle AU - Valters, Craig AB - This is a review by Danielle Stein and Craig Valters of the concepts and debates within ‘Theory of Change’ (ToC) material, resulting from a search and detailed analysis of available donor, agency and expert guidance documents. The review was undertaken as part of a Justice and Security Research Program (JSRP) and The Asia Foundation (TAF) collaborative project, and focuses on the field of international development. CY - London DA - 2012/// PY - 2012 DP - Zotero LA - en PB - LSE UR - https://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/UNDERSTANDINGTHEORYOFChangeSteinValtersPN.pdf Y2 - 2019/08/12/00:00:00 ER - TY - JOUR TI - Unpacking Mechanisms in Climate Resilient Agriculture Interventions AU - Barrett, Sam AU - D'Errico, Stefano AU - Anderson, Simon AU - Nebsu, Bayu T2 - New Directions for Evaluation AB - The investigation of causal mechanisms has the capacity to provide donors and implementing institutions with a greater understanding of people's reasoning and reactions as they work with interventions. This chapter contributes to the literature by identifying behavioral mechanisms generated through engagement with climate-resilient agriculture interventions within a larger livelihood project in Ethiopia. It works through the steps that enabled the study to unpack the black box between the climate-smart interventions and the outcome of crop production. The first step was a matching-based sampling design, following households over 3 years with six biannual surveys, and setting up a quasi-experimental evaluation setting. The second step used the data generated from the surveys in difference-in-difference models to assess the impact of the intervention on crop production. Third, to gain insight into the mechanisms at work, Focus Group Discussions (FGD) were conducted with smallholder farmer project beneficiaries. The FGDs revealed what the beneficiaries themselves considered the key mechanisms generated from the intervention, thus forming the bridge between the interventions and outcome. The result was an evaluation design enabling deeper insight into attribution claims. The findings offered novel insights for policymakers about how the climate-resilient interventions worked for the people themselves and shedding light on the inner workings of the climate-smart technologies. Finally, they provided key stakeholders (commissioning agency and implementing organizations) with a powerful means by which to learn about the last project, so to better plan for the next iteration and improve the climate resilience of smallholder farmers. DA - 2020/// PY - 2020 DO - 10.1002/ev.20423 DP - Wiley Online Library VL - 2020 IS - 167 SP - 115 EP - 130 LA - en SN - 1534-875X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20423 Y2 - 2021/02/18/14:26:35 ER - TY - RPRT TI - USAID/Colombia Introduces Political Economy Analysis to Better Adapt Programming to Local Contexts AU - Sharp, Preston T2 - CLA Case Competition CY - Colombia DA - 2015/// PY - 2015 SP - 2 PB - USAID UR - https://usaidlearninglab.org/sites/default/files/resource/files/cla_case_competition_casestory_25_usaidcolombia_colombia.pdf Y2 - 2018/09/22/00:00:00 ER - TY - RPRT TI - USAID Wildlife Asia as a case study in adaptive rigour AU - Ziegler, Jessica T2 - Briefing Paper AB - This briefing note looks at how the United States Agency for International Development (USAID) Wildlife Asia programme has operationalised the concepts of adaptive rigour and adaptive management as part of its approach to collaborating, learning and adapting. As described by the Global Learning for Adaptive Management (GLAM) initiative, adaptive rigour is about ensuring that the data, information, methods, processes and systems that underpin adaptive management are robust, systematic and high‑quality. Key messages When faced with programmatic complexity, it is important to take an adaptive approach driven by continuous and iterative monitoring, evaluation and learning (MEL). USAID Wildlife Asia, which works to advance regional action towards ending illegal wildlife trafficking in Southeast Asia and China, has embraced this approach as a way of operationalising adaptive rigour. Throughout programming, MEL approaches should reflect the characteristics of adaptive rigour: comprehensiveness, usefulness, practicality, timeliness and support. Utilising performance monitoring and research in order to test and revise technical approaches and employing mixed methods to collect both qualitative and quantitative data, as well as looking for relevant lessons generated by others, can ensure access to the most useful information for decision-making throughout implementation. For adaptive management, it is not enough to monitor, evaluate and learn; it is also essential to pause and reflect in order to analyse and process evidence gained through MEL with colleagues and stakeholders to reach the right conclusions and make good decisions. DA - 2020/// PY - 2020 DP - Zotero SP - 13 LA - en PB - ODI/GLAM ER - TY - JOUR TI - Useful Theory of Change Models AU - Mayne, John T2 - Canadian Journal of Program Evaluation AB - Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, flexible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent. DA - 2015/08/01/ PY - 2015 DO - 10.3138/cjpe.230 DP - DOI.org (Crossref) VL - 30 IS - 2 SP - 119 EP - 142 J2 - CJPE LA - en SN - 08341516 UR - https://cjpe.journalhosting.ucalgary.ca/cjpe/index.php/cjpe/article/view/294/230 Y2 - 2020/02/24/18:04:38 ER - TY - JOUR TI - Using a Participatory Theory Driven Evaluation Approach to Identify Causal Mechanisms in Innovation Processes AU - Blundo‐Canto, Genowefa AU - Devaux‐Spatarakis, Agathe AU - Mathé, Syndhia AU - Faure, Guy AU - Cerdan, Claire T2 - New Directions for Evaluation AB - Applied agricultural research institutes play different roles in complex agricultural innovation processes, contributing to them with other actors. To foster learning and usable knowledge on how research actions influence such lasting innovation processes, there is a need to identify the causal mechanisms linking these actions and the effects of the changes they enable. A participatory, theory-driven, ex-post evaluation method, ImpresS, was developed by the French Agricultural Research Center for International Development (Cirad). ImpresS reconstructs the innovation history and its impact pathway by analyzing behavioral mechanisms linked to stakeholders' individual reactions and responses, and underlying process mechanisms at a group level. ImpresS relies on iterative updating and refinement and on triangulating data sources and collection methods to ensure internal validity and to increase credibility by enabling different actors to express their viewpoints. Drawing on an in-depth case study, we discuss how ImpresS makes it possible to draw robust conclusions on causal mechanisms while posing challenges linked to the group dynamics and power imbalances commonly encountered in participatory methods. As demonstrated by the case study, ImpresS generates policy-relevant knowledge for future research projects. It also demonstrates how research actions can help coconstruct lasting dynamics that can survive fluctuating institutional support. Distinguishing between behavioral and process mechanisms benefits knowledge use as it makes it possible to disentangle the conditions that trigger changes in a given context while generating research questions concerning the external validity of mechanism hypotheses. DA - 2020/// PY - 2020 DO - 10.1002/ev.20429 DP - Wiley Online Library VL - 2020 IS - 167 SP - 59 EP - 72 LA - en SN - 1534-875X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20429 Y2 - 2021/02/18/14:26:32 ER - TY - RPRT TI - Using a ‘Partnership Rubric’ in Participatory Evaluations AU - Snijder, Mieke AU - Hicks, Jacky AU - Paul, Sukanta AU - Arulanantham, Amit AU - Apgar, Marina AU - Afroze, Jiniya AU - Karki, Shanta AU - Mareschal, Sophie AU - Prieto Martín, Pedro AU - Uddin, Forhad AU - Veitch, Helen T2 - CLARISSA Learning Note AB - Programmes that aim to tackle complex societal issues, such as the worst forms of child labour, require rich partnerships that bring together different perspectives. CLARISSA’s consortium partnership adopts an empowerment approach to the interventions we deliver and our ways of working together. Part of this approach involves ongoing reflection and learning about how we work together in our partnership, and how this can be adapted if needed. This learning note focuses on a method used in CLARISSA to both reflect on and strengthen how we work in partnerships – the partnership rubric. We found that using the rubric flexibly was key to mitigating some of the challenges of such a complex consortium. This included using it in different sizes of forum, with different levels of preparation. Periodically adapting it for country context and as new partners came on board also helped ensure a shared sense of our preferred ways of working as the project progressed. CY - Brighton DA - 2023/07/12/ PY - 2023 DP - opendocs.ids.ac.uk LA - en PB - Institute of Development Studies SN - 3 UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/18051 Y2 - 2023/10/16/13:06:53 ER - TY - JOUR TI - Using Actor-Based Theories Of Change to Conduct Robust Contribution Analysis in Complex Settings AU - Koleros, Andrew AU - Mayne, John T2 - Canadian Journal of Program Evaluation AB - The use of theories of change (ToCs) is a hallmark of sound evaluation practice. As interventions have become more complex, the development of ToCs that adequately unpack this complexity has become more challenging. Equally important is the development of evaluable ToCs, necessary for conducting robust theory-based evaluation approaches such as contribution analysis (CA). This article explores one approach to tackling these challenges through the use of nested actor-based ToCs using the case of an impact evaluation of a complex police-reform program in the Democratic Republic of Congo, describing how evaluable nested actor-based ToCs were built to structure the evaluation. DA - 2019/03/01/ PY - 2019 DO - 10.3138/cjpe.52946 DP - DOI.org (Crossref) VL - 33 IS - 3 J2 - CJPE LA - en SN - 1496-7308, 0834-1516 UR - https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/52946 Y2 - 2019/08/12/22:14:58 KW - Behavioural strategy KW - COM-B ER - TY - RPRT TI - Using adaptive monitoring, evaluation and learning in programme design AU - Nzegwu, Femi DA - 2018/08// PY - 2018 DP - Zotero SP - 5 LA - en PB - INASP ER - TY - JOUR TI - Using AI to disrupt business as usual in small evaluation firms AU - Sabarre, Nina R. AU - Beckmann, Blake AU - Bhaskara, Sahiti AU - Doll, Kathleen T2 - New Directions for Evaluation AB - While many knowledge workers may fear that the rise of artificial intelligence (AI) will threaten their jobs, this article argues that small evaluation businesses should embrace AI tools to increase their value in the marketplace and remain relevant. In this article, consultants from a research, evaluation, and strategy firm, Intention 2 Impact, Inc., make a case for using AI tools to disrupt business as usual in evaluation from theoretical and practical perspectives. Theoretically, AI may be another example of technology that was initially feared but is now ubiquitous in society. Using concrete examples, the authors describe how businesses and evaluators have evolved to keep up with changes in supply and demand. Lastly, it is posited that embracing AI will save time for those working in small businesses, which can ultimately increase added value and profitability. DA - 2023/// PY - 2023 DO - 10.1002/ev.20562 DP - Wiley Online Library VL - 2023 IS - 178-179 SP - 59 EP - 71 LA - en SN - 1534-875X UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20562 Y2 - 2023/12/11/09:47:54 ER - TY - RPRT TI - Using case studies to do program evaluation AU - Balbach, Edith D. AB - This paper, authored by Edith D. Balbach for California Department of Health Services is designed to allow evaluators to decide weather to use a case study evaluation approach. It also offers guidance on how to conduct a case study evaluation. Contents Using a Case Study as an Evaluation Tool 3 When to Use a Case Study 4 How to Do a Case Study 6 Unit Selection 6 Data Collection 7 Data Analysis and Interpretation 12 DA - 1999/// PY - 1999 PB - California Department of Health Services UR - http://case.edu/affil/healthpromotion/ProgramEvaluation.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - Using middle-level theory to improve programme and evaluation design AU - Cartwright, N AB - What can middle-level theory do? Middle-level theory (MLT) has several uses in development planning and evaluation. „ It helps predict whether a programme can be expected to work in a new setting. „ It offers insights into what design features are needed for success. „ It provides invaluable information for monitoring to see if the programme is on track and to fix problems that arise. „ It reveals the causal processes and related assumptions to be tested in an evaluation and helps identify evaluation questions. „ It helps in interpreting evaluation findings, assessing their relevance and locating a description of them that is useful for programme design and evaluation in other settings CY - Oxford DA - 2020/10// PY - 2020 M3 - Methods Brief PB - CEDIL UR - https://cedilprogramme.org/wp-content/uploads/2020/10/PDD10144-CEDIL-Template-WEB.pdf Y2 - 2021/02/18/14:55:21 ER - TY - JOUR TI - Using Participatory Process Evaluation to Understand the Dynamics of Change in a Nutrition Education Programme AU - Cornwall, Andrea T2 - IDS Working Papers AB - With roots in approaches to popular education and participatory action research that place the learner and the ‘beneficiary’ of development at the centre of enquiry and action, the participatory visualisation methods associated with Participatory Rural Appraisal have been widely used as tools for learning and accountability. In this article, I reflect on lessons learnt from using these methods in a participatory process evaluation of an educational programme aimed at addressing chronic malnutrition in an East African country. Building on this experience, I explore the educative and empowering dimensions of participatory visualisation methods, and consider the contribution that these methods can make to effective evaluation. DA - 2014/// PY - 2014 DO - 10.1111/j.2040-0209.2014.00437.x DP - Wiley Online Library VL - 2014 IS - 437 SP - 1 EP - 22 LA - en SN - 2040-0209 UR - https://onlinelibrary.wiley.com/doi/abs/10.1111/j.2040-0209.2014.00437.x Y2 - 2020/10/15/14:35:59 KW - Kenya KW - nutrition KW - participatory impact evaluation KW - process evaluation KW - randomised controlled trials ER - TY - JOUR TI - Using Participatory Statistics to Examine the Impact of Interventions to Eradicate Slavery: Lessons from the Field AU - Oosterhoff, P. AU - Bharadwaj, S. AU - Burns, D. AU - Raj, A. M. AU - Nanda, R. B. AU - Narayanan, P. AB - This CDI Practice Paper by Pauline Oosterhoff, Sowmyaa Bharadwaj, Danny Burns, Aruna Mohan Raj, Rituu B. Nanda and Pradeep Narayanan reflects on the use of participatory statistics to assess the impact of interventions to eradicate slavery and bonded labour. It deals with: (1) the challenges of estimating changes in the magnitude of various forms of slavery; (2) the potential of combining participatory approaches with statistical principles to generate robust data for assessing impact of slavery eradication; and (3) the practical and ethical questions in relation to working with people living within a context of modern slavery. The paper draws lessons from the realities of using participatory statistics to support the evaluation of a slavery eradication programme in North India. DA - 2016/02// PY - 2016 DP - opendocs.ids.ac.uk LA - en ST - Using Participatory Statistics to Examine the Impact of Interventions to Eradicate Slavery UR - https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/9582 Y2 - 2020/10/15/14:42:05 ER - TY - RPRT TI - Using political economy analysis in conflict, security and justice programmes AU - Denney, Lisa DA - 2016/03// PY - 2016 SP - 20 M3 - Toolkit PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10362.pdf Y2 - 2018/12/10/00:00:00 ER - TY - RPRT TI - Using Randomization in Development Economics Research: A Toolkit AU - Duflo, Esther AU - Glennerster, Rachel AU - Kremer, Michael AB - This paper is a practical guide (a toolkit) for researchers, students and practitioners wishing to introduce randomization as part of a research design in the field. It first covers the rationale for the use of randomization, as a solution to selection bias and a partial solution to publication biases. Second, it discusses various ways in which randomization can be practically introduced in a field settings. Third, it discusses designs issues such as sample size requirements, stratification, level of randomization and data collection methods. Fourth, it discusses how to analyze data from randomized evaluations when there are departures from the basic framework. It reviews in particular how to handle imperfect compliance and externalities. Finally, it discusses some of the issues involved in drawing general conclusions from randomized evaluations, including the necessary use of theory as a guide when designing evaluations and interpreting results. DA - 2006/12// PY - 2006 DP - National Bureau of Economic Research M3 - Working Paper PB - National Bureau of Economic Research SN - 333 ST - Using Randomization in Development Economics Research UR - http://www.nber.org/papers/t0333 Y2 - 2018/10/19/15:12:28 ER - TY - SLIDE TI - Using stories in evaluation: Participatory Narrative Inquiry and Sensemaking T2 - gLocal Evaluation Week 2022 AB - Sensemaking and Participatory Narrative Inquiry (PNI) are similar approaches, based on collecting stories about a real life experience from a large number of stakeholders on a topic of interest to the evaluation, and giving the storytellers themselves the responsibility for analyzing and making sense of the stories. Because a large number of stories are collected and self-interpreted, it is possible to conduct quantitative analysis of recurrent themes, perspectives and feelings emerging in the narratives. By combining elements of qualitative and quantitative research, these approaches help to make sense of complex and evolving realities. This session presented the general approach of Sensemaking and PNI. Panelists discussed their experiences with this method, recently used in IFAD and WFP’s evaluations and invited participants to share their own experiences with using stories in evaluation. DA - 2022/06/21/ PY - 2022 M3 - Webinar ST - Using stories in evaluation UR - https://www.evalforward.org/webinars/using-stories Y2 - 2022/07/26/12:40:02 ER - TY - SLIDE TI - Using Systemic M&E Tools in Feed The Future Uganda: Sensemaker ® - Webinar A2 - USAID Learning Lab AB - This webinar is part of a series organized by SEEP's Market Facilitation Initiative (MaFI), USAID's LEO project and the BEAM Exchange as a part of the "Learning with the Toolmakers" webinar series. As part of a new cycle of webinars focusing on inclusive market facilitation tools. In this series, Eric Derks and Leanne Rasmussen will share learning from this five-year, multi-million dollar USAID program, the objective of which is to increase Ugandan farmers’ use of good quality agro-inputs by fostering more inclusive systemic changes in the agro-inputs industry. Using Systemic M&E Tools in Feed The Future Uganda: Sensemaker ® This third webinar in the series, titled “,” will look at how the project team used this tool and method to monitor systemic change. Leanne and Eric will share their experiences with using the tool and give recommendations to other practitioners who are interested in applying it. What is Sensemaker ®? SenseMaker combines an innovative research methodology with patented software to collect and analyze large quantities of narratives in order to understand complex change. It brings together insights from complexity sciences, anthropology, and cognitive science. It uses participants’ narratives to uncover foundational attitudes and norms that inform and influence behavior. This webinar is organized by SEEP's Market Facilitation Initiative (MaFI), USAID's LEO project and the BEAM Exchange as a part of the "Learning with the Toolmakers" webinar series. DA - 2015/// PY - 2015 UR - https://usaidlearninglab.org/events/using-systemic-me-tools-feed-future-uganda-sensemaker-%C2%AE ER - TY - JOUR TI - Using Systemigrams in Problem Definition: A Case Study in Maritime Resilience for Homeland Security AU - Sauser, Brian AU - Mansouri, Mo AU - Omer, Mayada T2 - Journal of Homeland Security and Emergency Management AB - The concept of resilience has been used among a diversity of fields with a myriad of definitions. A systems approach to discovering the essence of resilience could aid in understanding its concept and provide opportunities to distinguish its systemic characteristics that can be identified, planned, or analyzed regardless of the field in which it is considered. In order to begin executing such an approach, a comprehensive study of the literature on the topic and circumstances in which resilience has been used or referred to in several fields of studies is necessary. In this paper, the conceptual common denominator of resilience is identified based on the results of a literature survey and according to the way it has been used in different contexts. This basis leads to the objective of this effort, and that is to recognize the systemic characteristics of resilience and demonstrate the defining of a universally accepted definition. Using a soft systems methodology and a supporting systemic diagramming technique entitled Systemigrams, we present the formulation of defining resilience in maritime homeland security. In this effort, many stakeholders contributed their thoughts and concerns on the meaning and operational use of resilience through a repetitive process to formulate a definition of the term. The result of this methodic approach is a general definition of resilience that has been refined in a collaborative environment via the application the Boardman Soft Systems Methodology. DA - 2011/01/27/ PY - 2011 DO - 10.2202/1547-7355.1773 DP - ResearchGate VL - 8 J2 - Journal of Homeland Security and Emergency Management ST - Using Systemigrams in Problem Definition ER - TY - BLOG TI - Utilization-Focused Evaluation AU - Better Evaluation T2 - Better Evaluation AB - Utilization-Focused Evaluation (UFE), developed by Michael Quinn Patton, is an approach based on the principle that an evaluation should be judged on its usefulness to its intended users. Therefore evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance. UR - https://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation Y2 - 2018/10/22/00:00:00 ER - TY - BOOK TI - Utilization focused evaluation: a primer for evaluators AU - Ramirez, Ricardo AU - Brodhead, Dal AB - Ramírez, R., & Brodhead, D. (2013). Utilization focused evaluation: A primer for evaluators. Penang, Malaysia: Southbound. Retrieved from http://evaluationinpractice.files.wordpress.com/2013/04/ufeenglishprimer... This book, authored by Ricardo Ramírez and Dal Brodhead, is designed to support evaluators and program managers implement Utilization-focused evaluation (UFE). It includes detailed discussion of the 12 steps for implementing UFE and also provides a number of case studies to guide the user. Excerpt "Throughout this Primer we refer to the value of having a mentor to assist an evaluator who is using UFE for the first time. Our collective experiences with UFE indicated having a mentor was, for many UFE participants, an essential support and it reflects how we learned and mentored UFE. Evaluators may use elements of a UFE in their work naturally, for example by engaging users in planning the process or in assisting them in the utilization of findings. This Primer, however, walks the reader through UFE by systematically covering all of the 12 steps. It reflects deeply on the UFE evaluation practice and builds from it." (Ramírez & Brodhead 2013) Contents What is Utilization Focused Evaluation? 1 The UFE Framework Summarized in Steps 3 The DECI Project 6 The UFE Steps Illustrated with Project Examples 9 Step 1 Assessing Program Readiness 10 Step 2 Assessing Evaluators’ Readiness 18 Step 3 Identifying Primary Intended Users 24 Step 4 Situational Analysis 31 Step 5 Identification of Primary Intended Users 38 Step 6 Focusing the Evaluation 44 Step 7 Evaluation Design 54 Step 8 Simulation of Use 58 Step 9 Data Collection 62 Step 10 Data Analysis 65 Step 11 Facilitation of Use 68 Step 12 Meta Evaluation 72 Summary About What Each Step Entails 76 What Benefit Does UFE Bring to Commissioners of Evaluation? 81 Take Away Lessons 83 Postscript 87 The UFE Checklist 89 Case Studies 103 Recommended Reading 110 CY - Penang, Malaysia DA - 2013/// PY - 2013 DP - Open WorldCat SP - 132 LA - English PB - Southbound SN - 978-983-9054-61-3 ST - Utilization focused evaluation UR - https://evaluationinpractice.files.wordpress.com/2013/04/ufeenglishprimer.pdf Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Utilization-focused evaluation for agricultural innovation AU - Patton, Michael Quinn AU - Horton, D T2 - ILAC Brief AB - Utilization-focused evaluation (UFE) is based on the principle that an evaluation should be judged by its utility. So no matter how technically sound and methodologically elegant, an evaluation is not truly a good evaluation unless the findings are used. UFE is a framework for enhancing the likelihood that evaluation findings will be used and lessons will be learnt from the evaluation process. This Brief, based on the book Utilization-focused evaluation, introduces this approach to evaluation, outlines key steps in the evaluation process, identifies some of the main benefits of UFE, and provides two examples of UFE in the context of programmes aimed at promoting agricultural innovation. DA - 2009/// PY - 2009 SP - 4 PB - CGIAR SN - 22 UR - https://cgspace.cgiar.org/bitstream/handle/10568/70056/ILAC_Brief22_Utilization_Focus_Evaluation.pdf?sequence=1&isAllowed=y Y2 - 2018/10/22/11:20:10 ER - TY - RPRT TI - Utilization-Focused Evaluation (U-FE) Checklist AU - Patton, Michael Quinn T2 - Evaluation Checklists Projects AB - Utilization-Focused Evaluation begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use. Use concerns how real people in the real world apply evaluation findings and experience and learn from the evaluation process. The checklist is based on Essentials of Utilization-Focused Evaluation (Patton, 2012, Sage Publications). All references in the checklist to exhibits and menus refer to this book. Step 1 Assess and build program and organizational readiness for utilization-focused evaluation. Step 2 Assess and enhance evaluator readiness and competence to undertake a utilizationfocused evaluation. Step 3 Identify, organize, and engage primary intended users. Step 4 Conduct situation analysis with primary intended users Step 5 Identify primary intended uses by establishing the evaluation’s priority purposes. Step 6 Consider and build in process uses if appropriate. Step 7 Focus priority evaluation questions. Step 8 Check that fundamental areas for evaluation inquiry are being adequately addressed. Step 9 Determine what intervention model or theory of change is being evaluated. Step 10 Negotiate appropriate methods to generate credible findings and support intended use by intended users. Step 11 Make sure intended users understand potential controversies about methods and their implications. Step 12 Simulate use of findings. Step 13 Gather data with ongoing attention to use. Step 14 Organize and present the data for use by primary intended users. Step 15 Prepare an evaluation report to facilitate use and disseminate significant findings to expand influence. Step 16 Follow up with primary intended users to facilitate and enhance use. Step 17 Metaevaluation of use: Be accountable, learn, and improve DA - 2013/01// PY - 2013 SP - 19 PB - The Evaluation Center UR - https://www.wmich.edu/sites/default/files/attachments/u350/2014/UFE_checklist_2013.pdf Y2 - 2018/10/22/00:00:00 ER - TY - RPRT TI - Value for Investment. A Practical Evaluation Theory AU - King, Julian AB - I am a public policy consultant from Auckland, New Zealand. My practice specializes in evaluation and economic analysis. In this booklet I set out a model for evaluating value for investment (VFI) in social programs. The model defines VFI as an evaluative question about an economic problem. It uses explicit evaluative reasoning to provide a clear answer to the VFI question. Methods are matched to context, to provide the specific mix of evidence (economic and/or other) needed to support a wellreasoned, well-evidenced evaluative judgment. CY - Auckland DA - 2016/// PY - 2016 PB - Kinnect Group UR - http://www.julianking.co.nz/wp-content/uploads/2014/09/160527-VFI-jk8-web.pdf Y2 - 2023/09/28/10:07:15 ER - TY - RPRT TI - Value for Investment: Application and Insights. Youth Primary Mental Health and Addictions Evaluation AU - King, Julian AU - Crocket, Alicia AU - Field, Adrian AB - What’s in the guide? This new document provides: An overview of the VfI approach, including the conceptual principles and processes underpinning it A worked example of the approach in action: evaluating the Youth Primary Mental Health and Addictions initiative in New Zealand Transferrable learning for others considering the use of the VfI approach. CY - New Zealand DA - 2023/06/22/ PY - 2023 DP - Zotero LA - en PB - Dovetail Consulting UR - https://juliankingnz.substack.com/p/new-vfm-guide-for-evaluators ER - TY - RPRT TI - Value for money and adaptive programming - Approaches, measures and management AU - Laws, Ed AU - Valters, Craig AB - - The United Kingdom’s Foreign, Commonwealth and Development Office (FCDO)’s standard economy, efficiency, effectiveness/cost-effectiveness and equity (4E) framework is still relevant for approaching, measuring and managing value for money (VfM) for adaptive programmes. • However, this framework needs to be reframed to capture and incentivise flexibility, learning and adaptation. • VfM appraisal and reporting should be done in a way that draws on beneficiary feedback and informs good decision-making, rather than just being a compliance exercise. • If VfM appraisal and reporting cannot be done appropriately for adaptive programmes, it should be avoided or minimised. There is a risk of diverting time and resources from more suitable tools and methods. CY - London DA - 2021/03// PY - 2021 M3 - Working Paper PB - ODI UR - https://cdn.odi.org/media/documents/odi-ml-rethinkingvfm-wp572-final.pdf Y2 - 2021/06/04/00:00:00 ER - TY - RPRT TI - Valuing peace: delivering and demonstrating mediation results - Dilemmas & options for mediators AU - Wadley, Ian T2 - Mediation Practice Series AB - Essential points for practitioners and donors • Mediation offers a cost-effective and proven method for resolving armed conflict. Between 1985 and 2015, 75 per cent of armed conflicts in the world were resolved through agreement rather than by force. In most cases these processes will have involved third party facilitation or support. • Professional mediators understand the high stakes involved in their work to prevent, mitigate and resolve armed conflict. In addition, they and their financial supporters are increasingly required to demonstrate ‘value-for-money’ to ensure continued funding. • However, traditional monitoring and evaluation (M&E) methods are not well suited to this task, typically imposing artificially linear project models on a dynamic conflict situation, as well as compliance reporting that moves attention away from real value. • Traditional M&E methods tend to focus on documenting the past and generating vast amounts of data, rather than enabling timely adaptation of the project in the present. • Traditional M&E approaches rely heavily on external evaluation consultants. Even in the best of cases this may interfere with the mediation process and impose a heavy time burden on the project team, leading to low acceptance of traditional M&E approaches by mediation practitioners. • In contrast, an ideal M&E approach for mediation should deliver useful insights in even the most dynamic and sensitive mediation environments, impose a light reporting burden, and be readily accepted by mediation teams. It should protect discretion and trust, enable rapid adaptation, and also provide some assurance that donor funds are being well spent. CY - Geneve DA - 2017/11// PY - 2017 PB - Centre for Humanitarian Dialogue UR - https://www.hdcentre.org/wp-content/uploads/2017/12/HDC_MPS7_EN-REV2-WEB.pdf Y2 - 2023/08/15/07:40:00 ER - TY - RPRT TI - Volume 3: Participatory Monitoring, Evaluation, and Learning in Complex Adaptive Environments AU - SACE AB - This volume is the last in a series of papers about systems approaches in complex environments, which includes the use of the collective impact model to address large-scale social problems, and the application of participant-driven MEL techniques across 17 networks of civil society organizations. It is based on the experiences of Root Change and Chemonics, two development partners working on a USAID civic engagement project in Nigeria (2013-2018), as well as hundreds of civil society organization partners. This is the final paper in the series and aims to highlight how the adaptation of participatory monitoring, evaluation, and learning techniques (e.g., most significant change, outcome mapping, and outcome harvesting) evolved and ultimately empowered cluster members. The first paper in the series presented a brief introduction to systems approaches in advocacy settings, the SACE theory of change, and the scope of Root Change’s work as technical lead on capacity building and measurement. The second volume aims to address the innovative use of the advocacy strategy matrix, adapted from work by the Center for Evaluation Innovation, for collective impact and the Collective Impact Model, an approach that engages multiple players in working together to solve complex social problems. DA - 2018/12// PY - 2018 PB - SACE Program UR - https://www.rootchange.org/wp-content/uploads/2019/11/SACE-Legacy-Volume-3-Branded.pdf Y2 - 2020/02/14/12:19:18 ER - TY - BLOG TI - Walking the adaptive talk AU - Denney, Lisa T2 - Devpolicy Blog from the Development Policy Centre AB - The first in a three-part series on adaptive aid.Lisa Denney clarifies the confusion. DA - 2018/09/11/T06:00:03+10:00 PY - 2018 LA - en-US UR - http://www.devpolicy.org/walking-the-adaptive-talk-20180911/ Y2 - 2018/09/24/08:15:47 ER - TY - BLOG TI - We created this MEL system for you, now please own it! | From Poverty to Power AU - Pasanen, Tiina AU - Yanguas, Pablo AB - External consultants, learning partners or critical friends -whatever we call them- can seldom change the system or organisational (learning) culture from outside. So, how can Monitoring Evaluation and Learning (MEL) consultants support real change instead of creating tools or processes that are quickly forgotten without any real institutional ownership? Consultants and learning partners can seldom change a learning system or organisational culture – it just does not work like that. What they can do is interrogate existing practices, introduce new ideas, and support change processes already underway to ensure lasting, transformative impact. They are a multiplier for homegrown change initiatives, not a substitute. Don’t hire MEL consultants to sell you tools or tell you what to do. Hire them because you need a critical friend along the path of organisational change. Own, then learn – not the other way round. DA - 2021/07/14/ PY - 2021 UR - https://frompoverty.oxfam.org.uk/we-created-this-mel-system-for-you-now-please-own-it/ Y2 - 2023/12/14/21:36:17 ER - TY - VIDEO TI - Webimar - Adaptive Management at the Strategy Level: Portfolio Reviews and Mid Course Stocktaking AU - USAID AB - On Wednesday, November 15, 2017 at 8:00 a.m. EST, the Bureau for Policy, Planning and Learning (PPL) held a one-hour webinar exploring Strategy-level Portfolio Reviews and Mid Course Stocktaking. DA - 2017/11/15/ PY - 2017 LA - en PB - USAID ST - Adaptive Management at the Strategy Level UR - https://usaidlearninglab.org/library/adaptive-management-strategy-level-portfolio-reviews-and-mid-course-stocktaking Y2 - 2019/02/25/12:30:28 ER - TY - BLOG TI - Webinar - Sustainability Ready: what it takes to support & measure lasting change AU - Cekan, Jindra T2 - Valuing Voices AB - On June 24th under GLocal’s UNConference, “Co-creating our future stories of hope and action”, Jindra Cekan, Holta Trandafili, and Isabella Jean presented their work on sustainability evaluations and exit strategies via local voices. We chaired a 2-hour discussion session on the following topics: Sustainability of global development projects and exit from them, The importance of valuing local partners’ and participants’ voices, How to embed ex-post evaluation of sustainability into the project cycle, How expectations, benchmarking, and early joint planning in exit strategies, as well as considering long-term ownership & relevance will support projects to be sustained locally, including questioning who will maintain results, Considering power dynamics between donors and ultimate ‘beneficiaries’ and the value of the impact of the project from a variety of perspectives, and more Here is the recording of our presentation or see just the PowerPoint presentation. We harvested lessons from our three presentations: Jindra Cekan: · Fear of learning about failure in our global development industry – INGOs are “waiting for a successful enough project” to commission ex-post project sustainability evaluation · It needs to be a culture of learning, not a culture of success. · Lack of transparency in sharing program evaluation results with communities and local government is widespread, and even more rarely do we come back after many years and share learnings · Participatory approaches are vital, listening to participants about sustained impacts is key · It is never attribution, always contribution. To isolate impacts, we need to look for project sites that haven’t had multiple other organizations overlapping through all phases. · Building sustainability planning throughout the project cycle is key – but often doesn’t happen Holta Trandafili: · Sustainability needs to be planned to be researched, including evaluating why or why not were project elements sustained, and why? What has the project done to enable communities to sustain improvements? · Expectations of sustainability need to be more modest (as most results are mixed good/bad) · We need to ask: How are you defining and measuring sustainability – for how long should the results last? Among how many participants? Have you set benchmarks for success? · We should expand your toolbox on methodology to investigate sustainability. Stories of success are one of a myriad of methods used, including mixed-methods, cost-benefit, etc. · Start with the need for learning not [just] accountability Isabella Jean: · Sustainability investigation/evaluation/learning should be mindful that this is NOT about projects. It is about people. · We have a system that focuses on gaps and needs to be filled vs. existing capacities’ structures to be reinforced. How can our work on measuring sustainability bring this to light and call it out, so that we change the norm? · Planning for sustainability requires the insight to integrate resources and experiences of outsiders with the assets and capacities of insiders to develop context-appropriate strategies for change If you would like to discuss this with any of us, please send us comments and we’re happy to respond. Thanks again to the G-Local UnConference team! Below please find our bios: Jindra Cekan/ova has worked in global development for 33 years focused on participatory design and M&E for global non-profits. She founded Valuing Voices 7 years ago. For details, see: Valuing Voices Founder Holta Trandafili is the Research, Learning, and Analytics Manager with World Vision US and has been leading field research, monitoring, and evaluation since 2007. She has led sustainability measurement studies for World Vision programs in Uganda, Kenya, Sri Lanka, India, Burma, and Bolivia. Her areas of expertise and interest include program and community groups’ sustainability measurements; agency-level measurements; empowerment approaches to development; integrated programming; local capacities for peace; gender analysis; and outcome monitoring. Currently, Mrs. Trandafili serves as an Advisory Committee member for InterAction’s Effectiveness and Program Evaluation Working Group and chairs one of the sub-working groups under The Movement for Community-led Development. Isabella Jean supports international and local organizations and funders to document promising practices, facilitate learning and strengthen capacities for conflict sensitivity, peacebuilding and humanitarian effectiveness. She has facilitated action research, collaborative learning and advisory engagements in over 25 countries, and serves as an advisor to policymakers, senior leadership and program teams. Isabella co-authored the book, Time to Listen: Hearing People on the Receiving End of Aid and developed practical guidance to support accountability to communities, listening and feedback loops, and responsible INGO exits. She teaches graduate-level courses on aid effectiveness, program strategies and M&E of peacebuilding at Brandeis University’s Heller School for Social Policy and Management. Previously, Isabella directed training at a community organizing network and conducted policy research for the Institute for Responsive Education, UNDP, and Coexistence International. DA - 2020/07/07/T23:07:04+00:00 PY - 2020 LA - en-US ST - Sustainability Ready UR - http://valuingvoices.com/sustainability-ready-what-it-takes-to-support-measure-lasting-change-webinar/ Y2 - 2020/10/01/09:34:19 ER - TY - BLOG TI - Werner Ulrich's Homepage AU - Ulrich, Werne AB - This website provides a variety of resources from Werner Ulrich's work in a range of philosophical and research areas with a particular focus on his own work in critical systems thinking and practice or, Critical Systems Heuristics (CSH). DA - 2018/// PY - 2018 UR - http://wulrich.com/ ER - TY - BLOG TI - What are the Simple Rules that guide our Strategies? AU - Green, Duncan T2 - From Poverty to Power AB - A management treatise on 'simplre rules' for companies facing unpredictable situations could provide useful guidance to aid organizations DA - 2019/11/21/T07:00:41+00:00 PY - 2019 LA - en-GB UR - https://oxfamblogs.org/fp2p/what-are-the-simple-rules-that-guide-our-strategies/ Y2 - 2020/10/01/10:27:20 ER - TY - JOUR TI - What are you afraid of: Collective leadership and its learning implications AU - Raelin, Joseph A T2 - Management Learning AB - In this provocation, the author attempts to cite the advantages of collective leadership while acknowledging the objections and fears of challengers. Collective leadership is seen as remote because it defies the traditional view of leadership as an individualistic attractive quality that not only protects us but is efficient when applied. Nevertheless, the collective alternative may not only be advisable but required in a connected world featuring a networked economy. The contemporary socio-politico-economic environment requires the contribution of, creativity from, and collaboration among multiple agents providing a dynamic concentration of management and knowledge. If we are to accept and recognize the contribution of a collective leadership, its development would require an entirely different learning model. In particular, collective leadership development occurs as an acute immersion into the practices that are embedded within in situ material–discursive relations—in other words, among people, objects, and their institutions. DA - 2018/02// PY - 2018 DO - 10.1177/1350507617729974 DP - DOI.org (Crossref) VL - 49 IS - 1 SP - 59 EP - 66 J2 - Management Learning LA - en SN - 1350-5076, 1461-7307 ST - What are you afraid of UR - http://journals.sagepub.com/doi/10.1177/1350507617729974 Y2 - 2023/05/22/13:07:23 ER - TY - RPRT TI - What does ‘adaptive programming’ mean in the health sector? AU - Tulloch, Olivia AB - • There is considerable interest in the concept of adaptive development and what it may look like in different sectors, including health. • Adaptive types of programming from the health sector are relatively advanced; as we work towards the Sustainable Development Goals (SDGs), experiences in health can provide useful lessons for other areas of development. • The international health community may not use the label ‘adaptive development’, but many are already conducting adaptive work. ‘Quality improvement’ is one such tried and tested approach. • Quality improvement is problem-driven, iterative and flexible. The methodology and principles can be used to identify, test and implement changes in any context or part of a health system. • Quality improvement is being used successfully, but there is still limited experience and evidence of how to apply its values and philosophy beyond the project level and embed it within national structures and systems. • Systems thinking and analysis of the political economy environment may help embed complex interventions like quality improvement and sustain their achievements DA - 2015/12// PY - 2015 SP - 9 PB - ODI UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10170.pdf Y2 - 2018/12/10/00:00:00 ER - TY - RPRT TI - What have we learned about learning? Unpacking the relationship between knowledge and organisational change in development agencies AU - Yanguas, Pablo AB - Development cooperation has spent decades wrangling over the merits, evidence, and implications of what we may term “the learning hypothesis”: the idea that increased knowledge by development organisations must logically lead to increased effectiveness in the performance of their development activities. Organisations of all stripes have built research and monitoring and evaluation (M&E) departments, adopted a multitude of knowledge management systems and tools, and tinkered with different ways to structure their organograms to stimulate knowledge sharing and learning. The topic of organisational learning is particularly significant as the global development community grapples with increasingly complex problems and the aspiration of evidence-based policymaking. This paper presents an analytical framework for interrogating “the learning hypothesis”, breaking it down into causal steps: knowledge causes learning, learning causes organisational change, change causes effectiveness. The framework focuses on the first two sub-hypotheses, mapping out the conceptual space around them by outlining potential relationships between different types of knowledge – tacit and explicit, internal and external – and between different types of learning – operational and strategic. This map provides a foundation for three key research questions: What impact has the rising knowledge agenda had on development organisations? Which factors appear to enable or inhibit organisational learning? What is the relationship between operational and strategic learning and organisational change? A review of available evaluations and studies, including two cases from former UK Department for International Development (DFID) and the World Bank, reveals that there is insufficient evidence to support the causal claim that knowledge leads to learning and thereby to organisational change in development agencies. Sources point to tacit learning prevailing while explicit knowledge management systems flounder, and external advocacy agendas appear more compelling than internal research and evaluation products. It is not entirely clear how, or indeed, whether operational and strategic learning intersect, with delivery-level lessons hardly aggregating into structural or policy shifts. Organisational change – even that aimed at enhancing learning – is rarely based on lessons learned from practice. More research is necessary to fully unpack the learning hypothesis, but what limited evidence is available disproves rather than confirms its central claim. This has significant implications for the future of learning in development agencies as advocated by thought leaders, researchers, and reformers. In particular, the latter should consider an evidence-based reassessment of the function and value-for-money of research and M&E in development practice, and a more critical examination of the politics of external advocacy efforts around innovative aid approaches like thinking and working politically, adaptive management, or results-based management. CY - Bonn DA - 2021/// PY - 2021 DP - DOI.org (Datacite) LA - en M3 - Discussion Paper PB - DIE - Deutsches Institut für Entwicklungspolitik SN - 9/2021 ST - What have we learned about learning? UR - https://www.die-gdi.de/discussion-paper/article/what-have-we-learned-about-learning-unpacking-the-relationship-between-knowledge-and-organisational-change-in-development-agencies/ Y2 - 2021/07/30/11:53:32 ER - TY - BOOK TI - What Is a Case?: Exploring the Foundations of Social Inquiry AU - Ragin, Charles AB - The concept of the case is a basic feature of social science research and yet many questions about how a case should be defined, selected, and judged are far from settled. The contributors to this volume probe the nature of the case and the ways in which different understandings of the concept affect the conduct and the results of research. The contributions demonstrate that the work of any given researcher is often characterised by some hybrid of these basic approaches, and it is important to understand that most research involves multiple definitions and uses of cases, as both specific empirical phenomena and as general theoretical categories. CY - Cambridge England ; New York, NY, USA DA - 2010/08/12/ PY - 2010 ET - 11th ed. edition SP - 254 LA - English PB - Cambridge University Press SN - 978-0-521-42188-1 ST - What Is a Case? UR - https://www.cambridge.org/gb/academic/subjects/sociology/research-methods-sociology-and-criminology/what-case-exploring-foundations-social-inquiry?format=PB&isbn=9780521421881 ER - TY - BLOG TI - What is a Theory of Change? Really. AU - Mack, Katelyn T2 - FSG AB - Recently thousands of evaluators came together in Chicago to celebrate the growing field, share exemplary practices, and to push our collective thinking on how evaluation responds to the complexity of social change efforts. DA - 2015/11/24/T16:39:52-05:00 PY - 2015 LA - en ST - What is a Theory of Change? UR - https://www.fsg.org/blog/what-theory-change-really Y2 - 2019/06/21/15:21:53 ER - TY - RPRT TI - What is democratic evaluation? AU - Picciotto, R. T2 - Evaluation Connections AB - This article from Robert Picciotto provides an overview of democratic evaluation, particularly with reference to its use in the European Union context. "Can evaluation strengthen the democratic process? Specifically can it help fill the democratic deficit (limited transparency; weak bureaucratic accountability, etc.) often ascribed to the European Union? First and foremost, evaluators operating in the European space should be committed to the ethical and democratic values of the European project and the promotion of its social inclusion and cohesion ideals. But two other sets of challenges must also be met: those that relate to evaluation in democracy and those that relate to democracy in evaluation. The former has to do with the macro positioning of evaluation processes in society. The latter has to do with the evaluation approaches and methods used in deliberative democratic decision making processes." (Picciotto, 2013) DA - 2013/06// PY - 2013 UR - http://www.czech-in.org/ees/ees-newsletter-2013-06-june-special.pdf Y2 - 2018/10/13/00:00:00 ER - TY - RPRT TI - What is impact? AU - Hearn, Simon AU - Anne, Buffardi AB - • Impact is a multi-dimensional concept. Some definitions focus on very precise understandings of impact, while others cast a much broader net. • How impact is defined and used has a significant effect on the design, management and evaluation of development programmes. • Development programmes should hold explicit conversations with different stakeholders about how impact is used and understood, in order to come to a shared understanding. • There are six dimensions of impact that may help development programmes be clearer about what they mean. DA - 2016/02// PY - 2016 UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10352.pdf Y2 - 2018/11/10/00:00:00 ER - TY - VIDEO TI - What is Outcome Harvesting? AU - Wilson-Grau, Ricardo AB - Ricardo Wilson-Grau, former Chair of the Board of Stewards of the Outcome Mapping Learning Community, introduces Outcome Harvesting, an approach for retrospective identification of outcomes based on the principles of Outcome Mapping. More details here: http://www.outcomemapping.ca/nuggets/outcome-harvesting C5 - Vimeo Video DA - 2014/09// PY - 2014 UR - https://vimeo.com/116856982 ER - TY - VIDEO TI - What is Rapid Action Learning and how was it developed? AU - Sanitation Learning Hub AB - To celebrate the publication of our latest Frontiers of Sanitation, we had a series of conversations with our colleagues and partners on our work on Rapid Action Learning so far. To download the publication in full, head to https://sanitationlearninghub.org/res... DA - 2020/09/22/ PY - 2020 DP - YouTube UR - https://www.youtube.com/watch?list=PLogOc8N6N-DtBEaMp7xNC7bwaq29U6KkM&v=cRUb8AVuyKo Y2 - 2020/10/16/13:59:51 ER - TY - RPRT TI - What is the Outcomes Star? AU - Triangle AB - The Outcomes Stars are a suite of evidence-based outcomes measurement and keywork tools, which drive an ‘enabling help’ approach to service delivery. They support a person-centred, collaborative and trauma informed approach and give service users, workers, managers and commissioners vital information about needs and progress. Since the first version was published in 2006, the unique features of the Outcomes Star contributed to its popularity and widespread use, with over 1,000 organisations with licenses to use the Star including over 500 charities and 170 local authorities CY - London DA - 2022/// PY - 2022 PB - Triangle UR - https://www.outcomesstar.org.uk/wp-content/uploads/Why-choose-the-Outcomes-Star-_-final1.pdf Y2 - 2023/04/27/13:26:17 ER - TY - JOUR TI - What Is This Thing Called a Mechanism? Findings From a Review of Realist Evaluations AU - Lemire, Sebastian AU - Kwako, Alexander AU - Nielsen, Steffen B. AU - Christie, Christina A. AU - Donaldson, Stewart I. AU - Leeuw, Frans L. T2 - New Directions for Evaluation AB - Realist evaluation has, over the past two decades, become a widely used approach in evaluation. The cornerstone of realist evaluation is to answer the question: What works, for whom, under what circumstances, and why. This is accomplished by explicating the causal mechanisms that, within a particular context, generate the outcomes of interest. Despite the central role of mechanisms in realist evaluation, systematic knowledge about how the term mechanism is conceptualized and operationalized is limited. The aim of the present chapter is to examine how mechanisms are defined and applied in realist evaluations. Informed by the findings of the review, further conceptual and practical developments for future applications of mechanisms in realist evaluation are considered. DA - 2020/// PY - 2020 DO - 10.1002/ev.20428 DP - Wiley Online Library VL - 2020 IS - 167 SP - 73 EP - 86 LA - en SN - 1534-875X ST - What Is This Thing Called a Mechanism? UR - https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20428 Y2 - 2021/02/18/14:26:33 ER - TY - BLOG TI - What, so what, now what? AU - Aston, Thomas T2 - Medium AB - Getting serious about systems change DA - 2022/03/18/ PY - 2022 LA - en UR - https://thomasmtaston.medium.com/what-so-what-now-what-4cef4d7e0281 Y2 - 2022/04/01/07:41:15 ER - TY - BLOG TI - What We Do: Getting Beyond Statistics AU - Sense Guide T2 - Sensemaker narrative monitoring AB - Our approach Understanding daily life reality with micro-narratives Narratives and more specifically micro-narratives are a fundamental and ancient way by which humans interpret their experience and make decisions. SenseMaker® provides the ability to capture and understand those narratives. Through the web or app environment the software allows the capture of pictures, recordings and writing in various combinations to reflect how the respondents are making sense of the world. Reducing cognitive bias with self signification In a patented method, the respondent then interprets their own story into a series of abstract constructs (‘signifiers’) that feel more like a game than a survey, but allow profound meaning to emerge. This interpretation adds layers of meaning rather than simply interpreting the story and provides quantitative data to detect visual patterns among stories. The patterns are linked back to the original material enabling a deeper dive into individual stories. If the researcher first look for patterns in the metadata using statistical or visual tools, he or she is less likely to be biased by content and prematurely converge on an interpretation. Evidence based insights that enables action The output of SenseMaker® is statistical data backed up by explanatory narrative. This means that advocacy is an integral part of the system. Numbers on their own appear objective but are not persuasive; anecdotes on their own may be persuasive but are not objective. SenseMaker® puts the two together and provides a powerful means of persuasion. It also enables action. Instead of saying “How do we create a culture of X?” we say “How do we create more stories like this and fewer stories like that?” Then, as actions are initiated, we see the impact in real time. DA - 2018/// PY - 2018 UR - http://senseguide.nl/en/sensemaker-narrative-monitoring/ Y2 - 2018/03/09/00:00:00 ER - TY - RPRT TI - What We Know About Traditional MERL Tech - Insights from a Scoping Review AU - Tilton, Zach AU - Harnar, Michael AU - Raftree, Linda AU - Perrin, Paul AU - Bruening, Gretchen AU - Banerji, Soham AU - Gordley, John AU - Foster, Hanna AU - Behr, Michele AB - This paper explores the peer-reviewed evidence base of “traditional” technology-enabled monitoring, evaluation, research, and learning (MERL Tech) in international development assistance from 2015 to 2019. The authors conducted a scoping review that searched seven databases, screened 3,054 reference titles and abstracts, coded 886 abstracts, and extracted and analyzed conclusions and recommendations from the full texts of 256 studies. The findings reveal the most frequently reported technologies, MERL activities, and the sub-sectors, and the geographies where those tech-enabled activities occur. Gaps in the evidence for specific technologies, MERL activities, and sectors are mapped. The data reveals which technologies are trusted more than others and reported barriers to effective MERL Tech implementation and areas that researchers suggest for further investigation. The results suggest that the evidence from peer-reviewed studies is not proportional to estimated MERL Tech activity, significant publication bias exists, and further knowledge synthesis of unindexed grey literature is needed to provide a more comprehensive and possibly accurate description of MERL Tech practice. DA - 2020/06// PY - 2020 DP - Zotero SP - 27 LA - en PB - Western Michigan University ER - TY - RPRT TI - When and how to develop an impact-oriented monitoring and evaluation system AU - Peersman, Greet AU - Rogers, Patricia AU - Guijt, Irene AU - Hearn, Simon AU - Pasanen, Tiina AU - Buffardi, Anne AB - This guidance note focuses on: • what an impact-oriented monitoring and evaluation system entails • why an organisation may want to establish such a system • when integrating an impact-orientation into an monitoring and evaluation system is most useful • what should be considered in developing the monitoring and evaluation system, or in tweaking an existing system, to become more impact-focused. The primary audience for this guidance note is internal and external monitoring and evaluation advisors involved in designing and implementing, and/or assessing monitoring and evaluation systems to include a focus on impact. It will also be useful for senior management of organisations who need to know how best to plan for a sustainable monitoring and evaluation system that supports impact assessment or to adapt an existing system to incorporate an impact perspective. DA - 2016/03// PY - 2016 UR - https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10327.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - When theory meets reality: assumptions, feasibility and implications of a complexity-informed approach AU - Buffardi, Anne AB - Over the last half century, repeated calls for adaptive learning in development suggests two things: many practitioners are working in complex situations that may benefit from flexible approaches, and such approaches can be difficult to apply in practice. • Complexity thinking can offer useful recommendations on how to take advantage of distributed capacities, joint interpretation of problems and learning through experimentation in complex development programmes. • However, these recommendations rely on underlying assumptions about relationships, power and flexibility that may not hold true in practice, particularly for programmes operating in a risk averse, results-driven environment. • This paper poses guiding questions to assess the fit and feasibility of integrating complexity informed practices into development programmes. DA - 2016/05// PY - 2016 SP - 16 PB - Methods Lab UR - https://www.odi.org/sites/odi.org.uk/files/resource-documents/10604.pdf Y2 - 2018/11/10/00:00:00 ER - TY - RPRT TI - Who counts reality? Participatory monitoring and evaluation: A literature review AU - Estrella, M. AU - Gaventa, John T2 - IDS Working Paper CY - Brighton DA - 1998/// PY - 1998 PB - IDS SN - 70 UR - https://www.ids.ac.uk/files/Wp70.pdf Y2 - 2019/09/01/22:26:39 ER - TY - BOOK TI - Who Owns Poverty? AU - Burt, Martin AB - This is the story of the one question about global poverty we never thought to ask: who owns it? It's a question with an unexpected answer, one that challenges everything that we thought we knew about what poverty is, and what we can do about it. This is a story of a powerful data-driven methodology being used in a dozen countries across 5 continents. It's a new approach that puts poor families in charge of defining and diagnosing their own unique, multidimensional poverty—who by owning the problem, own the solution. This book is for all the governments, development NGOs, charities, dreamers, thinkers, doers and leaders who are frustrated with limiting their aspirations to reducing poverty, or alleviating its effects—and the lack of progress we face in doing either. This is a book about unleashing trapped energy within poor families to do the unthinkable: eliminate global poverty once and for all. DA - 2019/09/01/ PY - 2019 DP - Amazon SP - 256 LA - English PB - Red Press Ltd SN - 978-1-912157-12-9 ER - TY - BOOK TI - Whose Reality Counts? Putting the First Last AU - Chambers, Robert AB - In this sequel to "Rural Development: Putting the last first" Robert Chambers argues that central issues in development have been overlooked, and that many past errors have flowed from domination by those with power.Development professionals now need new approaches and methods forinteracting, learning and knowing. Through analyzing experience - of past mistakes and myths, and of the continuing methodological revolution of PRA (participatory rural appraisal) - the author points towards solutions.In many countries, urban and rural people alike have shown an astonishing ability to express and analyze their local, complex and diverse realities which are often at odds with the top-down realities imposed by professionals. The author argues that personal, professional and institutional change is essential if the realities of the poor are to receive greater recognition. Self-critical awareness and changes in concepts, values, methods and behaviour must be developed to explore the new high ground of participation and empowerment."Whose Reality Counts?" presents a radical challenge to all concernedwith development, whether practitioners, researchers or policy-makers, in all organizations and disciplines, and at all levels from fieldworkers to the heads of agencies. With its thrust of putting the first last it presents a new, exciting and above all practical agenda for future development which cannot be ignored. CY - London DA - 1997/01/01/ PY - 1997 DP - Amazon ET - 2 edition SP - 320 LA - English PB - ITDG Publishing SN - 978-1-85339-386-0 ST - Whose Reality Counts? ER - TY - BLOG TI - Why are effective feedback mechanisms in cash transfers so important? AU - Jannat, Mahiratul T2 - CLARISSA AB - The CLARISSA Social Protection Intervention was set us as an innovative social policy intervention for tackling social ills, with a... DA - 2023/09/28/T14:53:21+00:00 PY - 2023 LA - en-US UR - https://clarissa.global/why-are-effective-feedback-mechanisms-in-cash-transfers-so-important/ Y2 - 2023/10/17/09:09:37 ER - TY - RPRT TI - Why Tackling Energy Governance in Developing Countries Needs a Different Approach AU - McCulloch, Neil AB - Global efforts to improve energy access and quality and to tackle climate change need a different approach to addressing poor energy governance. In 2015, leaders from around the world agreed to 17 Sustainable Development Goals (SDGs) to be achieved by 2030.1 The seventh goal (SDG7) is “Ensure access to affordable, reliable, sustainable and modern energy for all.” In the same year, the world’s leaders concluded the Paris Agreement to tackle climate change, which will require a global transition of the energy sector away from the use of fossil fuels. Yet, in many developing countries, despite growing investments in clean energy, the transition is happening much more slowly than needed to achieve SDG7 and avert damaging climate change. The central reason for this is poor energy governance. This paper outlines the size and nature of the energy challenge, with a focus on electricity. It describes the investments that are currently being made to improve the quality of power and access to electricity — and the growing evidence that investments often fail due to poor energy governance. The paper then delves more deeply into how bad governance influences the quality of and access to electricity, with specific country examples. It shows the importance of understanding how electricity fits into the political settlement of a country and how this affects the incentives of key actors in the sector. Unfortunately, donor projects designed to widen electricity access or to support reform of the power sector in developing countries often pay too little attention to the problem’s political nature; the same is true of measures to improve energy efficiency or to promote renewables. The paper outlines a new way of thinking about energy governance and shows how interventions can be better matched to the different governance challenges that they face. It concludes with recommendations for donors on how energy programs can be better designed and procured — as well as recommendations for implementors on how to improve the chances of successful implementation by adapting to the political realities of the contexts in which they operate. CY - London DA - 2021/06// PY - 2021 DP - Zotero SP - 30 LA - en PB - The Policy Practice & Chemonics ER - TY - BOOK TI - Why We Lie About Aid: Development and the Messy Politics of Change AU - Yanguas, Pablo AB - Foreign aid is about charity. International development is about technical fixes. At least that is what we, as donor publics, are constantly told. The result is a highly dysfunctional aid system which mistakes short-term results for long-term transformation and gets attacked across the political spectrum, with the right claiming we spend too much, and the left that we don't spend enough. The reality, as Yanguas argues in this highly provocative book, is that aid isn't – or at least shouldn't be – about levels of spending, nor interventions shackled to vague notions of ‘accountability’ and ‘ownership’. Instead, a different approach is possible, one that acknowledges aid as being about struggle, about taking sides, about politics. It is an approach that has been quietly applied by innovative development practitioners around the world, providing political coverage for local reformers to open up spaces for change. Drawing on a variety of convention-defying stories from a variety of countries – from Britain to the US, Sierra Leone to Honduras – Yanguas provides an eye-opening account of what we really mean when we talk about aid. DA - 2018/02/15/ PY - 2018 DP - Amazon SP - 275 LA - English PB - Zed Books SN - 978-1-78360-933-8 ST - Why We Lie About Aid ER - TY - BLOG TI - Windows on the world: The power of assumptions in uncertain times AU - Aston, Thomas T2 - Medium AB - In my last blog on theory-based Monitoring Evaluation and Learning (MEL), I explained why relationships matter, and how to assess change… DA - 2020/12/23/T09:55:16.686Z PY - 2020 LA - en ST - Windows on the world UR - https://thomasmtaston.medium.com/windows-on-the-world-the-power-of-assumptions-in-uncertain-times-b413e6f69720 Y2 - 2021/02/18/14:53:06 ER -