Causal inferences on the effectiveness of complex social programs: Nav igating assumptions, sources of complexity and evaluation design chall enges - Chatterji, Madhabi (Evaluation and Program Planning, 2016)
This paper explores avenues for navigating evaluation design challenge s posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on t he intervention’s effectiveness. A definition is provided of a CSP dra wing on examples from different fields, and an evaluation case is anal yzed in depth to derive seven (7) major sources of complexity that typ ify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, a lternative methodological strategies are discussed to navigate assumpt ions and counter the design challenges posed by the complex configurat ions and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems -informed logic modeling; and use of extended term, mixed methods (ETM M) approaches with exploratory and confirmatory phases of the evaluati on. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumpti on evaluation, question-framing, and selection of appropriate methodol ogical strategies, a multiphase evaluation design is recommended.
Keywords: Causal inferences; Complex social programs; Experimental designs; Impact evaluations; Mixed methods