AI Evaluation in the Social Sector - A living playbook for evaluating AI products in the social sector - Fund, Agency
A comprehensive framework for evaluating AI systems in development con texts. Features tools, case studies, and methodologies for responsible AI assessment. Comprehensive guidance, practical tools, and real-worl d case studies to help you implement effective AI evaluation in develo pment contexts. Continuous evaluation is a critical tool for AI produc t developers. Generative AI (GenAI) is a relatively new technology, an d as a result product development today is more of an art than a scien ce. By rapidly iterating through different AI models, architectures, p rompts, and knowledge bases, GenAI developers can steadily improve a p roduct or workflow. So it is no surprise that AI evaluation tools have gained significant attention from software companies, investors, and academics alike. But what should evaluations look like in the social s ector? One of the most compelling use cases for AI in the social secto r is its potential to cost-effectively deliver personalized decision-m aking support for millions of people. Done right, this technology can help individuals exercise greater agency over their lives and improve outcomes in meaningful, measurable ways. To bring clarity and structur e to the evaluation of AI services in the social sector – for both fun ders and service delivery organizations – we have introduced a four-le vel framework. We first shared the framework at a technical conference in Bangalore in March 2025. We later wrote about it in a blog post co -authored with the Center for Global Development (CGD) and J-PAL. This framework lays out four core questions that can guide AI evaluations in human development contexts: Introduction 1 Model evaluation: Does t he AI model produce the desired responses? 2 Product evaluation: Does the product facilitate meaningful interactions? 3 User evaluation: Doe s the product positively support users’ thoughts, feelings, and action s? 4 Impact evaluation: Does access to the product improve human devel opment outcomes?