Recently, a fascinating blog post made its way across social media, describing the unequal distribution of knowledge production around the world. The blog has a map that distorts each country based on its published research output (by impact factor), and the global north was disproportionately big, to put it mildly.
Spend much time reading up on “pop” evaluation, and it won’t take long to hear of the field’s most recognized celebrities: the randomistas, or proponents of the use of randomized control trials (RCTs) to solve some of development’s difficult questions. While there are plenty of critiques of experimental methods that have arisen in the past decade (many of which have their own counter-argument), I have found one downfall to be particularly compelling — they’re just not always appropriate.
I’ve been catching up on quite a few webinars lately, and I can’t help but notice the common emphasis they have been putting on the importance of children and youth in development (granted, I assumed that the World Bank’s live stream with Sesame Street would have carried such messages). Whether it has to do with WASH, family planning, or violence prevention — it’s certain that development must focus on the youngest generation to ensure that their efforts are not wasted in twenty years. And this focus has clear implications for the field of evaluation.
After Mexico introduced Progresa (now Oportunidades) and Brazil launched its Bolsa Escola (now Bolsa Familia) in the mid-nineties, development gurus were quick to hail conditional cash transfers (CCTs) as the poverty’s new silver bullet. The case of CCTs is particularly interesting to evaluation researchers, as it also ushered in an era of state-led evaluations, and blessed evaluators with the rare gift of evaluation forethought: most CCTs were rolled out randomly, so as to allow researchers to form an easy counterfactual (those who hadn’t received the program yet). Cue the flood of rigorous evaluations on a fascinating topic.
So, it has been two decades. What ever happened to the CCTs of Latin America? Click here for more.
Program evaluation, and particularly impact evaluation, is an exciting and growing area within the field of international development. Organizations are continually gathering new evidence that explains not only what works to alleviate poverty, but why it works. Little by little, evaluation is helping us make better decisions that actually make sense in the context of poverty.
This blog aims to make evaluation — its methods, trends, and results — accessible to anyone who wants to learn how to make effective change. New to the blog? Start here.
If you have just started reading up on development evaluation, it won’t take long to notice that many journal articles or reports start by outlining ‘The Evaluation Problem.’ In short, and without the mathematical notation, the evaluation problem is the problem of the non-existent counterfactual: that is, you don’t know what would have happened without the program, because the program did happen. But, in order to know if the program had any impact, it’s not enough just to do a before-and-after comparison. Things naturally improve (or worsen) over time. The true measure of impact is to know if things are different now than they would have been, had the program (often called an intervention) never existed.
And here’s the beauty of evaluation: done well, it allows us to jump to another dimension by simulating a counterfactual. If we have enough data, we can make a very educated guess about what would be happening in a world without our program, using a variety of methods.
Sound interesting? It absolutely is. Here’s how you can join in:
If you’re a development practitioner, use evaluations to find out what works before you start a new program. If you think your current program is successful, but need proof, learn about what data you should be collecting.
If you are a donor or policy-maker, find a Systematic Review on the type of interventions you support, and make sure they are successful.
For active citizens looking to improve your community, look up what has been successful in your area and context. Know of an NGO that is active in your community? Ask to see their evaluations and to participate in them.
Are you a student or development worker hoping to get into the field of evaluation? Read up! Find out where there are gaps in evaluation research and be the first to fill them.