First, a public service announcement: evaluation methods should be chosen according to the data and context of the intervention, and not the other way around. That said, coarsened exact matching (CEM) might be a perfect fit for your next impact evaluation, and here’s why:
I. How it works
CEM is a quasi-experimental method you can use when the unit of evaluation is not randomly assigned to the treatment (that is, the treatment and control groups are different from each other on the factors that influence your outcome). Imagine you’re in a room with a giant Rubik’s cube and a bucket full of ping pong balls, each representing an observation from your data.
Let’s say we’re working on an intervention for low-income women which attempts to raise their level of income. The first ping pong ball is Sara, with a set of characteristics: she’s Latina, she is a university graduate, and she’s in a caregiving field of work.
You notice that each colored axis of the Rubik’s cube represents these characteristics. What we would call the X axis represents race/ethnicity, with the border of each square marking the new category. We’ve decided our observations contain only three categories: Latina (white-hispanic), White (white, non-hispanic), and Black. The Y axis represents education. Three categories here are: Did not finish high school, high school graduate, and university graduate. The Z axis (running vertically) represents categories of jobs: caregiving, administrative, or manual labor.
When we throw the ping pong ball for Sara at the Rubik’s cube, it automatically sorts itself into the corresponding mini-cube (called a stratum in the literature). Sara would fit in the mini-cube which is at the bottom/back/left.
Now, some of these ping pong balls are from treatment group (let’s make them green), and some are from control (purple). Once you throw all of them at the Rubik’s cube and they are sorted, some of the green will be sharing mini-cubes with purples which match them on all of the specified characteristics. In an ideal scenario, all treatment observations would share a mini-cube with a corresponding control, and any controls that don’t match a treatment would just drop out of the Rubik’s cube. From there, we can conduct a simple t-test which compares the average income of the green ping pong balls with the purple ones inside the Rubik’s cube, since their characteristics are matched on all other relevant factors.
Obviously, this is a simplified scenario. Indicators usually have more than 3 divisions. This is fine – the Rubik’s cube can be longer than it is wide or tall. Some of them aren’t easy splits — income as an independent variable, for example, would be split into income ranges, much like we would do if we created a histogram. Usually outcomes are influenced by many more than 3 variables, as well. Also fine: the matrix doesn’t have to be a cube. It can be a higher dimensional figure. As long as we maintain transparency in justifying the divisions of the variables and include all relevant variables, the matching will work.
Sometimes our treatment variables have more than one match. Sometimes they have none. While trickier, these issues are all easily addressed (see the paper linked above and below) by weighting the control observations and specifying the treatment effect as a local sample average treatment effect on the treated (local SATT), respectively. As is the case with all evaluations, the evaluator need only be transparent about the decisions made to accommodate each challenge the data presents.
II. Why it’s good
Why is CEM a good method to consider? It has advantages for both evaluators who are concerned both about methodological validity, and accessibility to non-evaluators.
First, the method is part of a category of matching methods which avoids model dependence. Compared to propensity score matching, for example, we don’t risk continually running logit regressions with different functional forms until we achieve balance, and then justifying the model we used to predict treatment. Instead, we identify through literature the determinants of our outcome in question, and then exactly match without using a model. And of course, CEM maintains all the usual benefits of matching methods: you don’t need to have a treatment variable which is randomly assigned, and you don’t (necessarily) need baseline data.
Another highly attractive benefit is its simplicity. It’s not hard to explain to non-evaluators how exact matching works. Final reports will be accessible to stakeholders, donors, and any others who earlier might have been too intimidated to trudge through the methods section. And before you worry your economist self about using a method that seems too unimpressive, you can find peace in the quote often attributed to Da Vinci:
“Simplicity is the ultimate sophistication.”
III. Need more?
Ready to try it with R, SAS, or STATA? Look here for the packages.