I just finished an enlightening online session of the American Evaluation Association’s annual conference. The session covered the theme of culturally responsive evaluation, which to my knowledge is a rarely addressed topic in the field of impact evaluation.
The specific breakout session with was live-streamed was led by Dr. Melvin Hall of Northern Arizona University. While the session seemed largely focused on qualitative methods, I particularly appreciated Dr. Hall’s explanation some aspects of being a culturally-responsive evaluator with imagery of a 3D graph using X, Y, and Z axes. These axes represent different aspects of an evaluator’s position as a person from a separate culture. The idea is to pinpoint your position as an evaluator on this graph with respect to its ‘origin,’ or the population being researched. How far away from the origin are you, and how can you explicitly identify your variation from the origin in your research?
This is not a particularly new concept within the scope of ethnography, but broadening it and applying it to my role as a mostly quantitative impact evaluator has been a challenge. But it has made me reflect on some interesting cases that I, as a gringa in my current country of residence (Guatemala), have encountered while doing fieldwork. Below is one such case:
After having carefully calculated the stratified sampling strategy we were to use when choosing students for a literacy test evaluating the impact of an educational intervention, we implemented the sampling in the field. Students were chosen randomly but with regard to their gender and grade, and these students’ names were carefully written down and were to be tested over a period of 2 to 3 days.
The first day, we had randomly selected a little boy in 3rd grade to take the test. However, due to time constraints, we planned to test the boy on our 2nd day in the field. But when we returned the next day in search of this little boy, he was nowhere to be found. We asked his teacher about his absence, to which she responded, “He went north.”
At the time, as a new evaluator in a new context, I didn’t understand what this meant. My team carefully explained (while grinning at my naiveté, of course), that the little boy and his family had left to start the multi-week journey to the border of the U.S., hoping to cross. That very day, the family had finalized their US$15,000 loan with a local coyote, desperately putting up their home, land, and belongings as collateral, so they could receive help leaving Guatemala, crossing Mexico, and arriving at a potentially unguarded portion of the U.S. border where they might be able cross.
My first thought was: sample attrition before we even start the study?!
And then my second thought was:
That little boy and his family have an entirely different lived experience than I do. The feeling of being in such close proximity to a family’s decision to ‘go north,’ and the similarly difficult decisions that the community’s families regularly make… it all made me realize how foreign those day-to-day experiences are to me.
In today’s session Dr. Hall offered a word of caution to those wanting to contract an evaluator. He said, “An evaluator who comes to you and says ‘I am an expert at developmental evaluation’ or ’empowerment evaluation’: be wary. Pre-ordinate approaches to evaluation are akin to pre-ordinate methodology. They’ll use it whether it fits the situation or not. Culturally sensitive evaluation has to be co-constructed.”
Had we planned this literacy-intervention’s evaluation with the community, and pitched our idea of a long-term evaluation with required following up with the exact same representative sample after 6 months, the community probably would have warned us. We’re a highly migratory community, they might have told us, both within our country (for agricultural labor) and out (for economic migration). The likelihood is slim that your longitudinal study will provide you with the answers you need.
Well, now I know.
More reflections on culture and evaluations are sure to follow in this blog, but for an immediate fix, here’s a great list of resources to learn more.