Michael Clemens and Gabriel Demombynes, both friends and colleagues, sprang a provocative paper on me while on vacation. The basic argument: the Millennium Villages could and should be evaluated. No excuses. Sachs and Macarthur and others at the MV project respond here. And the debate would not be complete without a cannonade from Aid Watch, who suggest that the MV folks are moving their goalposts.
Let me stake out a different position. I've blogged about the Millennium Villages before, on what an evaluation could look like, but also why a rigorous evaluation might be infeasible. While guest blogging, Julian Jamison commented that it might actually be detrimental to a progressive agenda to evaluate too much.
Now, Michael and Gabriel write a good paper, and do a much more thorough job than I ever did, and more or less convince me that a rigorous evaluation is possible. The evaluation they propose, however, is one that measures the aggregate effect of the villages — e.g. "do the MVs reduce poverty?".
This should probably get done. But for me this is not really the interesting question.
First, if the Villages "work", why is that? Maybe it's the combination of schools plus clean water plus agricultural productivity gains plus clinics. Or maybe it's the fact that the local and district and state governments all know that the world is watching this one cluster of villages. So the state basically works the way it's supposed to: services get delivered, they don't send the idiot ag extension officers over there, and someone makes sure the police chief is not a thug.
If the Villages work, perhaps it's not the aid or the money or the technical expertise or all the other things that Westerners are in a position to give. Maybe it's because there's finally some accountability. And while we outsiders can do a decent job of making the government accountable for 10 villages, there's not much we can do for the other 9,990 in the country. That's up to the citizens themselves.
That to me is the interesting question. And to answer it, the alternative intervention we would need is not a "no Millennium Village intervention" but a "institutions and accountability" intervention. I call it the Villennium Millage project, and I am working to interest the Liberian government on something this as we speak. (I need all the luck you can wish me.)
The second interesting question is whether the whole is greater than the sum of its parts. This is (as I understand it) a fundamental premise of the MVs and (if I read him correctly) the fundamental development theory to which Jeff Sachs subscribes: that poverty is in part a bad equilibrium and needs a big push. Without coordinated action (schools plus clean water plus agricultural productivity gains plus clinics) you may as well be handing our right shoes to some villages and left shoes in others.
I don't subscribe to the strong version of that theory (I view poverty alleviation as something that works on the margin) but there are almost certainly positive interactions between aid interventions. That would be a useful thing to test and understand.
We could get halfway if we could at least figure out whether the Millennium Villages have a level effect (you give more X, and you get more income), or if there's a growth effect (you give more X, and you set a community off on a virtuous circle). Unfortunately I don't think there's enough statistical power (i.e. enough villages) to distinguish between level and growth effects.
Okay, so say we can't evaluate all these more interesting questions. The "do the Millennium Villages work at all?" question is still an important one, right? Yes, but as economists, we should be thinking in terms of opportunity cost. Research opportunities and funds are not limitless in the real world. What's the first and most important king of learning and research we'd like to see? I'm not convinced the answer is a randomized control trial.
The Millemium Promise CEO, John Macarthur, is also a friend of mine (we both worked for Sachs about ten years ago) and I still remember something he said to me: the learning that comes from the Millennium Villages is learning by doing. By trying to make these programs work together, and getting 10 or 15 villages right, the state learns how to do it better (mainly by screwing up along the way). It also shows to the rest of the country that it's doable. No one questions that delivery of public services like roads and schools and agricultural extension and clinics are important. That is a no brainer. The point is to get the government moving and learning and practicing and showing off, in the hopes that it will spread.
By this argument, the kind of research we need to get built around the Millennium Villages are process consultants (to learn how to deliver services better) or people who think about how policy and institutional change actually happens, so the lessons and examples can spread beyong the example villages. We also need lots of close observation so that we can see why some interventions work well together, and why some don't.
Speaking as someone who does rigorous program evaluation for a living, the idea that a rigorous program evaluation is the most useful evidence for most aid projects is kind of absurd. There are other paths to knowledge, especially practical knowledge, and most especially policy change.
When was the last time you met a prominent academic who said: "I study how good ideas actually get transformed into action."? It doesn't happen. The political economy of policy change is a field populated by probably six people we have never heard of. That is a sad state of affairs.
OK. Back to the MV project. They make a similar argument for process knowledge in their response to Michael and Gabriel, but they don't get points for quality of sales pitch. Personally I would be convinced if the MV hit me with a barrage of amazing things they learned about how to do aid better, or examples of how good practices are spreading. These would ideally get peer reviewed, and would have a degree of independence and impartiality, as a check on the enthusiastic nature of the anecdote. (Apologies to the MV folks if this is out there.) I would still like to see a rigorous evaluation, if only to keep those gushing anecdotes in check, but it's not my first concern.
Personally, I think this is a great debate that gets to the root of research in development, and where smart policy people should be focusing their energies (especially as the randomized control trial fad starts to fade).
Sadly, rumor has it that the open debate between Sachs and Clemens/Demombynes at the World Bank has been cancelled. (Aid Watch has that scoop.) If so, that is a shame. I would have liked to see that debate. I'd like to know why it was cancelled more.
In my experience, organizations like the Bank and the MVs operate with a "bad stink minimization" strategy, so calling attention to the quiet cancelling of an important public debate could have good effect. Pass it on, and hope for some courage and conviction to appear on all sides.
