The American organization Mathematica Policy Research (MPR) conducted a research study of a five year grant program condcucted in Armenia sponsored by the United States Millennium Challenge Corporation. The MPR researched “Water to Market” training courses organized for some 50,000 farmers in Armenia from 2007-2011. $16.6 million U.S. was spent from the program’s budget for this aim. Kenneth N. Fortson, director of the working group and principal investigator, sent the following letter to Hetq. 

''Dear Editor:

I write to you regarding Marine Madatyan’s article, “U.S. Policy Organization Calls $16.6 Million Farmer Training Program in Armenia a ‘Bust’; Ministry Disagrees” posted on hetq.am on January 13, 2014. I am the corresponding author of the Mathematica Policy Research(http://www.mathematica-mpr.com/publications/PDFs/international/Armenia_wp.pdf ) working paper cited in Madatyan’s article, and I am also the principal investigator of the research study onwhich the working paper is based.

Madatyan’s article mischaracterizes our research in several ways. The biggest mischaracterizations are that the article’s headline erroneously quotes us as saying the Water-to-Market training program was a “bust,” and the article implies that Vistaa scandalously earned great profits anyway.

Our study found that the WtM training was well-implemented with fidelity to the program’s design despite the plan to train over 50,000 farmers, an ambitious goal that Vistaa still achieved. I personally observed several training sessions and was impressed with the trainers’ breadth and depth of agricultural knowledge and their clear explanations of different techniques. Unfortunately, WtM training did not yield the desired longer-term effects despite being thoughtfully designed and implemented well. However, all parties learned a tremendous amount from these findings. Most importantly, we learned that the topics of training were probably too ambitious for the program participants, most of whom were small-scale farmers.

At the outset of the program, the Millennium Challenge Corporation (MCC), MCA-Armenia, and Vistaa were optimistic that WtM training would fill crucial gaps in Armenian farmers’ knowledge, spurring them to improve agricultural income. However, all parties also recognized that we did not know for sure that the program would succeed, because this was the first time that a program of this scope and magnitude was designed and implemented in Armenia.

To evaluate the design, implementation, and impact of WtM training, MCC contracted with Mathematica Policy Research to design and execute a rigorous impact evaluation so that we could learn whether the WtM training was effective and why. MCA-Armenia and Vistaa shared this objective to learn from the evaluation and were excellent partners. Without their cooperation, we would have learned nothing from the evaluation. Indeed, it was a great triumph to successfully implement a randomized controlled trial—considered the gold standard of evaluation designs—to evaluate this program. To our knowledge, this is the first randomized controlled trial of a social policy program in Armenia.

Training programs similar to WtM are implemented throughout the world, and these findings will influence the way such programs are designed in the future. We have already seen MCC adapt program designs for other countries in which it funds farmer training. Although we found that the impact of this particular training program was smaller than was anticipated, we should still seek other ways (and perhaps redesigned agricultural training programs) to improve wellbeing in rural Armenia. Whatever new approaches are attempted, however, should be subjected to similarly rigorous evaluations to be sure that we learn what approaches are truly effective.

Besides these broader issues, Madatyan’s article also includes smaller factual inaccuracies: First, the article states that “it is highly likely that the $16.6 million USD spent on the training program was profit this company earned.” This amount does not represent company profit; rather, it represents the revenue the company received. Much of that revenue was spent to pay salaries of the many trainers and other staff to develop training curricula, organize the training sessions, conduct training, and prepare printed materials; in addition, they had to pay for other program costs such as facilities, transportation, and so on. The total value of the contract also includes other tasks besides training; we estimate that $14.3 of the contract’s value was for training. We also note that the WtM program was implemented by a partnership between ACDI/VOCA and Vistaa, not just Vistaa alone.

Second, one of the direct quotes (in the English version but not the Armenian version, as I understand) omits the word “three.” The correct quote should read: “We find no evidence that, three years after it began, the training program had impacts on household income or consumption during the study period, nor do we find evidence that households adopted new farming practices or cultivated new crops that would potentially lead to longer-term impacts on income and consumption.”

I appreciate your attention in clarifying the mischaracterizations in the article. I would be happy to correspond with you further about any of these issues should you have questions.

Sincerely,

Kenneth N. Fortson, Ph.D.