Impact evaluations are an important tool for learning, in a precise way, about the extent and intensity of project effects as well as about the degree to which those effects can be said to have been caused by a specific intervention. USAID requires that impact evaluations be conducted, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. Government foreign assistance or other funding sources (i.e., a pilot intervention). Impact evaluations should be integrated into the design of a project or activity.

The term "impact evaluation" involves a specialized meaning of the word impact. In common usage, impact means everything from high level results, in a very general sense, to long term outcomes from a project, to people-level outcomes. When joined with the term evaluation, however, "impact evaluation" is defined by most international development donors as a type of evaluation that involves a structured test of one or more hypotheses underlying a program or project intervention that involves a comparison between those affected by the intervention and a comparable group or area that was not affected by the intervention.

USAID introduced “impact evaluation” as a rigorous learning method in 2011, and has since initiated this type of evaluation in most of the sectors in which it works. USAID’s Technical Note: Impact Evaluation provides a detailed discussion of this type of evaluation and the steps involved in undertaking one. USAID’s Evaluation Toolkit also includes information updates on impact evaluations.

Increasingly, impact evaluations that are being conducted in developing country environments are demonstrating that hypotheses testing that have advanced the development of effective medicines, agricultural practices can be cost-effectively applied to a much wider range of development assistance and partner government program interventions. The effect that this type of evaluation attributes to an intervention is the difference between these groups after exposure to the intervention. In some cases, differences between groups will be fully explained by an intervention; in others, both groups may improve as a function of factors to which everyone was exposed, but the group that received the project intervention will have improved more than the group that did not receive that intervention. Impact evaluation techniques are also being used to detect differences in the effects produced by two or more alternative approaches to solving a given development problem.

Key Considerations when Making an Impact Evaluation Decision

In its simplest form, an impact evaluation is an experiment in which one group or area receives a particular type of assistance (treatment group), while the group to which it will be compared other does not (comparison group). The status of each group on one or more variables of interest is measured both before and after assistance is provided. This provides USAID with the information needed to determine whether the assisted group is “better off” than the unassisted group at the end of the study period.

Concluding that USAID’s assistance explains whatever difference is observed depends upon several things that need to be considered when USAID makes its initial decision about whether to undertake an impact evaluation.

• How similar the two groups are before assistance is provided? This is where the term “randomized assignment” is important. When units are assigned to groups that will and will not receive assistance using a lottery or similar mechanism that depends on chance alone, they are considered to be equivalent at the start, with all of their known and unknown characteristics being equally distributed. These are called “experimental designs”. With any other method for setting up groups, including matching techniques, the groups are considered to be non-equivalent at the start, and the design is called “quasi-experimental” as the possibility of selection bias as this stage remains a threat to the validity of conclusions reached. USAID’s evaluation policy favors experimental designs that use “randomized assignment” to establish groups, where that is possible.

  • Take away: If USAID has enough control over the project design process for a project or activity where an impact evaluation is desired, it is always better to postpone site selection or the choice of districts, schools, etc., until the impact evaluation team is assembled and a deliberate decision can be made about randomizing units to the evaluation’s treatment and comparison groups.

• The scale of the experiment, or pilot project. If the area or group that receives assistance is quite small, it may not be possible to conduct the statistical tests that would determine whether the status of the evaluation’s treatment and comparison group area actually different on key variables after assistance is delivered.

  • Take away: If the pilot project or innovative intervention in a larger project the Mission is considering for an impact evaluation is very small, it may be worth checking with an Agency evaluation specialist to determine whether a different type of evaluation should be selected and a written explanation provided to management about why an impact evaluation will not be attempted in this instance, as is allowed in the ADS 201 statement of USAID’s evaluation requirements.

• Stability of the Intervention. Impact evaluations work best if the intervention to be examined is stable, meaning that what is being delivered will not be altered. In medical trials, dosage and frequency are often held constant over the life of an evaluation, and in agricultural trials, the same is done with planting schedules and the amount of fertilizer. When deciding whether to undertake an impact evaluation in another field the same kind of question about intervention stability needs to be asked. If the intervention is not stable, conclusions reached at the end may mislead decision-makers.

  • Take away: Before committing to an impact evaluation of a pilot project or innovative intervention, make sure that the intervention to be examined is clearly defined and will be delivered without change over the life of the evaluation. If this is not the case, other types of studies, such as operations or action research, that anticipate modifications over the life of a study, maybe better suited.

For additional information on impact evaluations see the website’s page on Impact Evaluation Designs and its Impact Evaluation Decision Tree diagram.