For projects where an impact evaluation will be undertaken, a parallel contractual or grant agreement needs to be established at the inception that includes sufficient resources for data collection and analysis. Under unusual circumstances, when a separate arrangement is infeasible, implementing partners may subcontract an impact evaluation of a project subcomponent.

When USAID decides, during the design phase of a project, to undertake a prospective Impact Evaluation that will involve a comparison, a number of preparatory steps may need to be undertaken fairly quickly. The timeline for the start-up of a prospective Impact Evaluation is driven by the requirement, in most such evaluations, for the comparison group to be selected, and baseline data on both the project’s target (treatment) group and the comparison group to be collected before delivery of the intervention that is to be evaluated begins. These requirements link to USAID’s reference to the need for parallel contractual arrangements for impact evaluations in some situations, as illustrated in the diagram below.

For many of the project-level interventions USAID determines warrant an Impact Evaluation, a timely start-up will involve completing half of the steps on the list of Impact Evaluation Process Steps shown listed below before USAID’s Implementing Partners for the project start delivering project services.

Impact Evaluation Process Steps

1. Confirm the Evaluability of the Intervention
2. Identify the Target Population, if not already determined
3. Prepare an Impact Evaluation Statement of Work (SOW), including Anticipated Method by Which Population Units will be Assigned to Groups
4. Identify and Engage an Impact Evaluation Team
5. Finalize Decisions about Method by Which Population Units will be Assigned to Groups (with the Impact Evaluation Team)
6. Prepare a Detailed Evaluation Design and Data Collection/Analysis Plan (Impact Evaluation Team)
7. Gather Baseline Data, on target population or a sample thereof (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner)
8. Implement the Method for Assigning Population Units to Treatment and Comparison Groups (Impact Evaluation Team)
9. Deliver the Intervention (USAID’s Implementing Partner)
10. Monitor Intervention Delivery and Treatment/Comparison Group Issues (Impact Evaluation Team)
11. Gather Endline Data on Groups (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner)
12. Analyze Results and Prepare Report
13. Disseminate Evaluation Findings
14. Act on Evaluation Findings to Enhance USAID and Partner Development Effectiveness

Parallel Contractual Arrangements for Some Impact Evaluations

In the figure below, Steps 3-12 above are displayed on a timeline that suggests how the timing of these steps can be linked to the timing of steps in a project implementation process. Given the work to be done before an Implementing Partner begins delivering the intervention the Impact Evaluation will examine, it is suggested that USAID issue a solicitation for an Impact evaluation team at the same time USAID issues the project-implementation solicitation. Some USAID Missions are already doing this and, as was done in solicitations related to a governance project in Nepal, they are sometimes cross-referencing these solicitations and making both the Implementing Partner and the Impact Evaluation team responsible for (a) the assignment of target population units to treatment and comparison groups and (b) the collection of baseline and endline data on key outcome measures clear in both solicitations.

Additional Considerations for Impact Evaluations

While many of the steps listed in the Impact Evaluation Process above parallel steps taken in conducting Performance Evaluations, several of these steps warrant special attention when an Impact Evaluation is being planned. These key features, which affect decisions about the extent of intervention monitoring and study groups that will be needed, and thus evaluation cost, include, but are not necessarily limited to:

• The Evaluability of the Intervention
• Unit of Analysis and Unit of Assignment
• Treatment and Comparison Group Size
• Assignment Method
• Protection of Human Subjects
• Attrition, Spillovers and Crossovers
• Implementation Fidelity

These Impact Evaluation features are discussed in greater detail in a short briefing paper entitled Key Features of Impact Evaluations that Warrant Consideration at the Planning Stage.

For USAID Missions that are considering undertaking an Impact Evaluation to examine causal linkages in a trade facilitation or other type of trade capacity building project, it may be worth noting that in 2008-2009, the OECD commissioned a review of these types of projects to determine whether or not existing approaches for conducting impact evaluations (that include an examination of the counterfactual) were adequate for trade projects or whether new tools were needed. USAID was among the donor organizations that were asked to review both the Statement of Work and the product of this research. The conclusion of this OECD study, which was not issued as an OECD publication was that the experimental and quasi-experimental designs already being used on a prospective as well as a retrospective basis to examine a wide range of interventions in developing countries were both adequate and appropriate for carrying out these types of impact evaluations in the aid for trade arena.

Additional guidance on planning and conducting Impact Evaluations is available in USAID’s TIPS on Rigorous Impact Evaluation, in the World Bank volume, Impact Evaluation in Practice, and on the websites of a number of organizations that conduct and provide guidance material on Impact Evaluations, including the J-PAL, IPA, 3ie, NONIE and DIME, which are described on the kit’s page on Locating Evaluations and listed under “useful links” this page. Two NONIE publications featured on this page are particularly useful as is USAID’s Microlinks web guide, What are the Steps in Implementing an Impact Assessment?