At the earliest step in the project design process, the development of a Concept Paper, a USAID design team is expected to identify 1-2 central questions to be evaluated over the course of project execution, considering those identified in the CDCS.

USAID's call for evaluation questions in a project Concept Paper iterates a parallel requirement for a CDCS. Both of these requirements reflect USAID's evaluation policy commitment to improving the degree to which evaluation is an integral element of the program cycle.

When developing project evaluation questions for inclusion in a Concept Paper, project Design teams may find it useful to use the Project Evaluation Questions Worksheet included in this kit section to identify questions by levels in a project's Logical Framework on or other issues, as illustrated below.

When developing questions that might be asked in project evaluations, it is important to remember that USAID undertakes two types of evaluations: performance evaluations and impact evaluations, as defined below.

• Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. Any activity within a project involving untested hypotheses or demonstrating new approaches that are anticipated to be expanded in scale or scope through US Government foreign assistance or other funding sources will, if feasible, undergo an impact evaluation.
• Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. Operating units are required to conduct at least one performance evaluation of each large project it implements; where large is defined as larger than the average for the unit's portfolio.

For performance evaluations, which are carried out at a single point in time, rather than over the course of an intervention, the timing of a proposed evaluation has implications for the types of questions to be asked. For example:

• In mid-project evaluations process questions may help identify opportunities for improving performance, as may questions about the realism of the project design, adherence to the project's budget and schedule, and early indications about the achievement of expected results and the validity of project assumptions.
• Questions about project results, including any differential impact on men and women, or their effects on youth or other target populations are likely to be answered more fully in an end-of-project evaluation, as are questions about a project's actual cost-effectiveness compared to project design stage expectations.
• Questions about the sustainability of project services and benefits can be asked at any stage, but must usually be adjusted to take evaluation timing into account. Thus, for example, in a mid-term evaluation a question about the existence of a sustainability plan and early action on that plan might be appropriate. An end-of-project evaluation could address questions about how effective a sustainability plan seems to be and early evidence concerning the likely continuation of project services and benefits after project funding ends. Only an ex-post evaluation, however, can provide empirical data about whether a project's services and benefits were sustained.