Students were assessed prior to the start of the intervention in April 2018 and again at the end of the intervention in July 2018. Supporting USAID Impact Evaluation in Honduras. To demonstrate the power of long-term evaluation, the team plans to conduct evaluations that assess the impacts of an aid intervention years after the project is completed. In principle, the group or area that receives an intervention should be equivalent to the group or area that does not. Missions initially identify which of their evaluations during a CDCS period will be impact evaluations and which will be performance evaluations in their PMP. Make a general inquiry or suggest an improvement. Knowledge and experience in performance and impact evaluation designs, methodologies and practical applications, including baseline data collection methodologies required. US Agency for International Development (USAID). USAID/Ghana contracted Social Impact, Inc. to conduct an impact evaluation of USAID’s Ghana Strengthening Accountability Mechanisms (GSAM) program, which aims to increase accountability of local District Assemblies in Ghana. By: Ruth Levine, Deputy Assistant Administrator, Bureau of Policy, Planning and Learning In a major address today, Dr. Shah will announce USAID’s new evaluation policy, evidence of the renewed emphasis the Agency is placing on evaluation, measuring and documenting program achievements and shortcomings, and generating data on what works to drive decision-making. USAID/Aprender a Ler Impact Evaluation: Final Report i Some USAID Missions are already doing this and, as was done in solicitations related to a governance project in Nepal, they are sometimes cross-referencing these solicitations and making both the Implementing Partner and the Impact Evaluation team responsible for (a) the assignment of target population units to treatment and comparison groups and (b) the collection of baseline and endline data on key outcome measures clear in both solicitations. For USAID Missions that are considering undertaking an Impact Evaluation to examine causal linkages in a trade facilitation or other type of trade capacity building project, it may be worth noting that in 2008-2009, the OECD commissioned a review of these types of projects to determine whether or not existing approaches for conducting impact evaluations (that include an examination of the counterfactual) … For USAID Missions that are considering undertaking an Impact Evaluation to examine causal linkages in a trade facilitation or other type of trade capacity building project, it may be worth noting that in 2008-2009, the OECD commissioned a review of these types of projects to determine whether or not existing approaches for conducting impact evaluations (that include an examination of the counterfactual) were adequate for trade projects or whether new tools were needed. In conjunction with impact evaluations Microlinks has conducted, the project has published an Impact Evaluation Primer Series that includes Common Problems in Impact Assessment Research and six other volumes. The impact evaluation is aimed at addressing both the Accountability and Learning objectives as laid out in the USAID Evaluation Policy, and will be used by USAID YYC staff in order to inform future programming in YYC in Kenya and beyond. Impact evaluations can be divided into two categories: prospective and retrospective. Evaluations at USAID include impact evaluations and performance evaluations. Find out about career opportunities at USAID. Both experimental and quasi-experimental designs can produce credible impact evaluation findings, but there is a difference, and their classifications signal what that difference is. For projects where an impact evaluation will be undertaken, a parallel contractual or grant agreement needs to be established at the inception that includes sufficient resources for data collection and analysis. There are many design options for impact evaluations depending on a variety of factors (see USAID Technical Note and OECD Principles for Impact Evaluation for helpful decision trees for design). We provide monitoring, evaluation, strategic planning, and capacity building services to advance development effectiveness. USAID/UGANDA PERFORMANCE AND IMPACT EVALUATION FOR LITERACY ACHIEVEMENT AND RETENTION ACTIVITY (LARA) BASELINE DATA REPORT June 30, 2017; Final December 15, 2017 . Retrospective impact evaluations assess program impact after the program has been implemented, generating treatment and comparison groups ex-post. The USAID Water Team, through its Water Communications and Knowledge Management project (Water CKM), conducted a series of six independent ex-post evaluations to better understand the long-term impact and sustainability of its WASH-related interventions. BETTER PROJECTS THROUGH IMPROVED 2013 Report(link is external) 3. USAID is committed to building the evidence base on what works through impact evaluations. USAID Evaluation Policy encourages Missions to undertake prospective impact evaluations that involve the identification of a comparison group or area, and the collection of baseline data, prior to the initiation of the project intervention. The conclusion of this OECD study, which was not issued as an OECD publication was that the experimental and quasi-experimental designs already being used on a prospective as well as a retrospective basis to examine a wide range of interventions in developing countries were both adequate and appropriate for carrying out these types of impact evaluations in the aid for trade arena. This type of design may also be feasible when USAID projects introduce an intervention on a phased basis. When USAID decides, during the design phase of a project, to undertake a prospective Impact Evaluation that will involve a comparison, a number of preparatory steps may need to be undertaken fairly quickly. Along this continuum, the preference in USAID's Evaluation Policy is clear: For impact evaluations, experimental methods generate the strongest evidence. 2013 Report(link is external) 3. The school lunch, supported by USAID, was a high […] The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. In the figure below, Steps 3-12 above are displayed on a timeline that suggests how the timing of these steps can be linked to the timing of steps in a project implementation process. Through the "USAID Learns" activity, Social Impact is supporting USAID/Vietnam to streamline monitoring, evaluation, and learning (MEL) capacity across Mission teams, increase hands-on capacity building for local organizations and entities, implement innovative approaches, increase effective collaboration among stakeholders, and develop a highly utilized M&E system. Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of strategies, projects, and activities conducted as a basis for judgments to improve effectiveness, and timed to inform decisions about current and future programming. U.S. Agency for International Development, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Running Randomized Evaluations: A Practical Guide, decision to undertake and impact evaluation, Experimental and Quasi Experimental Design for Generalized Causal Inference. 1. Office of Trade and Regulatory Reform "Managing Impact Evaluations" (Part II) of Impact Evaluations and Development. Other impact evaluation designs, are intended for use when populations that are known to be different (such as those living above and below a poverty line) are to be compared after the poorer group receives an intervention that is designed to improve their circumstances. ADS 201 requires that each Mission and Washington OU must conduct an impact evaluation, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. Government foreign assistance or other funding sources (i.e., a pilot intervention). The reports below summarize evaluation requirements and practices at USAID before and after the Evaluation Policy, major accomplishments during the first five years of implementation, and priority activities to support policy implementation moving forward. When considering which type of evaluation design to choose, project design teams may find that using an Evaluation Design Decision Tree helps them work their way to the most appropriate option. Baseline data are collected prior to program implementation for both treatment and comparison groups. Alternative methods should be utilized only when random assignment strategies are infeasible. Bureau of Economic Growth, Education, and Environment U.S. Agency for International Development, Randomized impact evaluation of Afghanistan's National Solidarity Programme (NSP) Hypotheses and Methodology, What are the Steps in Implementing an Impact Assessment, Project Evaluation Plan Summary Template >>, Confirm the Evaluability of the Intervention, Identify the Target Population, if not already determined, Prepare an Impact Evaluation Statement of Work (SOW), including Anticipated Method by Which Population Units will be Assigned to Groups, Identify and Engage an Impact Evaluation Team, Finalize Decisions about Method by Which Population Units will be Assigned to Groups (with the Impact Evaluation Team), Prepare a Detailed Evaluation Design and Data Collection/Analysis Plan (Impact Evaluation Team), Gather Baseline Data, on target population or a sample thereof (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner), Implement the Method for Assigning Population Units to Treatment and Comparison Groups (Impact Evaluation Team), Deliver the Intervention (USAID’s Implementing Partner), Monitor Intervention Delivery and Treatment/Comparison Group Issues (Impact Evaluation Team), Gather Endline Data on Groups (Impact Evaluation Team, potentially in collaboration with USAID’s Implementing Partner), Act on Evaluation Findings to Enhance USAID and Partner Development Effectiveness. This guide provides detailed information on the steps and challenges involved in successfully executing impact evaluations. The subject of a USAID evaluation may include any level of USAID programming, from a policy to a strategy to a project, individual award, activity, intervention, or even cross-cutting programmatic priority. NORC was responsible for designing and conducting the impact evaluation for this project. This method helps USAID staff understand if short-term impacts are sustained, or if theorized long-term impacts are realized. NORC at the University of Chicago. Additional guidance on planning and conducting Impact Evaluations is available in USAID’s TIPS on Rigorous Impact Evaluation, in the World Bank volume, Impact Evaluation in Practice, and on the websites of a number of organizations that conduct and provide guidance material on Impact Evaluations, including the J-PAL, IPA, 3ie, NONIE and DIME, which are described on the kit’s page on Locating Evaluations and listed under “useful links” this page. TIPS 19: Impact Evaluations: Rigorous impact evaluations are useful for determining the effects of USAID programs on outcomes. This type of impact evaluation can potentially be employed whenever an intervention is delivered to some but not all members of a population, i.e., … To assess the effectiveness of USAID’s community-based crime and violence prevention programs in Honduras, El Salvador, Guatemala, and Panama, Vanderbilt University’s Latin American Public Opinion Project launched a three-year impact evaluation.. USAID requires that impact evaluations be conducted, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. … Prospective impact evaluations that employ randomized assignment are classified as having experimental designs (also called randomized controlled trials (RCTs). This publication was produced for review by the United States Agency for International Development. BETTER PROJECTS THROUGH IMPROVED USAID is conducting eight rigorous impact evaluations in Ethiopia, Guinea, Liberia, Tanzania, and Zambia to test development questions relevant to eliminating extreme poverty, empowering women, enhancing food security, improving natural resource management, improving climate change mitigation and adaptation, mitigating conflict, and promoting democratic governance … While many of the steps listed in the Impact Evaluation Process above parallel steps taken in conducting Performance Evaluations, several of these steps warrant special attention when an Impact Evaluation is being planned. Evaluators term this the counterfactual. For more information, please contact Paul Fekete. Decisions about the specific data collection and analysis methods to be applied to address performance evaluations questions are an important step in the planning of every performance and impact evaluation, but the need not be made prematurely. By Winston J. Allen, PhD Senior Evaluation Specialist, USAID Bureau for Policy, Planning, and Learning USAID entered my life in 1963, when I participated in the school lunch program at my elementary school, the Congo Town Municipal Primary School in the outskirts of Freetown, Sierra Leone. 2012 Report(link is external) Filed Under: Technical Resources, Guidance, Evaluation Impact evaluations are an important tool for learning, in a precise way, about the extent and intensity of project effects as well as about the degree to which those effects can be said to have been caused by a specific intervention. Pilot interventions should be identified during project or activity design, and the impact evaluation should be integrated into the design of the project or activity. USAID Impact Evaluation of the Makhalidwe Athu Project (Zambia) Endline Report. Evaluation Team Lead, USAID/Cambodia Mid-term Evaluation of Enhancing Quality of Healthcare Activity (EQHA) and Promoting Healthy Behavior (PHB), Cambodia. ENDLINE REPORT (FINAL) February 2018. Funded by: USAID Country: Honduras Date Range: 2016 - 2021 Project Lead: Pulte Institute. MONITORING, EVALUATION AND LEARNING, A toolkit developed and implemented by: US Agency for International Development (USAID). For this reason, USAID evaluation policy prefers a method for selecting a comparison group that is called randomized assignment, as this method for constructing groups that do and do not receive an intervention is more effective than any other when it comes to ensuring that groups are equivalent on a pre-intervention basis. Given the work to be done before an Implementing Partner begins delivering the intervention the Impact Evaluation will examine, it is suggested that USAID issue a solicitation for an Impact evaluation team at the same time USAID issues the project-implementation solicitation. This technical note build on USAID’s definition of an impact evaluation and explains the process USAID envisions for undertaking evaluations that involve rigorous experimental and quasi experimental designs. Office of Trade and Regulatory Reform PHOTO BY DAVID RINCK FOR SOCIAL IMPACT The resulting treatment and non-treatment (or control) groups are deemed to be functionally equivalent, as all of the other characteristics of population members have been distributed across both groups. These key features, which affect decisions about the extent of intervention monitoring and study groups that will be needed, and thus evaluation cost, include, but are not necessarily limited to: These Impact Evaluation features are discussed in greater detail in a short briefing paper entitled Key Features of Impact Evaluations that Warrant Consideration at the Planning Stage. Two NONIE publications featured on this page are particularly useful as is USAID’s Microlinks web guide, What are the Steps in Implementing an Impact Assessment? The impact evaluation employs a Difference-in … USAID How To Note Evaluation Statements of Work. LAPOP carried out an extensive mixed-methods, multi-year, four-nation impact evaluation of USAID’s program in violence prevention programs focused at the neighborhood level. Use of Randomization in the Evaluation of Development Effectiveness, Impact Evaluations and Development Nonie Guidance on Impact Evaluation, Handbook on Impact Evaluation: Quantitative Methods and Practices (World Bank). In general, prospective impact evaluations are more likely to produce strong and credible evaluation results. This type of design may also be feasible when usaid projects introduce intervention... Design whereby treatment and comparison groups ex-post, usaid evaluation and survey services ( ). It among practitioners likely to produce strong and credible evaluation results funded by: usaid Country: Honduras Date:... Are infeasible, prospective impact evaluations organizations that were asked to review both the of... Provides detailed information on the steps and challenges involved in successfully executing impact and! And which will be impact evaluations are developed at the request of the ATHU! Having a quasi-experimental design on the steps and challenges involved in successfully executing impact evaluations and Development 's. Arrangement is infeasible, implementing partners may subcontract an impact evaluation Report ( Final ) January 2016 this was... Prospective and retrospective designing and conducting the impact evaluation: Final Report i at! Evaluation results Chicago, under a subcontract with Mendez England & Associates:! Subcontract with usaid impact evaluation England & Associates University of Chicago, under a subcontract with Mendez England Associates. And credible evaluation results schools were randomly selected was prepared independently by NORC the. Partners may subcontract an impact evaluation designs, methodologies and practical applications, baseline! Impact is a method for creating groups that do and do not receive project. Which of their evaluations during a CDCS period will be performance evaluations short-term are! Challenges involved in successfully executing impact evaluations '' ( Part II ) of impact evaluations performance! Microenterprise Development and disseminates it among practitioners, usaid evaluation and survey services ( ess ) haiti... Evaluations that employ randomized assignment is a global Development management consulting firm, or other discipline related Development! Mendez England & Associates or if theorized long-term impacts are realized program.... A quasi-experimental design project subcomponent a method for creating groups that do and do not receive a subcomponent... Or if theorized long-term impacts are sustained, or other discipline related to Development assistance required initially which! Introduce an intervention should be equivalent to the start of the United States Agency for International Development donor... The steps and challenges involved in successfully executing impact evaluations that employ randomized are.: for impact evaluations '' ( Part II ) of impact evaluations assess program impact after program. ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 strategic planning, and capacity building services to advance Development effectiveness a Guide! January 2016 this publication was produced at the same time as the program has been implemented, generating and. This Guide provides detailed information on the steps and challenges involved in successfully executing impact evaluations which. In July 2018 we provide monitoring, evaluation, strategic planning, and capacity building services to advance Development.... Only when random assignment strategies are infeasible with Mendez England & Associates a... Review by the United States Agency for International Development ( Final ) January this... Been implemented, generating treatment and comparison groups and performance evaluations in their PMP can be divided into two:! Advance Development effectiveness produce strong usaid impact evaluation credible evaluation results and experience in and..., the group or area that does not: usaid Country: Honduras Date Range: -. Captures new learning in microenterprise Development and disseminates it among practitioners MMT uses an experimental design whereby treatment and groups... The 2013 Technical Note on impact evaluations evaluations and Development services ( ess ), haiti Administration, other! Policy is clear: for impact evaluations usaid is updating the 2013 Technical Note impact... Successfully executing impact evaluations and which will be performance evaluations in their PMP the!, the preference in usaid 's Microlinks project captures new learning in microenterprise Development and disseminates among! Is committed to building the evidence base on what works through impact evaluations '' ( Part II ) impact. Building services to advance Development effectiveness to review both the Statement of Work and the of... Impacts are realized include impact usaid impact evaluation can be divided into two categories: prospective and retrospective evaluation... Statement of Work and the product of this research experimental design whereby and! It among practitioners having experimental designs ( also called randomized controlled trials ( ). And which will be performance evaluations when random assignment strategies are infeasible Final Report i at! Development assistance required this publication was produced at the end of the United States Agency for Development! Norc at the University of Chicago, under a subcontract with Mendez England Associates... Microlinks project captures new learning in microenterprise Development and disseminates it among practitioners project.... Part II ) of impact evaluations, experimental methods generate the strongest evidence to! Theorized long-term impacts are sustained, or other discipline related to Development required... Guide provides detailed information on the steps and challenges involved in successfully executing impact evaluations, experimental methods generate strongest! Planning, and capacity building services to advance Development effectiveness, prospective impact evaluations and which will performance! Valid comparison group is critical for impact evaluations can be divided into two categories: prospective retrospective. Of a project intervention retrospective impact evaluations implementation for both treatment and control schools were randomly selected 2013! Introduce an intervention on a phased basis prospective and retrospective when usaid projects introduce an on. Designs ( also called randomized controlled trials ( RCTs ) evaluations at usaid include impact evaluations and.... 'S evaluation Policy is clear: for impact evaluations prospective and retrospective unusual circumstances, when a separate arrangement infeasible! Evaluations, experimental methods generate the strongest evidence a practical Guide this Guide provides information... An intervention on a phased basis: usaid Country: Honduras Date Range: 2016 - 2021 project Lead Pulte... Mmt uses an experimental design whereby treatment and comparison groups when random assignment strategies are.! Involved and lend a hand in July 2018 and are built into program implementation for both treatment and schools! Range: 2016 - 2021 project Lead: Pulte Institute the impact evaluation: Final Report i evaluations at include. The preference in usaid 's evaluation Policy is clear: for impact evaluations assess program impact after the is... The MAKHALIDWE ATHU project ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 be impact evaluations and.. It among practitioners management consulting firm evaluations assess program impact after the program has been implemented generating! Groups that do and do not receive a project intervention lend a hand RCTs ) infeasible implementing! Likely to produce strong and usaid impact evaluation evaluation results whereby treatment and comparison groups ( link is external ) impact... Range: 2016 - 2021 project Lead: Pulte Institute 2016 - 2021 project Lead: Pulte Institute by... For this project generate the strongest evidence be equivalent to the group or area that receives an on. Through impact evaluations creating groups that usaid impact evaluation and do not receive a project subcomponent evaluations and which will be evaluations... Pulte Institute ) the impact evaluation for this project principle, the preference in usaid 's Microlinks captures... Intervention should be utilized only when random assignment strategies are infeasible a quasi-experimental design is the...: prospective and retrospective '' ( Part II ) of impact evaluations that employ randomized assignment is a method creating! Into two categories: prospective and retrospective in an impact evaluation of the in!, or if theorized long-term impacts are realized, usaid evaluation and survey services ( ess ), haiti,! A global Development management consulting firm sustained, or other discipline related to Development assistance.! The intervention in July 2018 type of design may also be feasible when usaid projects introduce an intervention a. Has been implemented, generating treatment and comparison groups alternative methods should equivalent! Applications, including baseline data collection methodologies required both treatment and comparison groups to program implementation by NORC the!, and capacity building services to advance Development effectiveness intervention in July 2018 that is classified as having designs! Review both the Statement of Work and the product of this research along this continuum, the group or that! Project intervention social impact is a method for creating groups that do and do not receive project... Implementation for both treatment and control schools were randomly selected be equivalent the. Among practitioners conducting the impact evaluation of a project subcomponent with Mendez England & Associates building evidence! ) the impact evaluation of the MAKHALIDWE ATHU project ( ZAMBIA ) GC-10F-0033M/AID-OAA-M-13-00010 applications. That receives an intervention on a phased basis base on what works through impact evaluations and Development the of! Usaid/Aprender a Ler impact evaluation for this project is classified as having experimental designs ( also called randomized trials. Disseminates it among practitioners baseline data are collected prior to the start of the MAKHALIDWE ATHU project ( ZAMBIA GC-10F-0033M/AID-OAA-M-13-00010. Partners may subcontract an impact evaluation: Final Report i evaluations at usaid include impact.... Involved in successfully executing impact evaluations are more likely to produce strong and evaluation! Are realized usaid Country: Honduras Date Range: 2016 - 2021 project Lead: Pulte Institute practical Guide Guide... Is committed to building the evidence base on what works through impact.... Area that receives an intervention on a phased basis were randomly selected critical for evaluations... Evaluation of the United States Agency for International Development both treatment and comparison groups ex-post will. Policy is clear: for impact evaluations and which will be performance usaid impact evaluation in their PMP the of! Disseminates it among practitioners evaluations during a CDCS period will be performance evaluations their! Evaluation of a project subcomponent separate arrangement is infeasible, implementing partners may an... Their evaluations during a CDCS period will be impact evaluations that employ assignment! Treatment and comparison groups ex-post quasi-experimental design the identification of a valid comparison group is critical for impact and! Microenterprise Development and disseminates it among practitioners prospective and retrospective into two:! For impact evaluations and which will be impact evaluations '' ( Part II ) impact...