What is included in an impact evaluation study?
Impact- if not measured or evaluated correctly, can be very difficult to understand and quantify. In this blog, we will see what an impact evaluation study is, what are the steps involved in completing an evaluation study, and how to communicate the study results to the stakeholders involved.
An ideal program or intervention functions with a goal and certain objectives in mind. Impact evaluations are necessary to gauge to what extent have the intended impacts of the intervention been achieved. They provide information about not just the intended objectives but also the unintended impacts. Impact evaluations give an understanding of how well a program has worked, and what are the areas that can be strengthened in it.
Let’s look at the steps for an effective impact evaluation study.
Step 1: Framing research questions
Research questions are questions that shape data collection, analysis, and reporting.
Evaluate your research question. Is it feasible? Is it interesting? Is it novel? Is it ethical? Is it relevant?
Step 2: Indicator development
An indicator is a variable that measures the key elements of a program. An indicator has 4 parts.
Make sure that the indicators you choose are easy to understand and will provide essential metrics to determine impact. They need to be cost-effective and fit within the program budget. Use a combination of quantitative and qualitative indicators. Only include indicators that are aligned with the program components and goals. Experts can be involved to get their insights on it.
Step 3: Sampling
Sampling is the process of selecting a set of observations from a population to estimate a program’s impact, for that population. The chosen sample should be a fair representation of the population. The types of sampling are:
Step 4: Research tools
The research design and tools need to be decided prior to the start of data collection. These tools should focus on garnering data that will provide rich insights into the external factors which enable or deter the achievement of impact and program objectives. Make sure each question adds value and drives survey responses that relate directly to the research goals and evaluation questions. Keep wording focused on the respondent and their opinions, rather than introducing a point of view of your own. Undertake a pilot study to understand possible response biases.
Step 5: Field research
A work plan needs to be made which sets out the various stages of data collection, analysis, and deadlines for the various deliverables. A qualified team needs to be chosen and trained to conduct the field research. There are several tools like Open Data Kit, SurveyCTO, KOBO, Google forms, Magpie, Track you Metrics, etc to collect data. Supervisors need to ensure the quality of data on the field along with the enumerators and monitor the data regularly.
Step 6: Reporting
The report of an impact study should cover the objectives of the evaluation and should be easy to understand. It should reflect the purpose and answer the key questions of the research. The components of a report are:
Table of Contents
List of Figures and Tables
List of Abbreviations
Results and Findings
Summary of Results according to OECD DAC
List of Annex
Step 7: Sharing findings with stakeholders and using the results for strategy
Just providing a comprehensive report to the stakeholders is not enough, communicating it in the best way possible is important too. The findings of the reports can be shared through presentations, summarised versions of the report, recorded videos, etc. Use existing online tools and software to create engaging and interesting content. Guide the stakeholders to use the findings of the report to bring about the necessary changes to the program.
Give real-time results to stakeholders to help them start working on it from the beginning.
Recommendations must be prioritized within a time frame to the extent possible so that the next measures can easily be taken.
The learnings must be incorporated into the overall programming cycle through an effective feedback system.