Linked e-resources
Details
Table of Contents
Front Cover
Contents
Preface
About the Authors and Contributors
Abbreviations
Part I The Basics of Impact Evaluation
Chapter 1 The Purpose of Impact Evaluation
Introduction
The difference between monitoring and evaluation
A definition of impact evaluation
Brief overview of types of impact evaluations
Other issues regarding impact evaluation
Conclusion
Notes
References
Chapter 2 How to Conduct an Impact Evaluation: Getting Started
Introduction
Step 1. Defining the program and the outcomes of interest
Step 2. Forming a theory of change to refine the evaluation questions
Step 3. Depicting a theory of change in a results chain (logic model)
Step 4. Formulating specific hypotheses for the impact evaluation
Step 5. Selecting performance indicators for monitoring and evaluation
Conclusion
Notes
References
Chapter 3 The Evaluation Problem
Introduction
Correlation does not imply causation
Potential outcomes and the evaluation problem
Observed outcomes and the gain from treatment
Parameters of interest
Conclusion
References
Chapter 4 Validity: Internal, External, and Trade-Offs
Introduction
Internal validity
External validity
Trade-offs and intermediate approaches
Conclusion
Notes
References
Chapter 5 Overview of Impact Evaluation Methods
Introduction
Using randomized controlled trials to evaluate program impacts
Impact evaluations based on nonrandomized and quasi-experimental data
Conclusion
Notes
References
Part II Experimental Methods
Chapter 6 Introduction to Randomized Controlled Trials
Introduction
The basic idea of a randomized controlled trial
How does randomization solve the evaluation problem?
What if some people assigned to the treatment group choose not to participate?.
Intention-to-treat effects
Intention-to-treat effects when effects spill over onto nonparticipants
Encouragement designs
Conclusion
Notes
References
Chapter 7 Regression Methods for Randomized Controlled Trials
Introduction
Estimating average treatment effects when no problems occur
Estimation when some in the treatment group are not treated
Complications caused by sample attrition
Methods for increasing precision of the estimates
Methods for obtaining correct standard errors
Other useful advice and recommendations
Conclusion
Notes
References
Chapter 8 Practical Advice for Implementing Randomized Evaluations
Introduction
Potential problems with randomized experiments, and possible solutions
Practical advice for randomizing into treatment and control groups
The use of pre-analysis plans in impact evaluations
Other practical advice
Increasing external validity
Conclusion
Notes
References
Chapter 9 Sample Size, Sample Design, and Statistical Power
Introduction
Statistical power as a criterion for choosing the sample design
Power and MDE calculations in more complex settings
Practical issues regarding power calculations
Further statistical issues
Conclusion
Notes
References
Chapter 10 Recommendations for Conducting Ethical Impact Evaluations
Introduction
Two frameworks for conducting ethical evaluations and research
Confidentiality
Ethics of randomized controlled trials
Conflicts of interest
Ethical research in practice
Conclusion
Notes
References
Part III Nonexperimental Methods
Chapter 11 Regression Methods for Nonrandomized Data: Cross-Sectional and Before-After Estimators
Introduction
Examples: Cross-sectional, before-after, and difference-in-differences estimators
Parameters of interest.
The cross-sectional estimator and sources of bias
The before-after estimator
Conclusion
Notes
References
Chapter 12 Regression Methods for Nonrandomized Data: The Difference-in-Differences Estimator and the Within Estimator
Introduction
The difference-in-differences estimator
Within estimators
Applications of difference-in-differences and within estimators
Conclusion
Notes
References
Chapter 13 Matching Methods
Introduction
Two simple examples
Cross-sectional matching
Implementation of propensity score matching estimators
Difference-in-differences matching
Additional topics for matching methods
Empirical applications of matching estimators
Conclusion
Notes
References
Chapter 14 Regression Discontinuity Methods
Introduction
Intuition for regression discontinuity methods
Identification of treatment effects under "sharp" and "fuzzy" data
Checking the validity of a regression discontinuity design
The Hahn, Todd, and van der Klaauw estimation method
Examples of regression discontinuity methods
Conclusion
Notes
References
Chapter 15 Instrumental Variables Estimation and Local Average Treatment Effects
Introduction
Two uses of instrumental variables estimation for impact evaluation analysis
Instrumental variables estimation of ATE and ATT
Using IV methods to estimate local average treatment effects
Conclusion
Notes
References
Chapter 16 Control Function Methods
Introduction
The basic idea of the control function approach
Methods for estimating control functions
Standard error calculations for control function estimation methods
Comparing control functions to matching methods and instrumental variables
Adapting the control function approach for estimating ATE(X) and ATE.
An application: The performance of public and private schools in Chile
Conclusion
Notes
References
Chapter 17 Quantile Treatment Effects
Introduction
The basic idea of quantile regression, with an example
Conditional and unconditional quantile treatment effect estimators
Conditional quantile treatment effect estimators
Unconditional quantile treatment effect estimators
Standard errors
Examples of applications
Conclusion
Notes
References
Part IV Data Collection and Project Management
Chapter 18 Designing Questionnaires and Other Data Collection Instruments
Introduction
General principles and recommendations
General advice on the design of questionnaires
Household questionnaires
Service provider questionnaires
Community (and price) questionnaires
Other data collection instruments
Paper questionnaires versus computer-assisted personal interviewing
Conclusion
Notes
References
Chapter 19 Data Collection and Data Management
Introduction
The steps involved in data collection and data management
Establish procedures for collecting and managing the data
Collect the data (including monitoring of data quality)
Further checks of data quality after the fieldwork
Create data files for analysis and dissemination
Establish a system to store, revise, and disseminate the data
Conclusion
Notes
References
Chapter 20 Survey Management
Introduction
Budgeting and developing an overall plan of activities
Human resources (personnel) management
Logistical coordination
Community relations
Lessons from unfortunate experiences
Conclusion
Note
References
Part V Related Topics
Chapter 21 Dissemination of Results and Working with Policy Makers
Introduction
What products should the impact evaluation deliver?.
Dissemination of the findings
Working with policy makers
Conclusion
Notes
References
Chapter 22 Qualitative Approaches, Data, and Analysis in Impact Evaluations
Joan DeJaeghere
Introduction
Contributions and challenges in using qualitative research in impact evaluations
Different purposes and types of qualitative approaches
The most common methods for collecting qualitative data
Exploratory and explanatory qualitative approaches
Practical suggestions for designing, gathering, and analyzing qualitative data
Conclusion
Annex 22A Questions for realist evaluations
References
Chapter 23 Cost-Benefit Analysis and Cost-Effectiveness Analysis
Introduction
Calculation of costs
A simple comparison of cost-benefit analysis and cost-effectiveness analysis
Cost-benefit analysis (valuing the benefits)
Cost-effectiveness analysis
Conclusion
Notes
References
Boxes
Box 1.1 Key organizations and agency departments that focus on impact evaluation in international development
Box 3.1 Requirements for answering the evaluation problem, "What is the causal impact of the program (or project or policy) on the outcomes of interest?"
Box 22.1 Case studies and comparative qualitative analysis: Example of Akazi Kanoze Youth Livelihoods Project (Alcid 2014)
Box 22.2 Longitudinal and theory-based design and analysis: Example of Learn, Earn, and Save Initiative of Youth Livelihoods Programs in Tanzania and Uganda
Figures
Figure 2.1 A results chain diagram: Basic layout and components
Figure 2.2 A more detailed view of what goes into a results chain (logic model)
Figure 2.3 Examples of a results chain's components: Education and health sectors
Figure 2.4 Example of a results chain: Mexico's PROGRESA program.
Figure 2.5 Example of a detailed, successive results chain: Regional vaccination program.
Contents
Preface
About the Authors and Contributors
Abbreviations
Part I The Basics of Impact Evaluation
Chapter 1 The Purpose of Impact Evaluation
Introduction
The difference between monitoring and evaluation
A definition of impact evaluation
Brief overview of types of impact evaluations
Other issues regarding impact evaluation
Conclusion
Notes
References
Chapter 2 How to Conduct an Impact Evaluation: Getting Started
Introduction
Step 1. Defining the program and the outcomes of interest
Step 2. Forming a theory of change to refine the evaluation questions
Step 3. Depicting a theory of change in a results chain (logic model)
Step 4. Formulating specific hypotheses for the impact evaluation
Step 5. Selecting performance indicators for monitoring and evaluation
Conclusion
Notes
References
Chapter 3 The Evaluation Problem
Introduction
Correlation does not imply causation
Potential outcomes and the evaluation problem
Observed outcomes and the gain from treatment
Parameters of interest
Conclusion
References
Chapter 4 Validity: Internal, External, and Trade-Offs
Introduction
Internal validity
External validity
Trade-offs and intermediate approaches
Conclusion
Notes
References
Chapter 5 Overview of Impact Evaluation Methods
Introduction
Using randomized controlled trials to evaluate program impacts
Impact evaluations based on nonrandomized and quasi-experimental data
Conclusion
Notes
References
Part II Experimental Methods
Chapter 6 Introduction to Randomized Controlled Trials
Introduction
The basic idea of a randomized controlled trial
How does randomization solve the evaluation problem?
What if some people assigned to the treatment group choose not to participate?.
Intention-to-treat effects
Intention-to-treat effects when effects spill over onto nonparticipants
Encouragement designs
Conclusion
Notes
References
Chapter 7 Regression Methods for Randomized Controlled Trials
Introduction
Estimating average treatment effects when no problems occur
Estimation when some in the treatment group are not treated
Complications caused by sample attrition
Methods for increasing precision of the estimates
Methods for obtaining correct standard errors
Other useful advice and recommendations
Conclusion
Notes
References
Chapter 8 Practical Advice for Implementing Randomized Evaluations
Introduction
Potential problems with randomized experiments, and possible solutions
Practical advice for randomizing into treatment and control groups
The use of pre-analysis plans in impact evaluations
Other practical advice
Increasing external validity
Conclusion
Notes
References
Chapter 9 Sample Size, Sample Design, and Statistical Power
Introduction
Statistical power as a criterion for choosing the sample design
Power and MDE calculations in more complex settings
Practical issues regarding power calculations
Further statistical issues
Conclusion
Notes
References
Chapter 10 Recommendations for Conducting Ethical Impact Evaluations
Introduction
Two frameworks for conducting ethical evaluations and research
Confidentiality
Ethics of randomized controlled trials
Conflicts of interest
Ethical research in practice
Conclusion
Notes
References
Part III Nonexperimental Methods
Chapter 11 Regression Methods for Nonrandomized Data: Cross-Sectional and Before-After Estimators
Introduction
Examples: Cross-sectional, before-after, and difference-in-differences estimators
Parameters of interest.
The cross-sectional estimator and sources of bias
The before-after estimator
Conclusion
Notes
References
Chapter 12 Regression Methods for Nonrandomized Data: The Difference-in-Differences Estimator and the Within Estimator
Introduction
The difference-in-differences estimator
Within estimators
Applications of difference-in-differences and within estimators
Conclusion
Notes
References
Chapter 13 Matching Methods
Introduction
Two simple examples
Cross-sectional matching
Implementation of propensity score matching estimators
Difference-in-differences matching
Additional topics for matching methods
Empirical applications of matching estimators
Conclusion
Notes
References
Chapter 14 Regression Discontinuity Methods
Introduction
Intuition for regression discontinuity methods
Identification of treatment effects under "sharp" and "fuzzy" data
Checking the validity of a regression discontinuity design
The Hahn, Todd, and van der Klaauw estimation method
Examples of regression discontinuity methods
Conclusion
Notes
References
Chapter 15 Instrumental Variables Estimation and Local Average Treatment Effects
Introduction
Two uses of instrumental variables estimation for impact evaluation analysis
Instrumental variables estimation of ATE and ATT
Using IV methods to estimate local average treatment effects
Conclusion
Notes
References
Chapter 16 Control Function Methods
Introduction
The basic idea of the control function approach
Methods for estimating control functions
Standard error calculations for control function estimation methods
Comparing control functions to matching methods and instrumental variables
Adapting the control function approach for estimating ATE(X) and ATE.
An application: The performance of public and private schools in Chile
Conclusion
Notes
References
Chapter 17 Quantile Treatment Effects
Introduction
The basic idea of quantile regression, with an example
Conditional and unconditional quantile treatment effect estimators
Conditional quantile treatment effect estimators
Unconditional quantile treatment effect estimators
Standard errors
Examples of applications
Conclusion
Notes
References
Part IV Data Collection and Project Management
Chapter 18 Designing Questionnaires and Other Data Collection Instruments
Introduction
General principles and recommendations
General advice on the design of questionnaires
Household questionnaires
Service provider questionnaires
Community (and price) questionnaires
Other data collection instruments
Paper questionnaires versus computer-assisted personal interviewing
Conclusion
Notes
References
Chapter 19 Data Collection and Data Management
Introduction
The steps involved in data collection and data management
Establish procedures for collecting and managing the data
Collect the data (including monitoring of data quality)
Further checks of data quality after the fieldwork
Create data files for analysis and dissemination
Establish a system to store, revise, and disseminate the data
Conclusion
Notes
References
Chapter 20 Survey Management
Introduction
Budgeting and developing an overall plan of activities
Human resources (personnel) management
Logistical coordination
Community relations
Lessons from unfortunate experiences
Conclusion
Note
References
Part V Related Topics
Chapter 21 Dissemination of Results and Working with Policy Makers
Introduction
What products should the impact evaluation deliver?.
Dissemination of the findings
Working with policy makers
Conclusion
Notes
References
Chapter 22 Qualitative Approaches, Data, and Analysis in Impact Evaluations
Joan DeJaeghere
Introduction
Contributions and challenges in using qualitative research in impact evaluations
Different purposes and types of qualitative approaches
The most common methods for collecting qualitative data
Exploratory and explanatory qualitative approaches
Practical suggestions for designing, gathering, and analyzing qualitative data
Conclusion
Annex 22A Questions for realist evaluations
References
Chapter 23 Cost-Benefit Analysis and Cost-Effectiveness Analysis
Introduction
Calculation of costs
A simple comparison of cost-benefit analysis and cost-effectiveness analysis
Cost-benefit analysis (valuing the benefits)
Cost-effectiveness analysis
Conclusion
Notes
References
Boxes
Box 1.1 Key organizations and agency departments that focus on impact evaluation in international development
Box 3.1 Requirements for answering the evaluation problem, "What is the causal impact of the program (or project or policy) on the outcomes of interest?"
Box 22.1 Case studies and comparative qualitative analysis: Example of Akazi Kanoze Youth Livelihoods Project (Alcid 2014)
Box 22.2 Longitudinal and theory-based design and analysis: Example of Learn, Earn, and Save Initiative of Youth Livelihoods Programs in Tanzania and Uganda
Figures
Figure 2.1 A results chain diagram: Basic layout and components
Figure 2.2 A more detailed view of what goes into a results chain (logic model)
Figure 2.3 Examples of a results chain's components: Education and health sectors
Figure 2.4 Example of a results chain: Mexico's PROGRESA program.
Figure 2.5 Example of a detailed, successive results chain: Regional vaccination program.