Readings and Presentations

Instructor: Laura Peck

Primary Readings:
Lemire, Sebastian, Allan Porowski, & Kaity Mumma. (2023). How We Model Matters: Visualizing Program Theories. Rockville, MD: Abt Associates. Available at: https://www.abtglobal.com/sites/default/files/files/insights/white-papers/2023/visualizing-program-theory.pdf

Peck, Laura R. (2020). Experimental Evaluation Design for Program Improvement, Chapters 1-3. Thousand Oaks, CA: SAGE Publishing.

 Additional Resources for Deeper Dives:

• For crafting research questions:
  Ratan, S. K., Anand, T., & Ratan, J. (2019). Formulation of research question–Stepwise approach. Journal of Indian Association of Pediatric Surgeons, 24(1), 15. Available at: https://pubmed.ncbi.nlm.nih.gov/30686882/

• Education-specific logic model resource:
  Kekahio, Wendy, Louis Cicchinelli, Brian Lawton, & Paul R. Brandon. (2014). Logic models: A tool for effective program planning, collaboration, and monitoring. National Center for Education Evaluation and Regional Assistance. Available at: https://files.eric.ed.gov/fulltext/ED544779.pdf 

• Resources on Theory of Change:
  https://www.theoryofchange.org/what-is-theory-of-change/

 Presentation

Instructor: Laura Peck

Primary Reading:
Litwok, Daniel, Douglas Walton, Laura R. Peck, and Eleanor Harvill. (2018). Health Profession Opportunity Grants (HPOG) Impact Study’s Three-Year Follow-Up Analysis Plan, Section 2 through 2.2.1 (pp.17-24). OPRE Report #2018-124, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Available at: https://www.acf.hhs.gov/opre/report/health-profession-opportunity-grants-hpog-impact-studys-three-year-follow-analysis-plan

 Suggested Reading:
Illustration: How does the U.S. measure hunger? https://www.ers.usda.gov/topics/food-nutrition-assistance/food-security-in-the-u-s/measurement/

Presentation

Instructor: Carolyn Hill

Hill, C. J., Scher, L., Haimson, J., & Granito, K. (2023). Conducting implementation research in impact studies of education interventions: A guide for researchers. (NCEE 2023-005). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from https://ies.ed.gov/ncee/pubs/2023005/index.asp

 Additional Reading

Hill, C. & Scher, L. (2024, August 7) Prioritizing and Selecting Context Features in Education Impact Studies: A 4R Lenses Approach [Webinar] Society for Research on Educational Effectiveness (SREE) and the Association for Public Policy Analysis & Management (APPAM)

Presentation

Instructor: Spyros Konstantopoulos

Bloom, H.S. (2005). Randomizing groups to evaluate place-based programs. In Howard S. Bloom (ed.) Learning More from Social Experiments: Evolving Analytic Approaches, pp. 115–172. New York: Russell Sage Foundations.

Hedges, L. V. (2024). Chapter 10: Cluster randomized designs. In The design of educational and social experiments. New York: Russell Sage Foundation.

Hedges, L. V., & Schauer, J. (2018). Randomised trials in education in the USA. Educational Research, 60(3), 265–275. https://doi.org/10.1080/00131881.2018.1493350

Hedges, L.V. & Hedberg, E.C. (2007). Intraclass correlations for planning group randomized experiments in education. Educational Evaluation and Policy Analysis, 29, 60–87. https://doi.org/10.3102/0162373707299706

Kirk, R. E. (2013). Experimental design: Procedures for the behavioral sciences (4th ed.). SAGE Publications, Inc. https://dx.doi.org/10.4135/9781483384733 Chapter 8: Randomized Block Designs. Chapter 11: Hierarchical Designs.

Konstantopoulos, S. (2011). Optimal Sampling of Units in Three-Level Cluster Randomized Designs An ANCOVA Framework. Educational and Psychological Measurement, 71(5), 798-813. https://doi.org/10.1177/0013164410397186

Konstantopoulos, S. (2012). The Impact of Covariates on Statistical Power in Cluster Randomized Designs: Which Level Matters More?. Multivariate Behavioral Research, 47(3), 392-420. https://doi.org/10.1080/00273171.2012.673898

Raudenbush, S. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://psycnet.apa.org/doi/10.1037/1082-989X.2.2.173

Instructor: Spyros Konstantopoulos

Hedges, L. V. (2024). Chapter 12: Multisite individually randomized designs. In The design of educational and social experiments. New York: Russell Sage Foundation.

Hedges, L. V. (2024). Chapter 13: Multisite cluster randomized designs. In The design of educational and social experiments. New York: Russell Sage Foundation.

Konstantopoulos, S. (2013). Optimal Design in Three-Level Block Randomized Designs With Two Levels of Nesting An ANOVA Framework With Random Effects. Educational and Psychological Measurement, 73(5), 784-802. https://doi.org/10.1177/0013164413485752

Raudenbush, S.W. (1993). Hierarchical linear models and experimental design. In L. K. Edwards (Ed.) Applied analysis of variance in behavioral science (pp. 459–496). New York: Marcel Dekker, Inc.

Rhoads, C.H. (2011). The implications of “contamination” for experimental design in education research. Journal of Educational and Behavioral Statistics, 36(1), 76–104. https://psycnet.apa.org/doi/10.3102/1076998610379133

PRESENTATION

Instructor:  Beth Tipton

Hedberg, E. C. & Hedges, L. V. (2014). Reference values of within-district intraclass correlations of academic achievement by district characteristics: Results from a meta-analysis of district-specific data. Evaluation Review, 38, 546-582. https://doi.org/10.1177/0193841X14554212

Hedges, L. V., & Borenstein, M. (2014). Conditional optimal design in three- and four-level experiments. Journal of Educational and Behavioral Statistics, 39(4), 257–281. https://doi.org/10.3102/1076998614534897

Hedges, L. V., & Hedberg, E. C. (2014). Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education. Evaluation Review, 37(6), 445-489. https://doi.org/10.1177/0193841X14529126 

Spybrook, J., Hedges, L., & Borenstein, M. (2014). Understanding Statistical Power in Cluster Randomized Trials: Challenges Posed by Differences in Notation and Terminology. Journal of Research on Educational Effectiveness, 7(4), 384–406. https://doi.org/10.1080/19345747.2013.848963

Presentation

Instructor: Jessaca Spybrook

Dong, N., & Maynard, R. A. (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24-67.
https://doi.org/10.1080/19345747.2012.673143.

 Spybrook, J., Kelcey, B., & Dong, N. (2016). Power for detecting treatment by moderator effects in two and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605-627.  https://doi.org/10.3102/1076998616655442

Spybrook, J., Bloom, H., Congdon, R., Hill, C., Martinez, A., Raudenbush, S., & TO, A. (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software. William T. Grant Foundation. Retrieved on November, 5, 2012.
            Section I: 1 and 2
            Section III: 7, 8, 9, and 10

Presentation

Instructor:  Beth Tipton

Baron RM, Kenny DA. The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986 Dec;51(6):1173-82. https://doi.org/10.1037//0022-3514.51.6.1173

Bryan, C.J., Tipton, E. & Yeager, D.S. Behavioural science is unlikely to change the world without a heterogeneity revolution. Nat Hum Behav 5, 980–989 (2021). https://doi.org/10.1038/s41562-021-01143-3

Presentation

Instructor:  Jessaca Spybrook

Presentation

Instructor:  Larry V. Hedges

Hedges, L. V. (2024). Excerpt from Chapter 6: Planning experiments. In The design of educational and social experiments. New York: Russell Sage Foundation.

Presentation

Instructor:  Elizabeth Tipton

Tipton, E. & Hartman, E. (2021) Generalizability and Transportability. Chapter in Handbook of Multivariate Matching and Weighting (Edited by Stuart, E., Rosenbaum, P., Small, D., & Zubizarreta, J.).

Tipton, E., & Olsen, R. B. (2022) Enhancing the Generalizability of Impact Studies in Education. (NCEE 2022-003). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from https://files.eric.ed.gov/fulltext/ED617445.pdf

Presentation

Instructor: Kylie Flynn

Farrell, C. C., Penuel, W. R., Coburn, C. E., Daniel, J., & Steup, L. (2021). Practice Partnerships in Education: The State of the Field. William T. Grant Foundation.  https://eric.ed.gov/?id=ED615899

Roschelle, J., Feng, M., Gallagher, H., Murphy, R., Harris, C., Kamdar, D., Trinidad, G. (2014). Recruiting Participants for Large-Scale Random Assignment Experiments in School Settings. Menlo Park, CA: SRI International. https://files.eric.ed.gov/fulltext/ED555574.pdf

 Suggested Reading:
Tipton, E., & Matlen, B. J. (2019). Improved Generalizability Through Improved Recruitment: Lessons Learned From a Large-Scale Randomized Trial. American Journal of Evaluation, 40(3), 414-430. https://doi.org/10.1177/1098214018810519 

Presentation

Instructor: Brooks Bowden

Bowden, A.B. (2023). Simplifying the Design of Cost-Effectiveness Research in Field Experiments. AERA Open.

Bowden, A. B., Shand, R., Belfield, C. R., Wang, A., & Levin, H. M. (2017). Evaluating Educational Interventions That Induce Service Receipt: A Case Study Application of City Connects. American Journal of Evaluation, 38(3), 405–419. https://doi.org/10.1177/1098214016664983

Gray, A., Sirinides, P., Fink, R., & Bowden, A.B. (2021). Integrating literacy and science instruction in kindergarten: Results from the efficacy study of Zoology One. Journal of Research on Educational Effectiveness. https://doi.org/10.1080/19345747.2021.1938313

Instructor: Larry Hedges

Campbell, M. K. et al. (2012). Consort 2010 statement: Extension to cluster randomized trials.British Medical Journal, 345, e5661. https://doi.org/10.1136/bmj.e5661

CONSORT Extension for Cluster Trials Checklist

Presentation

Instructor:  Ben Kelcey

Baron R. & Kenny D.A. (1986). The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037//0022-3514.51.6.1173

Krull, J.L., and MacKinnon, D.P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36(2): 249–77. https://psycnet.apa.org/doi/10.1207/S15327906MBR3602_06

MacKinnon, D. P., & Fairchild, A. J. (2009). Current directions in mediation analysis. Current Directions in Psychological Science, 18(1), 16–20. https://doi.org/10.1111/j.1467-8721.2009.01598.x

Suggested reading:

Bai, F., Kelcey, B., Xie, Y., & Cox, K. (2025). Design and Analysis of Clustered Regression Discontinuity Designs for Probing Mediation Effects. Journal of Experimental Education, 93, 2, 419-449. https://doi.org/10.1080/00220973.2023.2287445

Bai, F., Xie, Y., Kelcey, B., Ataneka, A., McLean, L., & Phelps, G. (2024). Design Parameter Values for Planning Mediation Studies With Teacher and Student Mathematics Outcomes. Journal of Research on Educational Effectiveness, 1-35. https://doi.org/10.1080/19345747.2024.2349670

Bauer, D.J., Preacher, K.J., and Gil, K.M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11(2): 142–63. https://doi.org/10.1037/1082-989x.11.2.142

Keele, L. (2015). Causal Mediation Analysis: Warning! Assumptions Ahead. American Journal of Evaluation, 36(4), 500–513. https://psycnet.apa.org/doi/10.1177/1098214015594689

Kelcey, B., Bai, F. & Xie, Y. (2020). Statistical Power in Partially Nested Designs Probing Multilevel Mediation. Psychotherapy Research Journal, 30, 8, 1061-1074.  https://doi.org/10.1080/10503307.2020.1717012

Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical Power for Causally Defined Indirect Effects in Group-Randomized Trials With Individual-Level Mediators. Journal of Educational and Behavioral Statistics, 42(5), 499-530. https://doi.org/10.3102/1076998617695506 

Kelcey, B., Spybrook, J., Dong, N., & Bai, F. (2020). Cross-Level Mediation in School-Randomized Studies of Teacher Development: Experimental Design and Power. Journal of Research on Educational Effectiveness, 13(3), 459–487. https://doi.org/10.1080/19345747.2020.1726540

Kelcey, B., Xie, Y., Dong, N., & Spybrook, J. (2020). Power and Sample Size Determination for Multilevel Mediation in Three-Level Cluster-Randomized. Multivariate Behavioral Research, 56,3, 496-513. https://doi.org/10.1080/00273171.2020.1738910

Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670. https://psycnet.apa.org/doi/10.1177/0049124112460380

Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719. https://psycnet.apa.org/doi/10.1177/1094428108327450

Presentation

Instructor:  Larry Hedges

Hedges, L. & Tipton, E. (2020). Addressing the challenges to educational research posed by COVID-19. Institute for Policy Research Working Paper (WP-20-47). 
https://www.ipr.northwestern.edu/documents/working-papers/2020/wp-20-47rev.pdf

Presentation