Seminars

James Druckman
Payson S. Wild Professor of Political Science and IPR Associate Director and Fellow

Location: IPR Conference Room, 617 Library Place

Abstract: One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over non-significant results of equal research quality. Recognition of this bias, along with increased calls for “open science,” has led to an emphasis on replication studies (i.e., repeating a study with new data). Yet, few have explored publication bias and its consequences in replication studies. We offer a micro-level model of the publication process involving an initial study and a replication. We use the model to describe three types of publication biases: 1) the aforementioned file drawer bias, 2) a “repeat study” bias against the publication of replication studies, and 3) a “gotcha bias” where replication results that run contrary to a prior study are more likely to be published. We then estimate the model’s key parameters with a large-scale vignette experiment conducted with political science professors teaching at Ph.D.-granting institutions in the United States. We find evidence of all three types of bias, although those explicitly involving replication studies are notably smaller. This bodes well for the replication movement. That said, the aggregation of all of the biases increases the number of false positives in a literature. We conclude by discussing possible antidotes.

Steven Franconeri
Professor of Psychology, Northwestern University

Location: IPR Conference Room, 617 Library Place

Abstract: Franconeri seeks a set of patterns in a dataset and later decides to communicate his findings to a non-expert audience in an intuitive way. Picking the right visualizations can vastly improve the speed, accuracy, and thoroughness of analysis and presentation by orders of magnitude. Franconeri will use interactive visual tasks to demonstrate the powerful capacity limits that arise when we extract structure and meaning from data visualizations. Understanding these limits produces guidelines for constructing effective visualizations for both visual analytics and visual communication of patterns in data, and shows how display designs and motivated cognition can bias interpretations of those patterns.

Paul Goren
Chief Strategist, School of Education and Social Policy

Location: 600 Foster St., Chambers Hall, lower level

Paul Goren was superintendent of schools for the Evanston/Skokie School District 65 from 2014-19. Prior to joining District 65, Goren served as senior vice president for program at the Collaborative for Academic, Social, and Emotional Learning (CASEL) in Chicago. He was the interim chief for strategy and accountability for Chicago Public Schools while working as executive director of the Consortium on Chicago School Research. He served for over a decade as senior vice president of the Spencer Foundation and as a program director for Child and Youth Development at the MacArthur Foundation

Cynthia Coburn
Professor of Human Development, Social Policy and Learning Sciences, and IPR Associate

Location: 600 Foster St., Chambers Hall, lower level

Cynthia Coburn studies the relationship between instructional policy and teachers’ classroom practices in urban schools. She has investigated this issue in a series of studies that tackle critical issues facing public schools: the relationship between reading policy and teachers’ classroom practice, the scale-up of innovative mathematics curricula, data use at the district level, and the relationship between research and practice for school improvement. In 2011, Coburn was awarded the Early Career Award from the American Educational Research Association.

Ronald T. Ackermann
Senior Associate Dean for Public Health, Director of the Institute for Public Health and Medicine, Professor of Medicine and Medical Social Sciences and IPR Associate

Location: IPR Conference Room, 617 Library Place

Abstract: Ronald Ackermann’s work focuses on ways to improve health and healthcare through the efficient coordination of health promotion and disease prevention activities across healthcare and community settings. He is considered a national expert in pragmatic research and natural experiments to improve the prevention and control of diabetes and other chronic conditions. Ackermann’s additional interests include advancing health equity, eliminating health disparities, healthcare provider interventions, quality improvement, cost effectiveness analysis, behavioral economics, and strategies for engaging community members and policy decision makers in the full spectrum of research. He is Director for Northwestern University’s Center for Community Health, which is committed to helping cultivate and support many forms of research that improve the health and healthcare of communities in Chicago and beyond.

Wendy Tam Cho
ChoProfessor of Political Science, Statistics, Mathematics, Asian Studies, and Law, University of Illinois at Urbana-Champaign

Location: IPR Conference Room, 617 Library Place

Abstract: Important insights into redistricting can be gained through an interdisciplinary approach that combines research from many fields, including statistics, operations research, computer science, high performance computing, math, law, and political science. Cho’s work integrates insights from all of these disciplines to create a novel approach for analyzing and reforming redistricting in a way that is tightly coupled with the legal frameworks articulated by the courts.

Blake McShane
Associate Professor of Marketing, Northwestern University

Location: IPR Conference Room, 617 Library Place

Michael Weiss
Senior Associate, MDRC

Location: IPR Conference Room, 617 Library Place

Abstract: Researchers face many choices when designing, analyzing data from, and interpreting results from multisite (blocked) individually randomized control trials (multisite RCTs). The most common parameter of interest in multisite RCTs is the overall average intent-to-treat (ITT) effect. But even this parameter is not simple; even defining it requires decisions. The researcher needs to determine whether to estimate the average effect across individuals, or the average effect across sites. Furthermore, the researcher can target the average effect for the experimental sample, or, by viewing those units as a sample from a larger population, target the average effect for a broader population. If treatment effects vary across sites, these estimands can differ. Once an estimand is selected, the researcher must choose among estimators to estimate their chosen estimand. Weiss and his colleagues describe 13 common estimators, consider which estimands they are most appropriate for, and discuss their properties in the face of treatment effect heterogeneity. Using data from 12 large multisite RCTs of educational and job training programs, they estimate the ITT effect and its associated standard error using each estimator and compare and contrast the results. This allows researchers to assess the extent that each of these decisions matter in practice. Guidance for applied researchers is provided.

Luke Miratrix
Assistant Professor of Education, Harvard Graduate School of Education

Location: IPR Conference Room, 617 Library Place

Abstract: Despite our best efforts, sometimes we are forced to use the interrupted time series (ITS) design as an identification strategy for potential policy change when we only have a single treated unit and no comparable controls. For example, with recent county- and state-wide criminal justice reform efforts, where judicial bodies have changed bail-setting practices for everyone to reduce rates of pre-trial detention while maintaining court order and public safety, we have no natural or plausible comparison group other than the past. In these contexts, it is imperative to model pre-policy trends with a light touch, allowing for structures such as autoregressive departures from any pre-existing trend to accurately assess the true uncertainty of our projections given our modeling assumptions. One way forward is to use simulation, generating a distribution of plausible counterfactual trajectories to compare to the observed. This approach naturally allows for incorporating seasonality and other time-varying covariates. It provides confidence intervals along with point estimates for the potential impacts of policy change.

Leah Welty
Associate Professor of Preventive Medicine (Biostatistics) and Psychiatry and Behavioral Sciences, Feinberg School of Medicine

Location: IPR Conference Room, 617 Library Place

Abstract: This talk will introduce a free, open-source program for conducting reproducible research and creating dynamic documents using Microsoft Word and Stata, SAS, and R. Called StatTag, this program was recently developed to address a critical need in the research community: There were no broadly accessible tools to integrate document preparation in Word with statistical code, results, and data. Popular tools such as knitR and Markdown use plain text editors for document preparation. Despite the merits of these programs, Microsoft Word is ubiquitous for manuscript preparation in many fields, such as medicine, in which conducting reproducible research is increasingly important. Furthermore, current tools are one-directional: No downstream changes to the rendered RTF/Word documents are reflected in the source code. We developed StatTag to fill this void. StatTag provides an interface to edit statistical code directly from Word, and allows users to embed statistical output from that code (estimates, tables, figures) within Word. Output can be individually or collectively updated in one click with a behind-the-scenes call to the statistical program. With StatTag, modification of a dataset or analysis will no longer entail transcribing results into Word. This talk will include worked examples, and will be accessible to many users.