Threats to validity

Content for Thursday, February 20, 2025

Readings

• Guido Imbens, “Statistical Significance, p-Values, and the Reporting of Uncertainty” (Imbens 2021). This is available on iCollege, and it’s also free here.
• Randall Munroe, “Significant”
• Alexander Coppock, “10 Things to Know About Statistical Power”
• Play around with FiveThirtyEight, “Hack Your Way To Scientific Glory”
• Chapter 9 in Impact Evaluation in Practice (Gertler et al. 2016)
• Chapter 11 in The Effect (Huntington-Klein 2021)

Slides

The slides for today’s lesson are available online as an HTML file. Use the buttons below to open the slides either as an interactive website or as a static PDF (for printing or storing for later). You can also click in the slides below and navigate through them with your left and right arrow keys.

View all slides in new window Download PDF of all slides

Tip

Fun fact: If you type ? (or shift + /) while going through the slides, you can see a list of special slide-specific commands.

Videos

Videos for each section of the lecture are available at this YouTube playlist.

• Introduction
• Construct validity
• Statistical conclusion validity
• Internal validity
• External validity

You can also watch the playlist (and skip around to different sections) here:

In-class stuff

Here are all the materials we’ll use in class:

• Session 6 FAQ slides (PDF)

• Tutoring CATE example

  library(tidyverse)
  library(readxl)
  library(broom)
  
  tutoring_data <- read_excel("tutoring_cate.xlsx", sheet = "raw_data")
  
  model_naive <- lm(outcome ~ program, data = tutoring_data)
  tidy(model_naive)
  
  model_adjusted <- lm(outcome ~ program + low_grade, data = tutoring_data)
  tidy(model_adjusted)

Art → Life expectancy DAG we made together

Art → Life expectancy

Confounding, unblocked

Confounding, blocked

Mediator

Collider

References

Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice. 2nd ed. Washington, DC: Inter-American Development Bank and World Bank. https://doi.org/10.1596/978-1-4648-0779-4.

Huntington-Klein, Nick. 2021. The Effect: An Introduction to Research Design and Causality. Boca Raton, Florida: Chapman and Hall / CRC. https://theeffectbook.net/.

Imbens, Guido W. 2021. “Statistical Significance, p-Values, and the Reporting of Uncertainty.” Journal of Economic Perspectives 35 (3): 157–74. https://doi.org/10.1257/jep.35.3.157.