Statistical versus practical significance.

Rote application of frequentist testing does not necessarily tell you answers to the questions you’re actually interested in. A partial list of the reasons for this are:

• Sampling variability is inadequate to describe genuine uncertainty
• Rejecting a null is not necessarily what you care about
• The standard statistics are built on untenable assumptions
• None of our modeling assumptions plausibly capture how the data was collected.

This will take the form of a class discussion.

For each example, discuss the following questions. Try to connect the statistical concept to the question you’re interested in. In some cases you may argue that there is no useful interpretation.

• How do you interepret the \(R^2\)?
• How do you interpret the point estimate of the coefficients?
• How do you interpret the confidence interval?
• How do you interpret the “statistical significance” (as reported by R)?
• Which set of modeling assumptions do you think are most reasonable?
• Are these assumptions matched by the statistical analysis performed by lm?

In cases where statistical tools fall short, suggest ways you might adequately represent and report uncertainty.

Bodyfat example

We want to find easy-to-use measurements that act as a proxy for bodyfat during routine visits.

bodyfat_df <- read.csv(file.path(root_dir, "datasets/bodyfat/bodyfat.csv"))
ggplot(bodyfat_df) +
  geom_point(aes(x=Abdomen, y=bodyfat))

bodyfat_reg <- lm(bodyfat ~ 1 + Abdomen + Weight + Age, bodyfat_df)
summary(bodyfat_reg)

Call:
lm(formula = bodyfat ~ 1 + Abdomen + Weight + Age, data = bodyfat_df)

Residuals:
     Min       1Q   Median       3Q      Max 
-11.5795  -3.2875   0.0381   3.1521  10.4377 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -45.951610   2.609749 -17.608  < 2e-16 ***
Abdomen       1.000456   0.066794  14.978  < 2e-16 ***
Weight       -0.151616   0.023850  -6.357  9.8e-10 ***
Age          -0.008198   0.026284  -0.312    0.755    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 4.464 on 248 degrees of freedom
Multiple R-squared:  0.7189,    Adjusted R-squared:  0.7155 
F-statistic: 211.4 on 3 and 248 DF,  p-value: < 2.2e-16

Microcredit example

We would like to use a large randomized controlled trial to estimate the causal effect of microcredit access on foot expenditures.

load(file.path(root_dir, "datasets/microcredit/microcredit_mx_final_project_data.Rdata"))

mx_df %>%
  filter(!is.na(treatment)) %>%
ggplot() +
  geom_histogram(aes(
    x=foodspend, fill=factor(treatment), group=factor(treatment)), bins=100) +
  facet_grid(treatment ~ .)

Warning: Removed 63 rows containing non-finite outside the scale range
(`stat_bin()`).

mc_reg <- lm(foodspend ~ 1 + treatment, mx_df)
summary(mc_reg)

Call:
lm(formula = foodspend ~ 1 + treatment, data = mx_df)

Residuals:
    Min      1Q  Median      3Q     Max 
 -889.4  -374.3   -89.4   210.6 17510.6 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  874.259      6.727 129.972   <2e-16 ***
treatment     15.176      9.524   1.593    0.111    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 611.6 on 16495 degrees of freedom
  (5026 observations deleted due to missingness)
Multiple R-squared:  0.0001539, Adjusted R-squared:  9.33e-05 
F-statistic: 2.539 on 1 and 16495 DF,  p-value: 0.1111

Kleiber

We are interested in fundamental laws governing the relationship between metabolism and animal size.

kleiber_df <-
  read.csv(file.path(root_dir, "datasets/kleiber/kleiber.csv"))

lm_log_fit <- lm(log10(Metabol_kcal_per_day) ~ 1 + log10(Weight_kg), kleiber_df)
summary(lm_log_fit)

Call:
lm(formula = log10(Metabol_kcal_per_day) ~ 1 + log10(Weight_kg), 
    data = kleiber_df)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.98739 -0.07576  0.03289  0.12581  0.48183 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)       1.88313    0.05118   36.79   <2e-16 ***
log10(Weight_kg)  0.65704    0.02747   23.92   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.2582 on 34 degrees of freedom
Multiple R-squared:  0.9439,    Adjusted R-squared:  0.9422 
F-statistic: 572.1 on 1 and 34 DF,  p-value: < 2.2e-16

ggplot(kleiber_df) +
  geom_point(aes(x=log10(Weight_kg), y=log10(Metabol_kcal_per_day),
                 color=Animal, shape=Comparable), size=3)