Practical Examples¶
Two sample t-test¶
Load the included Balsakhi data set, which we’ll use to estimate the control mean.
library(pwrcalc)
data(balsakhi)
control_data <- balsakhi[which(balsakhi$bal == 0), ]
control_mean <- mean(control_data$post_totnorm, na.rm = TRUE)
control_sd <- sd(control_data$post_totnorm, na.rm = TRUE)
Let’s inspect the results to make sure we’re all on the same page.
> print(control_mean)
[1] 0.4288781
> print(control_sd)
[1] 1.15142
Let’s say, based on other studies, that we expect an effect size of a tenth of a standard deviation. Now let’s calculate the sample size for our anticipated effect size.
expected_effect <- control_sd / 10
treated_mean <- control_mean + expected_effect
We can now calculate the sample size needed to test that hypothesis at the significance level of 0.05 and power of 0.8.
> twomeans(m1 = control_mean, m2 = treated_mean, sd = control_sd)
Two-sample t-test power calculation
m1 = 0.4288781
m2 = 0.5440201
n1 = 1570
n2 = 1570
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE:
m1 and m2 are the means of group 1 and 2, respectively.
n1 and n2 are the obs. of group 1 and 2, respectively.
Now imagine we anticipate an effect half as large as the previous example. In particular, we now expect 1/20 of a standard deviation.
smaller_expected_effect <- control_sd / 20
smaller_treated_mean <- control_mean + smaller_expected_effect
> twomeans(m1 = control_mean, m2 = smaller_treated_mean, sd = control_sd)
Two-sample t-test power calculation
m1 = 0.4288781
m2 = 0.4864491
n1 = 6280
n2 = 6280
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE:
m1 and m2 are the means of group 1 and 2, respectively.
n1 and n2 are the obs. of group 1 and 2, respectively.
Notice now we need four times as many observations as the previous example.
Two sample t-test with group clusters¶
Many designs randomize at the group level instead of at the individual level. For such designs, we need to adjust our power calculations so that they incorporate the fact that individuals within the same group may be subject to similar shocks, and thereby have correlated outcomes. Duflo et al. presents a modified parametric approach, which takes into account the intra-cluster correlation (ICC) that arises from randomization at the group level.
library(ICC)
icc_sample <- control_data[!is.na(control_data$divid) & !is.na(control_data$post_totnorm), ]
icc_sample$divid = as.factor(icc_sample$divid)
icc <- ICCest(divid, post_totnorm, data = icc_sample)
rho <- icc$ICC
> twomeans(m1 = control_mean, m2 = treated_mean, sd = control_sd) %>% clustered(obsclus = 10, rho = 0.3)
Two-sample t-test power calculation
m1 = 0.4288781
m2 = 0.5440201
n1 (unadjusted) = 1570
n2 (unadjusted) = 1570
rho = 0.3
Average per cluster = 10
Minimum number of clusters = 1162
n1 (adjusted) = 5809
n2 (adjusted) = 5809
sig.level = 0.05
power = 0.8
alternative = two.sided
NOTE:
- m1 and m2 are the means of group 1 and 2, respectively.
- n1 (unadjusted) and n2 (unadjusted) are the obs. of group 1 and 2 ignoring clustering.
- n1 (adjusted) and n2 (adjusted) are the obs. of group 1 and 2 adjusting for clustering.