The Tyranny of 0.05

A moral test of statistical significance. When the stakes are higher than Chrysler dealers, do we have cause to ignore arbitrary definitions of statistical significance?

From: Note on the presidential election in Iran, June 2009 Walter R. Mebane, Jr.

Nonetheless, pending the availability of less highly aggregated counts, it is easy to test the currently available data. A natural test is to check the distribution of the vote counts’ second signi?cant digits against the distribution expected by Benford’s Law (Mebane 2008).

Such a test for the full set of counts for each candidate shows no signi?cant deviations from expectations. In the following table, a test statistic ?2 2BL greater than 21.03 would indicate a deviation signi?cant at the .05 test level (taking multiple testing—four candidates—into account; the critical value for the .10-level test would be 19.0).

The single statistic value of ?2 2BL = 17.08 for Rezaei is signi?cant at the .05 level if the fact that statistics for four candidates are being tested is ignored. So a statistically sharp approach to statistical testing—taking the multiple testing into account—fails to provide evidence against the hypothesis that the second digits are distributed according to Benford’s Law. Tests based on the means of the second digits also fail to suggest any deviation from the second-digit Benford’s Law distribution. But arguably, in view of the ?2 2BL result for Rezaei, it’s a bit of a close call. Given the large aggregates being analyzed, such a close result warrents further examination.

Table 1: 2BL Test Statistics (Pearson chi-squared)
candidate ?2
2BL
Ahmadinejad 6.19
Rezaei 17.08
Karoubi 6.62
Mousavi 14.04

Nate Silver? Are you out there?