Simplifying the Juror Accuracy Analysis

Anne Reed at Deliberations wrote today about Spencer's analysis of the NCSC study data, summarizing some of the reactions across the blawgosphere. Having slept on the report, I'll address a couple of the criticisms. First, jury critics call for an end to the jury system. This analysis does not support that. Spencer estimates that juries are 87% accurate by assuming that judges are equally accurate. The analysis does not show that judges ("experts") are as accurate as juries, much less that they are more accurate than juries. Second, jury fans criticize Spencer for deciding what juries are right and what juries are wrong. This criticism is inappropriate. This 13% inaccuracy does not depend on whether we are interested in legal (procedural) guilt or factual ("omniscient") guilt. If we make three assumptions, the accuracy of juries can be mathematically estimated, using nothing more than high school math. The three assumptions are:

• that the judges and the juries in the NCSC study were equally accurate;
• that the NCSC data are correct (including that judges accurately reported their opinions); and
• that the judges' opinions and juries' verdicts were independent

If we assume that these are true, we can estimate their accuracy without any statistical math whatsoever, and without deciding whether we're talking about factual or legal guilt. As Scott Greenfield points out, the third assumption -- that judges' opinions and juries' verdicts were independent -- is probably not warranted:

[T]he impact of a judge on the likelihood of a jury erroneously convicting a defendant cannot be overlooked. Obviously, the judge has a huge impact on both the evidence to come before the jury by dint of his rulings, but also the jury's appreciation of that evidence because of the judge's tone, commentary, facial expressions and non-verbal cues.

In other words, a hanging judge (or even just a good ol' judge who thinks the defendant guilty) is going to impart that message to a jury, and by doing so will dramatically increase the likelihood of a wrongful conviction of an innocent defendant. The judge sets the tone, and that tone is a huge factor in how the jury will receive your evidence and argument.

The judges' opinions might also depend on the juries' verdicts. Sometimes judges might pretend that they agree with the jury's verdict; this would be positive dependence. Sometimes judges might pretend that they disagree with the jury's verdict; this would be negative dependence. (Judges are politicians, and they follow the wind while trying to appear tough on crime.) Here is the simple math that explains Spencer's estimate: There was disagreement between the judge and jury in 23.1% of (.231) cases. Disagreement between judge and jury happens either when the jury is right and the judge is wrong, or when the judge is right and the jury is wrong. If x is the jury's accuracy expressed as a decimal (1 = perfect accuracy), then the judge's accuracy is also x. Broadly speaking, the chance that the jury is wrong on a given case is x and the chance that the judge is right on a given case is 1-x. So the .231 cases represent x(1-x) cases where the jury was right and the judge was wrong and x(1-x) cases where the judge was right and the jury was wrong.

.231 = 2(x(1-x)) = 2x-2x^2.

-2x^2+2x-.231 = 0 -- A quadratic equation!

Plugging a=-2, b=2, and c=-.231 into the quadratic formula we get:

x= -2 +- sqrt (2.16)

-4

x= 3.47 / 4, 0.53 / 4

x= 0.87, 0.13

So the jury and judge are either 87% inaccurate or 13% inaccurate. There may be mathematical reasons to choose 13% over 87% -- algebra II was too long ago for me to recall -- but, for our purposes, we can note that if the jury and judge are 87% inaccurate, they are doing much worse than chance. If we assume that judges are better -- or worse -- than juries at determining factual guilt, or at determining legal guilt, then the numbers change. But as long as we assume that judges are as good as juries, we can estimate that both, in the NCSC study, dropped the ball on one case in eight.

Technorati Tags: blawgs, jury trial, statistics