united-states legal-terms statistics
Does &quot;statistically significant&quot; have a specific legal meaning?
<p>I occasionally write expert opinions as a statistician or econometrician for legal cases in the United States.</p>
<p>On a few different cases, I have been hired to determine whether a given sample should be considered statistically significant, usually at the request of a judge. In other words, does it have enough observations/records?</p>
<p>In standard statistical terminology, this is nonsense. A <em>sample</em> can't be statistically significant or not. Statistical significance is something that applies to the results of hypothesis tests, not samples, and the same sample can be used to conduct hypothesis tests that turn out statistically significant and also tests that turn out not to be.</p>
<p>So far, I've just assumed that what they really want is for me to calculate the margin of error, and I do so. I've had no complaints.</p>
<p>But seeing the same thing multiple times makes me wonder if I'm missing something. Is there a specific legal meaning of &quot;statistical significance&quot;, separate from the typical statistical meaning, that I should be looking at?</p>

<h2>Most judges went to law school</h2>
<p>Therefore, my null hypothesis is that they have next to no knowledge of statistics and couldn’t tell a median from a mean on their best, let alone, their average day - mean, median, or mode.</p>
<p>Courts turn to experts to fill in the gaps in their knowledge. If they are asking the wrong questions, then it is the expert’s job to, respectfully, tell them so and guide them to the right questions. However, you shouldn’t guess what the right question is.</p>
<p>“Excuse me, your honour, the term “statistical significance” can’t be applied to a data set, only to the result of an a priori null-hypothesis tested against that data set.” Feel free to explain p hacking if necessary.</p>
<p>Ask the judge to explain, in layman’s term, what they want from you. Feel free to admit that just like you don’t understand their legal jargon, you don’t expect them to understand your statistical jargon; if everyone speaks plain English, you can go and do your statistics thing and they can go and do their law-talking thing.</p>

<p>You may be interested in <a href="https://elibrary.law.psu.edu/cgi/viewcontent.cgi?referer=&amp;httpsredir=1&amp;article=1054&amp;context=fac_works" rel="noreferrer">this article</a>, which asks if proof of statistical significance is relevant in law (US) – it clearly is, but still, nobody understands it. It is packed full of relevant legal case citations.</p>
<p>Apparently, this peculiarity didn't exist until two Supreme Court cases in 1977, <a href="https://supreme.justia.com/cases/federal/us/430/482/" rel="noreferrer">Castaneda v. Partida</a>, 430 U.S. 482 and <a href="https://caselaw.findlaw.com/us-supreme-court/433/299.html" rel="noreferrer">Hazelwood v. US</a>, 433 U.S. 299. In the former case, the reasoning of the case relied on the observation that &quot;The discrepancy between the expected and observed values is more than 12 standard deviations&quot;, and in the latter case, that &quot;The difference between the observed and expected values was more than six standard deviations in 1972-1973 and more than five standard deviations in 1973-1974&quot;, citing the earlier Castenada case. It is worth noting that the court talks of statistics, and significant discrepancies, but does not use the expression &quot;statistically significant&quot;.</p>
<p>Then, in <a href="https://m.openjurist.org/690/f2d/1078" rel="noreferrer">Moultrie v. Martin</a>, 690 F.2d 1078, the court held (as &quot;the main legal conclusion&quot;, not a throw-away comment) a rule that &quot;in all cases involving racial discrimination, the courts of this circuit must apply a standard deviation analysis such as that approved by the Supreme Court in Hazelwood before drawing conclusions from statistical comparisons.&quot; (They also note the relationship between confidence level and statistical significance.)</p>
<p><a href="https://www.law.cornell.edu/rules/fre/rule_403" rel="noreferrer">FRE 403</a> is an evidentiary rule that can be used to preclude admitting relevant evidence if it confuses the issues or misleads the jury. (Of course, a witness can at most point out this connection to the relevant attorney who will then move to exclude, or not.)</p>

<p>The customary threshold for, and most common meaning of the phrase &quot;statistical significance&quot; in the sciences and social science is a difference of at least two standard deviations (a.k.a. two sigma a.k.a. a &quot;p-value&quot; of 0.05 or less) from a null hypothesis.</p>
<p>A related phase is that data that is not statistically significant is &quot;consistent with&quot; the null hypothesis.</p>
<p>The law doesn't have special definition of the term, but expert testimony could usually be adduced to that effect, for example, for the purpose of contract interpretation if a contract requires a statistically significant result without defining the term.</p>
<p>An obvious place for experts to duel would be over whether one is looking for &quot;local&quot; statistical significance considering only the data set at hand, or &quot;global&quot; statistical significance, which also takes into account the &quot;look elsewhere effect&quot; (i.e. the notion that if you run enough experiments, some of them will be locally statistically significant as a result of random chance even if the null hypothesis is true).</p>

<p>&quot;Statistical significance&quot; doesn't have any special legal meaning. It seems most likely that your interpretation is correct, as an inquiry into how reliable the data set is would speak to its reliability, which is part of the <em><a href="https://www.law.cornell.edu/wex/daubert_standard" rel="nofollow noreferrer">Daubert</a></em> test for admitting expert opinions.</p>

<p>If the legal definition of &quot;statistically significant&quot; is different from the statistical one, then it is very likely to be a deeply misleading one, and one that suggests the evidence is much stronger than it actually is.  In this case, described in the OP, the question about statistical significance is clearly from someone that has no idea what it actually means, and the best approach is to see if the question can be posed in layman's terms avoiding statistical jargon, so it is clear what they really want to know.  Dean's answer is a very good one.  However, telling people that they are wrong sometimes goes down badly, but I would <em>hope</em> that judges and lawyers would have the good sense to allow themselves to be corrected by a domain expert when they are wrong.</p>
<p>I thought it might be useful to explain <em>why</em> this phrase is likely to be misunderstood - it is because statisticians (at least those performing null hypothesis statistical tests) and laypersons are likely to have fundamentally different ideas of what is meant by a probability.</p>
<p>The original definition of a probability in statistics was similar to the everyday meaning - it is a numerical indication of the relative plausibility of a proposition, e.g. there is a probability of 30% that it will rain today.  However, there is a &quot;subjective&quot; element to this, as the plausibility may depend on your prior beliefs (or equivalently your state of knowledge).  A new branch of statistics came about, called &quot;frequentism&quot; in the 20th century that aimed to eliminate this &quot;subjective&quot; element and have a purely objective means of performing inference.  They did this by defining a probability <strong>only</strong> in terms of long-run frequencies.  This is perfectly reasonable, but it does mean you cannot assign a probability to a particular event, only to populations of events.  For instance, a frequentist fundamentally <em>cannot</em> tell you the probability of a particular DNA match being a  false-positive - it either is a false-positive or it isn't - it doesn't have a non-trivial long run frequency, so you can't assign a probability to it.  So what do we do?  Instead of &quot;probability&quot; we speak of &quot;confidence&quot; (as in &quot;confidence interval&quot;) or &quot;significance&quot; (as in &quot;statistically significant&quot;).  I think originally, this was intended to <em>highlight</em> that it <strong>isn't</strong> a probability and avoid misinterpretations, but sadly this is not how it has turned out.</p>
<p>The key problem is that we normally want to ask a question about a particular case (e.g. <strong>this</strong> DNA match), but a frequentist statistician cannot answer that question, so they substitute an answer about some (often fictitious) population (e.g. on average over a large number of DNA tests of which this one is representative in some way that is most often left explicitly unspecified).  The questioner hears this and interprets it as an old-fashioned (degree of plausibility) based answer about the specific case - because that is what they were expecting (not unreasonably).</p>
<p>So if there is a legal definition of &quot;statistical significance&quot;, it is very likely based on this &quot;degree of belief&quot; definition of probability, and be a bad misinterpretation of what was actually meant by the person that conducts the test.  This isn't particularly the fault of the legal profession, statisticians often shy away from explaining what things mean because the audience is often very hostile to it.</p>
<p>Bayesian statistics is likely to be a better match for the needs of the Law as it is based on the same basic definition of a probability that the layperson uses, and hence is less likely to be misinterpreted.</p>
<p>The way statistical hypothesis test <strong>should</strong> be used is as a minimal hurdle that prevents you from making a fool of yourself by getting carried away with your research hypothesis (the thing you <em>want</em> to be true).  A statistically significant outcome just means you have stumbled over the hurdle somehow and you can still talk about your research hypothesis.  If it wasn't significant, you should not talk about it as if it were true (yet).  It really doesn't mean very much, and is most useful to your when non-significant.</p>

<p>The definition in law - if it exists - would have to be compatible with what statisticians understand by that phrase. Otherwise statisticians would not be called in as expert witnesses in such cases.</p>
<p>But you <strong>must</strong> find a layman's phrasing for this concept so that everyone in court is on the same understanding as you.</p>
<p>It is also likely that the attorney engaging you will also need an explanation of what this phrase means (and indeed any other statistics jargon you propose to use during your expert witness statement) so he can counter any efforts by the opposing attorney to undermine your assertions.</p>
<p>It's hard at the beginning to explain to people from a humanities/arts/business primary degree background how statistics work. But it has to be done lest you end up tied in knots at court. Use everyday examples simple words to present the concept and take them through it slowly.</p>
