Disproving the Null Hypothesis

The Misconception

Although the null hypothesis cannot be proven true, it can be proven false. This is because science and hypothesis testing are based on the logic of falsification. If someone claims that all swans are white, confirmatory evidence (in the form of lots of white swans) cannot prove the assertion to be true. However, contradictory evidence (in the form of a single black swan) makes it clear that the claim is invalid.


Evidence That This Misconception Exists

The first of the following statements comes from an online document entitled “Converting Research Questions into Statistical Hypotheses.” The second statement comes from an article authored by a medical statistician at the University of Cambridge. The third statement comes from a university’s online study-skills document. (Note the phrases can disprove and to be disproved that appear in the second and third passages.)

"Remember we can never prove the null hypothesis. All we can prove is that there is a relationship or effect (H1) between two or more variables."

"The point is that we can disprove statements, but we can not prove them. This is the principle of disconfirmation, and it forms the basis for scientific inquiry . . . . Now, knowing that we can’t prove a hypothesis but can disprove it, we take the tact of attempting to disprove the null hypothesis. If we are successful then we have, in an admittedly backwards and somewhat convoluted manner, supported our real hypothesis, the alternative hypothesis. While you can’t prove that a statement or hypothesis is true, you can disprove that its opposite is true, thereby obtaining the desired result, provided that there are no possibilities other than your hypothesis and its opposite. It is really a rather ingenious system."

"A null hypothesis is a working hypothesis that is to be disproved by a statistical test in favour of the alternative hypothesis."

Why This Misconception Is Dangerous

The danger in thinking that null hypotheses are proven wrong if rejected is twofold. Both of these dangers are related to the word prove. Although this word has different meanings in different contexts (e.g., mathematics, printing, and cooking), most dictionaries indicate that we prove something when we establish its genuineness or authenticity. Proof, therefore, leaves no room for error. If you prove something, you and others can be 100% confident that your claim is true.

Those who think that null hypotheses can be proven false have mixed together, inappropriately, the logic of falsification and the statistical procedure of hypothesis testing. As will be indicated in the next section, nothing is truly falsified when a null hypothesis is rejected. The observation of one black swan is sufficient to falsify the claim that all swans are white. That single black swan proves that the claim is wrong. It is dangerous to accept or promote the belief that a rejected Ho has been proven wrong because sample data never constitute a black swan.

There is a second danger associated with the belief that null hypotheses can be proven wrong. This concerns the important scientific practice of replication. If a study’s null hypothesis were to be rejected, and if this rejection constituted proof that Ho is wrong, no replication would be necessary. Why bother to replicate a statistically significant finding if the tested null hypothesis has been proven to be wrong?


Undoing the Misconception

If you test a null hypothesis, reject it, and then think that you have proven Ho to be false, you have deceived yourself. To think that a “p < α” resultdisproves the null hypothesis is to forget completely that a Type I errorcan occur whenever the hypothesis testing procedure yields informationthat causes Ho to be rejected.

The only way a particular Ho can be proven false (or true) is to know the precise numerical value of the population parameter(s) specified in the null hypothesis. However, sample data do not provide that kind of information. Instead, summaries of sample data (e.g., the sample mean) are nothing more than estimates of population parameters (e.g., μ), and the two are likely to be different due to sampling error. Therefore, to think that sample-based information can prove Ho wrong is to disregard the inferential guesswork that’s involved in hypothesis testing.

If you flip a fair (i.e., unbiased) coin 10 times, the chances are about 2 in 3 that you’ll end up with somewhere between 4 and 6 heads. However, it’s clear that you might end up with a result that’s more lopsided than this. In fact, there’s about a 2% chance that your 10 flips will produce a 9-to-1 or 10-to-0 split between heads and tails. If you actually got one of these more extreme splits (for which p < .05), would it prove that the fair coin that you’ve been flipping was not fair? Of course not! (See Note 1.)

Researchers are encouraged to replicate their studies, and this concern for replication does not vanish simply because a researcher’s initial study leads to a rejection of the null hypothesis. The objective of replication is to see if the conclusions reached in the first investigation show up again in the second, replicated study. Clearly, the call for replication is based on the awareness that conclusions drawn from the first study might be erroneous. If the initial study had the ability to prove things, no replication would be needed.


Internet Assignment

Would you like to see some proof that Ho is not proven wrong if it is rejected? Would you like this proof to be both easy to understand and totally convincing? Would you like to generate the convincing evidence by means of an Internet-based interactive Java applet? If you would like to see and do these things, do this Internet assignment.

To begin this assignment, go to this book’s companion Web site
(http://www.psypress.com/statistical-misconceptions). Once there, open the folder for Chapter 8 and click on the link called “Disproving Ho.” Then, follow the detailed instructions (prepared by this book’s author) on how to use the Java applet. By doing this simple assignment that involves a computer-based coin-flipping activity, you will come to realize that sample data, by themselves, never can prove a null hypothesis to be false.


Note 1: The binomial distribution was used to determine the “chances” and p referred to in this paragraph.