The Proper Use of the Null Hypothesis When Discussing Transformers

Setting: James and Jessi are getting ready for work in the morning. Vinnie is sleeping in front of a space heater, being cute.

James: "No sacrifice, no victory! I live by the Witwicky motto!"

Jessi: "What?! Why would you base your decisions on Transformers?"

James: "Seriously? They are robots that are alive. AND they can transform. Why wouldn't I base all my decisions around them?"

Jessi: "Umm... Transformers are not real."

James: "Whoa. You cannot make such an outlandish definitive statement. All you can say is, 'There is not enough evidence to confidently claim Transformers are real'. Thus leaving the possibility open that they DO exist."

Jessi: "Sure. Whatever. There is not enough evidence to confidently claim Transformers are real. I'm leaving for work now."

James 1.

Jessi 0.

==================

What just happened there? Statistics 101. That's what.

The subtly of the debate is around statistical hypothesis testing.

We start with what Jessi foolishly thinks to be true: "Transformers are not real".

We then need to create the null hypothesis, which is the status quo, what Jessi assumes to be true unless we find evidence to suggest otherwise: "Transformers are not real".

If there is sufficient evidence suggesting otherwise, we reject the null hypothesis and accept the alternative hypothesis, which is the opposite: "Transformers are real".

The next step is to gather evidence to reject the null hypothesis.

Generally, the reason why the goal is to reject the null hypothesis is because normally things are being measured, like the weight of sand bags. The chances of finding every single bag to weigh exactly 50 pounds are so small you'll never be able to fully accept it. However, by looking to reject (sand bags are different than 50 pounds), all you need to do is show that there's a substantial difference in weight from 50 pounds ("substantial" being pre-defined by your tolerance for error). In all honesty, most groups doing these types of tests don't actually try to formally reject, they'll just say "sand bags are 50 pounds +/- 1 pound". As long as their sand bags fall within that range, life is good. They're inherently setting up the null hypothesis, just not technically saying it.

Anyways, the whole point of giving the rejection background is to say that typically the null hypothesis is testing something with a large enough accessible population to actually generate a sample and measure some aspect of it (there are lots of bags of sand, and we can measure their weight). The Transformer case is special because, as far as I know, there are not a lot of them around that are accessible, and we're not measuring some aspect on them, but actually counting them. The null hypothesis was not originally intended for this type of question, but it still gets applied regularly (i.e. "Does God exist?") and does kind of work - especially when harassing your wife.

OK here's what we have:

Null hypothesis: "Transformers are not real" (again, assumed as true by Jessi unless proven otherwise)
Alternative hypothesis: "Transformers are real" (must be true if null is rejected)

The next step, would be to set out and try to gather evidence to reject the null hypothesis. Now, we could go into a long winded and mind-blowing discussion on experimental design, but I'll spare you. For now, let's just look at what the two outcomes could be:

Reject the null hypothesis: You, via an elegantly designed experiment that was expertly conducted to minimize error, gather enough evidence to show a substantial difference from the null hypothesis (i.e. I saw a Transformer... on TV!). Therefore, I can confidently reject the null hypothesis and declare the alternative hypothesis, "Transformers are real."

Accept the null hypothesis: You cannot gather enough evidence to show a substantial difference (I know, sounds weird, but remember the original intent of hypothesis testing). Unfortunately for Jessi, the conclusion cannot be as strong as a rejection, "There is not enough evidence to confidently say Transformers are real." There still exists the possibility that a Transformer does exist.

You see, statistics can be a wonderful thing. You can make a claim. Then create a null hypothesis for that claim. You can then gather evidence to see if you can reject the null hypothesis. Now you can see why you want to reject the null hypothesis: you can make much stronger statements. That is why I can confidently claim that Transformers exist, and Jessi can only tentatively say they do not.

For you Atheists out there, I fully acknowledge that I did not provide any evidence either way for this particular debate. I simply walked through the process and reason why Jessi needed to choose her words wisely. There is still the burden of proof, but clearly any rationale person can't flat out deny the existence of Transformers.

Hopefully you also see why it's critically important to set up your null hypothesis correctly. Could you imaging doing it the other way? I would never be able to make a strong statement, and Jessi would never be able to fully check every corner of the Universe, thus never allowing her to make a strong statement either. Meaningless really.

[Image from themoviedb.org - also makes for an epic desktop wallpaper]