Bayesian Statistical Hypothesis Testing for HCI

01 Jan 2012 - Michael Tsikerdekis

Disclaimer (Please read this first!)

Evidence for and against the null hypothesis is possible :-)

I started this wiki so that I can try and gather as many procedures(and code) as I can that currently exists in Bayesian statistics. The goal is to create an easy to read, easy to apply guide for each method depending on your data and your design. Although this is geared towards HCI research, most of these methods can be applied in other scientific disciplines such as social sciences, psychology and others. The philosophy behind this guide is to always keep things simple. Just as I don't ask for my visitors on this website to understand HTTP requests, the same should apply for someone that wants to perform Bayesian statistics. You only need to know what is your input, and how to interpret the output. Therefore, the emphasis here is taken away from the math aspects of bayesian statistics.

My inspiration for developing such content was the site Statistics for HCI Research by Koji Yatani. It is an excellent guide for NHST analysis for HCI.

Keep in mind that I am not an expert of statistics. The contents provided here is basically what I learned from my experience of HCI research and by reading different online/offline materials. I always double-check the content before posting, but it still may be not 100% accurate or even wrong. So, use the contents on this website at your discretion. I own no responsibility on any kind of consequences, such as you have done a wrong analysis after reading my wiki or your papers do not get into a conference or a journal, or your adviser doesn't like your analysis. 

I also strongly recommend you get a second opinion on your analysis from other kinds of resources before you really perform a test. If you have found any factual errors, please email me([email protected]). Your comments would be greatly appreciated. Also, I am always looking for R(matlab,stata) code that can perform hypothesis testing so don't hesitate to let me know about it.

Basics of statistics (A quick introduction to things you need to know)

There are 4 types of variables that you need to know and identify. 

• Interval/Numerical/Ratio are ordered sets of data (usually numbers) that maintain equal distance between their space (e.g., the distance between 2 and 3 is equal to the distance between 3 and 4).
• Ordinal are ordered sets of data that do not show an equal distance between their elements. (e.g., "very strong" is definitely higher than "strong" and the same applies for "extremely strong" but the distance between this elements is not necessary equal.)
• Nominal/Categorical are sets of data with no order (e.g., countries is a good example).
• Dichotomous are categorical variables that have only two levels (e.g., sex can have values only male and female.)

You will also need a general understanding of the Bayes Factor. However, I have connected the link to every procedure's interpretation section as well.

Finally, Bayesian procedures have their pros and cons just as NHST analysis(guide development in progress) BUT the single most appealing thing for me is the power to provide evidence for the null hypothesis. Yes, with Bayesian methods you can do it!
 

What statistical test should I use?

While with NHST analysis answers are straight forward, Bayesian statistics is still a field under development. This is especially true when it comes to hypothesis testing. The following is a set of techniques that I managed to gather.