Statistical intuition
Statistical phenomena are notoriously difficult to get a good grasp of intuitively. Most people aren’t able to evaluate risks given in percent, nor given as probabilities. I don’t include myself in that group: I am aware of the intricacies and pitfalls involved in comparing probabilities, having always been fascinated by probabilities, so I’ll quickly resort to a calculator or pen and paper when comparing probabilities.
What I can’t deal with satisfactorily, even though I’ve taken a university course in statistics, is statistical testing. I would like to be able to create an A/B testing calculator (such as ABBA), and I’d certainly like to know intuitively under what conditions it’s safe to use:
- Can I continuously check results? (and upon “95% confidence” of the result, stop the test)
- How tightly coupled are the confidence and the number of conversions?
- What are the pitfalls when testing multiple versions at the same time?
- Is it possible to make small changes to the library to support multivariate testing? (i.e. testing multiple factors simultaneously)
And that’s just the beginning. When should I use a student’s t-test? What is regression analysis used for? How do I calculate confidence intervals? Which distributions should I use in which situations? Are any statistics “easy” to calculate (can I write a program to do it somewhat efficiently), or will I need dedicated programming libraries for that? Which libraries and tools should I use for my programming and everyday needs?
To me, statistical intuition means knowing what techniques to use in what situations, not only having a correct hunch about what probabilities mean. If I ever take another course in statistics, my goal will be to confidently be able to respond to all the questions above, and also others that I’m not yet knowledgeable enough to know to ask.