Difference between Bayesian and Frequentist inference with simplest analogy
There are many articles which claim that Bayesian is preferable over frequentist inference, which is highly questionable.
The basic problem lies in the improper application of these two concepts. The purpose of each of them is different.
Let’s dive into examples. Note the simple English words I am highlighting.
Phenomenon: Direction of Sun Rise
Theory 1: Sun rises in the east (OR)
Theory 2: Sun does not rise in the east.
Truth: Here only one of them is true(but not both).
How do you to test this? : You observe the sun (repeatedly every day). Only one of the theories turns out to be true
These are the kind of phenomenon (that only one of the hypotheses is true which we don’t know) that Frequentist setup deals with.
Probabilistic Interpretation: Now, can we say, ‘ when I observe on one day, Sun rises in the east and on some other day ‘sun does not rise in the East?’
No Right? I know you will punch me :-) if I say anything of that sort. So, the above phenomenon has no probabilistic interpretations(in other words it is not a random variable)
Note: If theory1 and theory2 are NOT mutually exclusive and exhaustive, there is one more test called likelihood ratio rest. We can discuss it another article.
Judgement error- Statistical conclusions from sample: If for example, someone observes the Sun to be in West at 5pm in the evening, but mistakenly updates the time as 5am — based on this sample, we might make wrong conclusions. There is a probability of error in statistical conclusions. We are at the mercy of the data. So, a large number of samples is recommended!!!
Phenomenon: Roll A die.
Event 1(**Not theory**): Observing even number.
Event 2(**Not theory**): Observing odd number.
Frequentist interpretation: Can we say, ‘when we roll a die, only one of them is true but not both’?
No Right? When we roll a die, ‘sometimes we get even, sometimes we get odd’ — that means there is a probability.
When the Hypothesis (Events) are not certain/have a probability, then Bayesian set up is an appropriate choice, if you know the prior probability of the events. That is why we have to take into account prior probability of an event into account. Because when you are doing an experiment, you never know which event you are going to get/draw.
In other words, Bayesian setup assumes that the hypothesis is a random variable (has probability)
Another Difference: Frequentist setup (Hypothesis test) is more focused on drawing conclusions — that the Sun rises in the East or not whereas Bayesian setup is more focused on probabilities of something happening.
Another Difference: Frequentist setup deals only with binary theories (either this or that) Bayesian can deal with multiple events (not just 2)
Scientific papers: Scientific papers work more on proving/disproving theories. So, you can see Hypothesis Test (Frequentist Setup) being prevalent in scientific papers.
