Logic of Hypothesis Testing

Logic of Hypothesis testing goes more inline with the Falsification principle, proposed by Karl Popper. It suggests that for theory to be considered scientific, it must be able to be tested and conceivably proven false. For example, the hypothesis that 'all swans are white' can be falsified by observing a black swan'.

Central question in the philosophy of science was distinguishing science from non-science.

For Popper, science should attempt to disprove a theory rather than attempt to continually support theoretical hypotheses. Popper replaced inductive reasoning with deductive reasoning for distinguishing scientific theory from non-science, as we do not observe the universe at all times and in all places.

Inductive Reasoning: observation > pattern > hypothesis > theory

Deductive Reasoning: theory > hypothesis > observation > confirmation

Poppers’ point is, no matter how many observations are made which confirm a theory, there is always the possibility that a future observation could refute it.  Induction cannot yield certainty. Science progresses when a theory is shown to be wrong and a new theory is introduced that better explains the phenomena. 

For Popper, the scientist should attempt to disprove his/her theory rather than attempt to prove it continually. Popper does think that science can help us progressively approach the truth, but we can never be certain that we have the final explanation.

(Thomas Kuhn argued that science does not evolve gradually toward truth. Science has a paradigm that remains constant before going through a paradigm shift when current theories can’t explain some phenomenon, and someone proposes a new theory).

Critics of Karl Popper, chiefly Thomas Kuhn, Paul Feyerabend, and Imre Lakatos, rejected the idea that there exists a single method that applies to all science and could account for its progress.

If you want to be able to test theories that predict no effect, or when you want to be able to falsify theory that predict an effect, it is important to be able to provide support for the null-hypothesis. 

Thus, Don't simply concluding that a p > 0.05 means there is no effect - instead, provide quantitative arguments for conclusion that there is no effect by using equivalence tests or Bayer Factors. 

You will always have to make assumptions about the alternative hypothesis, either by specifying an equivalence region consisting of a range of effect sizes you find meaningful, or by specifying a prior distribution.

You can use either Frequentist or Bayesian approaches to test absence of evidence. 

Reference:

Karl Popper: Theory of Falsification (simplypsychology.org)

Thomas Kuhn: Paradigm Shift Definition & Examples (simplypsychology.org)

Introduction | Improving Your Statistical Inferences (lakens.github.io)

'Module 1:7 - Statistical Inference I' in  "Significantly Statistical Methods'  a Free Online Statistics Course with JMP Software. Accessed at Significantly Statistical Methods Online Course | JMP