More hard for me to know

Further to reference 1, quite coincidentally, the very same correspondent has pointed me to a clutch of relatively accessible papers from the Royal Society stable about the very same p-values which came up in the course of that first post. That is to say, references 2, 3 and 4. A subject which must be big to make it to the Royal Society and which has also spawned a number of entertaining blogposts and videoclips. Readers are left to find those, for themselves, as an exercise.

So while I expect I would still be unable to form a view on the paper which was the subject of that first post, at least I can now feel that I am on top of the p-value game. I offer a few nuggets.

The basic scenario goes as follows: we have two populations, with one of the populations taking some new medicine and the other taking a placebo. After a while, you test all their blood for factor F, with the result of each test being some positive real number. It is understood that there is plenty of noise present, that there is going to be plenty of random, but hopefully modest, variation in all these measurements of F. So against this noisy and sometimes confusing background, the big question is whether the values of F are significantly different in the two populations, bearing in mind that a lot of money has already been spent and careers are hanging in the balance.

So one takes the two means and does stuff with the normal distribution or the Student-t distribution, with the offending statistical test being the computation of the probability that, on various statistical assumptions and the hypothesis that the new medicine is actually a placebo too, that the results observed would arise by chance. If this probability is small, then this hypothesis is false, the medicine works and we have hit the jackpot.

The basic problem is that this probability being small is actually quite a poor proxy for the likelihood that one’s new medicine is indeed a new medicine. But its computation is relatively easy, the technique is well established and people still use it, despite much muttering over the years from statisticians – with an early such muttering being found at reference 7. Helped along by an unhealthy dose of wishful thinking.

With one result being that lots of exciting stuff is being published which fails to replicate. The fine new medicine invented by laboratory A does not work in laboratory B. With a further risk being that people believe the exciting stuff without bothering with replication, replication not being particularly career enhancing when compared with exciting stuff.

The present papers are part of a drive to put all this right, with another part of this drive being a good dose of Bayes, a gentlemen not much talked of in my student days. See references 5 and 6.

PS: I have also learned, from Wikipedia, that the student of the famous Student-t distribution was the pseudonym of a bright young chemist from Oxford (snapped above) working for Arthur Guinness in Dublin at the beginning of the 20th century. Working on small sample aspects of the statistics of brewing the famous stout. I wonder now if I was ever told this in the course of my short foray into that kind of statistics, fifty years ago now. Certainly don’t remember any such thing, the lectures in question being notably dry, filling large blackboards with quantities of remarkably small writing. I think the idea was that we copied it all down.

References

Reference 1: http://psmv4.blogspot.com/2019/06/hard-for-me-to-know.html.

Reference 2: The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum? - Lewis G. Halsey – 2019.

Reference 3: The reproducibility of research and the misinterpretation of p-values - David Colquhoun – 2017.

Reference 4: An investigation of the false discovery rate and the misinterpretation of p-values - David Colquhoun – 2014.

Reference 5: http://psmv2.blogspot.com/2015/09/bayes-1.html.

Reference 6: http://psmv2.blogspot.com/2015/09/bayes-2.html.

Reference 7: Why Most Published Research Findings Are False - John P. A. Ioannidis – 2005. An early warning shot. Which I believe made quite a stir at the time, first came to my notice about three years ago – and than sank without trace. Maybe it will all stick this time.