Thanks for the article.

One of the important aspects here is terminology: a significance test giving a p-value of <0.05 doesn't mean that something is "true" - it means that the difference between the two things being compared was highly unlikely to have occurred by chance. The way that evidence becomes stronger is when the same thing is replicated in further studies. Clinical practice, for example, rarely changes on the basis of one study.

The other important thing in clinical studies is the difference between test-based results and patient-based results. For example, a certain medication may lower blood pressure by a result that is statistically significant, but makes no difference to the patients' clinical outcome (risk of stroke or heart disease, for example).

I'm not sure what your point is. There is nothing evil about p-values or hypothesis testing, although there is plenty of confusion about what they mean.

You take just a glancing blow at one very important question, that of the effect size of a statistically significant experimental result. You begin:

"So what’s the point in showing empirically that the null hypothesis is not true?"

First of all, one never shows "empirically that the null hypothesis is not true." Instead, what researchers do is to evaluate whether their experimental result is statistically significant. If it is, they reject the null hypothesis with confidence and they report the observed treatment effect. What's the point? The point is that while there will almost always be some observed treatment effect (as you said, researchers only test treatments that are likely to help - in humans, in fact, it would be unethical to do otherwise), it will not always be statistically significant. This step of assessing the null hypothesis is important!

Here's an example. A magician claims to have a special way of flipping coins that favors heads over tails. I give the magician a coin I have previously verified to be fair, and I watch her flip it 20 times. Heads comes up 12 of those times. Wow! Maybe the magician does have a trick up her sleeve, because heads came up 50% more often than tails. However, based on the known probabilistic behavior of coin-flipping, a single trial of 20 unbiased flips will result in 12 or more heads about a third of the time (i.e., p > 0.3), so after considering the mathematics of the situation, I decide that the preponderance of heads, even as big as it is at 50% excess, is not very telling. As the experimenter, I will say something like "I can't say that I've seen any evidence of magic here." On the other hand, if I had watched her flip the coin 2000 times and saw heads 1200 times, I would then say it looks like there is some magic (or some real effect, magical or not - sleight of hand, hypnotization of me, or something). In fact, I would stake my reputation on it.

You add: "Researchers who conduct clinical trials need to determine if the effect of treatment is big enough to make the intervention worthwhile, not whether the treatment has any effect at all."

They do. It's called effect size, practical significance, or clinical significance in textbooks, and there are many equivalent discipline-specific quantities that are used in reporting research. It's something completely different than the p-value. First let me clear up what is meant by it.

In both of the experiments I described above (20 and 2000 flips), the observed effect size was the same - heads turned up 50% more often than tails. However, the first result (heads 12 times out of 20 flips) was not statistically significant, while the second was. (The effect size is often measured in standard deviations. In this particular case, the effect size is 0.2 standard deviations.)

Was the effect size great enough that I would want to hire this magician for my traveling circus? I won't say, but I will tell you that when I interview magicians, I make sure I test them thoroughly enough to give myself a fighting chance of being able to notice if they have the magic, and I make sure to require enough confidence in what I see that I don't accidentally hire unmagical performers very often. What should count as "important" varies greatly by discipline. A very small effect size might be mission-critical in airplane design, but not in cavity prevention in adults, but in both disciplines, statistical significance is required. (The required level of confidence to deem a result statistically significant also varies by discipline.)

Good researchers definitely report and evaluate whether a treatment effect is big enough to be worthwhile or not. In fact, they design their experiments carefully (like I design my test of magicians) so that if there is a real and worthwhile effect, they are likely to detect it as a statistically significant result.

It's true that some researchers downplay the effect size, but that's not the biggest problem in the use and interpretation of statistics. (More often, the ones who downplay effect size are those with an interest in the financial success of a product, or publicity agents who write press releases. Journalists, either because they aren't well-educated about statistics, or because they're looking for a snappy headline, don't always go to the trouble to find and explain the effect size. Also worth noting is that many researchers analyze large retrospective studies done by others that are so large they can detect very insignificant findings but provide confidence that they are real. Good researchers don't report these results as important, but sometimes they end up getting reported.)

Are there better approaches? I'm not sure what you mean by "estimation," but I think it's something that follows very easily from the analysis done with significance and hypothesis testing. Researchers very often report an interval estimate of the real treatment effect. It's called a confidence interval, and it ranges above and below the exact effect observed in the test sample.

As for Bayesian analysis, yes, it's a powerful tool. A hint of what it's about is that while I might not have hired the magician after my single 20-coin experiment, I might have asked her back for a second interview. And Bayesian analysis will help address the question of how to analyze a situation where the underlying "probabilistic behavior" as I called it for the coin (or the underlying processes, in non-probabilistic situations) are unknown. In those situations, the assumptions behind significant and hypothesis testing can be problematic. But as useful as it is, Bayesian analysis has its own issues with designing experiments and explaining results, and it's not a magic bullet.

You wrap up with "Significance testing and hypothesis testing are so widely misinterpreted that they impede progress in many areas of science. What can be done to hasten their demise?" and mention "the evil p."

When something is misinterpreted and misunderstood, the solution is not to murder it. The solution is to understand it and interpret it better. We need to do that with p-values (which are by no means evil, when they come from a good experiment) and hypothesis testing.

I'm glad to see another call for researchers to think carefully about their statistical methods, and especially to consider Bayesian methods as possibly more appropriate for the answers they seek. Linked below is a very recent article for those who want a simple but thorough example of what Bayesian methods can do. The article is titled, "Bayesian estimation supersedes the t test." You can find it, and associated software and videos, at http://www.indiana.edu/~kruschke/BEST/

The article makes good points about the risk of misuse of hypothesis testing. (I recall a magazine article some years ago by a maths prof explaining H.T to the layperson. In his worked example he managed to invert the test completely, declaring 80% for a chance result when his test was for 20%.)
But I've always felt the major obfuscation is that the choice of p value is relied upon to address two unknowns: the prior probability (made explicit in Bayesian analysis) and the costs of error. The choice of criterion for a decision must not be divorced from the costs of a wrong decision. (The article partly addressed this topic in pointing out that the pharmaceuticals tester wants to know the strength of an effect, not merely whether there is one.) In practice, I suspect, most engineers and researchers pick a standard .05 or .01 out of the air, with no real understanding of the consequences.

Thanks
I am an engineer and as such have to deal with rare events and compound multivariate problems. I am regularly required to design systems with 95% confidence of outcome based on a data set of 1 or 2 points. Large data sets only become available in the event of dispute. I currently work in wastewater treatment. We are asked to design Million dollar treatment plants based on a single test. Often because it is impractical to test ore where there is nothing to test. And one must design in a competitive environment where lowest cost wins.

One becomes a data scrounger and Bayesian statistics is the only hope. It is possible and I wouldt say even safe to design on that basis as long as one has a reasonable first estimate and control methods to correct or modify cheaply designs based on partly wrong inputs. II see dodgy stats with p values stated to 2 or 3 significant figures regularly in the water industry testing for causes of disease outbreaks. Often buried in the background is a rigid 'statistical' basis for operation that would never have been allowed to persist if estimates of probabilities were continuously modified as new data arrived. The cause and likely risk of adverse outcome was often evident but not acted upon because the data set was 'small' and results not significant.

I am an ex-scientist who regularly used significance testing in analysing data, mostly from controlled trials where there was one intervention and on an individual experiment basis, the process seems to serve its purpose. If used appropriately, recognising the limitations, I do not really see a problem. I well understand the issues of a null hypothesis and the magnitude of an effect etc.

It would seem that a Bayesian approach is required when one is dealing with more complex, multi-factor whole population analysis where it is inappropriate to try to dissect out individual variables.