Some epidemiology for a change

John Cook has an interesting point:

When you reject a data point as an outlier, you’re saying that the point is unlikely to occur again, despite the fact that you’ve already seen it. This puts you in the curious position of believing that some values you have not seen are more likely than one of the values you have in fact seen.
 

This is especially problematic in the case of rare but important outcomes and it can be very hard to decide what to do in these cases.  Imagine a randomized controlled trial for the effectiveness of a new medication for a rare disease (maybe something memory improvement in older adults).  One of the treated participants experiences sudden cardiac death whereas nobody in the placebo group does. 

One one hand, if the sudden cardiac death had occured in the placebo group, we would be extremely reluctant to advance this as evidence that the medication in question prevents death.  On the other hand, rare but serious drug adverse events both exist and can do a great deal of damage.  The true but trivial answer is "get more data points".  Obviously, if this is a feasible option it should be pursued. 

But these questions get really tricky when there is simply a dearth of data.  Under these circumstances, I do not think that any statistical approach (frequentist, Bayesian or other) is going to give consistently useful answers, as we don't know if the outlier is a mistake (a recording error, for example) or if it is the most important feature of the data.

It's not a fun problem. 

Another reason observational epidemiology is hard

John D Cook:

And yet behind every complex set of rules is a paper showing that it outperforms simple rules, under conditions of its author’s choosing. That is, the person proposing the complex model picks the scenarios for comparison. Unfortunately, the world throws at us scenarios not of our choosing. Simpler methods may perform better when model assumptions are violated. And model assumptions are always violated, at least to some extent.
 

 One of the hardest things with simulation studies is that we get to develop our own set of assumptions.  So we actually know how to correctly model the phenomenon of interest. 

The problem is that we usually do not know how much error is introduced when the complex (and often non-linear) model fails.  On the other hand, it is amazing how far one can get with a clear set of rules of the thumb. 

I wonder if it would be better if we had a different person test the model than the one who proposed it? 

Evidence

John D Cook has a very nice post up about evidence in science:

Though it is not proof, absence of evidence is unusually strong evidence due to subtle statistical result. Compare the following two scenarios.

Scenario 1: You’ve sequenced the DNA of a large number prostate tumors and found that not one had a particular genetic mutation. How confident can you be that prostate tumors never have this mutation?

Scenario 2: You’ve found that 40% of prostate tumors in your sample have a particular mutation. How confident can you be that 40% of all prostate tumors have this mutation?

It turns out you can have more confidence in the first scenario than the second. If you’ve tested N subjects and not found the mutation, the length of your confidence interval around zero is proportional to N. But if you’ve tested N subjects and found the mutation in 40% of subjects, the length of your confidence interval around 0.40 is proportional to √N. So, for example, if N = 10,000 then the former interval has length on the order of 1/10,000 while the latter interval has length on the order of 1/100. This is known as the rule of three. You can find both a frequentist and a Bayesian justification of the rule here.

Absence of evidence is unusually strong evidence that something is at least rare, though it’s not proof. Sometimes you catch a coelacanth.

Now it is true that this approach can be carried too far. The comments section has a really good discussion of the limitations of this type of reasoning (it doesn't handle sudden change points well, for example).

But it is worth noting that a failure to find evidence (despite one's best attempts) does tell you something about the distribution. So, for example, the failure to find a strong benefit for users of Vitamin C on mortality, despite a number of large randomized trials, makes the idea that this drug is actually helpful somewhat less likely. It is true we could look in just one more population and find an important effect. Or that it is only useful in certain physiological states (like the process of getting a cold) which are hard to capture in a population based study.

But failing to find evidence of the association isn't bad evidence, in and of itself that the association is unlikely.

P.S. For those who can't read the journal article, the association between Vitamin C and mortality is Relative Risk 0.97 (95% Confidence Interval:0.88-1.06), N=70 456 participants (this includes all of the trials).