Wednesday, September 29, 2010

Ecological Fallacy

The recent debate on education has been pretty broad and I wonder if a key point has been overlooked. In the cross country comparisons, Mark has been arguing that "even if we grant that cross country comparisons are useful, they don't necessarily say what you think they say".

This has led to overlooking the key problem in these comparisons: it is nearly impossible to relate the performance of a specific country to that of individual students because the make-up of the students differ. The real question we want to ask is: if we adopted the educational system of country X then would our students do better? But that the student in country X do better than Americans is no proof -- maybe they are handicapped by an inferior educational system and would do even better in an American style educational environment.

This problem is a key limitation in epidemiological studies of all kinds. If we observe that the Japanese have fewer myocardial infarcts then we still don't know why. There are many exposures that could explain this finding. It is easy to go wrong. For example, in 2002-2003 47% of Japanese men smoking and 20% of American men smoked.

Yet in 2002, the age standardized death rate for American men from CHD was 216 per 100,000 whereas it was 54 per 100,000 in Japan. (now this association is cross sectional but most cross country educational debates are as well). Does this mean we should promote smoking to reduce heart disease?

If the answer is "no" then we should apply similar caution to other international comparisons.

No comments:

Post a Comment