Anti-orthogonality at Freakonomics

In one of the many recurring gags on the Beverly Hillbillies, whenever Jethro finished fixing the old flatbed truck, Jed would notice a small pile of engine parts on the ground next to the truck and Jethro would nonchalantly explain that those were the parts that were left over. I always liked that gag and the part that really sold it was the fact that the character saw this as a natural part of auto repair: when you took an engine apart then reassembled it you would always have parts left over.

Sometimes I find myself having a Jed moment when I read certain pop econ pieces.

"What's that pile next to your argument?"

"Oh, that's just some non-linear relationships, interactions, data quality issues and metrics that won't reduce to a scalar. We always have a bunch of stuff like that left over when we put together an argument."

I had one of those moments recently when I read this Freakonomics post by Dave Berri. Here's the key passage:

Despite what seems like a clear endorsement by the customers of this industry, the Avengers was ignored by the Oscars. Perhaps this is just because I am an economist, but this strikes me as odd. Movies are not a product made just for the members the academy. These ventures are primarily made for the general public. And yet, when it comes time to decide which picture is “best,” the opinion of the general public seems to be ignored. Essentially the Oscars are an industry statement to their customers that says: “We don’t think our customers are smart enough to tell us which of our products are good. So we created a ceremony to correct our customers.”

Andrew Gelman has already pointed out the odd mix of descriptive and normative here (and I think Joseph may have a post in mind that looks at underlying Randian attitudes about the rightness of the markets), but what struck me was how strange this seemed from a statistical standpoint.

Right now we have two metrics that measure related properties based on different data. Though correlated (lots of big hits like Titanic have won major Oscars; relatively few flops have been so honored), these metrics often produce different rankings. This strikes Berri as a problem.

Note, we're not talking  about the quality of these metrics, which are not that good (the Academy has serious issues while box office is confounded with factors like marketing, release date and number of screens), nor are we talking about the Academy's often discussed bias against certain genres.  Those would be valid grounds for criticizing the awards (though I'm not sure how they would figure into a pop econ framework).

Berri is saying that metric B should incorporate metric A to make B more consistent with A. From a statistical standpoint, this is simply a bizarre statement. Statisticians want different variables to tell us different things. Assuming we wouldn't be able to disaggregate the role of box office in these new Academy awards,  Berri's suggestion actually reduces the information in the system.

This is not an entirely abstract point. Movie goers do use the Oscars to make decisions as consumers.

Oscar-nominated films remain in theaters about twice as long as others, according to a report by Randy Nelson, professor of economics and finance at Colby College.
...

Nelson found that a nomination for Best Actor or Best Actress increases box office revenue by about $683,660 (we adjusted the values from the 2001 report to 2012 dollars). For Best Picture, the boost jumps to $6.9 million.
...

Taking home a big award has an even greater impact: Based on Nelson’s study, a Best Picture win boosts box office sales by $18.1 million, on average, and a Best Actor or Actress win by $5.8 million. Even a Supporting Actor or Actress award increases sales by $2.3 million.

Just to sum things up, Berri is suggesting that we should reduce the quality of a data source that consumers make extensive use of because, since the data sometimes doesn't align with consumers' previous revealed preference, that data is somehow insulting to those consumers.

In terms of the Oscars, this is a trivial discussion. (In terms of the Oscars, pretty much all discussions are.) Somewhat less trivial, however, is the accompanying discussion of the Freakonomics school of pop economics, currently one of the dominant influences on science writing for the mass audience. Writers of this school are noted for going into wide-ranging fields and finding interesting and unexpected results that often differ from the previous consensus.  Sometime, though, those results are based not on logical steps you haven't thought of, but on steps you wouldn't think of as logical.