What makes statistical modeling different from, say, new age methods of trying to predict the future?
Foretelling the birth of a child to a young woman used to be the domain of angels, but a few years ago the statistical models at Target started making those predictions too. When a teenager received diaper coupons in the mail, her father was outraged, then chagrined when it turned out Target knew more about his daughter than he did.
You may also remember when IBM’s Watson mistakenly decided Toronto was the response in the Final Jeopardy category “U.S. Cities.” In the long run, it was a puzzling quirk for an otherwise dominant Jeopardy! contestant. But it was still striking; even if you didn’t know the correct response, you’d know Toronto was not in the U.S.
Statistical models rely on correlations–when A happens, B also tends to happen. To be useful for prediction, there should be a causal connection between A and B. It can be indirect; buying unscented hand lotion and cottonballs doesn’t cause subsequent diaper purchases, but both (apparently) share an underlying cause in the form of a baby. When there’s no causal link, the correlation may be accidental or highly contingent and thus not useful for predictions. For example, there has been a correlation between which conference, NFC or AFC, wins the Super Bowl and which party’s candidate wins the following US presidential election. There’s no causal link there, so we can expect that once we’ve had enough Super Bowls in election years the correlation will go away, but until then we will like remain fascinated.
Statistical models and correlations are only as good as the data they reference. Since humans have access to more or different data, we know about different kinds of relationships and can draw different inferences. This is why statistical models sometimes give predictions or answers that seem ludicrous to humans. A psychic on the other hand may not make predictions from a rigorously validated model, but they will also never give an inhuman answer. They would have enough data on human psychology (at the very least, informal data based on a lifetime of interacting with other humans) to know that fathers of teenager daughters tend to be sensitive about subjects like pregnancy. They would also have a more nuanced understanding of how humans use categories than Watson had and so would never guess “Toronto.” Apparently Watson observed a weak overall correlation between Jeopardy! categories and correct responses and so largely disregarded the category names. With more data, it might have inferred that some categories, like “cities,” have fuzzy borders, while other categories, like “the United States,” do not.
Does that human touch make psychic predictions more reliable? No, but it probably makes them seem more appealing. When statistical models are wrong in inhuman ways, we probably weight those failures more heavily than human failures simply because they are surprising and inexplicable. To evaluate all predictions fairly, we need to be aware of our data limitations. In our experience, certain kinds of wrong answers may indicate limited education or intelligence, but maybe that’s just a coincidental correlation that only occurs when dealing with humans.
Andy has worn many hats in his life. He knows this is a dreadfully clichéd notion, but since it is also literally true he uses it anyway. Among his current metaphorical hats: husband of one wife, father of two teenagers, reader of science fiction and science fact, enthusiast of contemporary symphonic music, and chief science officer. Previous metaphorical hats include: comp bio postdoc, molecular biology grad student, InterVarsity chapter president (that one came with a literal hat), music store clerk, house painter, and mosquito trapper. Among his more unique literal hats: British bobby, captain’s hats (of varying levels of authenticity) of several specific vessels, a deerstalker from 221B Baker St, and a railroad engineer’s cap. His monthly Science in Review is drawn from his weekly Science Corner posts — Wednesdays, 8am (Eastern) on the Emerging Scholars Network Blog. His book Faith across the Multiverse is available from Hendrickson.
Leave a Reply