Hapgood

A quick follow-up to yesterday’s post on the supposed “death of theory” and its relation to MOOC research — the story thus far is that a number of people sincerely think the “why” doesn’t matter if our sample is big enough and the variables tracked are numerous enough. Here’s a typical quote from Thrun:

One day, Sebastian Thrun ran a simple and surprising experiment on a class of students that changed his ideas about how they were learning.

The students were doing an online course provided by Udacity, an educational organisation that Thrun co-founded in 2011. Thrun and his colleagues split the online students into two groups. One group saw the lesson’s presentation slides in colour, and another got the same material in black and white. Thrun and Udacity then monitored their performance. The outcome? “Test results were much better for the black-and-white version,” Thrun told Technology Review. “That surprised me.”

Why was a black-and-white lesson better than colour? It’s not clear. But what matters is that the data was unequivocal – and crucially it challenged conventional assumptions about teaching, providing the possibility that lessons can be tweaked and improved for students.

Note that last bit — “What matters is that the data was unequivocal”. This is how the End of Theory position appears in print. We don’t know *why* the students did better, but they did better, and the data was so “big” that that’s all that matters.

But the why does matter. Because without the why you can’t generalize from one situation to the next, and you keep repeating the same mistakes. In this case, we could hypothesize three alternate explanations of the phenomenon:

1. The lack of colors leads to a lack of distraction. Students watching the colored slides were processing the colors as meaningful (when they weren’t) and this was subtly hindering their comprehension and recall.
2. Students had a harder time reading the black and white slides. There’s recent research that indicates slight disfluencies in presentation can be desirable (jokingly dubbed “The Comic Sans Effect”), and these disfluencies aid in recall.
3. The cause was not that the slides were black and white, but that the black and whiteness was novel. We know from previous psychological research that the mind attends to novelty, the greater attentiveness led to greater retention.

So which one is it? Udacity doesn’t care. But they should. Because if it’s the second one, then writing slides in black and white is not exactly what you should be focused on. And if it’s the third — the novelty effect — then the impact of this is going to be very limited. This isn’t even getting into variable context, pedagogical aims, or path-dependence.

You see this problem in Big Data all the time. The Obama campaign was really a Small Data operation, but they did extensive A/B testing. And what they found one night was the Holy Grail of campaign email marketing:

It quickly became clear that a casual tone was usually most effective. “The subject lines that worked best were things you might see in your in-box from other people,” Fallsgraff says. “ ‘Hey’ was probably the best one we had over the duration.” Another blockbuster in June simply read, “I will be outspent.” According to testing data shared with Bloomberg Businessweek, that outperformed 17 other variants and raised more than $2.6 million.

The “magic formula”, right? Well, no:

But these triumphs were fleeting. There was no such thing as the perfect e-mail; every breakthrough had a shelf life. “Eventually the novelty wore off, and we had to go back and retest,” says Showalter.

Is this what is happening with Udacity’s black and white slides? Are the eternal truths they are unearthing merely statistically significant fleeting effects?

Udacity doesn’t care. But they should.