Why the Why Matters

A quick follow-up to yesterday’s post on the supposed “death of theory” and its relation to MOOC research — the story thus far is that a number of people sincerely think the “why” doesn’t matter if our sample is big enough and the variables tracked are numerous enough. Here’s a typical quote from Thrun:

One day, Sebastian Thrun ran a simple and surprising experiment on a class of students that changed his ideas about how they were learning.

The students were doing an online course provided by Udacity, an educational organisation that Thrun co-founded in 2011. Thrun and his colleagues split the online students into two groups. One group saw the lesson’s presentation slides in colour, and another got the same material in black and white. Thrun and Udacity then monitored their performance. The outcome? “Test results were much better for the black-and-white version,” Thrun told Technology Review. “That surprised me.”

Why was a black-and-white lesson better than colour? It’s not clear. But what matters is that the data was unequivocal – and crucially it challenged conventional assumptions about teaching, providing the possibility that lessons can be tweaked and improved for students.

Note that last bit — “What matters is that the data was unequivocal”. This is how the End of Theory position appears in print. We don’t know *why* the students did better, but they did better, and the data was so “big” that that’s all that matters.

But the why does matter. Because without the why you can’t generalize from one situation to the next, and you keep repeating the same mistakes. In this case, we could hypothesize three alternate explanations of the phenomenon:

  1. The lack of colors leads to a lack of distraction. Students watching the colored slides were processing the colors as meaningful (when they weren’t) and this was subtly hindering their comprehension and recall.
  2. Students had a harder time reading the black and white slides. There’s recent research that indicates slight disfluencies in presentation can be desirable (jokingly dubbed “The Comic Sans Effect”), and these disfluencies aid in recall.
  3. The cause was not that the slides were black and white, but that the black and whiteness was novel. We know from previous psychological research that the mind attends to novelty, the greater attentiveness led to greater retention.

So which one is it? Udacity doesn’t care. But they should. Because if it’s the second one, then writing slides in black and white is not exactly what you should be focused on. And if it’s the third — the novelty effect — then the impact of this is going to be very limited. This isn’t even getting into variable context, pedagogical aims, or path-dependence.

You see this problem in Big Data all the time. The Obama campaign was really a Small Data operation, but they did extensive A/B testing. And what they found one night was the Holy Grail of campaign email marketing:

It quickly became clear that a casual tone was usually most effective. “The subject lines that worked best were things you might see in your in-box from other people,” Fallsgraff says. “ ‘Hey’ was probably the best one we had over the duration.” Another blockbuster in June simply read, “I will be outspent.” According to testing data shared with Bloomberg Businessweek, that outperformed 17 other variants and raised more than $2.6 million.

The “magic formula”, right? Well, no:

But these triumphs were fleeting. There was no such thing as the perfect e-mail; every breakthrough had a shelf life. “Eventually the novelty wore off, and we had to go back and retest,” says Showalter.

Is this what is happening with Udacity’s black and white slides? Are the eternal truths they are unearthing merely statistically significant fleeting effects?

Udacity doesn’t care. But they should.

2 thoughts on “Why the Why Matters

  1. It’s a modernist approach to research…there is a Truth and a universality to things, so Big Data sets us free because it is, for all intents and purposes, God. This is the notion that the Universe is one big algorithm. But any good post-structuralist, postmodernist or individual who sees more than a Western notion of the Pilgrim’s Progress understands the importance of cohort and environment in any variable. There are lots of potential reasons why the B&W slides were better than the color ones. Without any sort of theoretical lens to view data by (or an understanding of learning theory, as the quote “crucially it challenged conventional assumptions about teaching, providing the possibility that lessons can be tweaked and improved for students” ironically shows), the information is meaningless. It’s less Richard Dreyfuss with a mountain of mashed potatoes in Close Encounters of the Third Kind and more Weird Al Yankovic with a mountain of mashed potatoes in UHF.

  2. Exactly! If we don’t have people thinking thoughtfully about the data it is blind in many ways.

    I’m so glad there are people like you blogging about this stuff and the problems of data. I think most people don’t think critically enough about the numbers that are thrown at them or don’t really know how to think critically about them.

    Your blog has taught me and continue to teaches me a lot. Thanks for this.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s