Synchronous Online Sucks

Synchronous online — the twisted mess of chat-rooms, video-conferencing, and screen-sharing we use for real-time online education — has sucked for a while now. To be sure we have products: it’s rare for a university to not have *some* web video-conferencing solution in place. And from what I understand, these products, at the enterprise level, are not cheap. There is commitment to supporting *some* level of functionality here.

But almost all of these products are designed for “pass-the-mic” lecture style classrooms, or worse, are sales presentation software that has been “adapted” for the classroom. And hence, all the techniques to which professors normally have access in face-to-face discussion (see, for example, the excellent work of Stephen Brookfield on structured peer discussion) are not really possible with modern video conferencing software.

But it doesn’t have to be this way. I’ve talked before about ideas like Ed Roulette, which attempts to structure online video conferencing using a peer instruction paradigm. That’s the tip of the iceberg. In a conversation with Devlin Daley at Open Ed, we ran through the variety of structured discussion styles used in the modern classroom, from the “speed dating” peer-review exercises I have my students do to get reader feedback to the Circular Response-style discussions that remain a favorite way to stop students from talking past one another.

All of these techniques can revolutionize your classroom. None of them are available to you online.

…But This Will Change This Coming Year

Why don’t we have technical options that borrow from the structure of the above models? Options that not only provide an opportunity to talk to one another, but also help structure the discussion? Part of it is the lack of interest in the synchronous online environment as a market. As Amy Collier noted to me in a recent conversation, up until recently online has been seen as a low-quality solution for people who do not have the time flexibility for scheduled class meetings, and as such the synchronous element has lagged behind the asynchronous piece. Who chooses an online class based on the synchronous experience, right?

But perhaps an even greater barrier has been the cost of serving real-time video. It’s huge. It requires a large, stable infrastructure and a sizable investment to make it work. Got an idea you’d like to try with online video? Maybe you want integrate deliberative polling into a issues discussion? Maybe you want a product that sets up project groups and allows you as the teacher to drop into them, annotate them, or put them on a timer. Great! All you need to get started is a server farm and a T1T3 line. Go to it!

Both these issues have changed in the past couple of years. On the culture side, we are increasingly seeing online as an option for students who are on-campus, and can commit to scheduled online meetings.

On the technology side, peer-to-peer web video is about to change the world. Following up a tip from both Devlin and Tim Owens, I recently looked into the new WebRTC technology/spec that is built into newer browsers. This technology allows programs to negotiate a peer-to-peer video connection, without having to serve up the actually video. You write the programming, and the individual participant computers (and browsers) handle the streaming. A great example of this is appear.in, a service that lets you set up a custom video chatroom in one step. Try it out — I’ve found the quality to be amazing, superior to what I get with Skype or Google Hangouts.

With the introduction of peer-to-peer video capabilities in newer browsers, much of the previous bandwidth barriers to innovation are disappearing — the “software-plus-infrastructure” problem of video-based synchronous online is quickly becoming a software-only problem. The market for synchronous online experiences is likely to grow. It’s hard to see this as anything but a perfect storm for rethinking the gun-metal-gray boredom of those Adobe Connect sessions we keep pushing on students. Hopefully we can replace it with something that allows us to mimic some of the fluid structures of the face-to-face class; at the very least we can get past the outdated assumptions still present in most of this software.

As readers of this blog know, Amy Collier and I have been making a year-long argument that MOOC community features, as currently designed, are often perceived by blended students as low-to-no-value substitutes for local interaction. That made this snippet of MOOC-runner Dan Ariely talking about his own class’s use of the MOOC rather interesting:

Dan Ariely:  …Let me tell you one more thing. I spend a lot of time creating these classes online, and then I try to use two of them in my regular face-to-face class. And I did what is called “reverse classroom,” so I asked the students to watch the video at home and then come to class to discuss it. It was incredibly successful. I think the students enjoyed the videos and then they enjoyed the discussion. The other thing I tried to do, again with my face-to-face class, was to get them to watch the videos and to have all the discussion about it online. And that one was not as useful.

So the students in my regular class basically had three versions: they had me in person; they had watched the video and come in to have just the discussion; and they watched the video and had the discussion online. And they basically rated them in that order. They said it was the most useful to have me in class. Not too far from that is to watch the video of the material and then have the discussion in class. Much less appealing was to have the video and then have an online discussion on that.

That’s part of the story — that you are losing something as you become more detached from the students and have less face-to-face time, but you have to figure a version of how this could work out because of the cost of the full-time colleges and universities.

Now caveats, caveats: Dan is an accomplished teacher in a face-to-face setting, and probably a much less accomplished online moderator. So maybe part of what we are seeing here is just that Dan’s strength in explanation can be captured online, but his strength in facilitation can’t.

Here’s the thing though — this is true of most face-to-face teachers. This is what they’ve become really good at,  year after year, and it’s the value they bring to the classroom. So any system of teaching that takes takes that core talent and throws it out the window isn’t likely to have a great success rate, no?

I think you can also argue that we privilege face-to-face communication more, assuming that it’s not inconvenient, and I think you can argue that online facilitation has to become a more common skill over time. But I’ve made enough points for today, so I’m done.

David Kernohan has a post up titled How I got a “first” on a FutureLearn MOOC with one weird old trick… over at FOTA, and it does just what it says on the tin. In the post, David details how he was able to get an 87.4% on a FutureLearn test for a course module without studying any materials. The “old trick” referenced is not actually one old trick, but the bag of tricks that most people come to learn taking multiple choice tests. You know, for example, that when “all of the above” appears, there’s a high chance that’s the correct option. You know that sentences that are not parallel in structure are usually wrong options. You know the longest, most qualified answer is usually right.

I’ve talked about this before, regarding a Coursera course my boss took, writing a scathing treatment of it in Sometimes Failure is Just Failure. In that case the malformed questions did not make it easy for a person with no knowledge to ace the test, but rather, it made it difficult for someone with good knowledge to pass it.

The reason these sorts of errors are so mind-blowing frustrating is:

1. You see it everywhere, in all these products.
2. It’s a really serious matter.
3. It could be solved with a four-page checklist.

It’s really hard to write excellent multiple choice questions, questions that go beyond simple recall and comprehension and test higher order, conceptual understanding. On the other hand, many decades of research have made writing good multiple choice questions relatively easy. You don’t even have to read the research — you just need to follow the rules distilled from the research.

I don’t know if the problem in this case was a FutureLearn designer or a subject matter expert they worked with. Perhaps one of the problems is that it’s unclear in the arrangement who exactly is responsible for test validity. But somewhere in the quality assurance chain there needs to be someone who can read the checklist and spend the amount of time necessary to reformat these questions. We can debate all day whether multiple choice is the best way to assess students, but making sure the multiple choice questions we do use are well designed is a no-brainer. Do like I do, and don’t trust your instincts. Print out the sheet and keep it by you while you write your questions. Check each question. Then get someone else to double check it for you.

When you’re rolling this out to 100 students, this sort of care is important. When you’re rolling it out to tens of thousands, this sort of care is essential.

A quick follow-up to yesterday’s post on the supposed “death of theory” and its relation to MOOC research — the story thus far is that a number of people sincerely think the “why” doesn’t matter if our sample is big enough and the variables tracked are numerous enough. Here’s a typical quote from Thrun:

One day, Sebastian Thrun ran a simple and surprising experiment on a class of students that changed his ideas about how they were learning.

The students were doing an online course provided by Udacity, an educational organisation that Thrun co-founded in 2011. Thrun and his colleagues split the online students into two groups. One group saw the lesson’s presentation slides in colour, and another got the same material in black and white. Thrun and Udacity then monitored their performance. The outcome? “Test results were much better for the black-and-white version,” Thrun told Technology Review. “That surprised me.”

Why was a black-and-white lesson better than colour? It’s not clear. But what matters is that the data was unequivocal – and crucially it challenged conventional assumptions about teaching, providing the possibility that lessons can be tweaked and improved for students.

Note that last bit — “What matters is that the data was unequivocal”. This is how the End of Theory position appears in print. We don’t know *why* the students did better, but they did better, and the data was so “big” that that’s all that matters.

But the why does matter. Because without the why you can’t generalize from one situation to the next, and you keep repeating the same mistakes. In this case, we could hypothesize three alternate explanations of the phenomenon:

1. The lack of colors leads to a lack of distraction. Students watching the colored slides were processing the colors as meaningful (when they weren’t) and this was subtly hindering their comprehension and recall.
2. Students had a harder time reading the black and white slides. There’s recent research that indicates slight disfluencies in presentation can be desirable (jokingly dubbed “The Comic Sans Effect”), and these disfluencies aid in recall.
3. The cause was not that the slides were black and white, but that the black and whiteness was novel. We know from previous psychological research that the mind attends to novelty, the greater attentiveness led to greater retention.

So which one is it? Udacity doesn’t care. But they should. Because if it’s the second one, then writing slides in black and white is not exactly what you should be focused on. And if it’s the third — the novelty effect — then the impact of this is going to be very limited. This isn’t even getting into variable context, pedagogical aims, or path-dependence.

You see this problem in Big Data all the time. The Obama campaign was really a Small Data operation, but they did extensive A/B testing. And what they found one night was the Holy Grail of campaign email marketing:

It quickly became clear that a casual tone was usually most effective. “The subject lines that worked best were things you might see in your in-box from other people,” Fallsgraff says. “ ‘Hey’ was probably the best one we had over the duration.” Another blockbuster in June simply read, “I will be outspent.” According to testing data shared with Bloomberg Businessweek, that outperformed 17 other variants and raised more than $2.6 million.

The “magic formula”, right? Well, no:

But these triumphs were fleeting. There was no such thing as the perfect e-mail; every breakthrough had a shelf life. “Eventually the novelty wore off, and we had to go back and retest,” says Showalter.

Is this what is happening with Udacity’s black and white slides? Are the eternal truths they are unearthing merely statistically significant fleeting effects?

Udacity doesn’t care. But they should.

At the MOOC Research Initiative conference Jeff Selingo gave what I thought was a capable presentation of the current landscape of higher education. People might quibble with a point or two, but overall it was a relatively balanced, hysteria-free overview of a market which is not necessarily “broken”, but is poised to undergo some relatively dramatic changes in the coming decade or two.

One stat from that talk bothered me though, and seeing it pop up again on Twitter today pushed me to try to explain the problem:

You hear variations of this stat all the time, and there’s no doubt in my mind we are undergoing a transformation of our student body. But here’s the important piece of information I never hear attached to such stats — is it a stock or a flow?

You see, there’s a couple ways to measure a phenomenon like this, and they have their strengths and weaknesses, and sometimes can give very different views of the data. The first way — measure the stock — is the first approach of most people. Is thyroid cancer a growing problem in a America? Control for population, compare the number of people with thyroid cancer with the number 50 years ago, and there you go — sorted.

But is the number of people living with thyroid cancer what you care about? Or is the number of people contracting thyroid cancer what you care about?

Those things sound like they’d give very similar views, but they don’t. Thyroid cancer used to be a uniformly deadly disease. People who contracted it would usually die, and die relatively soon after a diagnosis, so at any given time there was only a small population of people who had thyroid cancer. With the introduction of radioactive iodine as a treatment, the prognosis for most people with thyroid cancer today is quite good. Since cancer is considered a chronic disease (a person who contracts thyroid cancer will always be considered to have it, even in remission) the number of people who now have thyroid cancer has gone through the roof, but as a result of our success, not failure.

That’s why epidemiology has two different measures — one is prevalence, our measure of the “stock”. Prevalence asks how many people at a given time have a condition. But they also use an “inflow” measure: incidence. Incidence measures how many people per year move into a condition. Which measure we use will depend on what we aim to do with the information, but it is almost always appropriate when given one number to ask for the other as well. If the numbers tell two very different stories it’s time to sit down and parse out why.

So what about this 80% of students are non-traditional? Is that a stock or a flow?

My guess is it’s a stock, more akin to measures of prevalence than incidence (Apologies to Jeff if I am not right here). And if it is, that’s an interesting statistic, but I’m not sure it’s the right one to use. Here’s a simplified model to show why. Imagine a world that graduates two people from high school every two years. One always goes to college full-time and gets out in just under four years (just under because they graduate in May, not September). The other student goes part time, and finishes in eight years. Here’s a visual representation of how that plays out over time:

The blue arrows represent the tenure of our full-time students in college, and the red arrows the time spent by our part-timers.

Now here’s a question. Go to year eight and run down through the students that line bisects. Count up the students that are either starting college or currently enrolled. You’ll notice that there are two four-year students currently enrolled and four eight-year students. Now tally up your percentages. In this case we reach the conclusion that 66% of students are non-traditional eight-year students, compared to the measly 33% of the population who are in four-year programs. The “new normal” right?

But measures of flow tell a different story. In each graduating class, 50% of students go traditional, and 50% non-traditional. And asked whether flow or stock best represents what is a normal student experience here, I’d have to say it’s the measure of flow (50%). The stock measure is just too prone to outflow effects.

Of course, these are made up numbers. It could go the other way as well — if part-timers don’t persist at the rate of full-timers, the effect would run the other way, significantly minimizing the number of students who choose to attend part-time. Or perhaps the two trends balance one another out. The smaller point is that this discussion needs better numbers if we are to get an accurate sense of what is going on. The larger point is when presented with stats on a stock, always seek out a corresponding flow, and vice versa. If they give you deficit, ask for debt. If they give you number of people imprisoned, ask for a count of yearly incarcerations. And so on. The interesting story is usually where different measures seem to disagree, and you can’t get that from a single view.

I’ve just finished attending the MOOC Research Initiative conference and workshop, which felt not so much a conference on MOOCs to me as the beginning of something else. We kick Courdacity-style MOOCs around a  lot, but if all MOOCs did was bring this level of intelligence and insight into problems of online learning, it would be worth it all. There’s a way that that black hole of a term is pulling different universes together, and if the culture of MRI ’13 is what results from that, the future looks very bright.

Conferences are always a bit of a disembodied experience for me; I never feel grounded away from home. But this time the whole experience had a warm dream-like quality, and I’ve been trying to parse out why that was. Certainly, part of it was the psychological overload of seeing so many people who I admire in one place. Part of it might be the ice storm that swept in the night of day two, pushing us all together into smaller spaces.

But I think it was actually the conversations that struck me the most. You know how in a dream characters suddenly pop up and have bizarre yet fluid conversations that pull your mind in all sorts of new directions?  The conversations here were not that bizarre, I suppose, but they had that rapid fluidity of dream logic, where a sort of conversational shorthand shuffles you quickly past the normal pro/con nonsense and into something much bigger, and much more exciting and mysterious.

Kudos to George, Amy, Tanya, and everyone else that put this thing together. I’ll be processing the experience for weeks to come, and there’s no greater compliment for a conference than that.