Scoring Self-Study Quizzes Online, an IF-AT Model

Here’s my progress in a Stanford Online course:

The problem is that I didn’t really try on this. I just hit answers till I got it right, and it let me try again. I’ve been through online courses like this before — when the week gets heavy with other obligations the click-itis begins. You don’t learn, because you stop anticipating whether you are right, you stop investing yourself in the question. Expectation failure can’t happen because you stop expecting.

However, there’s a more elegant solution to this sort of stuff. It comes from some traditional techniques associated with IF-AT cards.

The way it works is this — on a 4 answer multiple choice question you get 5 points if your first guess is correct, 2 points if your second guess is correct, and one point if your third guess is correct. Zero points for getting it last.

I don’t have hard data handy on the learning impact of the technique, but I can tell you from direct observation that it radically changes student behavior. Students that shrugged off second and third guesses suddenly become very involved in the process.  Rather than going by gut instinct, they carefully think over their line of reasoning before taking the second guess.

The key, of course, is the diminishing returns — the fear of diminishing returns makes them think it over carefully, while the partial credit incentivizes them to keep going forward.

I’ve used these structures in my class, especially with team-based decisions, and I’m always amazed how it transforms detached students into highly engaged students, almost instantly. It’s one of those neat magic tricks that Project-Based Learning folks know about and most of the rest of the world doesn’t

Anyway — this is something that has annoyed me for a while about online quizzes (not just Stanford Online). I wish someone would implement a system like this, for my sake as an educator, instructional designer, and most especially, as a student.

You could do this with MOOCs too

It’s a Gates funded project, but it jives with how I’ve been thinking about MOOCs lately:

Once they’re in, Portmont students will meet up for a one-week, one-credit intensive orientation where ideally they’ll bond with their classmates and the personal “success coaches” that are part of Portmont’s faculty, before heading back home to work on a largely online curriculum. They’ll return once every eight weeks to take part in group-project-based learning and presentations. The idea is to combine the best of online learning–self-pacing, convenience, and an analytics-powered dashboard that provides instant feedback to teacher and student–with some of the benefits of face-to-face education, while still keeping total costs low. Portmont will cost $5,240 per student, an amount entirely fundable with need-based Pell Grants, so that students can graduate debt free.

Once you get past the corporate culture  (“success coaches”) and Valley buzzword  ickiness (“analytics-powered dashboard”) and concentrate on the core structure of the experience, you realize this is how we live right now outside of education. We often work separately from colleagues at other institutions — learning from each other online, through email and twitter and blogs and hangouts. But that just makes time at conferences — the face-to-face time — more intense. And in turn, the intensity of a good conference renews the meaning of the online work and depth of the connection between us.

I don’t know what the ratio of campus to home time should be, and the slacker in me still thinks the kids need at least enough time to drop acid on campus and spend at least some time in a campus radio station quoting some Derrida pomposity about hauntology while drinking Old Mil pounders — but the idea of short intensive residential spurts combined with online is a big part of the future, no matter what the ratio is. It’s only a matter of time. And my money says you could do this with MOOCs – xMOOCs, cMOOCs, dsMOOCs. And that we’re already sort of doing it, it’s just not institutionalized…

Coursera, CC-NC, and OCW

It’s interesting to see this on the front page of the Coursera course I’m taking:

Obviously what has happened here is that Johns Hopkins and Kevin Frick have negotiated additional rights for Coursera to this OCW material, which allows them to use it commercially. Which is as it should be — that’s how CC licences are supposed to work. A CC-NC license is not a blood oath; the license holder can negotiate exceptions. In fact, that’s the point — if you’re going to use this commercially, we want a cut. Or veto power on use. Or whatever.

Still, it’s interesting seeing it in there. As far as I can tell, there’s not much in this course so far that wasn’t in the original OCW material. I’m a week and a half in, and basically there are the audio lectures that they added on top of the slides. That’s pretty much it. The much-vaunted in-video quizzes are actually few and far between, and don’t show any particular genius in formulation. A segment that involved Kevin talking to the camera, which appears to be additional, was poorly executed and thought out — I ended up having to cover the video of him in order to follow along in the slides. The explanation of supply and demand doesn’t involve a single quiz, simulation, manipulation, or application. It’s just listening to audio while flipping through charts.

This isn’t to say it’s bad — the topic is interesting, and Kevin has a knack for staying out of the weeds in his presentation — it’s difficult to teach a one-off course interdisciplinary course like this, at the right level of complexity, and he nails it. But what value was there was there before Coursera arrived, and was available for free.

Again, what does Coursera add here? Rocket science? Amazing educational insights? Stanford-level brilliance about about how to tap networked learning?

Nope: time-released OCW (I cant see next week’s materials yet), a set of cohort/peer tools, and an audio track for the slides. Given all we know about online education, it’s pretty odd that this is the game changer that threatens traditional education. But for various social, economic, and political reasons, it is. How very, very, weird.

xMOOCs = OCW + Cohorts

I’m still going through the process of cleaning up some old posts damaged by the database, and tonight I found this one I wrote on OpenCourseWare from 2009:

Rise of the Cohort, Educational and Otherwise

Posted on January 9, 2009 

“Cohort” is a term used in sociology and education that refers to a group of people that experience a certain set of events simultaneously as they move through time.  Cohort isn’t a perfect term, but I wonder if we are coming to a point where we need a term that gets rid of the meddlesome baggage associated with a class, but preserves the idea that there’s a particular type of peer instruction that benefits from everybody being on the same lesson at the same time.


There’s some blue sky stuff in there about Netflix and radio as well as OCW, but the basic premise is that OCW would benefit from a cohort that could discuss the content as it is rolled out week by week via some serialization mechanism. 

What I think is missed in the hoopla about xMOOCs is — if you look at this long term — this is precisely what has happened. Right now, as we look at the first pass of these courses we are looking at new video, new pieces, etc. We think of it as a new course being “run”. But these courses will start to be rerun soon, and at that point it is basically OCW with a cohort. 

And in many cases, literally old OCW with a cohort experience wrapped around it. I was loooking at the coursera offerings the other day, and was surprised how many of them are actually older OCW projects which are getting a bit of a spit polish, dropped in an LMS, and serialized. See for example the Obesity Economics course in Coursera, and then check out the OCW site

What does this mean? I’m not quite sure, actually. You know, besides the fact I was right ;). 

I think the most obvious implication is that when xMOOCs are seen as serialized OCW + cohorts one realizes how quickly Coursera is going to be able to build a catalog…a second implication might be that you don’t need Coursera to do this for you — if Tony Hirst was serializing this stuff in 2009 with a couple hacked scripts, you could probably do this fairly easily in 2012….

How TED Culture Destroyed the World, Literally

From The Lomborg Deception:

From these and many similar statements, we can identify “Lomborg’s Theorem,” circa 2001, which asserts that the Earth and its environment are not threatened in any fundamental sense by human activity and, for the purposes of this volume, that man-made global warming is not the catastrophe that the environmental organizations claim. Lomborg’s book, with its illusion of serious scholarship, given the number of endnotes, was influential in the United States throughout the presidential tenure of George W. Bush, who held power during a critically important window of opportunity to reduce greenhouse emissions to prevent the worst impacts of global warming. Probably more than any single published source, Lomborg’s The Skeptical Environmentalist marked global warming as a threat that was “exaggerated” by environmentalists, and helped justify the inaction on greenhouse emissions by the Bush administration and the Republican-led Congress in the United States. Lomborg’s influence was such that in 2004 Time named him one of the world’s one hundred most influential people.4

It’s not just that TED gave this guy a platform in 2005, a fact that, were TED a journal or a newspaper, would require a retraction.

It’s much bigger. It’s the culture that surrounds TED. Because the culture of TED is what allows people like Lomborg to have more influence than actual experts.

Why? The idea of TED is that you’re smart enough to get it in 10 minutes or less, and the story that TED-ites love (b/c it supports that narrative) is the story of someone outside the “industry” or research area coming in from another area and declaring at a glance what everyone has missed.

So we get economists talking about global warming, game designers talking about learning, techies talking about political gridlock, and choreographers talking about physics. It’s so simple, they tell us.

Ah, say the TED-sters — us outsiders to the system, we can all see what the insiders can’t! Salman Khan is an expert because he knows nothing about learning theory. Lomborg is worth listening to because he doesn’t know about climate science. And I’m probably an expert too!

Everyone is happy. Ten years later, the planet is destroyed, and there’s no going back, but at least everyone felt smart for a bit.


UMW & Event-ness, Revisited

I have been slowly repairing the damage the great database corruption of 2010/2011 did to my blog posts, pulling them over from and trying to get them back into my blog. Today I came across this one from May 2010, which among other things contains this graf about what I saw on my 2010 visit to UMW:

All this should have been clear to me before, and it was, kind of, on some level. But I’ve been so blinded by Learning 2.0 rhetoric I think I’ve missed the obvious — many of the best course blogs engage with the outside world not to break down the traditional class-world distinction, but in some ways to enhance it. By engaging publicly with the outside world the class gels. The “we are this class and you are the world” distinction is stronger, but also conceptualized in a much more productive way.

It occurs to me how much of my thinking since then has been informed by that. In part, my focus on Residential Online (or local online, or whatever you want to call it) has been driven by the belief that the tension between the local face-to-face experience and the open online experience is far richer than either component on its own. And my obsession with wrapping large national online courses in a local, faculty-mediated experience is motivated not by the students needing “guidance” (this is no flipped classroom approach) — but again, by the productive tension that a subcohort exhibits when part of a large international cohort.

I still think its amazing that UMW paid me to speak there, considering how much I walked away learning. It’s almost unfair. Seeing what was going on there at the time in first person, talking to the students and the teachers involved — it was like looking three or four years into the future. And it led me to the the realization that I think people have still not fully grasped — that the true future of online education exists at the intersection of a localized experience and global community — and that combination will not undermine the local “event-ness” of courses, but supercharge it in ways we haven’t seen yet….the dawn of the open is also the rebirth of the local.


Cost Disease vs. The Fall of the Faculty

I read The Fall of the Faculty a couple months ago, partially because it was cited in an ongoing discussion on our campus about what is to blame for higher educational costs. My general operating assumption is that of all influences on educational cost, cost disease has the biggest (and most relentless) impact. I was hoping The Fall of the Faculty would provide a strong counter-argument, but what it provided instead was an unsupported hypothesis — the costs are all the result of administrative bloat, brought on by a 1980’s takeover of the university by pointy-headed management types.

I was reminded of that reading Bowen’s excellent lecture on Cost Disease and higher education, especially when looking at this graph:

These are privates, of course, but if you look at the graph (which covers costs from 1905 to 1965) it’s pretty obvious that any theory of increasing educational cost that treats it as a recent phenomenon is seriously flawed.

Most attacks on the cost disease theory I’ve seen fail to deal with this fact at all. Conservatives want it to be Pell Grants distorting market efficiencies, whereas liberals want it to be the evil corporatization of the university (The Fall of the Faculty takes this to the extreme). It’s hard to look at this graph and see either of those explanations as sufficient.