Scoring Self-Study Quizzes Online, an IF-AT Model

Here’s my progress in a Stanford Online course:

The problem is that I didn’t really try on this. I just hit answers till I got it right, and it let me try again. I’ve been through online courses like this before — when the week gets heavy with other obligations the click-itis begins. You don’t learn, because you stop anticipating whether you are right, you stop investing yourself in the question. Expectation failure can’t happen because you stop expecting.

However, there’s a more elegant solution to this sort of stuff. It comes from some traditional techniques associated with IF-AT cards.

The way it works is this — on a 4 answer multiple choice question you get 5 points if your first guess is correct, 2 points if your second guess is correct, and one point if your third guess is correct. Zero points for getting it last.

I don’t have hard data handy on the learning impact of the technique, but I can tell you from direct observation that it radically changes student behavior. Students that shrugged off second and third guesses suddenly become very involved in the process.  Rather than going by gut instinct, they carefully think over their line of reasoning before taking the second guess.

The key, of course, is the diminishing returns — the fear of diminishing returns makes them think it over carefully, while the partial credit incentivizes them to keep going forward.

I’ve used these structures in my class, especially with team-based decisions, and I’m always amazed how it transforms detached students into highly engaged students, almost instantly. It’s one of those neat magic tricks that Project-Based Learning folks know about and most of the rest of the world doesn’t

Anyway — this is something that has annoyed me for a while about online quizzes (not just Stanford Online). I wish someone would implement a system like this, for my sake as an educator, instructional designer, and most especially, as a student.

You could do this with MOOCs too

It’s a Gates funded project, but it jives with how I’ve been thinking about MOOCs lately:

Once they’re in, Portmont students will meet up for a one-week, one-credit intensive orientation where ideally they’ll bond with their classmates and the personal “success coaches” that are part of Portmont’s faculty, before heading back home to work on a largely online curriculum. They’ll return once every eight weeks to take part in group-project-based learning and presentations. The idea is to combine the best of online learning–self-pacing, convenience, and an analytics-powered dashboard that provides instant feedback to teacher and student–with some of the benefits of face-to-face education, while still keeping total costs low. Portmont will cost $5,240 per student, an amount entirely fundable with need-based Pell Grants, so that students can graduate debt free.

Once you get past the corporate culture  (“success coaches”) and Valley buzzword  ickiness (“analytics-powered dashboard”) and concentrate on the core structure of the experience, you realize this is how we live right now outside of education. We often work separately from colleagues at other institutions — learning from each other online, through email and twitter and blogs and hangouts. But that just makes time at conferences — the face-to-face time — more intense. And in turn, the intensity of a good conference renews the meaning of the online work and depth of the connection between us.

I don’t know what the ratio of campus to home time should be, and the slacker in me still thinks the kids need at least enough time to drop acid on campus and spend at least some time in a campus radio station quoting some Derrida pomposity about hauntology while drinking Old Mil pounders — but the idea of short intensive residential spurts combined with online is a big part of the future, no matter what the ratio is. It’s only a matter of time. And my money says you could do this with MOOCs – xMOOCs, cMOOCs, dsMOOCs. And that we’re already sort of doing it, it’s just not institutionalized…

Coursera, CC-NC, and OCW

It’s interesting to see this on the front page of the Coursera course I’m taking:

Obviously what has happened here is that Johns Hopkins and Kevin Frick have negotiated additional rights for Coursera to this OCW material, which allows them to use it commercially. Which is as it should be — that’s how CC licences are supposed to work. A CC-NC license is not a blood oath; the license holder can negotiate exceptions. In fact, that’s the point — if you’re going to use this commercially, we want a cut. Or veto power on use. Or whatever.

Still, it’s interesting seeing it in there. As far as I can tell, there’s not much in this course so far that wasn’t in the original OCW material. I’m a week and a half in, and basically there are the audio lectures that they added on top of the slides. That’s pretty much it. The much-vaunted in-video quizzes are actually few and far between, and don’t show any particular genius in formulation. A segment that involved Kevin talking to the camera, which appears to be additional, was poorly executed and thought out — I ended up having to cover the video of him in order to follow along in the slides. The explanation of supply and demand doesn’t involve a single quiz, simulation, manipulation, or application. It’s just listening to audio while flipping through charts.

This isn’t to say it’s bad — the topic is interesting, and Kevin has a knack for staying out of the weeds in his presentation — it’s difficult to teach a one-off course interdisciplinary course like this, at the right level of complexity, and he nails it. But what value was there was there before Coursera arrived, and was available for free.

Again, what does Coursera add here? Rocket science? Amazing educational insights? Stanford-level brilliance about about how to tap networked learning?

Nope: time-released OCW (I cant see next week’s materials yet), a set of cohort/peer tools, and an audio track for the slides. Given all we know about online education, it’s pretty odd that this is the game changer that threatens traditional education. But for various social, economic, and political reasons, it is. How very, very, weird.

xMOOCs = OCW + Cohorts

I’m still going through the process of cleaning up some old posts damaged by the database, and tonight I found this one I wrote on OpenCourseWare from 2009:

Rise of the Cohort, Educational and Otherwise

Posted on January 9, 2009 

“Cohort” is a term used in sociology and education that refers to a group of people that experience a certain set of events simultaneously as they move through time.  Cohort isn’t a perfect term, but I wonder if we are coming to a point where we need a term that gets rid of the meddlesome baggage associated with a class, but preserves the idea that there’s a particular type of peer instruction that benefits from everybody being on the same lesson at the same time.


There’s some blue sky stuff in there about Netflix and radio as well as OCW, but the basic premise is that OCW would benefit from a cohort that could discuss the content as it is rolled out week by week via some serialization mechanism. 

What I think is missed in the hoopla about xMOOCs is — if you look at this long term — this is precisely what has happened. Right now, as we look at the first pass of these courses we are looking at new video, new pieces, etc. We think of it as a new course being “run”. But these courses will start to be rerun soon, and at that point it is basically OCW with a cohort. 

And in many cases, literally old OCW with a cohort experience wrapped around it. I was loooking at the coursera offerings the other day, and was surprised how many of them are actually older OCW projects which are getting a bit of a spit polish, dropped in an LMS, and serialized. See for example the Obesity Economics course in Coursera, and then check out the OCW site

What does this mean? I’m not quite sure, actually. You know, besides the fact I was right ;). 

I think the most obvious implication is that when xMOOCs are seen as serialized OCW + cohorts one realizes how quickly Coursera is going to be able to build a catalog…a second implication might be that you don’t need Coursera to do this for you — if Tony Hirst was serializing this stuff in 2009 with a couple hacked scripts, you could probably do this fairly easily in 2012….

How TED Culture Destroyed the World, Literally

From The Lomborg Deception:

From these and many similar statements, we can identify “Lomborg’s Theorem,” circa 2001, which asserts that the Earth and its environment are not threatened in any fundamental sense by human activity and, for the purposes of this volume, that man-made global warming is not the catastrophe that the environmental organizations claim. Lomborg’s book, with its illusion of serious scholarship, given the number of endnotes, was influential in the United States throughout the presidential tenure of George W. Bush, who held power during a critically important window of opportunity to reduce greenhouse emissions to prevent the worst impacts of global warming. Probably more than any single published source, Lomborg’s The Skeptical Environmentalist marked global warming as a threat that was “exaggerated” by environmentalists, and helped justify the inaction on greenhouse emissions by the Bush administration and the Republican-led Congress in the United States. Lomborg’s influence was such that in 2004 Time named him one of the world’s one hundred most influential people.4

It’s not just that TED gave this guy a platform in 2005, a fact that, were TED a journal or a newspaper, would require a retraction.

It’s much bigger. It’s the culture that surrounds TED. Because the culture of TED is what allows people like Lomborg to have more influence than actual experts.

Why? The idea of TED is that you’re smart enough to get it in 10 minutes or less, and the story that TED-ites love (b/c it supports that narrative) is the story of someone outside the “industry” or research area coming in from another area and declaring at a glance what everyone has missed.

So we get economists talking about global warming, game designers talking about learning, techies talking about political gridlock, and choreographers talking about physics. It’s so simple, they tell us.

Ah, say the TED-sters — us outsiders to the system, we can all see what the insiders can’t! Salman Khan is an expert because he knows nothing about learning theory. Lomborg is worth listening to because he doesn’t know about climate science. And I’m probably an expert too!

Everyone is happy. Ten years later, the planet is destroyed, and there’s no going back, but at least everyone felt smart for a bit.


UMW & Event-ness, Revisited

I have been slowly repairing the damage the great database corruption of 2010/2011 did to my blog posts, pulling them over from and trying to get them back into my blog. Today I came across this one from May 2010, which among other things contains this graf about what I saw on my 2010 visit to UMW:

All this should have been clear to me before, and it was, kind of, on some level. But I’ve been so blinded by Learning 2.0 rhetoric I think I’ve missed the obvious — many of the best course blogs engage with the outside world not to break down the traditional class-world distinction, but in some ways to enhance it. By engaging publicly with the outside world the class gels. The “we are this class and you are the world” distinction is stronger, but also conceptualized in a much more productive way.

It occurs to me how much of my thinking since then has been informed by that. In part, my focus on Residential Online (or local online, or whatever you want to call it) has been driven by the belief that the tension between the local face-to-face experience and the open online experience is far richer than either component on its own. And my obsession with wrapping large national online courses in a local, faculty-mediated experience is motivated not by the students needing “guidance” (this is no flipped classroom approach) — but again, by the productive tension that a subcohort exhibits when part of a large international cohort.

I still think its amazing that UMW paid me to speak there, considering how much I walked away learning. It’s almost unfair. Seeing what was going on there at the time in first person, talking to the students and the teachers involved — it was like looking three or four years into the future. And it led me to the the realization that I think people have still not fully grasped — that the true future of online education exists at the intersection of a localized experience and global community — and that combination will not undermine the local “event-ness” of courses, but supercharge it in ways we haven’t seen yet….the dawn of the open is also the rebirth of the local.


Cost Disease vs. The Fall of the Faculty

I read The Fall of the Faculty a couple months ago, partially because it was cited in an ongoing discussion on our campus about what is to blame for higher educational costs. My general operating assumption is that of all influences on educational cost, cost disease has the biggest (and most relentless) impact. I was hoping The Fall of the Faculty would provide a strong counter-argument, but what it provided instead was an unsupported hypothesis — the costs are all the result of administrative bloat, brought on by a 1980’s takeover of the university by pointy-headed management types.

I was reminded of that reading Bowen’s excellent lecture on Cost Disease and higher education, especially when looking at this graph:

These are privates, of course, but if you look at the graph (which covers costs from 1905 to 1965) it’s pretty obvious that any theory of increasing educational cost that treats it as a recent phenomenon is seriously flawed.

Most attacks on the cost disease theory I’ve seen fail to deal with this fact at all. Conservatives want it to be Pell Grants distorting market efficiencies, whereas liberals want it to be the evil corporatization of the university (The Fall of the Faculty takes this to the extreme). It’s hard to look at this graph and see either of those explanations as sufficient.

Outflow issues and “Traditional Students”

I was just thinking about statistics on traditional students vs. non-traditional, and realized that there are huge outflow issues in the way they are often presented.

[For more on inflows, stocks, and outflows, read this short description]

It’s common to talk about a decline in traditional students by saying things like “Only x% of students in 2012 were traditional, full time students.” But that’s a highly deceptive formulation.

Imagine a world where there are three students – two traditional full time students and one part time student who takes eight years to graduate.

Most reasonable assessments of this world will say that 2/3 of students are “traditional”. But at any given time it will look like only 50% of “current students” are traditional. Check it out:


Full Time Students

Part Time

Ratio of current full-time to part-time














John (graduates)
















Mary (graduates)

Tim (graduates)



Of course it can get screwy the other way too. Quick-finishing community college students would be undercounted in any year-to-year percentages. The point is that year-to-year figures are so horribly distorted by outflow issues that they need to be approached with extreme caution.

Only 16% of Students “Traditional”? Not exactly.

I was flipping through Mark Taylor’s book on the Crisis in Higher Education when I found this startling statistic:

Though the fact is rarely noted, the traditional four-year college whose students are eighteen to twenty-two years old is rapidly becoming a thing of the past. Only 16 percent of all students2 currently fall into this category; the majority of students are now over twenty-two.

Follow that footnote and you’ll find it comes from a book called The Last Professors, which states that:

The image of an 18-to-22-year old, full-time student in residence at a traditional college, however, is now a figment of the past, only 16 percent of all undergraduates fit that…

So this adds an important piece — unlike Taylor’s quote, the 16 percent here refers to students in residence halls, which changes the meaning of the statistic completely.

But is this even true? The cite for this is to a book called Future of the Public University in America by a guy named Womack, who says something similar:

He cites the 2002 NCES report on The Condition of Education, and doesn’t give a specific page number, but rather, just cites the entire chapter on Nontraditional Students to back up his claim.

Which would be good, I suppose, but nothing in that chapter mentions 16 percent of anything, or deals in any quantitative way with what proportion of students do not live on campus.

However, I did find this handy chart in another publication, which is supposedly based off NCES data. The upshot? All of that list (full time, live on campus, between the ages of 18 and 22) could be replaced by “live on campus”:

What we see here are two trends (and remember this data encompasses the roughly 50% of students that are going to community college). Younger students live at home more than on campus, and older students live off-campus independently more than on campus.

When you look at entering freshman to four-year institutions, however, the picture changes dramatically. I am not sure what the numbers of actual freshman who opt for dorm life, but the overwhelming preference of freshmen is to live on campus:

This desire, of course, fades a bit over time. But it’s worth noting that the student that is counted as a senior as a “non-traditional student” for living off campus likely came in as a freshman very excited about an on-campus residence. If that’s the case, are we really looking at non-traditional students here — or perhaps just seeing a common pattern of students outgrowing the residence halls as they move through college?

In any case, I think rumors of the death of the traditional student are a bit exaggerated. Certainly there’s a bit more nuance to the story than some would have you believe.  I think the residential experience is ripe for reimagining — but we should start by admitting that it is still very much in demand, at least in the early years of traditional four year programs.

Productionist Models and Education

Farm Factory Wife, by csessums

A book I’m reading now, Food Wars, has this to say about “Productionism”, the paradigm that dominated food policy through the 20th century:

In the Productionist paradigm (Figure 1.3), health is portrayed as being enhanced, above all, by increasing production, which required investment in both monetary and scientific terms. Agriculture, the prophets of Productionism argued, deserved massive support if it was to move away from ‘peasant’, low-yield systems. (This, incidentally, was their rationale for the now much-derided subsidy system throughout the West.) As long as food could be adequately and equitably distributed, health benefits would result. This Productionist view of health saw the main problems as under-consumption, under-production and poor distribution. The health goal of public policy, therefore, should be to increase production of key health-enhancing ingredients such as milk, meat, wheat, and other ‘big’ agricultural commodities. Figure 1.3 shows how this policy relationship might connect inputs and outputs in health.

The health assumptions on which the Productionist paradigm was built were based on what today would be regarded as a very narrow understanding of nutrition and health. For example, the observation in the 1800s that animal protein aided human growth led to massive resources in countries such as the US and Europe being invested in the development of the dairy and meat industries. The agricultural and agribusiness focus of the Productionist paradigm has also been weakened by the shift of power and finance down the food supply chain to the retailing, trade and consumer industries such as food service, where most of the money from food is now made (a feature spelled out in Chapter 4). In the US, for example, about half of all food expenditure is on consumption outside the home. (Food Wars, page 34)

There’s obvious differences between food and education, but I couldn’t help seeing in this description of Productionism a parallel to focus on “access” in education today. In particular, it’s difficult to see how years of research on what sort of education works is being applied to the access debate – we are stuck in the equivalent of the “make more calories, make more protein” mindset.

That’s not to say Productionism is wrong – it’s a paradigm, not a theory. And in the 20th century, Productionism was fairly triumphant – we have avoided, by and large, the global famines and conflict that were predicted in the 1970s, largely due to moving away from those “low-yield” systems that Productionism set out to change. But Productionism is also behind much of our current ills – food policy’s relentless focus on calories over nutritional quality, for example, has massively distorted incentives for healthy eating, and the focus on a one-size-fits-all approach to agriculture has resulted in monocultures that many feel endanger the ecosystem as well as local economies.

If we accept that post WWII higher education policy operated largely under a Productionist paradigm, the question is whether the paradigm (if the paradigm does indeed capture our current “access” and “success” initiatives) has already become too reductive. When we talk all-or-nothing replacement of face-to-face education with online – and when we see online as a “delivery system” for our educational calories rather than a way to provide essential nutrients and foster a healthy interconnected sector – when the level of conversation is more about production and distribution, and less about how online education fits into our society as a whole and enriches the things we already value – then I think that maybe we are in the grip of 20th century Productionism…