Stinking Badges

(Ha, you thought I was going with the Brooks movie or Sierra Madre, right?)

Because this seems as good a place as any to note it, I think one person who doesn’t get credit enough on learning badges is Roger Schank. When I went to work for his company in 2000, he’d already been talking about a “merit badge” approach to assessment for years. And for certain projects with Harvard Business School Publishing and others, we essentially implemented those approaches.

Here’s Schank talking in 1999 about it:

“We won’t get rid of certification but perhaps we can contemplate new kinds of certification. Students should be certified as having accomplished something or as being able to do something. Like Boy Scout merit badges or Karate black belts or Truck Driver’s licenses, the proof should be in the pudding. A student should show his stuff, he should be able to do something and the attestation to the doing should be the certification.”

And if that leaves you thinking that “badges” was only one of many ways he expressed this idea, be assured by the time I joined the company in 2000, the merit badge metaphor was *the* metaphor used.

There were probably others at the time saying the same thing. I don’t know. But it seems like someone should give Schank some credit.

Incidentally, I have mixed feelings about badges. I was bullish on the concept even as late as 2007, but since then I have become more ambivalent.  I think I’ve become aware that while much of the badge talk centers around student learning, cognitive science, and motivation, the real reason badges are being pushed relentlessly is a lot of companies want to tap into some of that free taxpayer money for education, but don’t have the means or the patience to buy and run an accredited institution. The end game of all of this is that Facebook finds a way to skim money out of the Pell Program, or some Silicon Valley startup gets to take money from government-backed student loans.  That’s what badges are really about.

I’m not saying that we shouldn’t have new players in higher education, and I’m not saying badges aren’t a useful tool in our instructional design toolbox. It just makes me nervous when we discuss dismantling accreditation barriers through proxy issues like badges.

Mean vs. Median Student Loan Debt

I found this surprising (emphasis mine):

The average outstanding student loan balance per borrower is $23,300. Again, there is substantial heterogeneity in balances of individual borrowers. The median balance of $12,800 is roughly half the average level, which indicates that a small fraction of people have balances significantly higher than the median. About one-quarter of borrowers owe more than $28,000; about 10 percent of borrowers owe more than $54,000. The proportion of borrowers who owe more than $100,000 is 3.1 percent, and 0.45 percent of borrowers, or 167,000 people, owe more than $200,000. The distribution also varies by age group: for example, borrowers between the ages of thirty and thirty-nine have the highest average outstanding student loan balance, at $28,500, followed by borrowers between the ages of forty and forty-nine, whose average outstanding balance is $26,000 (see chart below).

Actually, there’s a surprise for me in almost every sentence here. First, the mean loan amount we hear tossed around all the time averages in only borrowers — students who do not borrow are not averaged in. To me, that’s deceptive. I can’t find the number right now, but I think between 25% and 33% of undergraduates take out no loans. If it is 25%, you can average in those zeroes by knocking 25% off that figure (I think that works, right?) and you get something like $17,500 — which isn’t as impressive.

Second, that is a huge difference between mean and median loan amount. That tends to indicate a very skewed distribution overall, with a few higher borrowers bringing up the average dramatically. That’s a different (and easier) problem than a high median debt.

Third, I don’t know what to make of the high borrowers (the top 3%). Maybe these are for medical programs, or something else I don’t understand. Could they even be for undergraduate education? $100,000? Maybe I’m missing something? Is this accumulated debt + (a lot of) unpaid interest?

If it is for normal, undergraduate programs, should we be encouraging debt of this sort?

Finally, the average debt of the different age groups. Actually, the heavily indebted forty year olds are not  that surprising. Your prototypical debt would be paid off between 24 and 34 years old.  I imagine a high percentage of cases in those older age groups are defaulters, people that couldn’t pay down the debt over twenty years, and that would drag the average up. The amount in the thirty year old age bracket surprises me though, because I would think that people almost done paying down debt would offset people that are defaulting. So there’s something weird going on there….will look into that….



The New College Completion Site and Disruptive Change

The new Chronicle College Completion Site may be met with groans from people who think the wrong metrics have been chosen or that necessary detail has been omitted, but I think it’s wonderful.

Sure, I’m not sure that cost per degree should be the central figure in the debate. I think cost per degree probably does not account enough for selectivity issues (take only students you *know* will graduate, and your cost per degree will go down). But even with that caveat, it provides a fascinating example of what many people have been trying to communicate to the press and public for years, to no avail: there really is no single educational productivity crisis, and measures which lump Dartmouth in with Podunk State are dangerously deceptive.

To understand why that is, take a bookstore (bookstores, which have been hit by the End of Retail phenomenon, are often used as an analogy for higher education). Bookstores have a given cost of operation, and that cost gets put into the markup on the book. Borders has failed, Barnes and Noble has not — but essentially the economics of both are pretty similar. We wouldn’t expect Barnes and Noble to have a cost of 32 cents per book sold and Borders to have a cost of three dollars per book sold. We’d expect these prices to track pretty close.

Education does not work like that. Education is more like a hospital system that routes the most difficult cases to hospitals that operate on pennies a day, and routes hangnails to research centers. Yale University spends $500,000 per degree. And they do that while being highly selective (96% complete within six years, median SAT 1472), which should technically reduce the cost. Fort Hays State University spends $29,000 per degree, with only 40% completing in four years (median SAT score: 990).

Then we have places like UMW, which spends $46,000 per completion, with a high completion rate (6 years: 75%), but a mid-range selectivity (SAT: 1170). Or Keene State, which is less selective than UMW (median SAT score: 1000), and pays for that with lower completion (56%) that results in a a higher cost per degree ($64,000).

We could go further into the data, but that’s probably enough to get the point. The obvious point is that the distribution of spending in education makes the U.S. health care system look positively socialist.

The less obvious point is that is that, once you get away from the Dartmouths in the equation we are actually pretty close to a workable system.

Let’s start with some idealism. If Fort Hays could double its completion rate while keeping costs down, the cost for a four-year degree would be $15,000. That’s the full cost, not just the student-side or subsidy side. At that rate we could give 40% of today’s 18 year olds a chance to earn a degree at a cost of less than 40 billion. And that’s with us picking up the total tab. Have students pay $2,000 a year, and the tab is $9,000. Not per year, mind you. For the entire degree.

At $9,000 per degree the cost to the U.S. government is about 22 billion to graduate 40% of the population. Another 20% would be served by two-year technical colleges and trade schools at even lower cost.

To give you an idea of the scale of 22 billion in terms of federal spending, the current Pell program is over 30 billion dollars a year.

People will jump on, I’m sure — hey, I’ve seen Fort Hays, I don’t like it! Or, better, how would you double the graduation rate? Well, if you don’t like Fort Hays, pick somewhere else. If we can’t double graduation rates, then increase them by 50%, or 40% and redo the math.

Or work the other way. Pick a place like Keene State or UMW, keep graduation rates constant, but reduce cost through a hybrid online/face-to-face approach. This is what University of Central Florida has been doing at a huge scale. It can be done elsewhere too.

Change the numbers in any reasonable way, and you’re still in the same ballpark. We are not at the point where we have to pursue zero-marginal cost options as end-to-end solutions.

People talk a lot about disruptive technology, but then assume that the disruption will look like MITx giving certificates out. When I look at these figures, that’s not what disruption looks like to me. Change the reward structure of higher education, and disruption looks like the University of Central Florida, or a souped-up UMW. There are admittedly huge institutional and cultural barriers to this — read The Innovative University for a summary of why we all aspire to be non-productive Dartmouths instead of efficient state universities — but evolution does not require that all institutions evolve. Given the right reward structure we only need a few institutions to show the way. I continue to think that’s possible.



College Completion Rates vs. College Attainment Rates

Regular readers will know I geek out about this stuff — the way different metrics within an industry can give radically different pictures of what is going on.  And so, with the recent focus on completion and attainment rates, it’s helpful to note that one does not imply the other. Here’s Arthur Hauptman on the subject:

In some other important ways, though, the attainment and completion debate in this country has taken some wrong turns over the past five years. One such misdirection is the assumption that higher completion rates automatically result in higher attainment rates. To understand the problem here, we need to recognize that completion and attainment rates do not measure the same thing. Completion rates are the percentage of students who finish the program they began, while attainment rates measure the share of the adult population who hold a degree.

Traditionally the United States has had high rates of bachelor’s-degree attainment compared to some other countries. But this ranking was not achieved by having high completion rates. Indeed, for a long time we have had comparatively mediocre completion rates because we expend more effort to increase access than many other countries that have more elitist systems of higher education. So the traditional U.S. leadership in bachelor’s-degree attainment has been a function of enrolling lots of students who, even with modest completion rates, produce very high rates of bachelor’s-degree attainment. The stimulus for the national debate we are having is that we now rank much lower on all measures of educational attainment.

This matters, quite a lot (emphasis mine):

But it is a mistake to assume that modest completion rates are the reason for our lower standing when it comes to attainment. Rather, the philosophy and the policies of countries, states, and systems of institutions are the primary determinant of completion rates. For example, a country with an elitist higher-education system that allows only 10 or 15 percent of its population to enroll in college is likely to have a much higher completion rate than a country that allows a much broader share of its population to enroll.

Similarly, it is also a mistake to use institutional completion rates as a measure of educational quality, because institutional selectivity is by far the principal predictor of completion rates. An open-access institution that graduates 50 percent of its students is most likely doing a much better job of educating its students than a highly selective institution that graduates only 60 percent of the students who enroll. So in assessing the record of institutions with regard to their quality, they ought to be compared with peer institutions with similar degrees of selectivity.

Does Dartmouth Affect New Hampshire’s Student Debt Rates?

I was looking at an older NY Times article about student debt, and noted this paragraph:

The report found that student debt loads vary substantially from state to state: New Hampshire students topped the list, owing an average of $31,048, while Utah’s students averaged half that. Students borrow more in the Northeast and Midwest, where there are more private nonprofit colleges, than in the West, where a greater share of students attend public universities.

I’d generally blamed the New Hampshire debt problem largely on our higher in-state tuition, and certainly that’s huge factor. But something occurred to reading that line — we are a small state with one of the world’s most expensive private schools in it. Published tuition at Dartmouth tops $41,000 a year; with room and board it’s about $55,000 a year. [Add net price caveat here, yadda, yadda]

Something like that could have an outsized effect in a small state. So just out of curiosity, what percent of New Hampshire students attend Dartmouth?

I wanted to limit it to undergrad, but I also wanted a quick answer. So for a rough guesstimate I took the table here, removed the community college system from it, and ignored the undergrad/graduate distinction.

And the interesting thing is that Dartmouth does have an impact on the state. There’s about 55,000 students in New Hampshire, Dartmouth has about 10% of them. Even if you were to filter out the post-bacc students across the board, Dartmouth would still approach 10%. To put it in perspective, Dartmouth has almost as many undergrads as Keene State.

So what’s the effect? Here’s where it goes in a direction I didn’t expect — Dartmouth students graduate with less debt… if we filtered out the effect of Dartmouth, our students would actually owe more than the average currently indicates.

Example COMPARABLE Analysis: College Graduate Debt

What follows is an example COMPARABLE analysis of the beginning of an article. You’ll notice that although the comparison in the article is really just a comparison between last year’s debt and this year’s debt it is sort of a structured examination of the number used and its implications.

The COMPARABLE framework is the framework I am writing my Making Fair Comparisons textbook around. This particular assignment is to apply it only to the visible part of the article as a pre-reading exercise that helps you think about the questions you need the rest of the article to answer. What is below is not a finished analysis, but a stream of consciousness of me working through the framework.

The neatest thing I found was that it helped me realize one or two things I might have not noticed without it. The second neatest thing was it was fun.

Article and COMPARABLE worksheet follow.


C: Comparison Groups: The main comparison here is to the previous year. Up 5% — it’s a longitudinal claim. But certain other comparisons are missing. First, it might be nice to compare this with debt in other countries. Second, it might be nice to know whether this is a particularly large gain or not. Was lat year’s gain bigger or smaller?

O: Operationalized/Defined: Debt is probably fair well defined. The students here are defined as 2010 graduates, which is fair enough. Are 2-year colleges included here? Not sure. If they were, that would make the price look lower. Oh – hmmm, one more interesting thing “Students who graduated from college with student loans” – it looks like students owing no debt are not averaged in. Why? That’s extremely odd. Also, do loan amounts include upfront interest? Also: why average? Why not median? Not sure as well – is this based on survey or summary federal figures?

M: Mental Experiment: The figure seems a bit high to me. If we assumed this was accumulated over four years, and the figure was indicative of real experience, that’s $6,000 a year. But that’s not just tuition, that’s room and board, I suppose. And interest, perhaps? And private colleges are averaged in, which might up it. One other experiment – if we make the average stay longer, say 6 years, the figure drops to the low $4,000s per year, which seems reasonable, even low – so knowing how long the average stay was might matter.

P: Pictorial/Graphical: None in this segment.

A: Accounted/Controlled for: It’s not clear that inflation was controlled for – although the debt was accumulated over four years, so I’m not sure how much last year’s inflation would account for. Some. Population doesn’t matter, this is already a per student number. Length of time in college might be interesting to control for – see above.

R: Randomness: This is actually a four or five year result (each year they accumulate more debt). So that helps protect against year to year randomness in net price. Plus, how much does price really zig-zag. As far as the result, it’s unclear to me from the section above whether it is a survey or derived from summary numbers in federal reports. Or somewhere in between. If it is a survey of individual students randomness would be more in play than if it is summary federal data.

A: Alternative measures: We have the amount, and the percentage here, which is good. Rate of change might be a useful level of abstraction. For instance, if last year’s was a 1% increase and this year is a 5%, that would represent acceleration of this trend. If last year was 5% it is a fairly level trend, etc.

B: Base rate? I’m actually unsure of how that would apply here, at least in base-rate-specific terms. Thinking in really broad terms, it might be nice to know how this compares to other debt. And again, in really broad terms, what is the “right” amount of debt? Tangential to base rate, but important. And since we are counting only students WITH college loans – what percentage DON’T have college loans?

L: Longitudinal/Cross-sectional: This is a longitudinal claim. As mentioned above, it might be good to get some cross-sectional comparisons – debt non-college students accumulate, perhaps. Probably more importantly it’s worth asking some of the key longitudinal questions – Are we too zoomed in, or not zoomed in enough? (Answer, we might want to zoom out more, one year is very prone to nontypical results due to other factors). There’s no cyclical issue I can think of – except maybe the recession? But again, with this representing 4 to 5 years worth of debt, these things are minimized.

E: Edges, Center, Distribution, Subpopulations: Our measure of center here is a mean, with zero values removed. So we are not taking into account the bottom edge of this, but we are counting the top. There’s no information about shape or spread in these first paragraphs.


Interesting findings:

  • This is only an average of students that have debt, which is not how most people would read it.
  • Would like more indication of how 5% increase compares to previous years.
  • Increased debt could be a function of increased cost, but also increased time in college. It would be nice to tease that out.
  • Not indexed for inflation –  5% is small enough that inflation could be significant.
  • Survey? As I read further in this article I’ll see if it was a survey. If it is a survey it may be more prone to bias, with people either over-reporting or under-reporting debt.


Repeat After Me: The Dose Makes the Poison

Today I came across this on the web:

why is everyone quitting gluten now?

There are a number of very valid reasons for the growing number of people having to pass on the pizza.

The short form: gluten is a poison (see below). We tolerate it, and tolerate it, like cigarettes in the lungs. And then. One day. It’s too much. Things tip over and BANG we have lung cancer. Or gluten intolerance. Or celiac’s disease.

But more detail…and they’re facts, mind.

Toxicology is a complex subject which I know little about, but I do know the first rule of it: the dose makes the poison.

Everything is toxic at some level: water, oxygen, peaches, brazil nuts. Everything.

The question is always how much, over what time period (and possibly in what pattern).  It makes no sense to talk about something as a poison apart from dose.

This seems to be one of the primary public misunderstandings of science. The idea is if a little of something is good for you, then a lot must be really good (see the harmful nonsense around antioxidants), and if a lot of something is bad for you, then even a little must be harming you in ways you don’t realize (see wheat, yeast, sugar, vinegar, and just about everything else).

I don’t doubt, by the way, that there are many people with horrible reactions to gluten. I know at least one person who has celiac’s disease. It’s horrible.

But in those cases, that’s due to personal factors, individual physiology. There’s really no reason that celiac’s disease is telling us something fundamental about our bodies any more than a latex allergy tells us some dark secret about latex.

I’m allergic to dogs, for example. Been hit so bad with it that I’ve ended up in the ER, and even after a mild attack I am messed up for quite a bit. So is every dog owner really just in denial about the slow death their dog is causing them?

My daughter is deathly allergic to peanuts. We found this out when she was two — she had a peanut butter cookie and in about 20 minutes looked like the Elephant Man. Her eyes swelled shut, her breathing got labored, and only Benadryl saved her. We now have an epi-pen around at all times.

So are peanuts a secret killer? (Someone should let India know!)

Once we get past edge cases, it’s all about dose. Star fruit contains a neurotoxin. Too much Vitamin A will kill you, and even a Vitamin E pill a day may radically increase your chance of prostate cancer. Fish contains mercury, and of course overdosing on water is not uncommon.

All these things are pretty good for you (or even necessary) in moderation. None of these facts tell us anything except that the cases at the edges are seldom some secret key to the center.

Does it matter? I think I agree with Ben Goldacre on this one — it matters, but not in the way you might think. Avoiding wheat is unlikely to kill you, but the magical reasoning invoked to explain why you are avoiding it may kill someone else. “Gluten is a toxin” is a gateway drug into anti-vax nonsense and global warming skepticism. And the flood of diet nonsense in popular magazines and on daytime TV support a notion that science is as simple as making a few logical connections and calling it a day.

If you don’t want to eat wheat, or if you are a person who really *is* harmed by wheat consumption, then please don’t eat wheat. But leave science alone, we need it for other things.