Comparing Porn Prosecutions

One of the things I like about the COMPARABLE framework is how nicely it can be used not only to evaluate existing comparisons, but to think through what a fair comparsion would look like where none is provided. For instance, today I saw this:

“Well you have to look at the proof that’s in the prosecution. Under the Bush administration, pornographers were prosecuted much more rigorously under existing law than they are under the Obama administration,” Santorum said. “My conclusion is they have not put a priority on prosecuting these cases, and in doing so, they are exposing children to a tremendous amount of harm. And that to me says they’re putting the un-enforcement of this law and putting children at risk as a result of that.”

The first thing is the habit of mind — when a student sees the word “more”, hopefully that triggers comparison mode. Honestly, getting that bell to ring is the hardest bit of this. Once we are there, we use our framework:

C: Comparison groups are the prosecutions under Bush vs under Obama. Fair enough.

O: Santorum talks about rigor, and priority — but the key claim here seems to be “un-enforcement” so the best variable seems like it might be number of prosecutions. But we’d also probably have to find some way to take into account severity of crime. A small prosecution on a fine-able offense should not equal a large prosecution with jail terms, etc.

M: Mental experiment on this is hard, since there are no numbers to run through. So we’ll skip it.

P: Again, without numbers, there’s nothing to do here.

A: We probably want to look at this not only from raw numbers, but as a percent of the DOJ’s total effort. There might also be other factors. Since there are limited resources of the DOJ, any large operation in a non-porn area that requires the same people might make resources scarce. If we do raw numbers, we probably also want to make sure we are taking into acount Obama has only been in office three years — a good comparison might be the last three years of the Bush administration to the first three years  of Obama. Finally, we might look at action controlled for the size of the porn industry — a bigger industry might require more regulation.

R: Randomness is not such an issue here as we’re not sampling. But there might be some year to year randomness in the number of prosecutable offenses.

A: Alternative measures might include looking at this as a trend. For example, were prosecutions declining year over year in the Bush administration, a decline under Obama might be a continuation of a historical trend. If they were increasing under Bush, even a stabilization under Obama would look like a redirection of resources.

B: Base rates — I’m not sure what relevant base rates are here. Again, we don’t have numbers. But obviously understanding whether a percentage increase or decrease is meaningful will require absolute numbers and an idea of how prosecutions relate to offenses. It might also be useful to see if the Bush administration is the historical anomaly here.

L: This is a longitudinal comparison. It might be interesting to go the other direction too, and look at how a country like 2011 Canada prosecutes this stuff.

E: Not sure how distribution effects this, although subpopulations is obvious — we’d want to look at how different types of crime account for the whole of prosecutions in each administration — the likelihood is that that breakdown would tell us far more than the headline statistic of prosecutions.

One thing to notice about COMPARABLE is it avoids going directly down the association rabbit-hole — in this case the “kids exposed to harm” piece. While that’s certainly important, I find that those questions end up being too nuanced for many of our students. The comparison question here is complex, certainly, but there is a certain concreteness to it that is helpful to the beginning student of QR.

Number Needed to Treat!

Number Needed to Treat is an aggregate measure of clinical benefit that medical study geeks love because it has a comprehensibility lacking in odds ratios and relative benefit percentages.

It represents the number of patients that would need to receive a treatment for one of the patients to avoid an adverse outcome (death, stroke, development of diabetes, high cholesterol). For instance, say we want to want to put people on a Mediterranean Diet after a first heart attack. How many heart patients would we have to put on a Mediterranean Diet to save one life? Here’s the answer from 30!

That’s pretty good, especially with no known harm. And it’s easy to conceptualize.

I was reminded of how few people ever come into contact with the NNT when I read the excellent thread in this NPR post.

If you don’t know the NNT metric, click that link and learn about it. It’s a great way of conceptualizing the benefit of interventions, and it may even help you think about your own work in a different way.

More Confused Credit Card vs. Student Debt Reporting

Student debt is a problem, I think. I’m pretty sure about that. But reporting like this isn’t helpful:

But this is a bigger problem than many realize. A recent study by the Federal Reserve Bank of New York found that outstanding student loans have surpassed the nation’s $693-billion credit card balance.

Even more eye-opening, nearly 80% of Americans held credit cards as of 2008, compared with 15% of consumers who now hold student debt. That illustrates just how small of a pool of Americans holds this huge pile of debt.

I don’t know where to start. Here’s a couple quick hits:

  • Student Debt is non-revolving debt. It gets borrowed in a big chunk and paid off over years. Credit card debt is revolving debt — expanding and contracting on a monthly basis. If student debt is similar to anything, it is probably similar to other non-revolving debt, like mortgages and car loans.
  • Student Debt has been rising, but the big reason it passed credit card debt recently is that credit card spending contracted massively during the recession.
  • According to the article, student debt is a problem because of it’s scale. But when it’s revealed that it is only held by 15% of the population, that’s… more proof it’s huge?
  • Supposedly the large amount combined with the small amount of holders shows something about how much debt the average loan holder owes. Except we already know how much the median holder owes — it’s around $13,000. Which is too high, sure, but also a directly citable figure that the article could provide.

In terms of the COMPARABLE framework, I think “M” is particularly important here. Do a mental experiment. Imagine instead the following headline, which inverts the comparison:

Credit Card Debt Soon to Exceed Student Loan Debt

Would we see that as a good thing? Or would we look at that and wonder why people don’t spend money on things with tangible long-term benefits, instead of buying junk?

Or consider this headline, which chooses an alternate comparator (“C” in the framework):

Student Loan Debt Exceeds Auto Loan Debt for First Time

Is that shocking? Or does it actually sound pretty sane? Because here’s the thing — auto loan debt is the same as credit card debt right now. And it’s hands down a better comparator — non-revolving debt that focuses on something of direct utility. It wouldn’t have *just* been exceeded by student loans, but the amount is functionally equivalent to credit card loans right now:

Straw Men

Occasionally when I argue against the claim that higher education is on the verge of catastrophic collapse, and warn that MOOCs are being advanced by many as a replacement for higher education instead of a supplement to it I’m told I’m arguing against straw men. Where are these people? Show me the articles!

The truth is the sentiment is so widespread no one bothers to to write an article about it.  It’s the assumed subtext of most of what you read. However, I couldn’t help but notice this comment on that article I just critiqued:

[Student debt] may not be a ‘bomb’ on the overall economy, but it is a precision guided device that will blow up higher education.  The sub-prime problem resulted in price inflation and excessive borrowing… sound familiar?  Sub-prime killed the housing market just as student debt will start a series of disruptive changes to what we  now recognize as higher education.  Higher education is over subscribed and over priced while powerful alternatives are visible on the horizon.  See article on Massive Open  Online Courses “MOOCs” in the NYT or last week’s 60 minute piece on Khan Academies.  The disruptive technologies are in place while student indebtedness is the trip wire.  It used to be that the quality of higher education could be correlated with brick & mortar square footage per student… does that make any sense today?   An education Tsunami is coming and the globe is flatter than it was… hold on!

This isn’t some brilliant education visionary showing up in the comments. This is a guy who saw a show on TV, read an article in the Times, and is parroting back their implicit meaning to whoever will listen. It’s not a novel interpretation, it’s a banal one. And it should bother anyone who believes the truth is more subtle than that.

The Difference Between Median Student Debt and What the Median Student Owes

There’s a great article in The Atlantic on student debt, and it’s well worth a read. But it makes what I believe is a common mistake on student loan debt. Check out this paragraph…

 When teenagers are forced to take out loans in order to pay for their education — the median graduate leaves school $12,800 in debt — it acts as a tax on their future wages. It postpones their ability to settle down, buy a home, and have children. That’s tragic for them, and it’s tragic for us, because it means less money will flow into other, more productive parts of the economy.

That figure comes from the Fed’s Liberty Street Economics:

The average outstanding student loan balance per borrower is $23,300. Again, there is substantial heterogeneity in balances of individual borrowers. The median balance of $12,800 is roughly half the average level…..

Notice, however, the difference.  This is the median balance — this is the median value of loans, not what the median student owes. That’s an important distinction, since only about 66% of students take out loans.

To consider how that might affect things, consider this set of numbers, where the median is 5.5:


Now what happens if we add our 33% of zero values to this? Well, if this set represents 2/3 of the values, then there are five zeroes to add in to fill it out:


Our median now drops from 5.5 to three. Now admittedly, our example is not a normal distribution, but we’d still expect a drop from the 50th percentile to the 33rd or so, which should be substantial.

This seems trivial — I agree that student loan debt is an issue, and the the number of students graduating with debt is a problem. But if we are to address the problem effectively, knowing the precise scale and shape of the debt is important.



Medical Student Debt is a Significant Subpopulation of Student Debt

From here:

The amount of student loans taken out last year crossed the $100 billion mark for the first time and total loans outstanding will exceed $1 trillion for the first time this year. Americans now owe more on student loans than on credit cards, reports the Federal Reserve Bank of New York, the U.S. Department of Education and private sources.

From here:

For almost three generations, debt has been a nearly inescapable part of becoming a doctor. Over 80 percent of each medical student class will graduate in debt; and while that percentage has remained unchanged for 25 years, the increase in the total amount owed has leapfrogged over all other economic reality checks, like inflation and the consumer price index.According to the Association of American Medical Colleges, which has been trying to address the problem for nearly a decade, young doctors who graduated from medical school last year had an average debt of $158,000, or$2.3 billion for the group as a whole. Almost a third of students owed more than $200,000, a number that will only increase with the addition of interest over payback periods of 25 to 30 years.

There’s a per year/per education problem with comparing any numbers here. But the scale is such that medical student debt probably accounts for a nontrivial amount of the difference between mean and median student debt. Law school debt is may be another factor, though I don’t have any good numbers on that yet.

Update: I actually wrote this before Anya K. confirmed it for me (I generally write a ton over nights and the weekend, but publish into a queue that spaces them out over the week). But thank you Anya for your reply — it’s good to get confirmation from someone that has been over the numbers in detail. What I’d really like to know is excluding this sort of debt, what does the debt picture look like? Sure I can find that somewhere….

Stinking Badges

(Ha, you thought I was going with the Brooks movie or Sierra Madre, right?)

Because this seems as good a place as any to note it, I think one person who doesn’t get credit enough on learning badges is Roger Schank. When I went to work for his company in 2000, he’d already been talking about a “merit badge” approach to assessment for years. And for certain projects with Harvard Business School Publishing and others, we essentially implemented those approaches.

Here’s Schank talking in 1999 about it:

“We won’t get rid of certification but perhaps we can contemplate new kinds of certification. Students should be certified as having accomplished something or as being able to do something. Like Boy Scout merit badges or Karate black belts or Truck Driver’s licenses, the proof should be in the pudding. A student should show his stuff, he should be able to do something and the attestation to the doing should be the certification.”

And if that leaves you thinking that “badges” was only one of many ways he expressed this idea, be assured by the time I joined the company in 2000, the merit badge metaphor was *the* metaphor used.

There were probably others at the time saying the same thing. I don’t know. But it seems like someone should give Schank some credit.

Incidentally, I have mixed feelings about badges. I was bullish on the concept even as late as 2007, but since then I have become more ambivalent.  I think I’ve become aware that while much of the badge talk centers around student learning, cognitive science, and motivation, the real reason badges are being pushed relentlessly is a lot of companies want to tap into some of that free taxpayer money for education, but don’t have the means or the patience to buy and run an accredited institution. The end game of all of this is that Facebook finds a way to skim money out of the Pell Program, or some Silicon Valley startup gets to take money from government-backed student loans.  That’s what badges are really about.

I’m not saying that we shouldn’t have new players in higher education, and I’m not saying badges aren’t a useful tool in our instructional design toolbox. It just makes me nervous when we discuss dismantling accreditation barriers through proxy issues like badges.