So we have this COMPARABLE framework I’ve been working on, where COMPARABLE is an acronym for the sorts of things you want to look at when presented with a comparison.

The “M” in the acronym stands for “Mental Experiment”, and it’s a reminder that a lot of sanity checking claims is about taking some guesstimate numbers and running them through the claim to see if they make sense.

A good example is this recent quote from candidate Rick Santorum:

He claimed that “62 percent of kids who go into college with a faith commitment leave without it,” but declined to cite a source for the figure. And he floated the idea of requiring that universities that receive public funds have “intellectual diversity” on campus.

There’s a lot of definition problems here — but given that there is no source, we we have to fill this stuff in. I am going to read it the way I think he meant it to be taken:

“college”: Any college, including community college
“leave”: Any way of leaving, including not finishing
“w/o it”: I’m assuming a loss of faith (not a change in denomination)

The premise of a mental experiment, in this case at least, is to ask what the world would look like if this was true. Let’s be charitable, and say that every single kid that goes to college is a person of faith. In that case, this model would predict that 62% of people who have attended college are atheists or agnostics.

So what rate of atheism would that predict in America? Here’s some data from the 2000 census from 25 year-olds and over:

• 21% of Americans had taken some college courses but had not earned a degree in 2000, compared with 18.7% 10 years earlier.
• 15.5% had earned a bachelor’s degree but no higher, compared with 13.1% in 1990.
• 8.9% earned graduate or professional degrees, compared with 7.2% earlier.

Rounding up, 21+16+9 = 46% of Americans who have “left college” in one way or another. Again, assuming that absolutely no kids went to college unreligious, 46 * 0.62 = 28.5. So what we should see in populations that have graduated college in the last 20 years is an atheism/agnosticism rate of about 28.5%.

Looking here, we find that about 22% of the 18-29 set is non-religious (if you include Deism, which is kind of suspect — isn’t a belief in God a faith commitment? But again, will give a charitable reading here).

So 22% does not equal 28.5%. And it’s not really that close. We’re doing some apples to oranges here (gen population to 18-29 year olds, etc.), but nothing that could possibly account for that gap, considering how generous we’ve been in other areas of the model. So we should be highly skeptical of this claim.

There’s one final nail in the coffin of this claim — remember that in order to get that 22%/30% comparison we had to attribute every nonreligious conversion to a college education (30% was just the level of non-religion predicted by college education, if nonreligion existed without college education, the level would be much higher).

So is that what we find?

No:

“Nones” in this context are people who identify under the “religion” question on the American Religious Identification Survey as “No Religion”. As you can see, there is a small association between a college education and nonreligion, but it’s very small. College graduates are 20% of the nones, and 17% of the population. That’s not much of a difference. And people who have “some college” college are actually more likely to be religious than the general population (24% of nones, 26% of population).

Actually, if you needed to answer the Santorum question, you’d really have to just show that graph above (and if I was writing a political post on this, rather than a Stat Lit post, that’s what I’d do). But I wanted to show the process of thought that leads one to ask the questions that leads to finding the graph. It’s a messy imprecise process that has a ton of stuff in it that is pretty questionable.

But that’s what mental experiments are supposed to be  — they don’t provide definitive answers, but they give you a handle on the questions raised — and once you know the right questions, you’re most of the way there. A mental experiment is your first fumbling attempt to get a grab on something solid.