This is from the University of Edinburgh. It’s being heavily circulated right now in edtech. I agree with most of it.
I find point two particularly a) true, and b) problematic. And if the truth of the manifesto is to be useful truth, that’s the piece that has to be solved institutionally first.
So we have this COMPARABLE framework I’ve been working on, where COMPARABLE is an acronym for the sorts of things you want to look at when presented with a comparison.
The “M” in the acronym stands for “Mental Experiment”, and it’s a reminder that a lot of sanity checking claims is about taking some guesstimate numbers and running them through the claim to see if they make sense.
A good example is this recent quote from candidate Rick Santorum:
He claimed that “62 percent of kids who go into college with a faith commitment leave without it,” but declined to cite a source for the figure. And he floated the idea of requiring that universities that receive public funds have “intellectual diversity” on campus.
There’s a lot of definition problems here — but given that there is no source, we we have to fill this stuff in. I am going to read it the way I think he meant it to be taken:
“college”: Any college, including community college
“leave”: Any way of leaving, including not finishing
“w/o it”: I’m assuming a loss of faith (not a change in denomination)
The premise of a mental experiment, in this case at least, is to ask what the world would look like if this was true. Let’s be charitable, and say that every single kid that goes to college is a person of faith. In that case, this model would predict that 62% of people who have attended college are atheists or agnostics.
So what rate of atheism would that predict in America? Here’s some data from the 2000 census from 25 year-olds and over:
- 21% of Americans had taken some college courses but had not earned a degree in 2000, compared with 18.7% 10 years earlier.
- 15.5% had earned a bachelor’s degree but no higher, compared with 13.1% in 1990.
- 8.9% earned graduate or professional degrees, compared with 7.2% earlier.
Rounding up, 21+16+9 = 46% of Americans who have “left college” in one way or another. Again, assuming that absolutely no kids went to college unreligious, 46 * 0.62 = 28.5. So what we should see in populations that have graduated college in the last 20 years is an atheism/agnosticism rate of about 28.5%.
Looking here, we find that about 22% of the 18-29 set is non-religious (if you include Deism, which is kind of suspect — isn’t a belief in God a faith commitment? But again, will give a charitable reading here).
So 22% does not equal 28.5%. And it’s not really that close. We’re doing some apples to oranges here (gen population to 18-29 year olds, etc.), but nothing that could possibly account for that gap, considering how generous we’ve been in other areas of the model. So we should be highly skeptical of this claim.
There’s one final nail in the coffin of this claim — remember that in order to get that 22%/30% comparison we had to attribute every nonreligious conversion to a college education (30% was just the level of non-religion predicted by college education, if nonreligion existed without college education, the level would be much higher).
So is that what we find?
“Nones” in this context are people who identify under the “religion” question on the American Religious Identification Survey as “No Religion”. As you can see, there is a small association between a college education and nonreligion, but it’s very small. College graduates are 20% of the nones, and 17% of the population. That’s not much of a difference. And people who have “some college” college are actually more likely to be religious than the general population (24% of nones, 26% of population).
Actually, if you needed to answer the Santorum question, you’d really have to just show that graph above (and if I was writing a political post on this, rather than a Stat Lit post, that’s what I’d do). But I wanted to show the process of thought that leads one to ask the questions that leads to finding the graph. It’s a messy imprecise process that has a ton of stuff in it that is pretty questionable.
But that’s what mental experiments are supposed to be — they don’t provide definitive answers, but they give you a handle on the questions raised — and once you know the right questions, you’re most of the way there. A mental experiment is your first fumbling attempt to get a grab on something solid.
For the stats text, I’ve been trying to think of/find rules that apply across a wide array of disciplines. Here’s one: control for cyclical effects. It applies here (summer gas spikes):
And here, with voting in elections — presidential elections (a cyclical event) boosts turnout every second congressional election):
And here is 2010:
The left-most column, incidentally, is Republicans, followed by Democrats in the second column. So did Democratic support collapse, dropping by nearly 50%? Of course not. It’s very difficult to make any comparison of a mid-term to a presidential cycle election, because in mid-terms different sorts of people tend to turn out. In general, midterms tend to pull out more anti-incumbency voters than pro-incumbency, and in this case the incumbency in 2010 was Democratic.
If you want to use the mid-term results to predict the general election you are going to have to come up with some model to account for the difference. Maybe you could look at subpopulations or some other measure. But the point is, like the gas prices, you have to control for cyclical effects.
I’d love to hear about additional examples of cyclical effects from your academic disciplines or the jobs you work. Shoot me an email, or post to twitter, facebook, google…
Gas prices are up again. What’s going on? Who’s to blame?
Of course big long term driver is China. China needs more gas, and its exploding demand has put pressure on the market. But what about the recent spike?
Apparently the big villian is… Chemistry:
Gas prices fluctuate seasonally — up in spring, down in fall — for a very specific reason: Butane. As Rapier wrote last year “Butane is a cheap ingredient in gasoline that boils at low temperatures. In winter, this isn’t a problem. But in summer, butane evaporates from gas, polluting the air while leaving us with less fuel in the tank than we paid for. As temperatures rise, refineries replace butane with more costly ingredients and draw down winter inventories just as beach season begins. Chemistry, not corporate conspiracy, limits supply.”
This happens every single year. You’d think people would have caught on by now. But sadly, no.
Prepare yourself for the shock when prices go down right before the elections. It’s a conspiracy!
These sorts of graphs are very in right now, as the format is perfect for showing change of distribution over time, and so much of our political discourse is dealing with questions of distribution. I used to see these relatively rarely, but in the last couple of years they are all over the place. Which is good! It’s an extremely compact way to show a trend and control for it.
One note on teaching graphs — I think there is a tendency with students to fall back on the idea that “Percentages should be done in pie charts”. As this shows, nothing could be further from the truth. Pie charts are a lousy form which people expect now, but there are so many more elegant was to deal with part whole relationships, especially ones we track over time.
Making some progress on the Making Fair Comparisons textbook. The preface is below.
One thing I’ve learned from reading cheesy self-help books: If you believe a skill will change a person’s life, you should say it. At the end of the book, the reader will know if their life is changed or not. There’s time to be cynical later. At the beginning of the book, let your passion show.
So anyway, here’s the cheesy intro to the text. I love it.
Why we compare
Which intersection in town is the most dangerous?
How much more expensive will college be if I graduate a year late?
Which product line has given our business the best overall return in the past two years?
How much more campaign money was spent in the election of 2008 compared to previous elections?
Comparisons don’t happen in a vacuum. Usually when someone is comparing things, they are comparing them for a reason. In the case of the intersection question above, maybe there is an action pending – if we are going to upgrade one intersection, which one should it be? Businesses may want to know what products have been the most profitable so they can pursue profitable avenues at the expense of the less profitable ones. A political scientist may be investigating the influence of money on elections, and trying to determine if that influence has increased over time.
Ultimately, comparisons have real world consequences. If you rightly determine which intersection is the most dangerous as an urban planner, perhaps you can save a life. Knowing which product lines have given a company a good return could be the key to keeping a business afloat, saving your job and the jobs of others. Determining whether money in elections is out of control or in line with historical trends can help us plot a course of action for our country that fixes what is wrong with our system while preserving what is right.
Depending on what profession you go into, you may use algebra or you may not. Some of you may use calculus or trigonometry. Some of you will be asked to use advanced statistical methods. Most won’t.
But every single one of you will be asked to compare things as an employee, consumer, and citizen. And whether you are able to compare things adequately will have dramatic effects on the success of your business, your family, and your community.
This is a book about how to use very simple statistical techniques to compare things. It is not so much about formulas as it is about critically thinking about numbers. We honestly believe this skill will be one of the most important skills you acquire in your college career. Mastering it will change your life for the better, and get you closer to being the sort of person you want to be.
I’m writing a intro stats book right now (a small one for students). This lecture really brings home the problem of the “garden path” solution, and how small changes in presentation could make a big difference.