What follows is an example COMPARABLE analysis of the beginning of an article. You’ll notice that although the comparison in the article is really just a comparison between last year’s debt and this year’s debt it is sort of a structured examination of the number used and its implications.

The COMPARABLE framework is the framework I am writing my Making Fair Comparisons textbook around. This particular assignment is to apply it only to the visible part of the article as a pre-reading exercise that helps you think about the questions you need the rest of the article to answer. What is below is not a finished analysis, but a stream of consciousness of me working through the framework.

The neatest thing I found was that it helped me realize one or two things I might have not noticed without it. The second neatest thing was it was fun.

Article and COMPARABLE worksheet follow.

C: Comparison Groups: The main comparison here is to the previous year. Up 5% — it’s a longitudinal claim. But certain other comparisons are missing. First, it might be nice to compare this with debt in other countries. Second, it might be nice to know whether this is a particularly large gain or not. Was lat year’s gain bigger or smaller?

O: Operationalized/Defined: Debt is probably fair well defined. The students here are defined as 2010 graduates, which is fair enough. Are 2-year colleges included here? Not sure. If they were, that would make the price look lower. Oh – hmmm, one more interesting thing “Students who graduated from college with student loans” – it looks like students owing no debt are not averaged in. Why? That’s extremely odd. Also, do loan amounts include upfront interest? Also: why average? Why not median? Not sure as well – is this based on survey or summary federal figures?

M: Mental Experiment: The figure seems a bit high to me. If we assumed this was accumulated over four years, and the figure was indicative of real experience, that’s $6,000 a year. But that’s not just tuition, that’s room and board, I suppose. And interest, perhaps? And private colleges are averaged in, which might up it. One other experiment – if we make the average stay longer, say 6 years, the figure drops to the low $4,000s per year, which seems reasonable, even low – so knowing how long the average stay was might matter.

P: Pictorial/Graphical: None in this segment.

A: Accounted/Controlled for: It’s not clear that inflation was controlled for – although the debt was accumulated over four years, so I’m not sure how much last year’s inflation would account for. Some. Population doesn’t matter, this is already a per student number. Length of time in college might be interesting to control for – see above.

R: Randomness: This is actually a four or five year result (each year they accumulate more debt). So that helps protect against year to year randomness in net price. Plus, how much does price really zig-zag. As far as the result, it’s unclear to me from the section above whether it is a survey or derived from summary numbers in federal reports. Or somewhere in between. If it is a survey of individual students randomness would be more in play than if it is summary federal data.

A: Alternative measures: We have the amount, and the percentage here, which is good. Rate of change might be a useful level of abstraction. For instance, if last year’s was a 1% increase and this year is a 5%, that would represent acceleration of this trend. If last year was 5% it is a fairly level trend, etc.

B: Base rate? I’m actually unsure of how that would apply here, at least in base-rate-specific terms. Thinking in really broad terms, it might be nice to know how this compares to other debt. And again, in really broad terms, what is the “right” amount of debt? Tangential to base rate, but important. And since we are counting only students WITH college loans – what percentage DON’T have college loans?

L: Longitudinal/Cross-sectional: This is a longitudinal claim. As mentioned above, it might be good to get some cross-sectional comparisons – debt non-college students accumulate, perhaps. Probably more importantly it’s worth asking some of the key longitudinal questions – Are we too zoomed in, or not zoomed in enough? (Answer, we might want to zoom out more, one year is very prone to nontypical results due to other factors). There’s no cyclical issue I can think of – except maybe the recession? But again, with this representing 4 to 5 years worth of debt, these things are minimized.

E: Edges, Center, Distribution, Subpopulations: Our measure of center here is a mean, with zero values removed. So we are not taking into account the bottom edge of this, but we are counting the top. There’s no information about shape or spread in these first paragraphs.

Interesting findings:

• This is only an average of students that have debt, which is not how most people would read it.
• Would like more indication of how 5% increase compares to previous years.
• Increased debt could be a function of increased cost, but also increased time in college. It would be nice to tease that out.
• Not indexed for inflation –  5% is small enough that inflation could be significant.
• Survey? As I read further in this article I’ll see if it was a survey. If it is a survey it may be more prone to bias, with people either over-reporting or under-reporting debt.