I’m re-reading my Course Signals post, and realizing how difficult it is to follow the numbers. So here’s an example that might make it clearer.

From this desk here, without a stitch of research, I can show that people who have had more car accidents live, on average, longer then people that have had very few car accidents.

Why? Because each year you live you have a chance of racking up another car accident. In general, the older you live, the more car accidents you are likely to have had.

If you want to know whether people who have more accidents are more likely to live longer because of the car accidents, you have to do something like take 40 year-olds and compare the number of 40 year-olds that make it to 41 in your high and low accident groups (simple check), or use any one of a number of more sophisticated methods to filter out the age-car accident relation.

The Purdue example is somewhat more contained, because the event of taking a Course Signals class or set of classes happens once per semester.  But what I am asking is whether

1. the number of classes a student took is controlled, for, and more importantly,
2. whether first to second year retention is calculated as
   1.  the number of students that started year two / the number of students who started year one (our car accident problem), or
   2.  the number of students who started year two / the number of students who finished year one (our better measure in this case).

I think it is the “car accident problem” statistic that is being quoted in the press. If it is, then it’s possible the causality is reversed: students are taking more Course Signals courses because they persist, rather than persisting because they are taking more Signals courses.