A Simple, Less Mathematical Way to Understand the Course Signals Issue

I’m re-reading my Course Signals post, and realizing how difficult it is to follow the numbers. So here’s an example that might make it clearer.

From this desk here, without a stitch of research, I can show that people who have had more car accidents live, on average, longer then people that have had very few car accidents.

Why? Because each year you live you have a chance of racking up another car accident. In general, the older you live, the more car accidents you are likely to have had.

If you want to know whether people who have more accidents are more likely to live longer because of the car accidents, you have to do something like take 40 year-olds and compare the number of 40 year-olds that make it to 41 in your high and low accident groups (simple check), or use any one of a number of more sophisticated methods to filter out the age-car accident relation.

The Purdue example is somewhat more contained, because the event of taking a Course Signals class or set of classes happens once per semester.  But what I am asking is whether

  1. the number of classes a student took is controlled, for, and more importantly,
  2. whether first to second year retention is calculated as
    1.  the number of students that started year two / the number of students who started year one (our car accident problem), or
    2.  the number of students who started year two / the number of students who finished year one (our better measure in this case).

I think it is the “car accident problem” statistic that is being quoted in the press. If it is, then it’s possible the causality is reversed: students are taking more Course Signals courses because they persist, rather than persisting because they are taking more Signals courses.

About these ads

4 Comments on “A Simple, Less Mathematical Way to Understand the Course Signals Issue”

  1. […] The Purdue researchers didn’t appear to have controlled for the number of courses taken. He explained it with an analogy: the older you live, the more car accidents you are likely to have had, and so people who have […]

  2. […] Put another way, “students are taking more … Signals courses because they persist, rather than persisting because they are taking more Signals courses,” Caulfield wrote. […]

  3. […] of this technology we’ve got a problem. Why? Well, because a lot has happened since 2010. Mike Caulfield pointed out six months ago, and Michael Feldstein re-iterated, the research claims of the effectiveness of […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 130 other followers