No More Secret Sauce Analytics

I’ve gotten two calls from reporters in the past week asking me about the “dangers of analytics” in higher education.

I’m always quite careful to say I think there’s a lot of promise for analytics in higher education. I can’t imagine a future where we’re not using analytics extensively to try and improve what we do. And I have no doubt those analytics will guide us on where to apply resources in some high stakes areas — how we spend advising time, where we apply remediation strategies, which students we select for special attention, what gateway courses we should fund.

But it’s precisely the power and potential of analytics that makes it so important we get this right. And what getting this right means, first and foremost, is we do it in the open, and avoid the “secret sauce” mindset we’ve tended to have about such things. As we move out of the institutional experimentation phase and into the commercialization phase of analytics, institutions are increasingly being sold a black box of formulas they can neither share nor explain. And more than anything else, it’s this “magic number” mentality which makes analytics dangerous.

The anti-analytics crowd may object that openness is not enough. We are, in fact, turning governance over partially to formula, and that’s frightening. Sugar-coat it all you like, no matter how much individual agency you preserve in your process of identifying at-risk students you wouldn’t be using analytics if they didn’t somehow reduce or otherwise impact your decisions. You can’t simultaneously claim that analytics are powerful and harmless.

But protestations that such a shift is unprecedented are unfounded. The truth is that we are ruled by formula more than most people would admit right now. The highway fund in the United States is a percentage of the gas tax. Your Social Security benefits are indexed to inflation. Cost of living is used to compute a number of benefits and your eligibility for various forms of student aid is a function of a poverty-level formula developed in the 1960s.

The difference with education analytics as it is being implemented is not that we are turning some of our adjudication over to formulas, but that those formulas are often shielded from our view.

If a set of students who should seem like they should be receiving aid are not, I can look at the poverty level formula and understand the ways in which that formula may be distorting public policy or excacerbating inequality because that formula is part of public record. We can have a debate about that.

But what if I am a student who is not selected for special intervention at a college, even though I clearly need help? If it is on the basis of written policy (say, advisors will contact all students with a first semester GPA of less than 2.3), we can debate that policy. If it is a matter of personal judgment, we can hold the individual responsible for that judgment accountable, and ask them to explain their reasoning.

If it’s a “secret sauce” algorithm on the other hand, what’s our option for public discourse on it? Where is our opportunity to examine it for racial bias, for bias against part-time students, or even for general error?

We’re left with “trust the programmers, they’re objective” in a world where the programmers have repeatedly proved that not only are they not objective, but quite often just plain wrong about the math. See, for example, Purdue’s Course Signals, or the strange and sad story of Google Flu Trends.

There’s absolutely much to be gained by using the data at our disposal to try and serve our students better. But deciding which students to expend extra resources on is not the same as guessing your likely personal rating for Pulp Fiction. Public policy, public money, and the public good require an accountability above and beyond individuals deciding whether they will give their $8 a month to Netflix, and it’s time we came to terms with that.

Analytics, yes. But no more secret sauce analytics. It’s time to open up these formulas to the light of day and get public comment on them. I understand that involves additional work on the part of institutions and corporations engaging in analytics. But it’s difficult to see how we hold institutions accountable to the public if we don’t make them show us the formulas producing their decisions.

About these ads

7 Comments on “No More Secret Sauce Analytics”

  1. mikecaulfield says:

    Oh, and someone need to fund more projects like this:

    Open Learning Analytics

    Thank you.

  2. giaythethao1 says:

    for bias against part-time students, or for just general error?

  3. jecgenovese says:

    Reblogged this on peakmemory and commented:
    “Analytics, yes. But no more secret sauce analytics.”

  4. […] saw jecgenovese of Peak Memory reblog this post from Mike Caulfield titled “No More Secret Sauce Analytics” which is of importance in science in general and this video on bioinformatics makes many of […]

  5. […] information” involved? And of course, what are the ideologies of algorithms – the “secret sauce” – that underlie adaptive technologies like […]

  6. We are a bunch of volunteers and starting
    a brand new scheme in our community. Your site
    provided us with valuable information to work on. You’ve
    done a formidable job and our whole group shall be thankful to you.

  7. […] But we don’t know. It’s a black box, the company’s “secret sauce.” […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 142 other followers