“Experimented On”

This “experimented on” phrase bothers me a bit:


The SJSU affair falls somewhere between educational research and a social experiment, and we are very much in need of better experiments in these areas. Most educational research is pretty abysmal. Most social policy goes untested. The lack of decently designed experiments in these areas generally allows the people with the most money and policy clout to determine what constitutes truth in this space. And people suffer because of that, every day.

So we need more experimentation. And we probably need better experiments than SJSU, where Udacity demonstrated negligence in offering students an experience they should have known to be inferior. I am not arguing that we should shrug our shoulders at a company that takes student failure too lightly, or directs policy interventions disproportionately at the powerless.

But “experimented on” somehow implies to me that the rest of us are not making choices every day on how we educate students. I used a team-based model in my third iteration of a statistical literacy class I taught, and I tracked its effectiveness. Results were mixed. Was I “experimenting on” my students? I introduced peer instruction another semester. I was certainly experimenting with my delivery — but was I experimenting “on” the class?

I’d say no. I was altering instruction to find out what worked best, and paying attention to the results. This, broadly, is what it means to be a professional. And I don’t think that changes at the institutional level. I was not using my power as a teacher to collect data on stress reactions to various forms of supersonic pitches, or on heart rate reactions to violent imagery. I was trying to do the best I could at what both society and the students were paying me to do.

I’m eliding a lot of concerns here for the sake of brevity. I’m happy to argue this more deeply in the comments. But “experimented on” sounds to me like the ickiness is coming from the use of a formal design. That’s wrong, in so many ways.

In reality, the ethical considerations in situations like SJSU are both more broad and more narrow. Such activities are unethical if the treatment received by either of the test groups is unethical, end of story. If the treatment of the Udacity students is unethical inside the experiment, it would likely be unethical had no experiment framed it — the fact that we’re tracking outcomes has little to do with it. Likewise, if the use of Udacity for this purpose is an acceptable policy option outside of an experiment, then the use of random assignment to assign it is ethically neutral.

To my ears, the phrase “experimented on” confuses that issue by imposing a particular set of ethical concerns that only exist once we decide to track outcomes, or use random assignment to allocate limited resources. So please — argue whether offering such courses as educational alternatives is ethical, and debate whether experimentation that tends to target those alternatives at poorer schools is socially just. But let’s not create the impression that it’s the presence of the experiment that makes these solutions ethically dubious.

5 thoughts on ““Experimented On”

  1. I’m glad you wrote about this. That phrase bothers me too. The ethical obligations run in both directions. Purdue, when they did their early Course Signals work, talked about their ethical obligation to try to do something with the data to help students. They asked, if we could potentially help the students, then don’t we owe it to them to try? This was typically positioned as the counterweight to the ethical obligation to student privacy that might be extended to the notion that one should never look at that data, but it could just as easily be positioned as a counterweight to the ethical obligation not to take unnecessary risks with student success. The Hippocratic oath exists, and yet it is possible to do medical research without violating that oath. That said, a lot of time and thought goes into ensuring compatibility between those two in the world of medicine, which should tell us something about the effort that we should be investing into thinking through our educational research ethics.

  2. I’m not going to argue for the hyperbolic rage behind the “experiments,” but I have to look at the Udacity/SJSU situation as more than a professional in a teaching space working with new pedagogy or a theoretical approach. The issue is a state one. Jerry Brown sees higher education as a crisis in need of fixing, and through a pedagogical process of calling Sebastian Thrun because his name was the one he got to the quickest, he settles on Udacity as the means to fix the problem. They roll out these MOOCs in a manner of weeks, despite generations of distance education research showing both a need for time in instructional design as well as a nagging of history of not being so good for underrepresented learners. Predictably, the Udacity students did worse. The sort of thing I go through in IRB did not happen at any process with Udacity. And that is in part because it is not a research project but a classroom learning model. But someone should have stopped along the way and said, “Hey, I make people spend months in a review board to see if they can engage individuals in a new instrument…shouldn’t something similar happen with education’s magic bullet?” But Udacity was looked at more as the MagicBullet (TM) rather than a Magic Bullet.

    • I think I would argue however it’s not an IRB issue. Let’s say everything mentioned above happens EXCEPT we don’t track anything, no experiment. Still the the Brown-Thrun marriage, still the sloppiness, still the cronyism, still the uneasy influence of potential campaign money deciding what educational model to pursue, still the valuing of business over students.

      How much does the fact that it’s not an “experiment” change things? I really can’t see a single way it makes it better, and I can see a lot of ways it makes it worse.

      • I hope I was not arguing for IRB to be involved. Rather I opined for someone along the way to be a voice akin to IRB saying, “Hey, for stuff that matters one one-thousandth as much as this we require numerous hoops and paperworks…shouldn’t someone at least do a lit review here?” I agree there were lots of places this could have gone south had more hands been in the mix, because it was going to happen based on political power alone. But at least according to what we know, there was no voice of dissent, and that is troubling for an institution that prides itself on rigor.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s