Hapgood

Yesterday, Roger Schank (famous cognitive scientist and learning theorist) released some of the work he and his team did for Trump University back in 2005 or so.

You didn’t know that Roger Schank was involved in Trump University? Well, he was. Trump University’s online materials were designed by the artificial intelligence pioneer and learning theorist who founded one of the first online learning labs in the early 1990s (Northwestern’s ILS) and had spent (by the time of Trump U.) almost two decades researching what works in technology-mediated education.

Quick disclosures: I worked for Roger Schank’s company Cognitive Arts 15 years ago (but not for Trump University), and I despise Donald Trump and everything he stands for. More on these points at the bottom of this post.

The Online Courses for Trump University Were Well-Designed

So here’s what I want to tell you about the online portion of Trump University: It probably puts your university’s online courses to shame. It certainly makes a mockery of what Silicon Valley darlings Coursera and Udacity call courses.

Here’s a link to some of the online materials.

I think I built enough of these courses at Cognitive Arts to walk you through the design of this one. This particular course is similar, if somewhat less ambitious, than what we used to produce there.

Walking Through the Course

Schankian learning experiences are often structured as simulations. On a high end project (e.g. a million dollars or so) these simulations might involve branching video narratives that allow a user to play out very realistic scenarios that develop the emotional involvement that promotes deep learning. You get to know a cast of characters through video intereactions, watch situations blow up as you make wrong decisions, and become deeply invested in a multimedia storyline.

Smaller projects involved different types of simulation, but one of the favorite lower-end models was email simulation. (If I remember correctly, this method was partially developed from techniques Schank’s colleague Chris Reisbeck used in his computer classes).

In an email simulation, you place the student in a role, usually that of a newer worker, who is asked to create or fix some artifact for a job. You learn about these projects via a set of carefully sequenced emails and assignments. In a programming class, early emails may ask you to take a quick peek at a piece of code and figure out why it’s hanging when it runs. Later emails have you reviewing specifications documents, client emails, and peer submissions and building new code.

In the older language of “Goal-based scenarios” you’d have a variety of elements, structured to make the simulation meaningful. Here’s a list of what you need, from some of Schank’s early 1990s work:

• Learning goals – target skills or understandings the student will grapple with
• Mission – objective the student will pursue
• Cover story – a fictional background story that provides a motivational context for the activity
• Role – the character a student will play in the simulation
• Scenario operations – the activities the student will perform to achieve mission, presented in optimal sequence.
• Resources – the reference materials the student will use to solve the problems presented
• Feedback – provided by experts (usually through video) but in just-in-time segments, usually telling stories about related experiences

I don’t know enough about these subjects to evaluate the learning goals chosen, but the learning goals are exceptionally clear here. Schank’s focus on what it is you want the student to be able to do works well with this material.

• EN001: Mini-Course: Write a Business Plan
• EN002: How to Find Start-Up Funding
• EN003: How to Start a Business on a Shoestring Budget
• EN101: Think Like An Entrepreneur
• EN101S: Critically Evaluate Your Business Plan
• EN201CS: Design a Winning Business Based on Your Idea

Within each module, the approaches are mixed. Having worked on these sorts of things before, I can tell you the information in each module is based on the critical skills experts need to master and the common misconceptions novices tend to have about the subject. That is, the design team for these things will spend literally hundreds of hours talking to experts talking about how things are done and why things fail, and design the tasks and feedback on what they discover.

Design a Winning Business Based on Your Idea

If we look at the last module (“Design a Winning Business”) we can see that the scenario operations are structured in the natural sequence you would go about the activity, with a focus on what is done at each point (e.g. not “Introduction to Competitive Analysis” but “Analyze the Competition”).

Each of these scenario operations (Refine the Business Concept, Validate the Market, etc) takes place in an email simulation.

In this module you have been invited to partner with someone who thinks they have a great idea for a business — a new trend in pet care called “mobile grooming” where the groomer travels to the dog for an in-house appointment. This is a bit of a capstone mission, where you’ll use skills and knowledge you’ve learned in previous modules in a novel context.

It starts with an email from Susan, who gives you the basics of her idea:

You can see the resources attached. These are meant to be realistic resources from which the student has to gather relevant information. For instance, there’s an article on the mobile dog grooming trend. Buried in the articles are hints on the consumer profile, level of demand, and competition.

Also attached is a business questionnaire that your partner Susan has gotten from her friend Tom at the Small Business Association. She’s started to fill it out, but found she doesn’t have answers to a lot of the questions. You need to help her work through the resources and answer those questions:

Those seemingly simple questions lead to complex activities like competitive analysis, and understanding barriers to entry, etc. You’re asked to make sense of a variety of resources to determine whether the business as imagined is truly viable (and if it’s not, how can it be altered to be more successful). Here’s a cluster diagram of some of your survey results.  What does this tell you about your current plan? Have you selected the correct target market?

As you move through these activities (usually done through free form entry in a comments box), just-in-time support is available to help you understand the methods and meaning of what you are doing. Crucially, the learning is structured so that you are presented the information after the scenarios have led you to the question — information  is presented only after it has value to your task.

After your work on a segment of the scenario, you get feedback from the experts as well. What did that cluster diagram show you about your ideal target market? How did the competitive analysis flesh out which of those markets were already being served?

Following best pedagogical practice, you don’t see what the experts think until you submit your own analysis, and following best practice with feedback, you see their feedback immediately after you submit, in video format. (Although other ADA compliant formats are available).

Each scenario operation builds on the next. After defining your market you need a marketing plan. You are given a spreadsheet to fill out and submit:

To help fill out that spreadsheet, you are given a bunch of email attachments about the demographics of each potential media buy. These are structured to look like the actual media kits you would get from radio stations and newspapers, and the information you need isn’t always directly there — you sometimes have to derive it. It’s a good example of using authentic resources that don’t map easily to inputs, to test whether students have a deep or shallow understanding of a domain.

As you progress through the each scenario operation, you submit your work to your peers in the class to evaluate. You are frequently submitting an artifact (spreadsheet, etc), but after submission you are asked to reflect on your decisions and defend them.

Your submission goes to peers for eventual grading, but in the meantime you receive instant feedback from experts who tell you how they would have done the activity. You apply that expert information to evaluate the work of your peers, and in turn you look at the feedback your peers have given you.

The modules are often structured around complicating actions. A simple analysis is done and submitted, but new information comes to light: costs are higher than expected, new competitors have emerged, etc. Your analyses grow in the complexity they accommodate.

Here Susan sends you an email noting that she has realized that because of the time that travel to homes takes, there is an upper limit on customers that can be served in a day (oops!). Is there any way to salvage the plan? Given the market, can they maybe shrink the geographic area served? Make money on accessories? Does using a “window” instead of an appointment time affect the convenience and price quadrant we’re serving?

I pose these questions here, but of course the important thing about way the exercises are structured is that you have to develop those questions yourself. In other words, Susan suggests that you adjust area served to reduce travel time, and you have to think through the aspects of your model that affects. Susan mentions the introduction of the “window” approach to appointments, and it’s up to you to translate that in your head into a price and convenience quadrant question.

By the end of the module you are creating an operating model including start-up, fixed, and variable expenses using excel spreadsheets, choosing specific products based on profit margins and customer sensitivity to quality. You’re mapping out expansion plans — does it make sense to expand to cats? To open in another city?

Each segment involves the creation of authentic artifacts: spreadsheets, marketing plans, competitive analyses. Often stages are sequenced and scaffolded, with Susan’s friend from the SBA helping you fill out things partially in the beginning, but as you progress you start doing the work without any substantial scaffolding.

Evaluation

I’ve covered only a fraction of the course here, a couple segments of a couple pieces of one mission in a course. And in certain places I’ve had to intuit what goes on: the version Schank has made available is an HTML only version and does not support the server sessions needed to progress past artifact submission. But given the stuff I’ve worked on in the past I’m pretty confident about my assumptions.

Depending on what you teach, you may not like the model here. It’s not a great model for a course in Issues in French Colonialism or other less applied material. But for something like this, where someone wants to learn to do something specific in an area where there is consensus on what constitutes good practice, the method is a perfect fit.

And apart from being an example of Schank’s approach to education (simulations, war stories, just-in-time expertise, etc.) it’s well designed from a more traditional standpoint as well. It’s well scaffolded, expertly sequenced, and focused on application of knowledge in authentic contexts. It uses peer evaluation guided by expert feedback. It makes clear to the student what is being taught, and why it is important beyond the classroom.

For various reasons, these strictly sequenced classes are not the sort of classes I’m working with anymore, and this is not my particular style of design for what I currently do. I’m an open pedagogy guy, looking for ways for students to do real work, not simulations.

But different classes and objectives require different approaches and for what this class is attempting to do, this is a perfect approach. It is difficult to see the design of the Trump University online experience as anything but exemplary. In fact, if you encountered something this well designed and thought out in traditional online classes you would be stunned.

More than that, consider that Coursera and Udacity built million-dollar businesses years after these courses came out. And their materials were not (and still are not) even close to this in terms of quality and research-informed design. Not even a fraction as well designed.

More recently, ASU’s Habitable Worlds course is (rightly) getting a lot praise for its structure, which involves a task focus with just-in-time instruction and authentic resources. But here we have that approach in a well-developed example ten years ago, at Trump University.

What the Hell Went Wrong?

Of course that last line (“Ten years ago at Trump University!”) is a bit of an unfair dig. Five years before these classes were being made for Trump University we were making them for Columbia University in an initiative that was called Columbia Online. I worked on these classes and can tell you they were better quality than the Trump U. classes, but used the same methods and approaches. Here’s a paragraph from the press release:

Columbia and Cognitive Arts plan to build a program which will focus on courses traditionally offered by Columbia Continuing Education, including courses for Columbia’s Computer Technology and Applications (CTA) Program and American Language Program (English as a second language). Columbia will also work with Cognitive Arts to develop courses in more traditional academic disciplines, such as basic economics and psychology. The courses will be Columbia courses, powered by Cognitive Arts proprietary learning technology that uses problem-based learning in the context of real life situations. Cognitive Arts’ approach is based on the theoretical work and design strategies of its founder, chairman and chief technology officer, Roger Schank.

This was in 2000. So it’s been amusing to watch the Udacity, Coursera, ASU hype about scientifically designed, scalable classes when sixteen years ago I was working for a pioneer of cognitive science, building classes based on a decade of research that were designed to scale to millions.. That’s the year 2000, when a eighteen year-old Britney Spears was topping the charts, MTV was still playing music videos, and an unknown show named Survivor was debuting on network television.

It’s even funnier (by which I mean sad) to see former CogArts partner Columbia University hyping up cutting edge courses in ColumbiaX today where students engage in cutting edge activities like this:

How did we get here? I suppose I’ll leave that to another post, but the short answer is designing quality scalable online courses takes a lot of money, a lot of custom programming, a lot of design expertise, and no one wants to pay. Except, apparently, Donald Trump.

Go ahead, you can weep now.

Disclosures

Disclosure #1:

I worked for Schank’s company Cognitive Arts from 2000-2003. I went there explicitly because I was, at the time, a true-believer in his approach to online pedagogy, which seemed miles ahead of what anyone else was doing. My position at Cognitive Arts ended a few years after the first education dot-com crash when NIIT, an Indian company, acquired us in a nasty-ish leveraged buyout.

I could have maybe signed on at that point to the new entity, but I was exhausted with that life at that point; years of life in a crashing startup had temporarily broken me. I took a detour out of education for a bit and built search interfaces for newspaper archives for a bit. Other people in my unit went with Schank to Socratic Arts and worked on Trump University as a project. Apart from the occassional Facebook comment, I haven’t really kept up with them.

I got back into educational tech in 2005, with a renewed focus on two earlier interests of mine: open pedagogy and how participants in online communities learn from one another. In general, I’m still proud of the work we did at CogArts, my passions are just elsewhere now.

Disclosure #2:

I despise Trump, and think he endangers our republic with his brand of xenophobic authoritarianism. So let me say this: nothing I’ve said here defends the seminars that were staged by Trump, or the course when it switched over to a real estate focus. Nothing here defends the use of the name “University”, the price charged of people, or promises that may have been made.

So frankly, you can believe that the online materials were surprisingly good, but that the University as a whole was a scam. I’d kind of prefer that actually, because I think Trump is a very scary person, and people should say whatever it is they have to to keep him out of the White House.