Infant mortality and choice of a base

If I have 10 kids in my class and two failed last year and one failed this year, I can say two equivalent things:

  • 50% less students failed my course this year
  • 10% more of my students passed.

The odd thing is most students refuse when looking at such figures to believe they are equivalent statements. In fact, they are prone to believe that if

  • 10% more of my students passed, then
  • There were 10% less failures

The key is what I chose for a base to calculate the percentage from. I can choose

  • total students: 10% more of my students passed,
  • failing students: 50% less students failed
  • or passing students: 12.5% more students passed

as the base, and each will give me a different percentage. It’s a stunningly easy sort of manipulation that is used all the time to great effect.

Apparently students aren’t the only ones confused. Here’s a paper making a similar error on infant mortality.

Tutoring at Scale Sighting

From The Chronicle, Tenured Professor Departs Stanford U., Hoping to Teach 500,000 Students at Online Start-Up:

Eventually, the 200 students taking the course in person dwindled to a group of 30. Meanwhile, the course’s popularity exploded online, drawing students from around the world. The experience taught the professor that he could craft a course with the interactive tools of the Web that recreated the intimacy of one-on-one tutoring, he said.

I still believe the major technological paradigm that is going to reshape education is tutoring at scale, and it’s interesting to see that for those that succeed in that realm that’s exactly how the experience feels to them.

Are we winning or losing the “War on Cancer”?

If your answer was that war is the wrong metaphor, you win the prize, I suppose. Still, I found this exercise from a medical stats textbook rather interesting:

17.1. A major controversy has occurred about apparent contradictions in biostatistical data as researchers try to convince Congress to allocate more funds for intramural and extramural investigations supported by the NIH. Citing improved survival rates for conditions such as cervical cancer, breast cancer, and leukemia, clinicians claim we are “winning the war” against cancer. Citing increased incidence rates for these (and other cancers), with minimal change in mortality rates, public-health experts claim that the “war” has made little progress, and we should focus on prevention rather than cure.

17.1.1. What explanation would you offer to suggest that the rising incidence of cancer is a statistical consequence of “winning” rather than “losing” the battle?

17.1.2. What explanation would you offer to reconcile the contradictory trends for survival and mortality rates, and to suggest that both sets of results are correct?

The pessimistic explanation is that early detection detects more benign cancers, leading to inflated five-year survival stats. But the question is looking for the optimistic answer, which seems harder to grok.

Off the top of my head, one thing that comes to mind is that people can get multiple types of cancer. If you survive thyroid cancer at 60, you might die of colon cancer at 70. Since we all die of something, greater survival rates would also lead to greater incidence, but wouldn’t necessarily reduce mortality. In a world of miracle cancer cures, for instance, you might start racking up cancer later in life the way people rack up colds in their youth. 

Of course, I don’t have a teacher’s edition here, so any input would be welcome — alternate theories?

The Numbers Game

We often talk of social statistics, especially those that seem as straightforward as age, as if a bureaucrat were poised with a clipboard, peering through every window, counting; or, better still, had some machine to do it for them. The unsurprising truth is that, for many of the statistics we take for granted, there is no such bureaucrat, no machine, no easy count, we do not all clock in, or out, in order to be recorded, there is no roll call for each of our daily activities, no kindergarten 1, 2, 3.

What there is out there, more often than not, is thick strawberry jam, through which someone with a bad back on a tight schedule has to wade—and then try to tell us how many strawberries are in it.

I’m reading Blastland and Dilmot’s The Numbers Game right now, and it is brilliant so far. I love that it starts with one of the fundamental quantitative reasoning questions: What did you count and how did you count it?  

===

The book is tangentially related to the long running BBC radio show More or Less, which you can can listen to for free here.

Milo Schield’s short paper Teaching the Social Construction of Statistics deals with “strawberry jam” issues, and is well worth a read.

Keynes, Anti-Semite? Really?

Pardon the intrusion — but I find this interesting. There’s a diary entry of Keynes being circulated around that supposedly proves Keynes was an anti-Semite. This is meant to be a brilliant rebuttal to Paul Krugman’s “Keynes was Right” column on how Keynes’s theoretical model of macroeconomics has been vindicated.

In some ways it doesn’t matter. Milton Friedman did some terrible things, but that doesn’t effect whether his theories were solid. Add to that that anti-semitism in turn-of-the-century Britain was the norm, and it’s hard to see what this tells us about Keynes’s character anyway.

But I couldn’t help noticing that they were all quoting from the same source — same elipses, etc. Classic sign of people not checking original sources.  

So I went to the original source of the quotes (with the tell-tale ellipses). Humorously it comes from a years-old Marxist critique of Paul Krugman’s support of Keynes. From long enough ago that he was at MIT.

That part is kind of funny, no? That conservatives are cribbing off of Communist attacks (and not fact checking them!).

Now, Marxists are wrong about just about everything, but they do love their citations, and they provided a full cite to the quote. So I went one step backwards, and got to the paper on Scribd. And I read a little further. Here is what I found:

Keynes was a Zionist

When we put Keynes’ derogatory remarks in a dialogue with his political acts concerning the targeted individuals, a stark contrast between them emerges. Rather than analyzing Keynes’ policy toward the targeted groups, let us follow the main thesis of this analysis and focus on his political acts that might possibly compensate for the burden of his disturbing anti-Semitic remarks, whose nature I have already partially addressed. Keynes` support of Zionism remains a theme so far largely unnoticed by historians. Annand Chandavarkar was probably the first historian to call attention to this topic. It is hardly known that Keynes was the only non-Jewish member of a high-powered advisory committee responsible for preparing a report on Zionist efforts to establish a national home in Palestine, which was to be presented in Paris at the Peace Conference in February 1919.

Keynes was a vocal critic of 1930s German policy against the Jews

Later during the persecution of Jews in Germany in 1930s, Keynes decisively stood on the side of the Jews. The letter written to Professor Spiethoff, who was arranging the publication of a German translation of “National Sufficiency,” supports this claim:

“Forgive me for my word about barbarism. But the word rightly indicates the effect of recent events in Germany on all of us here…It is many generations in our judgment since such disgraceful events have occurred in any country pretending to call itself civilized…If you tell me that these events have taken place, not by force but as an expression of the general will…that in our view would make some of the persecutions and outrages of which we hear…ten times more horrible.”

Keynes even suggested making an offer to Germany to make organized arrangements for all German and Austrian Jews who wished to emigrate and be naturalized elsewhere. He readily intervened in favor of interned Jewish economists. According to editors of The Collected Writings, Keynes was one of the most active in succoring the Jewish refugees.

Again, to repeat, in some ways it doesn’t matter. Keynes could be a rascist and be right about the economy, just as Friedman was right about many things but made a horrendous mistake in Chile. 

But honestly, if you are going to cite sources, read downpage in that source at least a little. It’s embarassing…

A Herd Immunity to Nonsense

Mark Pagel on the internet and our cultural evolution:

A tiny number of ideas can go a long way, as we’ve seen. And the Internet makes that more and more likely. What’s happening is that we might, in fact, be at a time in our history where we’re being domesticated by these great big societal things, such as Facebook and the Internet. We’re being domesticated by them, because fewer and fewer and fewer of us have to be innovators to get by. And so, in the cold calculus of evolution by natural selection, at no greater time in history than ever before, copiers are probably doing better than innovators. Because innovation is extraordinarily hard. My worry is that we could be moving in that direction, towards becoming more and more sort of docile copiers.

If you go to that link above, which is something you should do right now, you’ll see evolutionary biologist Mark Pagel sorting through some free-ranging and often contradictory ideas on how the internet is shaping our evolution — or to be more precise, how it has taken the 200,000 year trend toward copying over innovation and thrown it, perhaps, into hyperdrive. If you read one thing today, read Pagel. It’s wonderfully free of polemic, and fantastically interesting. It focuses on the copier vs. innovator divide, and explores whether the incentives to innovate are so low in a hyperconnected world that we’re in deep trouble. 

I’d like to focus on something smaller than that — the “docile” part of Pagel’s equation. If you put together Pagel’s insight with Dan Kahneman’s insight that the true value of teaching critical thinking is not that you keep yourself out of trouble, but you can keep others out of trouble, I think you have an interesting argument for the importance of critical consumption to the survival of our species in a connected world.

How do I mean? In a system where people are purely docile copiers, nonsense spreads as quickly as insight. If you want your system to truly float valuable ideas to the top and let the nonsense sink, people have to do more than share — they have to engage with material, annotate it, note reservations, and in most cases simply share nonsense at a considerably lower rate than insight. They have to be able to spot cognitive bias and flaws in reasoning in the three minutes between when the tweet comes in and when they decide to retweet it (or not). That requires, as Kahneman points out, not so much a set of enlightened creators or leaders, but a culture of skilled critics. 

Again, as I noted in my last post, this is a different way of looking at education — instead of seeing it only about the impact on a student’s ability to do specific work, we are looking at more like we look at vaccination — it’s not only about the individual, but about developing a herd immunity to nonsense — getting our collective critical capacity to the point where the dumb ideas spread less widely than the smart ones. 

To do that, what do you need? A background in quantitative reasoning, critical thinking. A recognition of common biases. Critical reading skills, and a good intuition about authoritative sources. You need, essentially, a broadly rethought liberal arts education…

Cognitive Bias and Education as a Public Good

A strange but true exhortation from Dan Kahneman, the guy who, with Amos Tversky, basically invented the field of cognitive bias. After forty years of looking into the weird world of bias he says the only effective way to get around your own biases is to create a society of people skilled enough to correct you:

From the end of Thinking, Fast and Slow:

What can be done about biases? How can we improve judgments and decisions, both our own and those of the institutions that we serve and that serve us? The short answer is that little can be achieved without a considerable investment of effort. As I know from experience, System 1 is not readily educable. Except for some effects that I attribute mostly to age, my intuitive thinking is just as prone to overconfidence, extreme predictions, and the planning fallacy as it was before I made a study of these issues. I have improved only in my ability to recognize situations in which errors are likely: “This number will be an anchor…,” “The decision could change if the problem is reframed…” And I have made much more progress in recognizing the errors of others than my own.

The way to block errors that originate in System 1 is simple in principle: recognize the signs that you are in a cognitive minefield, slow down, and ask for reinforcement from System 2. This is how you will proceed when you next encounter the Müller-Lyer illusion. When you see lines with fins pointing in different directions, you will recognize the situation as one in which you should not trust your impressions of length. Unfortunately, this sensible procedure is least likely to be applied when it is needed most. We would all like to have a warning bell that rings loudly whenever we are about to make a serious error, but no such bell is available, and cognitive illusions are generally more difficult to recognize than perceptual illusions. The voice of reason may be much fainter than the loud and clear voice of an erroneous intuition, and questioning your intuitions is unpleasant when you face the stress of a big decision. More doubt is the last thing you want when you are in trouble. The upshot is that it is much easier to identify a minefield when you observe others wandering into it than when you are about to do so. Observers are less cognitively busy and more open to information than actors. That was my reason for writing a book that is oriented to critics and gossipers rather than to decision makers.

Emphasis mine.

I think this is a pretty interesting way of thinking about education as a public good. Think of training in critical thinking or quantitative reasoning like CPR training — the person you are most likely to help with it is not yourself but somebody else. The most important role may be that of the informed observer, because, as Kahneman observes, what keeps us on track best is not our own abilities, but a culture of critical thought.

Evidence-Based and the Marginal Cost of Zero

If you can conceive of a solution to a problem that has a marginal cost of zero due to cheap replication and economies of scale, then that’s good. If you’re doing that by going into the digital space, where cost of experimentation is low, even better. 

Many elements of education are best seen through the marginal cost of zero lens, and it’s that dream of essentially free education, based on economies of scale, that tends to drive a lot of philanthropy. 

Where there’s dissonance, though, is that we are also told that what we do has to be evidence-based. And I think sometimes there is this show that granting agencies and philanthropic institutions make that they are going to cut through the bull and be scientific about this stuff, in a way that we have not been.

The problem is that most of the time these things don’t line up. The projects that get the most attention focus on the economies of scale (Khan, MITx) with very little focus on evidence-based practice. 

That seems to me a problem. Are we letting our dream of a completely free education undermine the more modest (and evidence-supported) goals of dirt cheap technology-assisted education? Maybe not. But if you looked at what gets covered in the press and funded on the ground, you could be forgiven for thinking that.

Semantic Mapping vs. Pictorial Cues

From A Theory-Based Meta-Analysis of Research on Instruction by RJ Marzano:

The next two techniques displayed in Table 7.2 employed the information processing function of idea representation.  Techniques that provided students with metacognitive strategies for using visual memory had an effect size of 1.04, indicating a percentile gain of 35 points.  Presumably, these strategies help students represent information they are reading in nonlinguistic form…  From these findings, one might infer that idea representation is a key aspect of the reading process…

[R]eading studies [that] addressed techniques that attempt to enhance the idea representation information processing function during reading using pictorial aids…were considered as a group in themselves (as opposed to grouping them with the idea representation techniques in Table 7.2) because they did not employ the metacognitive system.  Rather, they were considered manipulations of the environment designed to stimulate idea presentation in students.  These studies had an average effect size of .46 (n=16; SD=.20), indicating a percentile gain of 28 points.  Table 7.4 displays the differential effects of techniques within this category

I may be misreading this, but what it seems to say is that interventions that use pictures as cues presented to students while reading underperform interventions that teach students to make up their own pictorial representations by a lot (0.46 is a fairly average effect size seen in numerous interventions, whereas 1.0 and above is a fairly rare effect size).

The key seems to be that information organized for the students into pictorial form allows them to disengage from metacognitive strategies, whereas organizing their ideas into pictorial form themselves engages metacognitive processing. These findings are in line with Hattie’s findings on audio-visual aids and the like. 

But I’ve only just skimmed this at this point, and the focus is not higher education here, so one must tread carefully. I’ll come back to this tomorrow I guess. 

This all relates to a crazy idea for the Quantitative Literacy course next semester…