Two Feeds, Two Scarcities

I’ve put my tweets on a rolling auto-delete, which probably means I’ll be doing ocassional shorter pieces in this space in addition to longer pieces. For posterity, or something.

Anyway, a thought for the day. As we think about the firehose of the Stream — that never-ending reverse-chronological scroll of events that has become the primary metaphor of the web, via Facebook, Twitter, Instagram, and who-knows-what-else — it’s worth noting that the Stream was originally a solution for scarcity, not abundance. That is, the reason that Facebook made the News Feed was that people got tired checking out all of their friend’s Facebook walls only to find there were no updates. So Facebook borrowed a lesson from RSS, that had solved this problem years earlier: serialize contributions from different places into a single reverse chronological feed. This made sure that when ever you logged into Facebook you were guaranteed there was some activity with which to engage.

To repeat, the Stream here was a solution for too little activity. By pooling activity and time-ordering it, a sense of abundance was created.

We’ve talked about this before on this blog (I should find the link, but I’m being lazy at the moment).

What I don’t think I recognized before now was that this motivation was behind the first web stream as well — that granddaddy of all feeds, the NCSA “What’s New” page:

what'snew

The What’s New page was there for a bunch of reasons — making things findable being the big one, and creating a sense of WWW momentum being another. But the biggest reason why it was there was scarcity: Without it, people would log in and find nothing new to do. I mean look at it — you have an average of one or two servers — one or two servers — coming online each day. We’re not talking information overload here.

I don’t really have a point here. I just find it interesting that the feeds that we now portray as a solution to organizing abundance grew out of needs to deal with scarcity.

 

Google Should Be a Librarian, not a Family Feud Contestant

I’ve been investigating Google snippets lately, based on some work that other people have done. These are the “cards” that pop up on top sometimes, giving the user what appears to be the “one true answer”.

What’s shocking to me is not that Google malfunctions in producing these, but how often it malfunctions, and how easy it is to find malfunctions. It’s like there is little to no quality control on the algorithm at all.

Other people have found dozens of these over the past couple days, but here’s a few I found goofing off yesterday while half watching Incorporated on Syfy.

Prodded with the right terms, Google will tell you that:

  • Sasha Obama was adopted
  • Lee Harvey Oswald didn’t shoot JFK
  • GMOs make you sick

Want some screenshots? Today’s your lucky day!

oswald

C6c1glUVQAAYdXM

gmos and health.PNG

Now I’m sure that Google will reply that the results are the results. And I’m sure that other people will ask why I’m being such a special snowflake and stamping my iron boot on the neck of results I don’t like. (Their mixed metaphor, not mine!)

(By the way, trivia fact: one technique of populist dictatorships is to portray the opposition as simultaneously weak and effete while being all-powerful and brutal. Just some facts for your next pub trivia night…)

The truth is, however, that I have a fairly simple definition of a fact, and I would hope that a company who’s stated mission is “to organize the world’s information” would as well. For me a fact is:

  • something that is generally not disputed
  • by people in a position to know
  • who can be relied on to accurately tell the truth

And so, not to be too Enlightenment era about this, but all these snippets fail that test. And not just fail: they fail spectacularly.

The person writing about the GMO health risks has no science background and is considered such a sham by the scientific community that when he appeared on Dr. Oz scientists refused to share the stage with him, fearing even that would be too much normalization of him.

The site writing about Sasha and Malia being adopted, “America’s Freedom Fighters”, is site specializing in fake news to such an extent that Google autosuggests “fake news” if you type it into the search box.

aff.PNG

And the JFK conspiracy theory is — well, a conspiracy theory. It’s literally the prototypical modern conspiracy theory. It’s the picture in the dictionary next to the word “conspiracy theory”.

The truth is in cases like these cases Google often fails on all three counts:

  • They foreground information that is either disputed or for which the expert consensus is the exact opposite of what is claimed.
  • They choose sites and authors who are in no position to know more about a subject than the average person.
  • They choose people who often have real reasons to be untruthful — for example, right-wing blogs supported by fracking billionaires, white supremacist coverage of “black-on-white” crime, or critics of traditional medicine that sell naturopathic remedies on site.

Google Should Not Be Family Feud

I never really got the show Family Feud when I was a kid. That’s partially because my parents mostly put me on a diet of PBS, which made anything higher on the dial look weird. But it’s also because it just didn’t jive with my sense of why we ask questions in the first place.

For those that haven’t seen Family Feud, here’s how it works. The host of Family Feud asks you a question, like “What builds your appetite?” You try to guess what your average American would answer.

You win if you guess something in the top five of what most people would say. So a lot of people say “smelling food” so that ranks in the list. No one says “not eating” so that doesn’t rank.

Watching this as a kid I’d always wonder, “Yes, but what actually builds your appetite the most?” Like, what’s the real answer? Don’t we care about that?

But Family Feud doesn’t care about that. It was never about what is true, it was about what people say.

I don’t think Google’s purpose is to aspire to be a Family Feud game show team, but it’s sometimes hard to tell. For example, a principle of “organizing the world’s information” has to be separating reliable sources from unreliable ones, and trying to provide answers that are true. But it’s clear that in many cases that’s not happening — otherwise quality control would be flagging these misfires and fixing them. The snippets, which create the impression of a definitive answer while feeding people bad science, conspiracy, and hate speech, make matters worse.

It should not be that hard to select good sources of information. For example, there is an excellent National Academies report on genetically engineered crops that was written by a mix of corporate and anti-corporate scientists and policy analysts. Here’s the conclusion of that study on health effects:

gene

On the basis of its detailed examination of comparisons between currently commercialized GE and non-GE foods in compositional analysis, acute and chronic animal-toxicity tests, long-term data on health of livestock fed GE foods, and epidemiological data, the committee concluded that no differences have been found that implicate a higher risk to human health safety from these GE foods than from their non-GE counterparts. The committee states this finding very carefully, acknowledging that any new food—GE or non-GE—may have some subtle favorable or adverse health effects that are not detected even with careful scrutiny and that health effects can develop over time.

That’s actually what science looks and sounds like — having reviewed the data available, we find no evidence but are aware that, since impacts may take time to develop, there may yet be adverse impacts to appear.

If you went to a competent health sciences librarian and asked for material on this, this is what you’d get back. This report as one of the definitive statements to date on GMO safety. Because the librarian’s job is not to play Family Feud, but to get you the best information.

Google instead gives you the blog of a man with no medical or scientific training who claims GMOs cause infertility, accelerated aging, and organ damage. But “survey says!” that’s true, so it’s all good.

The world right now is in a post-truth crisis that threatens to have truly earth-shattering impacts. What Google returns on a search result can truly change the fate of the entire world. What Google returns can literally lead to the end of humanity as we know it, through climate change, nuclear war, or disease. Not immediately, but as it shapes public perception one result at a time.

I’m not asking Google to choose sides. I’m not asking them to put a finger on the scale for the answers I’d like to see. I’m asking them to emulate science in designing a process that privileges returning good information over bad. I’m asking that they take their place as a librarian of knowledge, rather than a Family Feud game show contestant. It seems a reasonable request.

Doubt Versus a Bayesian Outlook

There’s lots of primary causes of the recent assault on truth that are non-technological. In fact, most causes have very little to do with technology. I’d point people to the excellent book The Merchants of Doubt which details the well-funded and and well-planned corporate assault on science that began as early as the 1950s around the issue of whether cigarettes cause cancer. There was a simple but profound realization Big Tobacco had 50 years ago — they didn’t have to refute the conclusion of the science that clearly, even back then, pointed to tobacco as a primary cause of lung cancer. They just had to introduce doubt.

The neat thing about doubt is it makes you look and feel like a pretty deep thinker. America loves doubt. Every four years we run an election for 18 months and then treat the people who haven’t decided until the last week of the election as if they were some sort of free-thinkers rather than the most politically ignorant population in the country. The mythology of doubt is strong.

Reporter:  “So what do you think about the election, Bob?”

Independent: “Well, I’m not sure. Clinton has some good points, but Trump seems like a strong leader. I like to take my time thinking about these things.”

Reporter: “Well, it’s quite the important decision. Back to you, Maria!”

The mythology of doubt is that we have things which need to be “proven”, and until they get proven we we are in a state of doubt: we really don’t know what to believe. Who can say?

But doubt is not actually what you want. Doubt is just certainty from another direction, and these two orientations — doubt and certainty — form a binary worldview that promotes polarization, narrow thinking, and poor policy outcomes.

What you really want is not doubt. What you want, for lack of a better word, to be Bayesian in your outlook. The famous statistician and epidemiologist Jerome Cornfield, responsible for much of the revival of Bayesian approaches in epidemiology in the 1960s and beyond, used to talk about the “Bayesian Outlook”.

The Bayesian Outlook is at its heart simple, but it’s also profound. Here’s Cornfield:

The Bayesian outlook can be summarized in a single sentence: any inferential or decision process that does not follow from some likelihood function and some set of priors has objectively verifiable deficiencies. The application of this outlook is a largely extra-mathematical task, requiring the selection of likelihoods and priors that are appropriate to given problem situations, with the determination of what is appropriate requiring, in Fisher’s words (in another context), ‘responsible and independent thinkers applying their minds and imaginations to the detailed interpretation of verifiable observations. (Cornfield, 1969)

There’s a field of Bayesian statistics that is fairly developed and beyond the scope of this post. But as Cornfield notes, Bayesian approaches are not really about the math — they are about a way of looking at the world. And given that I think it’s possible to talk about having a “Bayesian outlook” when it comes to fact-checking.

What does this mean in practice? As an example, I use this tweet occasionally in my presentations:

Is the part about the Nazis true? It’s either true or not, of course. But we can only view that truth through an array of probability.

When I first see something like this, my immediate reaction is it has a good chance of being true. Why?

Well, there are priors. I know Schumer is Jewish, of European descent. And I know that the Nazis and their collaborators killed a substantial portion of of that population, maybe about 40%. I also know you have, by definition, eight great-grandparents. The chances that at least one of the eight great-grandparents might have died in WWII at the hands of Nazis or Nazi collaborators is something that had a reasonable chance of being true before this tweet.

We call these the priors: they exist before this tweet makes its way to me. One key component of Bayesian analysis is that we begin with a set of priors, and pay careful attention to the selection of those priors before assimilating new information.

Now as to the new information: the fact that someone tweeted this claim makes the claim more probable, to some extent. This is a specific claim. It came to me through a feed where I weed out the worst misinformation offenders pretty regularly. The second statement, about Trump’s father, is true.

It seems plausible. But I follow my prime habit with social media: check your emotions. Never reflexively tweet something that factual that feels “perfect”.

A quick search shows there’s a 1998 article from the New York Times that says that “aides say” seven of nine of his great-grandmother’s children were killed by Nazis. That’s good, and raises the likelihood it’s true. The old priors plus this new information become our new priors. We’ve moved from plausible to probable.

But I want to hear it from Chuck Schumer’s mouth, not some unnamed aides responding to a campaign attack in 1998.

And when you start to try to find Schumer saying it it gets less clear. There is Holocaust after Holocaust event that Schumer has attended — and yet this fact never makes the papers or his speeches:

schumer

Absence of evidence is not strong evidence of absence. But it is evidence, especially as it starts to pile up. With each failed attempt to find support for this, my disposition towards this fact inches down, moving from likely and sinking back towards plausible.

Then, at some point, I change my search terms. One of the unreliable sites on this question — a forum post —  mentions a “porch” where his great grandmother was killed. That’s a specific detail that is likely to get me closer to the event. So I throw it in and look what comes up:

schumer

And when we go to that top result we find testimony from Schumer at a congressional hearing on reparations for Holocaust survivors:

Senator Schumer. Now I am going to give my opening  statement, and first I want to start by thanking our Chairman,  Chairman Leahy, for letting me have the gavel today in order to  explore this exceptionally important topic: how to resolve what  I hope, what we all hope are among the last remaining  reparation claims stemming from the murder of 6 million Jews during the Holocaust. We all know the horror of the Holocaust.  My great-grandmother, who was the matriarch of her family, was told to leave her home. She and her family had gathered on the front porch. They refused to leave, and they just machine-gunned all of them down in 1941. So, obviously, I have personal experience with the horrors of the Holocaust, but the horrors are just awful.

Sometimes we refer to the horror as “unspeakable.” But unspeakable is exactly what the Holocaust must never become.  Those who perpetrated it, those who benefited from it want us not to speak. But we are here to speak and to have this hearing.

Now that’s a good source — official testimony from Schumer himself. From a written statement. The weight of this evidence outweighs everything prior, but is still added to it. It’s not just that Schumer is telling a story here, but that he is telling a story about an event that was plausible to begin with.

Is it bulletproof? No. Schumer could, of course, be lying, or exaggerating. He might have heard or remembered the story told him wrong. But right now, the best information we have is this testimony plus the remarks of others (such as aides) over a 20 year period. We have enough here, in absence of other evidence, to call this claim true.

But unlike “doubt” or “certainty”– the demand that something anything less perfect knowledge one way or another must leave us in a useless middle ground, we end up, with each step, getting better, more informed priors even as our decisions on what is true vacillate. By the end we call this true, because to overcome what we know here would require strong evidence that currently doesn’t seem to exist. But we’d be excited to get new information, even if it contradicted this, because it would build a better set of priors, both for this and other related claims.

This post is pretty nascent stuff — and maybe I’ve bit off a bit more than I can chew here. But I suppose what I’m saying is that fact-checking on a complex claim looks a bit like this:

truth.png

We’ll get this together in a better presented post at some later time. But I do think one of the primary goals of fact-checking is to get students to think about truth in more nuanced ways, and this is the sort of direction I see that going, instead of the cynical skepticism we often peddle.

 

 

 

How “News Literacy” Gets the Web Wrong

I have a simple web literacy model. When confronted with a dubious claim:

  • Check for previous fact-checking work
  • Go upstream to the source
  • Read laterally

That’s it. There’s a couple admonitions in there to check your emotions and think recursively, but these three things — check previous work,  go upstream, read laterally — are the core process.

We call these things strategies. They are generally usable intermediate goals for the fact-checker, often executed in sequence: if one stops panning out, then you go onto the next one.

The reason we present these in sequence in this way is we don’t just want to get students to the truth — we want to get them there as quickly as possible. The three-step process comes from the experience of seeing both myself and others get pulled into a lot of wasteful work — fact-checking claims that have already been extensively fact-checked, investigating meaningless intermediate sources, and wasting time analyzing things from a site that later turns out to be a known hoax site or conspiracy theory site.

To give an example, here’s a story from Daily Kos:

hunter

And here’s what students will say, when confronted with this after years of “close reading” training:

  • Who is this Hunter guy?
  • Hunter is a pseudonym, which is bad. How do we know who he really is? Suspicious!
  • What is this Daily Kos site?
  • Who owns Daily Kos? Liberals? Really?
  • There’s a lot of comments which is good.
  • The spelling and punctuation on this page is good, which makes it credible.
  • The site looks well designed.
  • The site is very orange.
  • There’s anti-Trump language on the page which makes it not credible and slanted.
  • The picture here isn’t of the Russians, it doesn’t match, which is fishy.

They might even go to Hunter’s about page and find that the most recent story he has recommended has, well, a very anti-Trump spin on it:

hunter2

They can spend hours on this, going to the site’s about page, reading up on Hunter, looking at the past stories Hunter wrote. And in my experience, students, when they do this, are under the impression that this time and depth spent here shows real care and thought.

Except it doesn’t. Because if your real goal is to find out if this is true, none of this matters.

What matters is this:

disk.JPG

What you see above, in the first paragraph of the story, is a link to the Wall Street Journal, the source of the claim. This “Hunter” might be a Democrat with a pseudonym invoking an 80s police procedural series, but he follows good, honest web practice. He sources his fact using something called “hypertext”. It’s a technology we use to connect related documents together on the web.

And once we see that — a way to get closer to the actual source of the fact, all those questions about who Hunter is and what his motives are and how well he spells things on this very orange looking site don’t matter, because — for the purposes of fact-checking — we don’t give a crap. We’re going to move up and put our focus on the Wall Street Journal, not Daily Kos.

(Disclosure — I used to write a bit on Daily Kos, I know certain front-pagers there, and yes I know that Hunter’s name is not really a reference to the uniquely forgettable Fred Dryer television series).

Once we get to the Wall Street Journal, we’re not done. We want to make sure that the Wall Street Journal‘s report on this is not coming from somewhere else as well. But when we get to the Wall Street Journal we find this is original reporting from the Journal itself:

The younger Trump was likely paid at least $50,000 for his Paris appearance by the Center of Political and Foreign Affairs. The Trump Organization didn’t dispute that amount when asked about it by The Wall Street Journal.

“Donald Trump Jr. has been participating in business-related speaking engagements for over a decade—discussing a range of topics including sharing his entrepreneurial experiences and offering career specific advice,” said Amanda Miller, the company’s vice president for marketing.

So going upstream comes to an end for us, and we move on to our next strategy — reading laterally about the site. Now in this case, we all might skip that — it is the Wall Street Journal we have here — but the truth is that students might not know whether to trust the WSJ. So we execute a trusty domain search: ‘-site:wsj.com wsj.com’, which tells Google to get all the pages that are talking about wsj.com that aren’t from that site itself:

not

And when we do that we see that there is a Wikipedia page on this site that will let us know that the WSJ is the largest newspaper in America by circulation and has won 39 Pulitzer prizes.

We do note, looking at the WSJ article, that Hunter has tweaked the headline here a bit. The WSJ says that Trump Jr. was likely paid $50,000, whereas Hunter’s headline is more strident about that claim. But apart from that the story checks out.

Do we trust this WSJ article 100%? No, of course not. But we trust it enough. It’s tweetworthy. And after we’ve confirmed that fact we can go back down to the Daily Kos page and see if that article by Hunter has any useful additional analysis. Over time, if you keep fact-checking Hunter’s stories, and they keep checking out, you might start considering him a reliable tertiary source.

If you use this process, you’ll notice a couple of things. The first one is that it’s pretty quick — the sort of thing that you can execute in the 90 seconds before you decide to retweet something.

But there’s another piece here too — rather than the fuzzy analysis of a single story from a single source you have here a series of clearly defined subgoals with defined exit points: check for previous work until there is no more previous work, get as close to the original as you can until you can get no closer, and read laterally until you understand the source. These goals are executed in an order that resolves easy questions quickly and hard questions efficiently.

That’s important, because if you can’t get it down to something quick and directed then students just think endlessly about what’s in front of them.  Or worse, they give up. They need intermediate goals, not checklists.

Fact-Checking the Mailman

Recently the Digital Polarization Initiative has been getting a lot of press, and as a result a lot of people have been sending me alternative approaches to fake news.

Most aren’t good. I’ve already talked about the reasons why CRAAP is ineffective. I’ve been more hesitant to talk about elements of a popular program from the News Literacy Project called Checkology, which is less obviously bad. But in past days I’ve seen more and more people talking about how Checkology might be a solution to our current problem.

Unfortunately, news literacy isn’t the big problem here. Web literacy is. And the Checkology curriculum — at least as I see it reported — doesn’t really address this.

As an example, here’s a Checkology checklist:

1. Gauge your emotional reaction: Is it strong? Are you angry? Are you intensely hoping that the information turns out to be true? False?

2. Reflect on how you encountered this. Was it promoted on a website? Did it show up in a social media feed? Was it sent to you by someone you know?

3. Consider the headline or main message:

a. Does it use excessive punctuation(!!) or ALL CAPS for emphasis?

b. Does it make a claim about containing a secret or telling you something that “the media” doesn’t want you to know?

c. Don’t stop at the headline! Keep exploring.

4. Is this information designed for easy sharing, like a meme?

5. Consider the source of the information:

a. Is it a well-known source?

b. Is there a byline (an author’s name) attached to this piece?

c. Go to the website’s “About” section: Does the site describe itself as a “fantasy news” or “satirical news” site?

d. Does the person or organization that produced the information have any editorial standards?

e. Does the “contact us” section include an email address that matches the domain (not a Gmail or Yahoo email address)?

f. Does a quick search for the name of the website raise any suspicions?

6. Does the example you’re evaluating have a current date on it?

7. Does the example cite a variety of sources, including official and expert sources? Does the information this example provides appear in reports from (other) news outlets?

8. Does the example hyperlink to other quality sources? In other words, they haven’t been altered or taken from another context?

9. Can you confirm, using a reverse image search, that any images in your example are authentic (in other words, sources that haven’t been altered or taken from another context)?

10. If you searched for this example on a fact-checking site such as Snopes.comFactCheck.org or PolitiFact.com, is there a fact-check that labels it as less than true?

Now, there’s some good things in here. I think their starting point — check your emotional reaction — is quite good, and it’s similar to some advice I use myself. Thinking about editorial standards is good. Reverse image search is a helpful and cool tool. Looking for reports from other sources is good.

But if you include subquestions, there are twenty-three steps to Checkology’s list and they are all going to give me conflicting information of relatively minor importance. What if there are no spelling errors but there is also no current date? What if the about page says the site is a premier news source, but it has no links back to original sources? What if it cites a variety of things but doesn’t hyperlink?

Even more disturbingly, this approach to fact-checking keeps me on the original page for ages. What if I get all the way through the quarter of an hour that the first twenty-two questions take only to find out on question twenty-four that Snopes has looked at this and it’s complete trash?

This isn’t hypothetical. Given the current reaction time of Snopes to much of the viral stuff on the web you could probably give Student A this long list and Student B a piece of paper that says “Check Snopes First” and the Snopes-user would outperform the other student every time.

And even if there is no Snopes article on the particular issue you are looking at, what good is it going to do you to look this deeply at the article in front of you if it is not the source. Consider our Hunter article:

hunter

Let’s answer the questions using Checkology:

1. Gauge your emotional reaction:

Is it strong?  Yes.

Are you angry? Yes.

Are you intensely hoping that the information turns out to be true? Yes.

False? No.

2. Reflect on how you encountered this.

Was it promoted on a website? Facebook

Did it show up in a social media feed? Yes.

Was it sent to you by someone you know? Yes

3. Consider the headline or main message:

a. Does it use excessive punctuation(!!) or ALL CAPS for emphasis? No.

b. Does it make a claim about containing a secret or telling you something that “the media” doesn’t want you to know? No.

c. Don’t stop at the headline! Keep exploring. Ok.

4. Is this information designed for easy sharing, like a meme? No.

5. Consider the source of the information:

a. Is it a well-known source? Maybe?

b. Is there a byline (an author’s name) attached to this piece? Kind of but fake.

c. Go to the website’s “About” section: There is no About section.

Does the site describe itself as a “fantasy news” or “satirical news” site? There is no About section.

d. Does the person or organization that produced the information have any editorial standards? Not sure how I find this?

e. Does the “contact us” section include an email address that matches the domain (not a Gmail or Yahoo email address)? Looked for the contact us page for a couple minutes but could not find it.

f. Does a quick search for the name of the website raise any suspicions?  Yes! It is listed on a site called “fake news checker”!

6. Does the example you’re evaluating have a current date on it? Yes

7. Does the example cite a variety of sources, including official and expert sources? No

Does the information this example provides appear in reports from (other) news outlets? Yes

8. Does the example hyperlink to other quality sources? Yes

In other words, they haven’t been altered or taken from another context? No

9. Can you confirm, using a reverse image search, that any images in your example are authentic (in other words, sources that haven’t been altered or taken from another context)? The image doesn’t match, it’s old!

10. If you searched for this example on a fact-checking site such as Snopes.comFactCheck.org or PolitiFact.com, is there a fact-check that labels it as less than true? No results.

Ok. So now we’ve spent ten to fifteen minutes on this article, looking for dates and email addresses and contact emails. Now what? I have no idea. The site appears on a list of fake news sites and doesn’t have a contact page. But it does have a date on the story and Politifact and Snopes don’t have stories on it. The image for the article is an old image (fake!).

And conversely, the article links to the Wall Street Journal as the source of the claim.

Hmm.

Are you starting to get the feeling we just spent a whole lot of time on a checklist that we are about to crumple up and through in the trash?

To put this in perspective, you got a dubious letter and just spent 20 minutes fact-checking the mailman. And then you actually opened the letter and found it was a signed letter from your Mom.

“Ah,” you say, “but the mailman is a Republican!”

How does this make any sense?

Staying On the Page

If you want to read how badly this fails, you can look at some of the stories about the program as it is used in the classroom. Here’s a snippet about some folks using Checkology (Update: Checkology has contacted me to make clear that the newspaper article does not represent their entire curriculum).

The students’ first test comes from Facebook. A post claims that more than a dozen people died after receiving the flu vaccine in Italy and that the CDC is now telling people not to get a flu shot. [One student] is torn.

“I mean, I’ve heard many rumors that the flu shot’s bad for you,” [she] says. But instinct tells her the story’s wrong. “It just doesn’t look like a reliable source. It looks like this is off Facebook and someone shared it.”

Cooper labels the story “fiction.” And she’s right.

This drives me nuts. It worked out this time, of course, because the story is false. But relying on your intuition like this, based on no real knowledge other than how a claim looks, is not what we should be encouraging here.

Worse, you see the biggest failing here — in any curriculum based around asking questions of a text, the student is not actually doing anything other than asking questions. They are looking at a text and seeing what feelings come from it after asking the questions.

Here’s another student on the same flu story:

Her classmate takes a different path to the same answer. When he’s not sure of a story, he says, he now checks the comments section to see if a previous reader has already done the research.

“Because they usually figure it out,” [he] says. And, indeed, he wasn’t the first to question the vaccine story’s veracity. “Like one comment was, ‘I just fact-checked this, and it doesn’t appear to be true. Where else do you see this to be true?’ “

I’m not attacking the student here — they are doing exactly what most curricula tell them to: looking at a page and asking questions about it. But you can see here that we just had a student use comments on an article to fact-check an article. Comments!

Comments can be useful, of course. When the trail has gone cold tracing a story to its source, often it’s a comment from someone that points the way to the original story. Sometimes a person points to an article on Snopes or Politifact.

But to get to the truth quickly, comments are usually the worst place to look. At this point, almost every anti-Trump story online has someone under it calling it “fake news”. What do you do with that? How does it help?

Again, this is not what a web literate person does when they hear that the flu vaccine may be bad. A web literate person finds the original source of the claim and then asks the web what it knows about the source. All this other stuff is mostly beside the point.

More than Fiction, Less than Fact

Which brings me to my second (third? tenth?) pet peeve here: there’s a muddling here of the issues of claim, story, and source.

Take that claim on Facebook that over a dozen deaths were caused by the flu vaccine and the CDC banned it. “Fiction,” said the student.

cdc

And it’s true that the source she was reading and the story that she was reading were misinformation. But is the story complete fiction? Let’s do a search:

capture

I’m guessing the student read the Natural News story down towards the bottom — Natural News is one of the big suppliers of anti-vaxx propaganda on Facebook. But the story here is not cut from whole cloth. Just reading the two blurbs at the top of the search results I get a pretty good idea what happened. The Wall Street Journal reports that on December 1 Italy suspended, pending an investigation, the use of two batches of the flu vaccine. This was apparently due to 12 people dying shortly after receiving it.

On December 3rd, the BBC reported that Italy had completed its investigation and cleared the vaccine as safe. A bit of domain knowledge tells me that what probably happened is what often happens with these things — flu vaccine is administered to a population that is relatively old and has a higher chance of dying due to any cause. Eventually those sort of probabilities produce a bunch of correlations with no causation.

By the way, this ability to read a page of results and form a hypothesis about the shape of a story based on a quick scan of all the information there — dates, URLS, blurbs, directory structure — that’s what mastery looks like, and that’s what you want your employees and citizens to be able to do, not count spelling errors.

So is this vaccine story “fiction”? I suppose so. It’s not true that the vaccine killed these people, and the CDC certainly didn’t cancel the vaccine. If we were doing a Snopes ruuling on this I’d go with a straight up “False” as the ruling.

But I’d also note there was a brief panic over a series of what we now know to be unrelated deaths, followed by an investigation that ruled the vaccines safe.

You are not going to get that if you stare at a page looking for markers that the story is true or false. You are only going to get that if you follow the claim upstream.

The Checkology list declares that students should “use the questions below to assess the likelihood that a piece of information is fake news.” In that instruction you have a dangerous conflation of source and claim, which is only furthered by confusing questions like “Does the example have a date on it?”

News as source, and news as claim. It’s an epistemological hole that we put our students in, and to help them out of it we hand them a shovel.

The Ephemeral Stream

How do programs like these get these issues wrong? The intentions are good, clearly. And there is a ton of talent working on it that’s had a lot of time to get it right:

The News Literacy Project was founded nearly nine years ago by a Pulitzer prize-winning investigative reporter with the Los Angeles Times, Alan C. Miller. The group and its mission have been endorsed by 33 “partner” news organizations, including The Associated Press, The New York Times and NPR.

Fundamentally, these efforts miss because what’s needed is not an understanding of news but of the web. 

As just one example, the twenty-plus questions that students are asked to ask a document seem to assume that

  1. Sources are scarce and we must absolutely figure out this source instead of ditching it for a better one.
  2. Asking the web what it knows about a source is a last resort, after reading the about page, counting spelling errors, tallying punctuation, and figuring whether an author’s email address looks a bit fishy.

The web is not print, or a walled garden of digital content subscription. Information on the web is abundant. And yet the strategies we see here all telegraph scarcity, as if the website you are looking at was a book you were handed in the middle of a desert to decipher.

The approach also does not come to terms with the Stream — that constant flow of reshareable information we confront each morning in Twitter or Facebook. You don’t have fifteen minutes to go through a checklist in the Stream. You have 90 seconds. And your question isn’t “Should I subscribe to Natural News?” — your question is “Did a dozen people die of flu vaccine?” Whether news folks want to admit it or not, the stream tends to erode brand relationships with providers in favor of a stream of undifferentiated headlines.

Above all, the World Wide Web is a web, and the way to establish authority and truth on the web is to use the web-like properties of it. Those include looking for and following relevant links as well as learning how to navigate tools like Google which use that web to index and rank relevant pages. And they include smaller things as well, like understanding the ways in which platforms like Twitter and Facebook signal authority and identity.

In short, we need a web literacy that starts with the web and the tools it provides to track a claim to ground. As we can see from the confused reactions of at least some of the students in the Checkology program, that’s may not be happening now, and “news literacy” isn’t going to fix that.


If you’re interested in alternative, web-native approaches to news literacy, you can try my new, completely free and open-source book Web Literacy for Student Fact-Checkers.

You should also read Sam Wineburg and Sarah McGrew’s Why Students Can’t Google Their Way to Truth, and the results of their Stanford study which showed that the major deficits of students with regard to news analysis were issues of web literacy and use.

The News Literacy Project has responded to this piece with one clarifying the scope and nature of their curriculum, which is not fully captured by the newspaper article and the checklist. I’ve made some small edits to the article to reflect this is not a review of their entire curriculum, but only of certain elements. On the other hand, I can not review elements that are hidden, and the elements that are available do reflect, in my opinion, some errors in thinking about the problem.