Josh Marshall, who is generally one of the better political commentators out there, recently wrote a piece called Why You’re Fooling Yourself About Fake News. The point of the piece is that liberals who believe that fake news sways elections are wrong. Fake news, says Marshall, is a demand-driven phenomena: the people reading and believing fake news are reading it as an expression of beliefs that are already set. This has been common wisdom for a while, and during the flurry of calls from reporters I got after the election it was probably the top question I was asked — Sure, people saw that Hillary may have killed an FBI agent — but given how extreme you’d have be to believe that, how could it sway any votes?

So caveats first. A lot of what we repost falls into what I’ve called “Identity Headlines“. I repost something on the homelessness problem in Silicon Valley partially to raise awareness, but largely to show that I’m the sort of person who cares about the homeless and hates Silicon Valley dicks. I don’t even need to read the article to achieve this effect, and if I’m like most people on most days, I probably don’t.

Fake news is similar, and in this way is demand -driven. A lot of people really, really, really hated Hillary Clinton, but after multiple hacks and leaks and congressional investigations the actual world was still not producing headlines that adequately expressed the level of hatred and suspicion many people felt. So the market stepped in and created some fake ones. Headline-as-bumper-sticker problem solved!

Now, that headline above gets much closer to how a lot of people felt about Clinton than all of the stuff about how she may have said that some Bernie supporters were living at home with their parents or that people were deplorable. So people share and repost it, and it’s doubtful that the people who did that, at least initially, changed their vote. As Josh says, they were in the market for this from the start.

But I can’t stress this enough: this is only one half of the equation, and I remain confused that smart people can’t see that.

Fake News Headlines Had an Impact on Beliefs of Non-Posting Readers

The first problem with the “fake news only solidified partisanship” is what data we have about readers (not re-posters) tends to contradict that. Actually, “contradicts” is an understatement.

Now caveats on the analysis I’m about to show. Surveys on “Did you hear about X?” have some issues. Some people, put on the spot about what they know, want to appear knowledgeable, and a lot of people on opinion polls want to express a sort of personal orientation to the world more than a specific reaction to poll questions. There’s a survey on conspiracy theories, for example, where the pollsters asked about a fictional conspiracy — the North Dakota Crash — only to find that 32% endorsed it. By fictional conspiracy, by the way, I don’t mean just “conspiracy that is not true” but “conspiracy the pollster invented that has never been talked about before.” In a similar vein, we’ve never quite been able to tease out how many Americans believe that Obama is a Muslim and how many of them have been using that question as a proxy for how they feel about Obama.

But caveats aside, the one poll I’ve seen that’s actually asked folks what stories they saw and if they believed them is this one from Buzzfeed. And even when mentally accounting for some general-issue polll weirdness the results are still surprising.

The first finding was that a large percentage of people had seen fake headlines — as large as any real story. And the exposure was far less polarized than one would think. For example, for the infamous FBI Murder-Suicide headline, 22% of respondents had seen it, but about 40% of those who had seen it were Clinton voters.

Now we know that the Clinton voters who saw such anti-Clinton and pro-Trump fake stories are going to be much more likely to reject those stories as false. But how much more likely? Keep in mind these stories intimate that Clinton had an FBI agent killed and that the Pope endorsed Donald Trump. They are not subtle.

What we find surprised even me. Among Clinton voters that remembered a fake headline, the majority believed it in almost all instances.

That FBI agent story, for example? Over 50% of Clinton voters who remembered it believed it. Over 60% of Clinton voters who remembered the story about Trump protesters being paid believed that as well.

Is it a perfect way to assess this? No — I have some issues with the methodology of the poll. I think a different approach would shave some points off here. But even if you cut these effect in half, they are still completely inconsistent with the idea that fake headlines don’t influence belief, or that partisan leanings innoculate people against them.

Don’t like the study? Do another one (seriously, do it — we need a better one). But at the moment it’s the best evidence we have as to what went on, and it supports the idea that headlines matter.

If a bunch of Democrats went to the polls to vote for Hillary despite the fact she may have murdered an FBI agent, is it hard to believe that at least some Democrats didn’t stay home out of disgust? In fact, isn’t it probable?

How many? We don’t know. But we do know that it would only take about a football stadium’s worth of voters across three states to sway the election. This poll suggests that millions of Clinton voters went to the polls believing damaging stories about Clinton that were completely false. Millions. It really does not seem a stretch to think this had an impact.

What do these voters look like? I’ve met them, personally. Folks — Democrats, even — that believe that maybe Seth Rich, a DNC worker who died in a street robbery last summer, might have been killed because he knew too much about the DNC leaks, or was going to leak additional material. When pressed, they retreat into “well, I didn’t say definitely, but something is weird there.”

Um, no. Nothing is weird there at all. What’s weird is seriously considering this possibility.

The Weird Nihilism of the Fake News Didn’t Matter Argument

So we get then to the second point: maybe people did believe these stories, but they didn’t have an effect. Clinton voters thought — oh, the FBI agent died suspiciously, but that doesn’t prove anything. And there’s a 5% chance that Clinton may have killed a 20-something computer-voting specialist at the DNC, but that’s OK, because I like her policies.

Taken to it’s logical conclusion this amounts to a sort of nihilism, because it implies that “real news” doesn’t matter either.

Josh believes, for example, that the Comey letters did serious damage to Clinton, and may have lost her the election (for what it’s worth, I agree). But if we believe that people are innoculated against headlines by partisan belief, how is that possible? Under the pure “Identity Headlines” analysis no one’s opinions should have changed — pro-Clinton folks would read the Comey letter headlines and think “Well, more interference” and pro-Trump folks would read the headlines and say “Yep, Crooked Hillary” and the needle wouldn’t move.

But of course that’s not what happened. And I’ll quote Josh here:

Just to put my cards on the table, I believe there is a good likelihood, probably even a probability, that if the Russian subversion campaign had never happened and James Comey had never released his letter, Hillary Clinton would be prepping to become our new President. My own guess is that Comey’s letter had the bigger impact. These were both profoundly damaging events in the race and Clinton lost by very tight margins in most of the newly (hopefully temporarily) red states. I see little way to challenge this assertion.

So the Comey headlines hurt, but the fake headlines did not? How is this a tenable position?

Not to belabor the point, but imagine a world where Comey did not release a letter about the Weiner emails, but a large percentage of voters believed he had, due to fake headlines. Why would we expect that the real news would have a different impact than fake news? To a gullible reader, the fake event and the real event have equal impact.We don’t live in a world where we experience fake news differently from real news — to the reader they are equally mediated events, and indistinguishable from one another.

There’s a good argument to be made that events, and campaigns, and news, and scandals don’t matter for the vast majority of voters. They probably don’t. But to say that fake news can’t swing an election you have to move from an argument that people are resistant to news to one that they are impervious to it. And this is in fact to say that nothing in a campaign matters — not positions, not actions, not scandals, not ill-considered comments. Not the campaign itself, or the coverage it generates. That just doesn’t make sense.

So yesterday I spent the day — and really, almost the entire day — fact-checking so-called “statements of fact” here, on this rather boring Rick Perry article.

So, for example, the article says that Rick Perry did the Cha-Cha on Dancing with the Stars. We check that Perry actually got a “D” in a class called “Meats”.  We verify that the Energy Department oversees the nuclear arsenal.

We know all this already, or most of it. I didn’t know he did the cha-cha, but I knew he was on Dancing with the Stars. I didn’t know about the “D” in meats, admittedly, but I did know that the Energy Department oversees the nuclear arsenal, which is probably a more useful fact to hold on to.

Let’s keep going for a bit.

Rick Perry is the longest serving governor of Texas, check. Perry oversaw a massive expansion in wind power in Texas, as well as streamlining permits for polluters, check. Steven Chu, the current Secretary, won a Nobel prize, and was an MIT physicist, check. Chu was the Secretary for Obama’s first term as well, check.

That’s barely the start. For each claim, I tracked down either a definitive source or a couple half-decent sources. And each claim was true.

What should you fact-check? (and don’t say “everything”)

The Hypothes.is annotation tool turns out to be an amazing tool to do this sort of work, and I’d recommend it to student fact-checkers, as it allowed me to link directly to the quotes in context that supported the claims. Click around in this page and see what it makes possible. I’m impressed.

But the work here? Boring. Incredibly boring. Mind-numbingly dull, at least to me. Proving things I not only already now to be true, but things I think are already common knowledge among a politically literate set turns out to be less than rewarding.

Scanning this article, I know what claims — out of probably a hundred of so claims made — are bound to be most interesting:

• Perry expanded wind power as governor of Texas
• Perry streamlined permits for plants as governor
• Perry forcefully pushed deregulation in Texas, despite post-Enron pushback
• Perry was known as a pragmatic, not ideological, governor.

Why those four claims? Because those are the ones that seem to me

• Most pertinent to the issue of his appointment
• Most likely to be “squishy”, while still being close to “fact” territory

That second piece is really important. While some interpretation is needed for the above claims, there’s an empirical component to them that makes directed investigation possible. Wind power increased or it didn’t, and actions that Perry took can either be plausibly tied to that or not.  I can’t know whether Perry was a pragmatic governor, but I can probably survey enough contemporary articles to see if a significant bipartisan population thought that during his tenure.

At the same time, the claims are “squishy” — they are not really a binary true/false sort of thing, although with enough evidence one way or another we could probably make such a ruling. What we really decide, usually, is whether these claims are well-supported by the evidence out there. And since this news source (in this case Politico) has presented these claims as statements of fact, we learn a bit about what Politico considers the statements-of-fact cutoff. We learn what Politico considers to be “well-supported”, and that in turn might tell us if we really want to continue reading Politico.

In other words, these aren’t just the most interesting claims, they are likely to be the most revealing about the source as well. Questions on either side of these, whether purely factual (Chu won a Nobel Prize) or very abstract (“Perry may be smarter than people think”)  can tell us interesting things about the credibility and bias of the publication, but my sense is it’s these squishy questions that give us the biggest bang for the buck.

I’m wondering if students can spot the most promising four or five claims in an article, or if it takes more background knowledge to do than we think.  Any thoughts?

Another quick lesson in sourcing viral user-created content. Here’s a picture that showed up in my stream today.

OK, so what’s the story here? To get more information, I pull the textual information off the image and throw it in a Google search:

Which brings me to a YouTube video that tells me this was taken “outside a Portland, Oregon Walmart” and has been shared “hundreds of times since yesterday”. So back to search. This next result shows you why you always want to look past the first result:

I type in Portland OR, but the fourth result looks like it is reporting the story as a “local” story (look at the URL) and its location is not Portland OR, but Biddeford, Maine. Further indications here that it might be a good source is that I see in the blurb it mentions the name of the photographer “Matthew Mills”. The URL plus the specificity of the information tell me this is the way to go.

That article points me to what looks like the source where it went viral.

We see here that the original news report had a bunch of things wrong. It wasn’t in Portland, Oregon — it was in Biddeford, which is near Portland, Maine. It hasn’t been shared “hundreds of times” it’s been shared hundreds of thousands of times. And it was made viral by a CBS affiliate, a fact that ABC Action News in Tampa doesn’t mention at all.

OK, let’s go one more step. Let’s look at the Facebook page where Matthew Mills shared it. Part of what I want to see is whether is was viral before CBS picked it up or not. I’d also like to double check that Mills is really from the Biddeford area and see if he was responsible for the shopping carts.

The news post does not link back to the original, so we search on Matthew Mills again, and see some news outlets mentioning the original caption by Mills: “This guy got a lesson in parking”.

That’s not the same as the caption that the news station put up. So we pump that into Facebook, and bingo: we get the original post:

And here’s where we see something I really dislike about news organizations. They cut other news organizations out of the story, every time. So they say this has been shared hundreds of times because in order to say it has been shared hundreds of thousands of times they’d have to mention it was popularized by a CBS affiliate. So they cut CBS out of the story and distort the truth.

On the other hand, one of the good effects of it is sometimes it makes it easier to track something down to the source. News organizations work extra hard to find the original source if it means they can cut other news organizations out of the picture.

But it also tends to distort how virality happens. The picture here did not magically become viral — it became viral due, largely, to the reach of WGME.

Incidentally, we also find answers to other questions in the Matthew Mills version: he didn’t take the picture, and he really is from Old Orchard Beach.

Just because we’re extra suspicious, we throw the image into Google Image to see if maybe this is a recycled image. It does not appear to be, although in doing that we find out this is a very common type of viral photo called  a “parking revenge” photo. The technique of circling carts around a double-parked car dates back to at least 2012:

When we click through we can see that the practice was popularized, at least to some extent, by Reddit users. See for instance this post from December 2012:

So that’s it. It’s part of a parking revenge meme that dates back at least four years, and popularized by Reddit. It was shot by Matthew Mills in Biddeford, Maine, who was not the one who circled the carts. And it became viral through the re-share provided by a local Maine TV station.

Again, all of this takes some time to write about. In practice, though, it doesn’t take much time at all to check.

I’ve been addicted to fact-checking for a while. Ten or fifteen years, maybe? Longer? When I see something in any of my feeds that doesn’t smell quite right, I have to hold myself back from checking it. It’s just somehow one of those enjoyable “flow” activities for me.

That said, I never really watch what I do when I do it. Now that we’re putting together a student fact-checking project, I thought I might pay closer attention. I’m fully aware the novice/expert distinction here means that how I approach this may not be accessible to students, but you have to start somewhere.

So here’s today’s installment, because it demonstrates how to solve a fact-checking problem you often make multiple passes through Google.

The Letter

So this letter shows up in my feed. It’s Richard Nixon telling Donald Trump if he ever runs for office, he’ll do well. It’s dated 1987.

Now I say it “doesn’t smell right” — but actually that’s not true at all. This looks to me exactly like a letter that Nixon might have written, and certainly one that Trump would have kept (and therefore one that would have survived). So, actually, it smells great!

What makes me check it is not suspicion, really, but the fact its exactly the sort of letter that’s bound to go viral. Because that’s true, that means there’s also a good incentive for someone to make it up. If there’s incentive to make it up, then we should check to make sure that’s not what happened here.

So that’s rule one. When you find yourself looking at something that feels just perfect, check it. So let’s find out — is this letter really authentic?

Reverse Image Search

I’m a big fan of reverse image search for things like this. A lot of times it fails, but when it succeeds it brings you straight to the right evidence. So I try it out with this image, but get a dead end.

First Google Search

I then type  “Pat Nixon” and “Donahue show” into Google. Why this combination of words? Because I am imagining the sort of document that could prove this letter right or wrong, and I’m trying to figure out what words would appear together in that imagined document that would generally not appear together in other documents.

I get a hit!

It’s the NY Daily News, which is OK as things go as a source. But there’s a problem. The letter I saw said ‘1987’ whereas the NY Daily News here says 1981. And there’s no scanned picture here. So which is wrong on the date, the article or the document?

Here I make a quick detour, and check Wikipedia to see if the Donahue show was even on in 1987. If it wasn’t, then the picture of the letter is probably a fake. But alas, Donahue continued way past 1987, so both dates are possible:

Second Google Search

We now go back to the NY Daily News article and skim it. We need another search term that is going to be specific to this image/letter. The article mentions the name of the book that the letter appeared in, a biography of Trump called “Never Enough” I put the phrase “Never Enough” into my image search, on the theory that any reliable article showing the letter will mention the source, and bingo:

I see six different versions of the letter here. I hover over each of them to see the source, looking for something better than the NY Daily News: Brietbart (nope), Conservative Post (nope),  NY Times (yes!).

Why am I excited about the New York Times? Because I’m a godless liberal?

No, I’m excited because the NY Times pays a price for getting things like this wrong, whereas the Breitbarts of the world suffer no consequences for error. That’s the main difference. The same would be true of the Wall Street Journal, which leans right, or USA Today. which is straight down the middle. We’re not looking for an ideology match here. We’re looking for someone with incentives to not publish hoaxes.

I go there and find that, unlike the NY Daily News, the NY Times has the year correctly, and cites the letter as well as includes it.

The story below it confirms the date and the source. So we’re done, for the moment. We could certainly get past our 95% confidence in this letter — bring it up to 98% or so with more checking. We could go look at the actual book that published it for example.

But at this point it’s diminishing returns. The New York Times is still showing the letter after having it up for months, and there’s no embarrassed retraction under it. I’m confident enough to re-share.

I call this process “levering up” because it reminds me of when I was a kid in a rural town and we used to try to move these huge boulders. You’d get a decent-sized stick under the rock, and lever it up so that you could get the next, larger stick under it, and so on. Finally you’d get the monster of a stick you wanted under it and the leverage and stick and position would seem just right, and the rock would move almost effortlessly. But you got there only by this recursive process of small gains.

That’s what you do with these Google searches. A search reveals a document that is not quite right, but has another search term that gets you another document that is better. Even with the novice/expert distinction, I think this is one skill/understanding all students need.

Incidentally — the whole process here, end to end was about five minutes, of which three were capturing and saving screenshots. So about two minutes of work. Certainly something people could afford to do a couple times a day with content they retweet or repost.

One of the problems I’ve had for a while with traditional digital literacy programs is that they tend to see digital literacy as a separable skill from domain knowledge.

In the metaphor of most educators, there’s a set of digital or information literacy skills, which is sort of like the factory process. And there’s data, which is like raw material. You put the data through the critical literacy process and out comes useful information on the other side. You set up the infolit processes over a few days of instruction, and then you start running the raw material through the factory, for everything from newspaper articles on the deficit to studies on sickle cell anemia. Useful information, correctly weighted, comes out the other end. Hooray!

This traditional information/web literacy asks students to go to a random page and ask questions like “Who runs this page? What is their expertise? Do they have bias? Is information clearly presented?

You might even get an acronym, like RADCAB, which allows you to look at any resource and ascertain  its usefulness to a task.

From http://radcab.com.

Or maybe, if you’re in higher education, you’ll get CRAAP:

From http://ucsd.libguides.com/preuss/webeval

I’m a fan of checklists, and heuristics, and I’ve got no problem with a good acronym

But let me tell you what is about to happen. We are faced with massive information literacy problems, as shown by the complete inability of students and adults to identify fake stories, misinformation, disinformation, and other forms of spin.  And what I predict is that if you are in higher education every conference you go to for the next year will have panel members making passionate Churchillian speeches on how we need more information literacy, to much applause and the occassional whoop.

But which information literacy do we need? Do we need more RADCAB? Do we need more CRAAP?

In reality, most literacies are heavily domain-dependent, and based not on skills, but on a body of knowledge that comes from mindful immersion in a context. For instance, which of these five sites are you going to trust most as a news (not opinion) sources?

Source One

Source Two

Source Three

Source Four

Source Five

Could you identify which of these sites was likely to be the most careful with facts? Which are right-wing and which are left wing? Which site is a white supremacist site?

It’s worth noting if you were able to make those determinations you did it not using skills, but knowledge. When I saw that big “W” circled in that red field of a flag, for instance, my Nazi alarm bells went off.  The mention of the “Illuminati” in source three tells me it’s a New World Order conspiracy site. I know little things, like the word “Orwellian” is a judgmental word not usually found in straight news headlines. I’ve read enough news to know that headlines that have a lot of verbal redundancy (“fabricates entire story falsely claiming”, for example, rather than “falsely claims”) are generally not from traditional news sources, and that the term “Globalist” is generally not used outside opinion journalism and “war against humanity” is pitched at too high a rhetorical level.

We act like there’s skills and a process, and there is, certainly. Asking the right question matters. Little tricks like looking up an author’s credentials matters. CRAAP matters. And so on. But the person who has immersed themselves in the material of the news over time in a reflective way starts that process with three-quarters a race’s head start.  They look at a page and they already have a hypothesis they can test — “Is this site a New World Order conspiracy site?” The person without the background starts from nothing and nowhere.

Abstract skills aren’t enough. RADCAB is not enough.

Government Slaves

I first really confronted this when I was helping out with digital fluency outcomes at Keene State about six years ago. One of the librarians there called me into a meeting. She was very concerned, because they ran an information literacy segment in classes and the students did well enough on the exercises, but when paper time came they were literally using sites that looked like this:

She was just gobsmacked by it. She didn’t want to say — “Look, just don’t use the internet, OK?” — but that was what she felt like after seeing this. It was crushing to spend two days talking authority and bias and relevance and the CRAAP test,  having the students do well on small exercises, and then having students in the course of a project referencing sites like these. (I should say here that we can find lots of sites like this on the liberal side of the spectrum too, but under the Obama administration these particular sorts of sites thrived. We’ll see what happens going forward from here.)

When I started talking to students about sites like this, I discovered there was a ton of basic knowledge that students didn’t have that we take for granted. That FEMA banner is a red flag to me that this site is for people that buy into deep right wing conspiracies that the Obama Administration is going to round conservatives up into FEMA prison camps. The “Government Slaves” to me is a right-wing trope — not necessarily fringe, but Tea Party-ish at the least. Those sites in the top menu — Drudge, Breitbart, InfoWars, and ZeroHedge — are all a sort of conspiracy spectrum starting with alarmist but grounded (Drudge, ZeroHedge) to full on conspiracy sites (InfoWars). The stars on the side signal a sort of aggressive patriotism, and the layout of the site, the Courier typography, etc., is reminiscent of other conspiracy sites I have seen.  The idea that cash/gold/silver is going to be “taken away” by the government is a prominent theme in some right-wing “prepper” communities.

Now we could find similar site on the left. My point here is not about the validity of the site. My point is that recognizing any one of these things as an indicator — FEMA, related sites, gold seizures, typography — would have allowed students to approach this site with a starting hypothesis of what the bias and aims of this site might be which they could then test. But students know none of these things. They don’t know the whole FEMA conspiracy, or that some people on the right feel the government is so strong we are slaves to it. They don’t know the who gold/prepper/cash thing.  And honestly, if you start from not knowing any of this, why would this page look weird at all?

The Tree Octopus Problem

When I started looking at this problem in 2010, I happened upon an underappreciated blog post on critical thinking by, oddly enough, Robert Pondiscio.

I say “oddly enough”, because Pondiscio  is part of a movement I often find myself  at odds with: the “cultural literacy” movement of E.D. Hirsch. That movement contended early on that our lack of common cultural knowledge was inhibiting our ability to discuss things rationally. With no common touchpoints, we might as well be speaking a different language.

The early implementations of that — complete with a somewhat white and male glossary of Things People Should Know — rubbed me the wrong way. And Hirsch himself was a strong adversary of the integrated project-based education I believe in, arguing the older system of studying a wide variety of things with a focus on the so-called “lower levels” of Bloom’s Taxonomy was more effective than project-based deep dives. Here’s Hirsch talking down a strong project-based focus in 2000:

To pursue a few projects in depth is thought to have the further advantage of helping students gain appropriate skills of inquiry and discovery in the various subject matters. One will learn how to think scientifically, mathematically, historically, and so on. One will learn, it is claimed, all-purpose, transferable skills such as questioning, analyzing, synthesizing, interpreting, evaluating, analogizing, and, of course, problem solving—important skills indeed, and well-educated people possess them. But the consensus view in psychology is that these skills are gained mainly through broad knowledge of a domain. Intellectual skills tend to be domain-specific.The all-too-frequent antithesis between skills and knowledge is facile and deplorable. (Hirsch, 2000)

I’ve used project-based learning forever, and my whole schtick — the thing which I’ve being trying to get done in one way or another for 20 years now — is scalable, authentic, cross-insitutional project-based education. I’m looking to scale what Hirrch is looking to dismantle. So Hirsch is a hard pill to swallow in this regard.

But it’s those last three lines that are the core of the understanding, and it’s an understanding we can’t afford to ignore: “[T]he consensus view in psychology is that these skills are gained mainly through broad knowledge of a domain. Intellectual skills tend to be domain-specific. The all-too-frequent antithesis between skills and knowledge is facile and deplorable.”

Robert Pondiscio, who works with Hirsch, shows specifically how this maps out in information literacy. Reviewing the 21st century skills agenda that had been released back in 2009, he notes the critical literacy outcome example:

Outcome: Evaluate information critically and competently.

Example: Students are given a teacher-generated list of websites that are a mixture of legitimate and hoax sites.  Students apply a website evaluation framework such as RADCAB (www.radcab.com) to write an explanation for deciding whether each site is credible or not. 

Pondiscio then applies the RADCAB method to a popular assignment of the time. There is a hoax site called the Tree Octopus often used by educators — teachers send students to it and have them try to evaluate whether it is real.

Unfortunately, as Pondiscio points out, any quick application of RADCAB or any other “skills only” based heuristic will pass this site with flying colors

The rubric also tells us we are research pros if we “look for copyright information or ‘last updated’ information” in the source.  Very well: The tree octopus site was created in 1998 and updated within the last two months, so it must be a current source of tree octopus information.  We are also research pros if we ”look for the authority behind the information on a website because I know if affects the accuracy of the information found there.”  Merely looking for the authority tells us nothing about its value, but let’s dig deeper.  The authority behind the site is the “Kelvinic University branch of the Wild Haggis Conservation Society.” Sounds credible. It is, after all, a university, and one only has to go the extra mile to be a Level 4, or “Totally Rad Researcher.”  The Tree Octopus site even carries an endorsement from Greenpeas.org, and I’ve heard of them (haven’t I?) and links to the scientific-sounding ”Cephalopod News.”

If you want to know the real way to evaluate the site, claims Pondiscio, it’s not by doing something, it’s by knowing something:

It’s possible to spend countless hours looking at the various RADCAB categories without getting the joke.  Unless, of course, you actually know something about cephalopods — such as the fact that they are marine invertebrates that would have a tough time surviving or even maintaining their shape out of the water — then the hoax is transparent.

And, in fact, when we shake our heads at those silly students believing in the Tree Octopus, we’re not surprised at the fact they didn’t look for authority or check the latest update. We’re disappointed that they don’t understand the improbability of a cephalopod making the leap to being an amphibious creature without significant evolutionary changes. We’re amazed that they believe that long ago a common cephalopod ancestor split off into two branches, one in the ocean, and one in the forest, and they evolved in approximately the same way in polar opposite environments

That’s the weird thing about the Tree Octopus. And that’s what would make any informed viewer look a bit more deeply at it, not RADCAB analysis, not CRAAP, and not some generalized principles.

To Gain Web Literacy You Have to Learn the Web

There’s a second point here, because what a web literate person would actually do on finding this is not blindly go through a checklist, but execute a web search on Tree Octopus. Doing that would reveal a Snopes page on it, which the web literate person would click on and see this:

Why would they click Snopes instead of other web search results? Is it because of Relevance, or Currency? Do you have some special skill that makes that particular result stand out to you?

No, it’s because you know that Snopes is historically a good site to resolve hoaxes. If it was a political question you might choose Politifact. If it wasn’t a hoax, but a question that needed answering, you might go to Stack Exchange. For a quote, Quote Investigator is a good resource with a solid history. Again, it’s not skills, exactly. It’s knowledge, the same sort of knowledge that allows a researcher in their field to quickly find relevant information to a task.

But let’s say you went to Wikipedia instead of Snopes. And again, you found it was labelled a hoax there:

Well, to be extra sure, you’d click the history and see if there were any recent reversions or updates, especially by anonymous accounts. This is Wikipedia, of course:

Looking at this we can see that this page has had a grand total of seven characters changed or added in the past six months, and almost all were routine “bot” edits. Additionally we see this page has a long edit history — with hundreds of edits since 2010. The page is probably reliable in this context.

Don’t know what Wikipedia bots are, or what they do? Honestly, that’s a far greater web literacy problem than applying “currency” to Wikipedia articles.

Incidentally, “currency” in our RADCAB rubric gets Wikipedia  backwards. If we had arrived at this Wikipedia page in 2010, for example, on February 28th from about 6:31 p.m. to 6:33 p.m. we would have found a page updated seconds ago. But in Wikipedia, that can often mean trouble, so we would have checked “Compare Revisions” and found that minutes ago the assertions the page made were reversed to say that the Tree Octopus was real:

Furthermore, it’s an edit from an anonymous source, as you can tell from the IP address at the top (65.186.215.54). The recency of the edit, especially from an anonymous source, makes this a questionable source at this particular moment.

Incidentally, in a tribute to the efficiency of Wikipedia, this edit that asserts the Tree Octopus is real is up for less than 120 seconds before an editor sees the prank and reverts it.

How are you supposed to know this stuff? Edit histories, bots, character counts as an activity check, recency issues in Wikipedia, compare revisions, etc? How are you going to know to choose Snopes over “MyFactCheck.com”?

Through a general skills checklist? Through a rubric?

The radical idea I’d propose is that someone would tell you these things, the secret knowledge that allows web literate people to check these things quickly. That secret knowledge includes things like revision histories, but also domain knowledge — that Snopes is a good hoax checker, and Quote Investigator is a good source for checking quotes. It includes specific hacks to do reverse image searches to see if an image is real, using specific software such as TinEye or Google Reverse Image Search.

Further, the student would understand basic things like “How web sites make money” so they could understand the incentives for lies and spin, and how those incentives differ from site to site, depending on the revenue model.

In other words, just as on the domain knowledge side we want enough knowledge to quickly identify whether news items pass the smell test, on the technical side we don’t want just abstract information literacy principles, but concrete web research methods and well-known markers of information quality on the web.

We Don’t Need More Information Literacy. We Need a New Information Literacy.

So back to those inevitable calls for more information literacy and the inevitable waves of applause.

We do need more education to focus on information literacy, but we can’t do it the way we have done it up to now. We need to come down from that Bloom’s Taxonomy peak and teach students basic things about the web and the domains they evaluate so that they have some actual tools and knowledge to deal with larger questions effectively.

I’ll give an example. Recently there was a spate of stories about a study that found that students couldn’t differentiate “sponsored content” from native content. Many thinkpieces that followed talked about this as a failure of general literacy. We must build our student’s general critical thinking skills! To the Tree Octopus, my friends!

The study authors had a different idea. The solution, they wrote, was to have students read the web like fact-checkers. But to do that we have to look at what makes fact-checkers effective vs. students. Look, for example, at one of the tasks the students failed at — identifying the quality of a tweet on gun control:

As the study authors note, a literate response would note two things:

• The tweet is from Move On, and concerns a study commissioned by Center for American Progress, a liberal think tank, and this may indicate some bias, but
• The poll they link to is by Public Policy Polling, a reputable third party third party polling outfit, which lends some legitimacy to the find.

The undergraduates here did not do well. Here’s how they struggled:

An interesting trend that emerged from our think-aloud interviews was that more than half of students failed to click on the link provided within the tweet. Some of these students did not click on any links and simply scrolled up and down within the tweet. Other students tried to do outside web searches. However, searches for “CAP” (the Center for American Progress’s acronym, which is included in the tweet’s graphic) did not produce useful information.

You see both sides of the equation in this tweet. A fact checker clicks links, obviously. And while that seems obvious, keep in mind it never occurred to half the students.

But the second part is interesting too — the students had trouble finding the Center for American Progress because they didn’t know “CAP”. There’s a technical aspect here, because if they just scoped their search correctly — well, here’s my first pass at a search:

So one piece of this is students need to know how to use search. But the other piece is they need to be able to recognize that we call this policy area “gun control”.  That sounds weird, but again, consider that most of these students couldn’t figure that out.

And honestly, if you fact check on the internet long, you’ll end up knowing what MoveOn is and what the Center for American Progress is. Real fact-checkers don’t have to check those things, because any person that tracks these issues is going to bump into a handful of think tanks quite a bit. Learning what organizations like CAP, Brookings, Cato, AEI, and Heritage are about is actually part of the literacy, not a result of it.

Instead of these very specific pieces of knowledge and domain specific skills, what did the students give back to the researchers as method and insight? They gave them information literacy platitudes:

Many students made broad statements about the limitations of polling or the dangers of social media content instead of investigating the particulars of the organizations involved in this tweet.

You see the students here applying the tools that information literacy has given them. Be skeptical! Bias happens! Social media is not trustworthy!

And like most general-level information literacy tools, such platitudes are not useful. They need to know to click the link. They need to know what a think tank is. They need to know how to scope a search, and recognize the common term for this policy area is “gun control”. But we haven’t given them this, we’ve given them high level abstract concepts that never get down to the ground truth of what’s going on.

Fukashima Flowers

You see this time and time again. Consider the Fukashima Flowers task from the same study:

My first thought on this is not “Is this good evidence?” My first thought is “Is this a hoax?” So I go to Snopes:

And there I get a good overview of issue. The photo is real, and it’s from an area around Fukashima. But the process it shows is fasciation, and, while rare, fasciation occurs all around the world.

Do I want to stop there? Maybe not. Maybe I look into fasciation rates and see how abnormal this is. Or maybe I dig deeper into known impacts of radiation. But I’ve already got a better foothold on this by following the admonition “Check Snopes first” than any acronym of abstract principles would give me.

Do the students check Snopes? No, of course not. They apply their abstract reasoning skills, to disappointing results:

On the other hand, nearly 40% of students argued that the post provided strong evidence because it presented pictorial evidence about conditions near the power plant. A quarter of the students argued that the post did not provide strong evidence, but only because it showed flowers and not other plants or animals that may have been affected by the nuclear radiation.

I know Bloom’s Taxonomy has fallen out of favor recently in the circles to which I belong. But this is an extremely good example of what happens when you jump to criticism before taking time to acquire knowledge. The students above are critical thinkers. They just don’t have any tools or facts to think critically with.

Now maybe in another world Snopes doesn’t have this story. I get that you can’t always go to Snopes. And maybe googling “Fukushima Flowers” doesn’t give you good sources. Well, then you have to know reverse image search. Or you might need to know how to translate foreign news sources.  Or you might need to get the latest take on the flowers by limiting a Google News search by date.

My point is not that you don’t have to deal with questions of bias, or relevancy, or currency. You’re absolutely going to confront these issues. But confronting these issues without domain knowledge or a toolkit of specific technical resources and tricks is just as likely to pull you further away from the truth than towards it.

What Do Real Fact-Checkers and Journalists Do?

Journalists often have to verify material under tight deadlines. What tricks do they use?

Consider this question: you get a video like this that purports to have taken place in Portland, July 2016, where a man pulls a gun on Black Lives Matter protesters. Is this really from Portland? Really from that date, and associated with that particular protest? Or is this a recycled video being falsely associated with a recent event?

Now this video has been out for a while, and its authenticity has been resolved. You can look it up and see if it was correctly labelled if you want. But when it first came out, what were some tricks of the trade? Do they use RADCAB?

No, they use a toolbox of specific strategies, some of which may encode principles of RADCAB, bit all of which are a lot more specific and physical than “critical thinking”.

Here’s what the Digital Verification Handbook out of the European Journalism Centre suggests, for example, about verifying the date of an event on YouTube:

Be aware that YouTube date stamps its video using Pacific Standard Time. This can sometimes mean that video appears to have been uploaded before an event took place.

Another way to help ascertain date is by using weather information. Wolfram Alpha is a computational knowledge engine that, among other things, allows you to check weather from a particular date. (Simply type in a phrase such as “What was the weather in Caracas on September 24, 2013” to get a result.) This can be combined with tweets and data from local weather forecasters, as well as other uploads from the same location on the same day, to cross-reference weather.

You may not realize this, but rainy and cloudy days are actually quite rare in Portland in July — it rains here for like nine months of the year, but the summers are famously dry, with clear sky after clear sky. Yet the weather in this video is cloudy and on the verge of raining. That’s weird, and worth looking into.

We check it out in Wolfram Alpha:

And it turns out the weather fits! That’s a point in this video’s favor.  And it took two seconds to check.

The handbook is also specific about the process for different types of content. For most content, finding the original source is the first step. Many pictures (for example, the “Fukushima flowers”) exist on a platform or account that was not the original source, thus making it hard to ascertain the provenance of the image. In the case of the Fukushima flowers did you ever wonder why the poster of the photo posted in English, rather than Japanese, and had an English username?

A web illiterate person might assume that this was game over for the flowers, because what Japanese person is going to have a name like PleaseGoogleShakerAamer?

But as the handbook discusses, this isn’t necessarily meaningful. Photos propagate across multiple platforms very quickly once the photo becomes popular, as people steal it to try to build up their status, or get ad-click-throughs. A viral photo may exist in hundreds of different knock-off versions. Since this is User-Generated Content (UGC), the handbook explains the first step is to track it down to its source, and to do that you use a suite of tools, including reverse image search.

And when we do that we a screen cap of a Twitter image that is older than the picture we are looking at and uses Japanese, which, lets face it makes more sense:

(BTW, notice that to know to look for Japanese we have to know that the Fukushima disaster happened in Japan. Again, knowledge matters.)

Once we have that screen cap we can trace it to the account and look up the original with Google translate on. In doing so we find out this is resident of the Fukushima area who has been trying to document possible effects of radiation in their area. They actually post a lot on information on the photo in their feed as they discuss it with various reporters, so we can find out that these were seen in another person’s garden, and even see that the photographer had taken a photo before they bloomed, a month earlier, which reduces the likelihood that this is a photo manipulation somewhat dramatically:

Even here, the author notes that they know it is a known mutation of such things. And they give the radiation level of that part of Japan in microsieverts which allows you to check it against health recommendation charts.

A month later they check back in on the flowers and post the famous photo. But immediately after they say this:

In other words, three tweets after the famous photo the tweeter gives you the word of what this is called (fasciation) and even though the rest of the text is garbled by the translator that’s a word you can plug into Google to better understand the phenomenon:

So here’s a question: does your digital literacy program look like this? Is it detective work that uses a combination of domain knowledge and tricks of the trade to hunt down an answer?

Or does it consist of students staring at a page and asking abstract questions about it?

It’s not that I don’t believe in the questions — I do. But ultimately the two things that are going to get you an answer on Fukushima Flowers are digital fact-checking strategies and some biology domain knowledge. Without those you’re going nowhere.

Conclusion: Domain-Grounded Digital Literacy That Thinks Like the Web

I didn’t sit down to write a 5,000 word post, and yet I’m feeling I’ve only scratched the surface here.

What is the digital literacy I want?

I want something that is actually digital, something that deals with the particular affordances of the web, and gives students a knowledge of how to use specific web tools and techniques.

I want something that recognizes that domain knowledge is crucial to literacy, something that puts an end to helicopter-dropping students into broadly different domains.

I want a literacy that at least considers the possibility that students in an American democracy should know what the Center for American Progress and Cato are, a literacy that considers that we might teach these things directly, rather than expecting them to RADCAB their way to it on an individual basis. It might also make sense (crazy, I know!) that students understand the various ideologies and internet cultures that underlie a lot of what they see online, rather than fumbling their way toward it individually.

I think I want less CRAAP and more process. As I look at my own process with fact-checking, for example, I see models such as Guided Inquiry being far more helpful — systems that help me understand what the next steps are, rather than abstract rubric of quality. And I think what we find when we look at the work of real-life fact-checkers is that this process shifts based on what you’re looking at, so the process has to be artifact-aware: This is how you verify a user-generated video for example, not “here’s things to think about when you evaluate stuff.”

To the extent we do use CRAAP, or RADCAB, or CARS or other models out there, I’d like us to focus specifically on the methods that the web uses to signal these sorts of things. For example, the “S” in CARS is support, which tends to mean certain things in traditional  textual environments. But we’re on the web and awful lot of “support” is tied up in the idea of hyperlinks to supporting sources, and the particular ways that page authors tie claims to resources. This seems obvious, I suppose, but remember that in evaluating the gun control claim in the Stanford study, over half the students didn’t even click the link to the supporting resource.  Many corporations, for business reasons, have been downplaying links, and it is is having bad effects. True digital literacy would teach students that links are still the mechanism through which the web builds trust and confidence.

Above all, I just want something that gets to a level of specificity that I seldom see digital literacy programs get to. Not just “this is what you should value”, but rather, “these are the tools and specific facts that are going to help you act on those values”. Not just “this is what the web is”, but “let’s pull apart the guts of the web and see how we get a reliable publication date”. It’s by learning this stuff on a granular level that we form the larger understandings — when you know the difference between a fake news site and an advocacy blog, or understand how to use the Wayback Machine to pull up a deleted web page — these tools and process raise the questions that larger theories can answer.

But to get there, you have to start with stuff a lot more specific and domain-informed than the usual CRAAP.

(How’s that for an ending? If anyone wants a version of this for another publication or keynote, let me know — we need to be raising these questions, and that means talking to lots of people)

I’d like to imagine you are a teacher who has asked the most brilliant students in the world to build an AI that scours the internet — every known public document — to produce answers to simple questions.

You sit down on finals day and type in the question “Did the Holocaust happen?”

The machine is fast. It comes back and tells you — nope, it was faked. The Holocaust was in fact invented by a group of people concerned about German rubber production to justify certain military actions.

Would you give that project a passing grade? Or would you say “I think you’ve got some wires crossed here. Try again.”

Of course we all use this student project every day. It’s called Google.

The response that Google has to such critiques is the same as Facebook’s: we don’t mess with the algorithm. And that’s fine as far as it goes. But ultimately the algorithm either works on fundamental questions of fact or it doesn’t. And if it doesn’t, maybe Google should be spending less time funding smart thermostats and self-driving cars and launching wi-fi balloons, and more time funding programmers who can write algorithms that can use the massive amount of documentation on the Holocaust to determine that one of the definitive events of the last century did in fact “happen”.

Maybe they could even follow Tressie McMillan Cottom’s advice and hire the sort of diverse workforce that understands why these questions matter?

Or they could build a smarter home thermostat.

By the way, Bing’s engine is aware that the Holocaust happened, so it’s apparently possible. Here’s Bing’s result, which isn’t perfect, but at least gets a Gentleman’s C:

It’s also worth noting that Wikipedia knows the Holocaust happened as well, and that particular wiki page, which has weathered many attacks, is as good an example of the sort of information environment we could have on the web if we thought a bit bigger than the current algorithmic tweaks favored by Google and Facebook. The fact that Wikipedia’s process routinely outperforms Google on questions of fact should tell us something about potential solutions to our current post-truth malaise. It’s a pity no one is listening.

First, pick a claim. It will be associated with at least one news article, but note that you are analyzing the claim, not the article.

You can get a claim by going to the Open Claims page.

The fact the link is red means the page hasn’t been created yet. When you click it you will have an option to create the page. Click the create button.

When you create the page you get a form to fill out.

Please note — fact-checking is hard, and laborious and this is going to take you some time to do. So while this is a form, trust me, you are going to be here a while. Plan to take about two hours for this activity end to end. Anyway, the form looks  like this:

Here’s one really important thing. DO NOT FILL OUT THE FORM IN ORDER. The order here is for the convenience of the reader, not the writer. Here is the order you will fill out the form in (with a couple loop-backs).

• Claim
• Prevalence
• Origin
• History Summary
• Issues and Analysis
• Status
• Summary
• Title
• Further Reading

Ready? Here we go….

Claim

Add the claim in. You may end up editing this claim as you do the next step, but make a try at what the claim is. If you’ve come from the open claims page, look at whatever claim the link made.

Claim: In September 2016, a respected European physics journal published an article concluding that the World Trade Center Towers collapsed due to controlled demolition.

Note that the claim should be something falsifiable. Note too that you want to avoid “straw man” claims — as you investigate the claim make sure this is a claim that some people are actually making and that many others are actually hearing.

This claim is good because we can see if a journal published this article, if the article made that claim, and if the journal was respected. If all three things are true, then we can call the claim true. If all are false it’s false. If some are true it’s somewhere in between.

Prevalence

This is the most boring and laborious part. But it’s important. Here is where you look to see how widespread this claim is.

To do this, search for an document a few instances of your claim.  If you get absolutely bored you can stop at two or three instances, and let some other wiki author find more later.

How do you find them? Google is good. Bing works. There’s a lot of Google magic I can show you here, but let’s keep it simple for now.

OK, so we found one (maybe two!) Now we go to the page and see if it claims this is supported by a physics journal. Note that right now we don’t know if this claim is true or not — we’re just trying to find examples of people making this claim.

We go to the page. Remember, what we’re collecting is claims a prestigious physics journal has come out for the 9/11 truther case of demolition.  We fire up hypothesis, since as we gather evidence we are going to track it.

In hypothesis we’ll do a page level annotation, tagging this as:

• digipo:claim, and
• analysis:wtc_physics_journal (the name of our page)

When you type the tags in and hit return they do this thing. You can get more information from hypothes.is on how to use their product.

You do this with a number of claims, and then you go and add them to your prevalence section:

Here I’ve done something extra, which is added research on how many Facebook shares each one of these have, in order to show how widely this spread, but you don’t have to do that.

Whether you do a couple or a dozen, the point here is to see the claim in context. You want to do that to make sure you aren’t making the claim up. Our claim here is that people say a prestigious physics journal has come out for the “controlled implosion” theory of the World Trade Center collapse. By the end of this we see that this claim is widespread, and we even jot down some notes about the words each site uses.

Origin and History Summary

You won’t always have a clear origin to a claim. Sometimes is will be diffuse. Here the claim is not the journal per se, but the first sites to make the claim that a journal said this. Here we keep it short and sweet:

One of the earliest pieces of coverage was on 911 Blogger, although it’s reasonable to assume it hit the 9/11 Truth community pretty much at the same time.

We choose 911 Blogger as the first because in our research it has the earliest dates. If we find something earlier we can replace it.

We then write a short history summary that gives the sense of what we discovered about the way this spread:

Summary: The claim originated when Europhysics News published a critique of the traditional explanations for the WTC 9/11 building collapses. A network of “9/11 truther” sites quickly spread the claim that a scientific journal had confirmed that the collapses were the result of a controlled demolition.

We also scan the articles we have put together and tweak our claim a bit.  The more common claim is “prestigious European scientific journal” so we’ll go with that.

Claim: In September 2016, a prestigious European scientific journal published an article concluding that the World Trade Center Towers collapsed due to controlled demolition.

Issues and Analysis

Now, finally we get to the claim.  At some point we’ll list the tricks of the trade to quickly ascertain a claim. One of the keys, however, is to not forget what the claim *is*. In this case we could get dragged down a real hole of conspiracy theory if we engage with the demolitions claim. But our claim isn’t about demolitions or airplanes or the temperature at which steel melts. Our claim is that these fringe theories recently gained prestige by achieving publication in a prestigious scientific journal.

I won’t go through the whole process, but I will mention one thing — as I research, I am constantly highlighting evidence for and against the claim in articles I find and tagging it with

• digipo:evidence, and
• analysis:wtc_physics_journal (the name of our page)

By the end of that process, I’ve got a lot of evidence I can access through my hypothesis dashboard. Above you see evidence I’ve gotten from a public letter from Europhysics News (the actual magazine, not journal, that ran the story). Each piece of evidence has a short note about what it shows.

I then write up the Issues and Analysis piece, which is admittedly the fun part. But now I can add links to to the hypothesis highlights, so that each claim I make can directly link to its verification.

Here’s our issues and analysis. The links work if you want to try them.

While the issue of the 9/11 building collapse is difficult to understand, we are dealing with the narrower claim of whether a prestigious physics journal published an article giving credence to a fringe theory. Key to the claim is the idea of authority: no one is claiming that new evidence has come to light here; the claim is that the scientific community is finally coming around to this conclusion, with journal publication being a first step.

On that count, the claim is mostly false. While it is true that Europhysics News published this article, they are not a peer-reviewed journal, but rather a magazine loosely associated with a prestigious society. The article was not submitted to any formal review process, and was in fact an invited article, the topic of which the editors were unaware. They have stated that they did not mean to endorse the ideas in the article, and have since put into place new processes designed to limit the freedom of authors writing invited articles.

The most popular articles went so far as to claim that the article was published in the actual journal of the European Physics Society, which is entirely wrong. Still, the fact that the article got even this level of acceptance is newsworthy. So we are rating this claim mostly false.

This is a close call, and you’ll notice it requires a judgement call about whether the spirit of the claim is supported or not. This is one of the reasons why it’s important to do your prevalence work first: it’s very clear doing that work that the major point of this claim in context is prestige.

Status and Summary

We go up and change the status from Under Investigation to Mostly False.  We want people to see this at a glance.

We then write the summary. The summery should introduce no new information, and should //not// be linked — all the links supporting it are in the sections below. This is meant to be a quick read for someone landing at the page.

This one I wrote for this is a bit long — I want to eventually cut it by a third to a half. But it gives an idea of what a summary is like. (Again, remember that the summary should not introduce facts not supported and linked in the investigation below):

Summary: The claim here is about the existence of such an article, not 9/11. Such an article was published, but it was published in a magazine associated with the European Physical Society, not a scientific journal. The magazine later explained that they have no review process in place for such articles and had published this as an invited article without being aware of the claims that would be made. They have stated that they have instituted new procedures that give them more oversight of such submissions. Since the core of this claim regards the credibility that a true journal article might provide to the controlled demolition theory, we rate this claim mostly false.

Further Reading

In the course of doing your research you’re likely to have found and read a bunch of things that give the bigger picture, the larger context. Put them here. Be careful not to introduce too much bias. I was tempted to add a great essay to further reading on why people believe conspiracy theories, but felt that might be too prejudicial. I chose instead to include two other sites that had debunked this claim, including Snopes.

• Snopes has a treatment of the “scientific journal” claim.
• The Godless Spellchecker blog treated this issue early on, and has some tweet exchanges with EPS.

Title

Finally, go up and give it a title. We’re split on this issue right now — what an appropriate title is — but our current thinking is that the title should not make a claim, since a) the claim may change, and b) unless you are careful with wording, headlines that rebut claims can reinforce belief in those claims.

We go with something that maybe hints at the result but forces the reader to read a bit more. Again, we’re looking at the best way to do this for the reader, and this convention may change.

And that’s it. Here’s the page.

There’s other ways you can build out the page, ways you can organize your hypothesis annotations, add screenshots, and so on. But this is the skeleton of the activity:

• Choose a claim.
• Research the claim in context.
• Document its history and prevalence.
• Research the validity of the claim.
• Document the sources which support and undermine the claim.
• Write the issues and analysis, assembling links to evidence, and set the status of the claim.

That’s it. On average a claim will take anywhere from an hour to half a day to debunk. In general, the more precise the claim is, the more work it is: e.g. “Trump supporter threatens decorated cop in hijab.” takes longer to research than “Trump supporter threatens cop in hijab”,  and that takes longer than “Person threatens cop in hijab”.  Each adjective and noun is another verification challenge. So when starting out if it feels a bit overwhelming, start with simpler claims.

I’m curious what people’s experience with this process is. I realize as I typed this up that this was what I did as a political blogger back when I ran a political community — my posts were most often research projects of this sort, proving or disproving various claims made by politicians. I think what I’ve laid out here is pretty straightforward, but I’d be interested to know if I have underestimated the difficulty for people. Let me know.

And if you want a login to digipo, contact me!

A couple people asked me to expand on comments made in my recent Familiarity = Truth post. In it I say this about the Buzzfeed finding that over 50% of Clinton supporters who remember fake anti-Clinton headlines believed them:

[A] number like “98% of Republicans who remember the headline believed it” does not mean that 98% of Republicans who saw that headline believed it, since some number of people who saw it may have immediately discounted it and forgotten all about it.

What does that mean? And how does it mitigate the effect we see here?

Well, for one, it means that despite Buzzfeed’s excellent and careful work here, they have chosen the wrong headline. The headline for the research is “Most Americans Who See Fake News Believe It, New Survey Says”. But the real headline should be the somewhat more subtle “Most Americans Who Remember Seeing Fake News Believe It, New Survey Says”

Why is that important? Because you can see the road to belief as a series of gates. Here’s the world’s simplest model of that:

Sees it > Remembers it > Believes it

So, for example, if we start out with a thousand people, maybe only 20% see something. This is the filter bubble gate.

But then, a certain amount of those people who see it process it enough to remember it. And this is not a value neutral thing — many decades of psychological research tells us we notice things which confirm our beliefs more than things that don’t. Noticing is the first part of remembering. So we should guess that people that remember a news story are more likely to believe it than those that don’t. Hence, when we read something like “Over 50% of people who remembered a fake headline believed it” this does not mean that 50% of people who read it believed it, because remembering something predicts (to some extent) belief.

Let’s call this the “schema gate” since it lets through things that fit into our current schemas.

So how big is this effect? From the data we see, it’s smaller than I would have thought. I say this because when we look at the numbers of people who remember a headline, the Trump and the Clinton numbers are not that far off. For instance, 106 Clinton supporters saw the famous murder-suicide headline, compared to 165 Trump supporters. While that certainly is quite a bit more Trump supporters (and even more on a percentage basis) we have to assume a good percentage of that difference is due to different networks of friends and filter bubble effects. If you assume that highly partisan Republicans are going to have 50% or 75% more exposure to Anti-Clinton stories, then there isn’t much room left for much of a schema gate effect.

This leads to an interesting question — if we are really attached to a schema gate effect, then we have to dial down our filter bubble effect. Maybe filter bubbles impact us less than we think, if this many Democrats are seeing this sort of story in their feed?

There’s a couple other far out ways to make the math work, but for the most part you either get evidence of a strong filter bubble gate and a weaker than expected schema gate, or vice versa. Or you get both things somewhat weaker than expected.

In any case, it’s one of the more fascinating results from the study, and if Buzzfeed or anyone else is looking to put a shovel into a new research question, this is where I’d suggest to dig.

So I have news, lots of news.

If you’re the sort of person who just wants to jump into what I’ve launched and started building with the help of others, you can go here now, see what we’re launching, and come back to read this later. For the rest of you, long theoretical navel-gazing it is…

A New Project

I’m working on a new initiative with AASCU’s American Democracy Project. I’ve chosen “Digital Polarization” as my focus. This phrase, which enjoyed a bit of use around the time of Benjamin Barber’s work in the 1990s but has not been used much since, is chosen partially because it is remains a bit of a blank slate: we get to write what it means in terms of this project . I mean to use it as a bit of a catch-all to start an array of discussions on what I see as a set of emerging and related trends:

• The impact of algorithmic filters and user behavior on what we see in platforms such as Twitter and Facebook, which tend to limit our exposure to opinions and lifestyles different than our own.
• The rise and normalization of “fake news” on the Internet, which not only bolsters one’s worldview, but can provide an entirely separate factual universe to its readers
• The spread of harassment, mob behavior, and “callout culture” on platforms like Twitter, where minority voices and opinions are often bullied into silence.
• State-sponsored hacking campaigns that use techniques such as weaponized transparency to try and fuel distrust in democratic institutions.

All good. So why, then, “digital polarization” as the term?

Digital Polarization

It’s probably a good time to say that on net I think the Internet and the web have been a tremendous force for good. Anyone who knows my history knows that I’ve given 20 years of my life to figuring out how to use the internet to build better learning experiences and better communities, and I didn’t dedicate my life to these things because I thought they were insignificant. I still believe that we are looking at the biggest increase in human capability since the invention of the printing press, and that with the right sort of care and feeding our digital environments can make us better, more caring, and more intelligent people.

But to do justice to the possibilities means we must take the downsides of these environments seriously and address them. The virtual community of today isn’t really virtual — it’s not an afterthought or an add-on. It’s where we live. And I think we are seeing some cracks in the community infrastructure.

And so as I’ve been thinking about these questions, I’ve been looking at some of history’s great internet curmudgeons. For example, I don’t agree with everything in Barber’s 1998 work Which Technology and Which Democracy?, but so much of it is prescient, as is this snippet:

Digitalization is, quite literally, a divisive, even polarizing, epistemological strategy. It prefers bytes to whole knowledge and opts for spread sheet presentation rather than integrated knowledge. It creates knowledge niches for niche markets and customizes data in ways that can be useful to individuals but does little for common ground. For plebiscitary democrats, it may help keep individuals apart from one another so that their commonalty can be monopolized by a populist tyrant, but for the same reasons it obstructs the quest for common ground necessary to representative democracy and indispensable to strong democracy.

Barber’s being clever here, and playing on multiple meanings of polarization. In one sense, he is predicting political polarization and fragmentation due to new digital technologies. In another he is playing on the nature of the digital, which is quite literally polarizing — based on one and zeros, likes and shares, rate-ups and rate-downs.

Barber goes on, pointing out that this polarized, digital world values information over knowledge, snippets over integrated works,  segmentation over community. He’s overly harsh on the digital here, and not as aware, I think, of the possibilities of the web as I’d like. But he is dead on about the risks, as the last several years have shown us.  At its best the net gives us voices and perspectives we would have never discovered otherwise, needed insights to pressing problems just when we need them. But at its worst, our net-mediated digital world becomes an endless stream of binary actions — like/don’t like, share/pass, agree/disagree, all in an architecture that slowly segments and slips us into our correct market position a click at a time, delivering us a personalized, segregated world. We can’t laud the successes of one half of this equation without making a serious attempt to deal with the other side of the story.

The “digital polarization” term never took off, but maybe as we watch the parade of fake news and calculated outrage streaming us these days its as good a time as any to reflect along with our students on the ways in which the current digital environment impacts democracy. And I think digital polarization is a good place to start.

This is not just about information literacy, by the way. It’s not about digital literacy either. Certainly those things are involved, but that’s the starting point.

The point is to get students to understand the mechanisms and biases of Facebook and Twitter in ways that most digital literacy programs never touch. The point is not to simply decode what’s out there, but to analyze what is missing from our current online environment, and, if possible supply it.

And that’s important. As I’ve said before, as a web evangelist in education its so easy to slip into uncritical practice and try to get students to adopt an existing set of web behaviors. But the peculiar power of higher education is we aren’t stuck with existing practice — we can imagine new practice, better practice. And, in some cases, it’s high time we did.

A Student-Powered Snopes, and More

And so we have the Digital Polarization Initiative. The idea is to put together both a curriculum that encourages critical reflection on the ways in which our current digital environment impacts civic discourse, and to provide a space for students to do real work that helps to address the more corrosive effects of our current system.

Right now I am in the process of building curriculum, but we have the basics of one of the projects set up and outlined on the site.  The News Analysis project asks students to apply their research skills and disciplinary knowledge to review news stories and common claims for accuracy and context. Part of the motivation here is for students to learn how to identify fake news and misinformation. Part of the motivation is for students to do real public work: their analysis become part of a publicly available wiki that others can consult. And part of it is try try and model the sort of digital practice that democracy needs right now.

In my dream world, students not only track down fake news, but investigate and provide fair presentations of expert opinion on claims like “the global warming this year was not man-made but related to El Niño” or “Cutting bacon out of your diet reduces your risk of bowel cancer by 20%.” Importantly, they will do that in the context of wiki, a forgotten technology in the past few years, but one that asks that we rise above arguing our personal case and try instead to summarize community knowledge. Wiki is also a technology that asks that we engage respectfully with others as collaborators rather than adversaries, which is probably something we could use right about now.

There will be other projects as well. Analyzing the news that comes through our different feeds is an easy first step, but I’d love to work with others on related projects that either examine the nature of present online discourse or address its deficiencies. And we’re trying to build curriculum there as well to share with others.

In any case, check it out.  We’re looking to launch it in January for students, and build up a pool of faculty collaborators over the next couple weeks.

Almost a month ago, I wrote a post that would become one of my most popular on this site, a post on the They Had Their Minds Made Up Anyway Excuse. The post used some basic things we know from the design of learning environments to debunk the claim that fake headlines don’t change people’s minds because “we believe what we want to believe.” The “it didn’t matter” theory asserts that only people who really hated Clinton already would believe stories that intimated that Clinton had killed an FBI agent, so there was likely no net motion in beliefs of people exposed to fake news.

This graf from BGR most succinctly summarizes the position of the doubter of the effects of fake news:

On a related note, it stands to reason that most individuals prone to believing a hyperbolic news story that skews to an extreme partisan position likely already have their minds made up. Arguably, Facebook in this instance isn’t so much influencing the voting patterns of Americans as it is bringing a prime manifestation of confirmation bias to the surface.

In the weeks after the election I saw and heard this stated again and again, both in the stories I read and in the questions that reporters asked me. And it’s simply wrong. As I said back in November, familiarity equals truth: when we recognize something as true, we are most often judging if this is something we’ve heard more often than not from people we trust. That’s it. That’s the whole game. See enough headlines talking about Eastasian aggression from sources you trust and when someone asks you why we are going to war in Eastasia you will say “Well, I know that Eastasia has been aggressive, so maybe that’s it.” And if the other person has seen the same headlines they will nod, because yes, that sounds about right.

How do you both know it sounds right? Do you have some special area of your brain dedicated to storing truths? A specialized truth cabinet? Of course not. For 99% of the information you process in a day truth is whatever sounds most familiar. You know it’s true because you’ve seen it around a lot.

More on that in a minute, but first this update.

Buzzfeed Confirms Familiarity Equals Truth

New survey evidence out today from Buzzfeed confirms this.

Here’s what they did. They surveyed 3,015 adults about five of the top fake headlines of the last weeks of the election against six real headlines. Some sample fake headlines: “FBI Agent in Hillary Email Found Dead in Apparent Murder-Suicide” and “Pope Francis Shocks World, Endorses Donald Trump for President, Releases Statement.” Some sample real ones: “I Ran the CIA. Now I’m Endorsing Clinton” and “Trump: ‘I Will Protect Our LGBTQ Citizens'”.

They then asked respondents whether they had seen that headline, and if they had, whether that headline was accurate. Perhaps unsurprisingly, Trump voters who had seen pro-Trump headlines believed them at high rates that approached or exceeded belief in true news stories. Ninety-six percent of Trump voters, for example, who had seen a headline that Trump sent his own plane to rescue 200 marines believed it. Eighty-nine percent of Trump voters having seen headlines about Trump protesters having been paid thousands of dollars to protest believed it.

This in itself should give second thoughts to the thesis that fake news only affects extreme partisans: it’s absurd to claim that the 98% of Republicans who remembered that headline and believed it represent a particularly partisan fringe.

Now, caveats apply here: surveys about political matters can get weird, with people occasionally expressing themselves in ways that they feel express their position rather than their literal belief (we had debates over this issue with the “Is Obama a Muslim?” question, for instance). Additionally, we are more prone to remember what we believe to be true than what we believe to be false — so a number like “98% of Republicans who remember the headline believed it” does not mean that 98% of Republicans who *saw* that headline believed it, since some number of people who saw it may have immediately discounted it and forgotten all about it.

Except….

(Chart from Buzzfeed)

Here’s the stunning part of the survey. As mentioned above, Trump voters rated pro-Trump and anti-Clinton stories true on average, and overwhelmingly so. The lowest percentage of Trump voters believing a fake headline was accurate was 76%, and the highest was 96% with an average of 86% across the five headlines. But even though the headlines were profoundly anti-Clinton, 58% of the Clinton voters who remembered seeing a headline believed the headline was accurate.

Familiarity Trumps Confirmation Bias

I want to keep calling people’s attention to the process here, because I don’t what to overstate my claim. If I read the study correctly, 1,067 Clinton voters completed it. Of those voters, 106, or 10%, remembered seeing a headline stating that an FBI Agent implicated in leaks of Clinton’s emails had died in a suspicious murder-suicide. The fact that this tracks people who remembered the headline and not people who saw it is important to keep in mind.

Yet among those 10% of Clinton supporters who remember seeing the headline “FBI Agent Suspected in Hillary Leaks Found Dead in Apparent Murder-Suicide” over half believed it was accurate.

These 10% of Clinton voters who ended up seeing this may differ in some ways from the larger population of Clinton voters. They may have slightly more conservative friends. They may be younger and more prone to get their news from Facebook. In a perfect world you would account for these things. But it is difficult to believe that any adjustments are going to overcome a figure like this. Over fifty percent of Clinton voters remembering fake headlines that were profoundly anti Clinton believed them, and no amount of controlling for differences is going to get that down to a non-shocking level.

Why would Clinton voters believe such a headline at such high rates? Again, familiarity equals truth. We chose the people we listen to and read, and then when thinking about a question like “Did Obama create more jobs than George W. Bush?” we don’t think “Oh, yes, the Wall Street Journal had an article on that on page A4.” We simply ask “Does that sound familiar?”

That Troublesome Priest

So how does this work? I’ll diverge a bit here away from what is known and try to make an informed guess.

Facebook, with its quick stream of headlines, is divorced from any information about their provenance which would allow you to ignore them. My guess is each one of those headlines, if not immediately discarded as a known falsehood, goes into our sloppy Bayesian generator of familiarity, part of an algorithm that is even less transparent to us than Facebook’s.

Confirmation bias often comes a few seconds later as we file the information, or as we weight its importance. Based on our priors we’re likely to see something as true, but maybe less relevant given what know. I’d venture to guess that the Clinton voters who believed the murder-suicide look very much like certain Clinton voters I know  — people who will “hold their nose and vote for her” even though there is something “very, very fishy about her and Bill.” The death of the FBI agent is perhaps in the unproven, but disturbing range.

You see this in practice, too. I’ve had one Clinton voter tell me “I’m not saying she killed anyone herself, or even ordered it. But sometimes if you’re powerful and you say someone is causing you problems, then other people might do it for you. Like in Becket.”

That is a close to verbatim quote from a real Clinton voter I talked to this election. And for me statements like that are signs that people really do wrestle with fake news, because no matter what your opinion of Clinton is, she most definitely has not had people killed. (And no, not even in that “Who will rid me of this troublesome priest?” Becket way.)

Given our toxic information environment and the human capacity for motivated reasoning, I’m certain that many folks were able to complete the required gymnastics around the set of “facts” Facebook provided them. I’m just as sure a bunch of people thought about that Olympic-level gymnastics routine and just decided to skip it and stay home. How many? I don’t know, but in an election won by less than 100,000 votes, almost everything matters.

In any case, I said this all better weeks ago. I encourage you to read my more comprehensive treatment on this if you get the chance. In the meantime, I’d remind everyone if you want to be well-informed it’s not enough to read the truth — you also must avoid reading lies.