JSON-Based Transclusion, and WordPress as the Universal Reader

Dave Winer wrote a recent post on, roughly, how to reboot the Blogosphere with JSON. I read it last night and thought I understood it, then read it again this morning and realized I’d missed the core idea of what he was saying. Here’s the relevant graf:

But there is another approach, to have WordPress accept as input, the URL of a JSON file containing the source code for a post, and then do its rendering exactly as if that post had been written using WordPress’s editor. That would give us exactly what we need to have the best of all worlds. Widespread syndication and control over our own writing and archive. All at a very low cost in terms of storage and CPU.

Maybe I was just a bit tired last night but it’s worth staying on how this is different from other models. The idea here is that your data doesn’t have to be stored in WordPress at all. Dave Winer can make his editor, Ward Cunningham can make federated wiki — but should they want to publish to WordPress site —

See, that’s not quite it either. Because the idea here as I read it is pull, not push. (Dave, correct me if I misunderstand here). The idea is, given certain permissions and compliance with a WordPress API, a WordPress site I have can go out and fetch Dave or Ward’s content and render it up for me in my own blog dynamically.

I’m not sure Dave is going this far, but imagine this as a scenario — I link to Dave on my WordPress blog, but the link makes a double pass.

First, it sees if it can just grab the JSON and render Dave’s post in my WordPress blog. If it can, great. It renders Dave’s page with the links. Links I click on there also attempt to render in my default WordPress environment.

Sometimes links won’t return nice requests for JSON. Those ask me if I want to go outside my reading environment to someone else’s site. If history is any guide, these sites don’t get much traffic, because the answer to that question is often no.

Links that render into your environment could be acted on by your theme functions. Maybe you take a snapshot of something, repost it, index it, annotate it. Over time, there is a market for themes that play nice with other peoples content, or allow people to make the best sense of it.

And of course if you add in feeds….

What this does is move from a situation where we have a couple online RSS readers to a world where every WordPress theme (and there are tensof thousands of WordPress themes)  is potentially an RSS reader of sorts. It moves from a world where every theme is potentially a new Facebook or Twitter as well.

It does this because it solves part of the problem Facebook solved for people — it lets us read other people’s stuff in a clean, stable environment that we control. (There are other things as well, but you have to start somewhere).

So why not try this? Turn themes — the killer feature of WordPress — into a way to read other people’s content, and see what happens. WordPress has already made a stab at being the universal publisher, but it could be the Universal Reader as well, not through providing a single UI, but by supplying an endlessly customizable one.


Capable Clients and the WordPress API

Update: If you read the comments below you’ll see one of the API developers has responded; there are some issues with private information in short codes being exposed.

I still wish all the smart-quotification, m-dashing, and paragraphing could be more easily disabled, but I’m very grateful for the quick and thoughtful response.


I wasted another afternoon looking for a way that I won’t have to write my own custom WordPress JSON API. After all, the WordPress REST API folks have spent many, many hours producing a core API, and writing my own just seems silly.

But it looks like I will have to write my own, and I thought I’d explain why. Maybe the WordPress API folks will read this and understand a whole set of use cases they have missed.

Here’s the deal. I store Markdown in my WordPress database. Not the processed stuff. Not Markdown that turns to HTML on save. Markdown.

I do this because I believe in what I call “capable clients”. I want anyone else to be able to make their own client and represent my data in the best way they see fit.  I want people to be able to fork my source text. I want people to be able to remix my stuff with stuff from other servers.

The same is true about the short codes that go in for things like images. I want my reader’s client to get the source and render it, or move it to their server to render.

I don’t think this is such a weird thing. This is, after all, how the web worked in 1991, before graphic designers and relational database designers mucked the whole thing up with their magazine designs and fifth normal forms.  Back in 1991, if you saved a file from Tim Berners-Lee’s NeXT server to your laptop, you had an exact copy of what Tim’s server had. You had the source. It was a level playing field.

APIs + JSON gives us a chance to return to that world, a world of capable clients. A world where clients of my platform can potentially build a better view of my data than I had imagined.  A world where I let you fork raw data from my server and reuse it on yours, git-style. A world where people can remix data from multiple servers in new and illuminating ways, the way they did in the early days of RSS.

That’s the real promise of JSON development — the permissionless innovation that characterized the early web, but brought up to date.

So why would you only allow raw content — the unprocessed HTML or Markdown source — to be accessed by people who are logged in as editors?

Is what I typed originally into the text field of the WordPress editor such a secret? I suppose it could be, but why not give people the option to see exactly what I entered into the editor? Why fix it with paragraph tags and smart quotes, when you don’t even know if it’s HTML, or Markdown, or LaTeX stored there? Why run always run the shortcode filters, even when the client wants the data — not the useless processed output?

There’s a huge opportunity here to unleash a level of innovation we haven’t seen for years. But it starts with allowing capable clients to consume clean data, even if they only have read permissions.

In a system that truly values client-side development, clean data is a right, not a privilege. Why not give us that right?






The Future of Empowerment Is On the Client

I’m excited about Brave, the new browser coming out with privacy and content payment features built into the core browser code. I won’t detail it here, but you need to check it out.

The piece that people miss about all these debates about Facebook-ization and evil tracking and big data is that the Web we got is largely a function of the structure of browsers and protocols we inherited. As a simple example, a browser has an idea of what your IP is, but no concept of you as a user, which means you need to rely on big central servers like Facebook to supply an identity to you. (Compare email, where identity is federated, and central to the system protocols).

As another example, more pertinent to Brave, the third-party cookie hack available in browsers sprouted a culture of Surveillance as a Business Model.

I think two approaches to this mess have emerged. The first idea is since browsers cede all your power to servers (at least when you want to do something interesting) — the idea is you should own a server, because that’s where the power is.

I think the less publicized idea is to move more power back to the client. Let the browser (or the JavaScript running in the browser) make more choices on how to interact with the web, supplying the layers that never got built, the identity, commerce, and syndication gaps that companies like Google and Facebook have made a fortune filling in.

Both the “Own your own domain” approach and the “Power to the client” approach to a better web are complementary, but I actually believe that it is this second path — exemplified by projects such as Brave and Calypso — that has the best chance of broad adoption.

See also Calypso is the Future of Personal Cyberinfrastructure


Quick Edit Functionality in Wikity

If you’ve started a new site on Wikity, you’ll notice that it has a new interface. We’ve taken some cues from people who loved the index-cardiness of federated wiki and from others who urged us to embrace the “like Pinterest for text” elevator speech and JUST GO WITH IT.


These are your latest posts, with special first two cards.

The first card has your site name and description, as well as a number of functions we’ll get to in a moment. The second card, which only appears when you are logged in as an author or editor, is a “quick create” window, and it turns out to be awesome.

So let’s make a page using the quick create interface.

We’re going to make this page using Markdown. While Markdown has a but of a learning curve to it, it turns out to be easier to learn and less overwhelming than many graphical WYSIWYG interfaces. It also turns out to be a killer way to compose posts on your phone as well as make sure that the site is accessible to people who may not be able to operate a mouse or trackpad, or may have a vision impairment which makes selection of text time-consuming. In the past few weeks, I’ve come to see Markdown support in terms of universal design, and it’s a good fit.

So we make a page. Here we’re going to take some notes for a class on colonialism. We’re researching the 1931 Paris exhibition (one of the last of the “human zoos”) and we want to research and record what we can on the full-scale model of Angkor Wat which was constructed in Paris for the event.



We fill out the title and write an introductory paragraph. Then we decide we want to add a video, so we copy a YouTube URL.



And then paste it in our text box as a bare URL



Now we’ll add a blockquote, using the Markdown “angle bracket” syntax, and the asterisk syntax to italicize the title:


Want to add a picture? We use the markdown syntax with a twist — put in a Markdown image reference to an external image, and when you save the file the server will go out and fetch that image for you and upload it to your server, replacing your image URL with the new one. (Make sure you cite the author!)

Here we put an image at the top of our post:


Now we add our works cited, and related pages using the hash heading syntax (here we’ll do heading level 3):



And here’s what that page looks like — formatted, with an embedded YouTube and a cited image that is now being served from your local server:


Want to edit it after you post? Just click any card in your “card stack”, and it goes back to edit mode.

I know this may seem like it’s complex, but I’ve shown this to a few people now, and the pain of having to learn a bit of Markdown is made worth it by never having to enter into the Dashboard interface, even when doing a complicated multi-part formatted document like this.

Moreover, I am not kidding you when I say it is possible to compose a document like this using any size smartphone — there’s no modifier keys needed (alt, cntrl), no complex download and upload behaviors, no ribbon menu bars, no right click. I personally believe the laptop is the most perfect machine ever invented, but I have done edits on the fly on my phone, and even started an article or two. The ability to do this on mobile can supplement laptop use in nice ways.

Anyway, next post will be on the new feature called “paths”, which was inspired by Vannevar Bush, but can also form the basis of an OER strategy.



Wikipedia Is the One Impossible Thing

Wikipedia turned 15 today, a day where I happen to have my back against the wall on some deadlines. Turns out faculty want to nail down their edtech plan by week three of the class. Who knew?

But in any case, I didn’t want to let the anniversary go by without saying something.

The thing I’ve come to realize about Wikipedia is this: it’s the one impossible thing the Web has given us.

Name the other things you encounter on the Web on a daily basis: email, online reading, shopping, media viewing, self-paced education, online hobby communities, self-publishing. Cooperative music making.

For any of these things I can find you a prediction somewhere between 1968 and 1978 that describes these functions as an inevitable part of the future. In many cases (e.g. online education and shopping) I can show you early 1970s technologies *in use* that are not that different from what we do today.

If, at the dawn of the web, I was to take a list of things the web would bring about and show them to a researcher, they might disagree on the level of interest people would have in things (what’s with the cat pictures, spaceman?) but there’d be little there to surprise them except for one item: the most used reference work in the world will be collaboratively maintained by a group of anonymous and pseudonymous volunteers as part of a self-organizing network.

Nobody predicted that in 1971 or 1991. In fact, in 2001, the only people predicting it were a small group of people who had been using wiki.

I’m sure people will reply that I’ve missed thing X or Y, and there’s an argument to be made that “the one impossible thing” is hyperbole. But if it is, it’s not by much, and most of the things you’d mention were partially inspired by Wikipedia in any case.

It would be nice if on this day, as we marvel about the rise of Wikipedia, we could turn some of our attention to the Wikipedias of the future. Where are opportunities for this mode of collaboration that we’ve missed? Why are we not confronted by more impossible things? How can we move from the electronic dreams of the 1970s to visions informed by the lessons of wiki and Wikipedia?

Some people might think we’ve already done that. But I’m pretty sure we’re barely getting started.

The Recommender’s Paradox

A recent-ish study looked at film recommendation systems and found an interesting result: perceived novelty of the recommendations was negatively correlated with user satisfaction, and user satisfaction was correlated adoption.

One of the most striking things we found is that the novelty of recommended items has a significant negative impact on users’ perception of a recommender’s ability to satisfactorily meet their information needs. This effect was particularly strong in its impact on the user’s first impression of an algorithm, and was present even though we restricted the depth of the long tail into which our algorithms could reach.

And so you end up with what I’ll call the recommender’s paradox. People turn to recommendation systems to find things they wouldn’t have thought of themselves, but they choose recommendation systems that largely provide them choices that don’t surprise them. Aspects of trust in the recommender relationship undermine efficacy.

One approach to the paradox is to introduce novelty slowly, after trust is built, but there’s some bad news there as well:

Increasing the novelty of recommendations as the user gains more experience with the system and has had more time to consider its ability to meet their needs may provide benefit, but our results cannot confirm or deny this. The users in our study are experienced with movie recommendation in general and MovieLens in particular (the median user has rated 473 movies), and their first impressions were still heavily influenced by novelty.

One other way to think of this is that people think they want new films recommended to them but actually they really want to be reminded of films they had already been thinking about seeing but had forgotten about, or notified about familiar looking films that had slipped under their radar. My guess is the most successful engine (based on user satisfaction) would probably provide you with a wide range of films by directors and actors you know you like but had forgotten about, in genres that you know work for you.  These results would seem tailored to you, but at the same time be things you would not have thought of yourself.

Obvious implications here on Open Educational Resource recommendation algorithms, Big Data, the Filter Bubble, and the problem of resistance to novelty in education more generally.

Join Code for Wikity

We got our first piece of spam on Wikity, so we’ve rotated out the join code. It used to be “peloton”. It is now “copies”.

If you are reading this post more than a month after it was posted, you probably need to look for a more recent post on Wikity, with a new join code. We’ll rotate it as often as we have to to keep spammers out.


An Idle Thought on Differential Identity

I work a lot with teachers that don’t want to expose themselves to risk, argument, or harassment on the web. They create crazy good stuff that would be useful to other people for their classes, but the thought of sharing it on the web where they would be prone to lawsuits or complaints from wingnuts is unappealing to them or their bosses, so into the Blackboard Learning Management System it goes to die behind a password.

What sort of things are they afraid of? Well, maybe they are afraid that the readings of their class on gender identity will get picked up by wingnut media, and their legislators will be pushing for them to be fired. Maybe they are afraid that the page they wrote on the economics of medical care, in which they use a graphic under fair use, will get hit by a copyright suit. Maybe they are afraid that some unfriendly pedant out there will see hastily composed lecture notes and scoff at a mistake they make in presenting kin selection algorithms.

All these things have low probability overall, but present enough risk that people won’t share. As open educators we’ve fought, I think, a prolonged battle to cut through the FUD and get people to embrace the small level of risk.

I’ve lately been wondering if there are other solutions. Almost all the risks come from people you don’t know encountering your stuff and making your life miserable. So the solution has been to cut off access to your stuff by people you don’t know, which reduces network effects.

But another way to do this is to let anyone access the stuff you create (worksheets, explanations, activities, etc) but only let people who know the creator see the identity and institution of the creator. In other words, you sort of password protect the identity of the person doing it, not the asset itself.

I think (maybe) with hashing this might not even be hard — identity is stored in a hash, when I find materials I like they are checked against my list of email contacts to see if the hashes could be results of any of those, and if they are I can see the person’s identity. If not then they are Anon689aef6, or whatever.

Just playing around with the idea. It would be a scheme mostly for people wanting to release Creative Commons licensed material for other to fork while minimizing exposure.

Why Facebook Won, and Other Hard Truths

A lot of people have been tweeting and emailing me and DM-ing me the recent Guardian piece by Iran’s “blogfather”.

You should read it yourself, but in short it is the story of a man sent to jail for blogging in Iran at the height of blogging’s influence and coming out of jail many years later to find that Facebook, Twitter, and Instagram have “killed the web.” And this is true. And it’s horrific. You need to read the article to remember why this fight matters.

In a related conversation yesterday we were talking about the New York Times article featuring a cast of merry Luddites talking about escaping the endless grind of Facebook and the stream.

And yes, The Stream Won. I get it. But when you ask many people *why* it won the answer you get, more or less, is that Evil Facebook and Twitter and Instagram hid the truth from the larger population and fooled them into thinking they wanted Evil Facebook and Instagram and Twitter instead of the exhilarating open web.

But that’s delusion at best. Facebook, Instagram, and Twitter won because they solved problems that the open web repeatedly failed to solve. To some extent, they became evil because they won; they didn’t win because they were evil.

If you want to make a difference in this fight it’s important to understand what those issues they solved were, and to provide open solutions to them. A lot of the things that made Facebook what it is revolve around gaps in HTML and HTTP that once the web got up to speed never got addressed.

What we have learned over time is that people don’t want to go to 80 different sites, with different interfaces, layouts, and ads to read. They want, mostly, to go to one site with a standard layout and look at things there.

Maybe you love to go to 80 different sites a day. Maybe the visual diversity and experience of learning a new interface every five minutes is super-appealing to you. Maybe you think everyone is just being lazy IT’S NOT THAT HARD GUYS AMIRITE?

Great. You get a gold star. But you’re why Facebook won. For most people that “visual diversity” looks like 1970s commercial sprawl:


This is what browsing the web feels like today.

Do you know what Facebook feels like to people? It feels like this:


It feels like a carefully crafted downtown, full of many different things but where building codes have enforced a standardization of design.

Before you protest much, do you remember what your day looked like at the height of blogging? I remember what it looked like for me — it was Google Reader, all the time. And what Google Reader did was strip out all that graphic design we had worked so hard on so we didn’t have to deal with it, and we could read in a peaceful, quiet manner, in a standard interface.

Or engage in a thought experiment. Imagine that every email you got during a day had different fonts, headings, layout, navigation, and scrolling bar behavior. Wouldn’t that be fun? Or even better, every email forced you to click, and go read it on a beautiful custom-designed website. Then you would reply by making that person come to your website and emailing them a link to your new GeoCities creation. Wouldn’t that be awesome?

No? Well here’s the thing. People read the web now at the level they read email — they look at a lot of stuff. And what they want (and what many people continue to shame them for) is a standard interface that allows them to do that without feeling stressed.

You want to win against Facebook? Let go of the idea of people reading your stuff on your site, and develop or support interfaces that put your readers in control of how they view the web instead of giving the control to the people with the servers. Support people looking into federated recommendation systems. Make friends with the idea of full copies of your stuff flowing across the web instead of links.

There’s a way out of this that brings back a diversity of viewpoints on the web. But we will never get there if we require people to read our stuff on our site.

NOTE: People that don’t understand federation or syndication may think I am saying that we just have to make peace with the Facebooks and Twitters of the world. I’m not. I’m saying that we have to build a federated system with less ego, one that has the affordances of Facebook but on a distributed platform. Read my old posts on Federated Wiki if you want to see what that looks like.

Markdown added to Wikity

Markdown support has been added to Wikity (more specifically GitHub-flavored Markdown). I’ve done this for some very good reasons, and I’ve also done this a bit differently than most implementations, and I thought I’d explain why.

First, a brief description of what Markdown is, for the uninitiated. Markdown is a markup format produced in 2004-ish by Jon Gruber with some input from Aaron Swartz. The idea of Markdown was to create a markup syntax that balanced human-readability with machine readability. Like wiki markup before it, it drew heavily from conventions of a format called setext which was used in Usenet newsgroups and email communications.

Here’s a comparison of what markup looks like in HTML and Markdown:

text using Markdown syntax the corresponding HTML produced by a Markdown processor

### Another deeper heading
Paragraphs are separated
by a blank line.

Leave 2 spaces at the end of a line to do a  
line break

Text attributes *italic*, **bold**, 
`monospace`, ~~strikethrough~~ .

Shopping list:

  * apples
  * oranges
  * pears

Numbered list:

  1. apples
  2. oranges
  3. pears

The rain---not the reign---in

A [link](http://example.com).


<h3>Another deeper heading</h3>

<p>Paragraphs are separated
by a blank line.</p>

<p>Leave 2 spaces at the end of a line to do a<br />
line break</p>

<p>Text attributes <em>italic</em>, <strong>bold</strong>,
<code>monospace</code>, <s>strikethrough</s>.</p>

<p>Shopping list:</p>


<p>Numbered list:</p>


<p>The rain&mdash;not the
reign&mdash;in Spain.</p>

<p>A <a href=http://example.com&#8221;>link</a>.</p>

Why Markdown and Reusability Go Together

“So what?”, you might think. “Markdown is moderately easier to read and work with than HTML but I use an HTML editor so I don’t ever see that HTML gobbledygook!”

Well, we’ve set it up so you can use the HTML editor side by side in Wikity next to Markdown. So you don’t need to change a thing. But I’d like you to consider a couple reasons why you might try working in Markdown.

Markdown keeps you honest. Visual editors pander to people who want to control how something looks rather than what it means. But in our multi-device world you can’t really control how something looks. When you do silly things like declare the size of a heading in absolute terms, you create a reusability mess that adversely effects everyone down the line.

Now it’s true, the WordPress editor has gotten very good at saving people from their worst instincts, but then there’s the copy problem…

Markdown solves the copy/paste problem. When you copy text from another website into your WordPress editor kittens die. Lots of kittens. Even if the text you pasted in looks good to you in your browser it looks horrible in someone else’s, and it’s probably uneditable to boot.

What happens is this. The website you go to has a databased version of your text. It looks at your browser, your platform, your screen size and the surrounding template and encodes that all into a set of HTML probably keyed to a specific style sheet served up separately. Now you go and copy that text into a new context that has to serve multiple browsers and platforms and screen sizes from a template that works entirely differently and uses its own, different style sheet.

What you’ve done is coded the use context into the content, and that is very bad.

With Markdown, you copy and paste the text in, and then quickly add the semantic markup, indicating links, blockquotes, italics, etc. Your text comes over clean.

Markdown is friendly to the mobile composer. I’ll be honest: I think mobile devices are the wrong devices for composing things. But I know I’m in the minority here.

Now if you’ve ever tried to blockquote or bold text with a mobile device, or tweak which words in a phrase should be part of a link you’ll have realized that a big problem with text editing on these devices is that 30 years of WYSIWYG development has assumed that there is a mouse attached to your machine.

Markdown solves this problem by making all formatting text-based. If you can type on your phone you can format.

Markdown will make you faster at everything. Add this to the “hard-to-learn but easy-to-use” category. There is a small learning curve with Markdown. However, because Markdown allows you to compose and format without periodically breaking concentration to look for the tool bar, grab the mouse, select text and find the right button it will make you faster in everything you do. There’s a reason that programmers, the laziest people on the planet, have gravitated to Markdown for documentation and composition.

Special bonus for teachers: keep students focused on the content. I’ll just add this in here, from personal experience. I have been teaching with various site making tools for more than a decade and there’s a definite balance to be struck with student design on projects that are not about student design.

I think, for example, that your history or stats class will enjoy being able to choose a WordPress template for their page, a background image, maybe a custom header, etc. These things build a sense of ownership with a site.

But I’ve also seen projects with Google Sites and other tools go awry, where a certain group gets so fixated on the issue that they can’t get the picture to float-left just the right way or get the blockquote in a certain section to just the right size that they forget they are supposed to be doing history or stats or sociology.

In fact, I’d argue that part of technical literacy these days is understanding in a multi-device accessibility-focused world that this pixel-level focus is a sign of technical illiteracy, and we must do our best to get students to understand the separation between content and styling that makes the modern communications ecosystem possible.

There are more reasons, but damn this post is long. I was going to go into a long discourse (diatribe?) on the history of WYSIWYG and how it killed text reusability, but this post is too long already.

Final Note

We did the implementation differently than JetPack, becuase JetPack has different goals.

JetPack lets you type in Markdown which is then converted to HTML which then gets saved. This is because the assumption in blogging engines is that once you post you will likely never edit that page again, so there’s no benefit in maintaining Markdown editability relative to the cost of the system having to interpret Markdown each time a viewer looks at your page.

Needless to say, we see things differently. The primary use to us of Markdown is as a storage format, so that when someone forks your page they get a nice clean Markdown version of it (if that’s the way you wrote it) that they can easily edit and update.

So we hacked in a module called Parsedown, and rolled our own wrapper for it.

What this means is if someone writes Markdown-based pages you should be able to fork those pages. Ideally, what I’d like to see is that a few communities that emerge on Wikity might make community rules which either require or forbid the use of Markdown for materials that they produce.

There is a Markdown Cheatsheet here, and of course you can go get an account on Wikity here. The join code for the moment is “peloton”, but if it changes you can always reach me via Twitter (@holden).