March 2nd, 2020

Hard Stare

"Social networking: it's new but it isn't News" (from an old Inquirer article of mine)

(30th June 2007 on The Inquirer)

THERE'S ANOTHER NEW social networking site around, from the guy behind Digg. It's called Pownce, it's still invitation-only and if they're offering anything genuinely new and different they aren't shouting about it. In particular, nobody's talking about the feature I want to see.

Get connected

There are myriads of social networking-type sites these days; Wikipedia lists more than ninety. Some of the big ones are MySpace, Bebo, Facebook and Orkut. Then there are "microblogging" sites like Twitter and Jaiku. Then of course there are all the tired old pure-play blogging sites like LiveJournal and Blogger. I have accounts on a handful of them - in some cases, just so I can comment, because OpenID isn't as well-supported as it deserves to be.

They all do much the same sort of thing. You get an account for free, you put up a profile, maybe upload some photos, tunes, video clips or a blog, then you can look up your mates and "add" them as "friends". Mainly, this allows you to get a summary list of what your mates are up to; secondarily, you can restrict who can see what that you're putting up.

Doesn't sound like much, but these are some of the biggest and most popular websites on the Internet. That means money: News International paid $580 million for MySpace and its founders are asking for $12.5 million a year each to stay on for another couple of years.

The purely social sites, like Myspace, sometimes serve as training wheels for Internet newbies. You don't need to understand email and all that sort of thing - you can talk to your mates entirely within the friendly confines of one big website. After all, there's no phonebook for the Internet - it's hard for friends to find one another, especially if they're not all that Net-literate.

A lot of the sites try to keep you in their confines. MySpace offers its own, closed instant-messenging service, for example - so long as you use Windows. Another way is that when someone sends you a message or comment on MySpace or Facebook, the site informs you by email - but the email doesn't tell you what the actual message was. You have to go to the site and sign in to read it.

Buzzword alert

Some sites aren't so closed - for example, the email notifications from Livejournal tell you what was said and let you respond from within your email client, and its profiles offer basic integration of external IM services. On the other hand, Facebook offers trendy Web 2.0 features, like "applications" that can run within your profile and can be rearranged by simple drag&drop, whereas LJ or Facebook owners who want unique customisations must fiddle with CSS and HTML or use a third-party application.

As well as aggregating your mates' blogs, many social networking sites let you syndicate "web feeds" from other sites. A "feed" - there are several standards to choose from, including Atom and various versions of RSS - supplies a constantly-updated stream of new stories or posts from one site into another. For instance, as I write, fifteen people on LiveJournal read The Inquirer through its LJ feed.

(If you fancy this aggregation idea but don't want to join a networking site, you can also do this using an "feed reader" on your own computer. There are a growing number of these: as well as standalone applications such as FeedReader or NetNewsWire, many modern browsers and email clients can handle RSS feeds - for example, IE7, Firefox, Outlook and Safari.)

But even with feeds, the social networking sites are still a walled garden. If you read a story or a post syndicated from another site, you'll probably get a space to enter comments - but you won't see the comments from users on the original site and they won't see yours. The same goes for users anywhere else reading a syndicated feed - only the stories themselves get passed through, not the comments.

A lot of the point of sites like Digg and is the recently popular concept of "wisdom of crowds". If lots of people "tag" something as being interesting and the site presents a list of the most-tagged pages, then the reader is presented with an instantaneous "what's hot" list - say, what the majority of the users of the site are currently viewing.

There are sites doing lots of clever stuff with feeds, such as Yahoo Pipes, which lets you visually put together "programs" to combine the information from multiple feeds - what the trendy Web 2.0 types call a "mashup". What you don't get through a feed, though, is what people are saying.

Similarly, the social networking sites are, in a way, parasitic on email: you get more messages than before, but for the most part they have almost no informational content, and in order to communicate with other users, they encourage you to use the sites' own internal mechanisms rather than email or IM. Outside a site like Facebook, you can't see anything much - you must join to participate. Indeed, inside the site, the mechanisms are often rather primitive - for instance, Facebook and Twitter have no useful threading. All you get is a flat list of comments; people resort to heading messages "@alice" or "@bob" to indicate to whom they're talking. Meanwhile, the sites' notifications to the outside world are a read-only 1-bit channel, just signals that something's happened. You might as well just have an icon flashing on your screen.

In other words, it's all very basic. Feeds allow for clever stuff, but the actual mechanics of letting people communicate tend to be rather primitive, and often it's the older sites that do a better job. The social sites are in some ways just a mass of private web fora (it's the correct plural of "forum), with all their limitations of poor or nonexistent threading and inconsistent user interfaces. Which seems a bit back-asswards to me. Threaded discussions are 1980s technology, after all.

Going back into time

Websites have limits. Email may be old-fashioned, but it's still a useful tool, especially with good client software. Google's Gmail does some snazzy AJAX magic to make webmail into a viable alternative to a proper email client - its searching and threading are both excellent. An increasing number of friends and clients of mine are giving up on standalone email clients and just switching to Gmail. The snag with a website, though, is that if you're not connected - or the site is down - you're a bit stuck. When either end is offline, the whole shebang is useless.

Whereas if you download your email into a client on your own computer, you can use it even when not connected - if it's in a portable device, underground or on a plane or in the middle of Antarctica with no wireless Internet coverage. You can read existing emails, sort and organize, compose replies, whatever - and when you get back online, the device automatically does the sending and receiving for you. What's more, when you store and handle your own email, you have a major extra freedom - you can change your service provider. If you use Gmail or Hotmail, you're tied to the generosity of those noted non-profit philanthropic organizations Google and Microsoft.

The biggest reason email works so well is that it's open: it's all based on free, open standards. Anyone with Internet email can send messages to anyone with an Internet email address. Even someone on one proprietary system, say Outlook and Exchange, can send mail to a user on another, say Lotus Notes. Both systems talk the common protocols: primarily, SMTP, the Simple Mail Transfer Protocol. Outside the proprietary world, most email clients use POP3 or IMAP to receive messages from servers - and again, SMTP to send.

Now here's a thought. Wouldn't it be handy if there was an open standard for moving messages between online fora? (It's the correct plural of "forum", not "forums".) So that if you were reading a friend's blog through a feed into your preferred social networking site, you could read all the comments, too, and participate in the discussion? If it worked both ways, on a peer-to-peer basis, the people discussing a story on Facebook could also discuss it with the users on Livejournal. If it was syndicated in from Slashdot, they could talk to all the Slashdot users, too.

Now there is a killer feature for a new, up and coming social networking site. Syndication of group discussions, not just stories. It would be a good basis for competitive features, too - like good threading, management of conversations and so on.

The sting in the tail

The kicker is, there already is such a protocol. It's called NNTP: the Network News Transfer Protocol.

The worldwide system for handling threaded public discussions has been around for 26 years now. It's called Usenet and since a decade before the Web was invented it's been carrying some 20,000 active discussion groups, called "newsgroups", all around the world. It's a bit passé these days - spam originated on Usenet long before it came to email, and although Usenet still sees a massive amount of traffic, 99% of it is encoded binaries - many people now only use it for file sharing.

You may never have heard of it, but there's a good chance that your email system supports Usenet. Microsofties can read newsgroups in Outlook Express, Windows Mail and Entourage, or in Outlook via various addons; open sourcerers can use Mozilla's Thunderbird on Windows, Mac OS X or Linux. Google offers GoogleGroups, which has the largest and oldest Usenet archive in the world. There are also lots of dedicated newsreaders - on Windows, Forté's Agent is one of the most popular.

Usenet is a decentralised network: users download messages from news servers, but the servers pass them around amongst themselves - there's no top-down hierarchy. Companies can run private newsgroups if they wish and block these from being distributed. All the problems of working out unique message identifiers and so on were sorted out a quarter of a century ago. Messages can be sent to multiple newsgroups at once, and like discussion forum posts, they always have a subject line. Traditionally, they are in plain text, but you can use HTML as well - though the old-timers hate it.

There are things Usenet doesn't do well. There's no way to look up posters' profiles, for example - but that's exactly the sort of thing that social networking sites are good at. Every message shows its sender's email address - but then, the social networking sites all give you your own personal ID anyway.

Big jobs, little jobs

It would be a massive task to convert the software driving all the different online discussion sites to speaking NNTP, though. It isn't even remotely what they were intended for.

But there's another way. A similar problem already exists if you use a webmail service like Hotmail but want to download your messages into your own email client. Hotmail used to offer POP3 downloads as a free service, but it became a paid-for extra years ago. Yahoo and Gmail offer it for free, but lots of webmail providers don't.

Happily, though, there's an answer.

If you use Thunderbird, there's an extension called Webmail which can download from Hotmail as well as Yahoo, Gmail and other sites. Like all Mozilla extensions, it runs on any platform that Thunderbird supports.

But better still, there's a standalone program. It's called MrPostman and because it's written in Java it runs on almost anything - I've used it on Windows, Mac OS X and Linux. It's modular, using small scripts to support about a dozen webmail providers, including Microsoft Exchange's Outlook Web Access; it can even read RSS feeds. Its developers cautiously say that "Adding a new webmail provider might be as simple as writing a script of 50 lines."

And it's GPL open source, so it won't cost you anything. It's a fairly small program, too - it will just about fit on a floppy disk.

MrPostman shows that it's possible to convert a web-based email service into standard POP3 - and for this to be done by a third party with no no access to the source code of the server. Surely it can be done for a forum, too? And if it's done right, for lots of fora? It doesn't need the help or cooperation of the source sites, though that would surely help. More to the point, if it was done online, the servers offering the NNTP feeds can be separate from those hosting the sites.

What's more, there's a precedent. For users of the British conferencing service CIX, there's a little Perl program called Clink, which takes CoSy conferences and topics and presents them as an NNTP feed, so that you can read - and post to - CIX through your newsreader.

It sounds to me like the sort of task that would be ideal for the Perl and Python wizards who design Web 2.0 sites, and it would be a killer feature for any site that acts as a feed aggregator.

Rather than reading contentless emails and going off to multiple different sites to read the comments and post replies, navigating dozens of different user interfaces and coping with crappy non-threaded web for a, you could do it all in one place - as the idea spread, whichever site you preferred.

And, of course, the same applies to aggregator software as well. When you download this stuff to your own machine, you can read it at your leisure, without paying extortionate bills for mobile connectivity. Download the bulk of the new messages on a fast free connection, then just post replies on the move when you're paying for every kilobyte over a slow mobile link.

What's more, in my experience of many different email systems, it's the offline ones that are the fastest and offer the best threading and message management. It could bring a whole new life to discussions on the Web.

All this, and all I ask for the idea is a commission of 1 penny per message to anyone who implements it. It's a bargain.