It must be an odd thing to run a company in the midst of a debate around the idea of nationalizing your core technology. In a Venezuelan moment, the Gillmor Gang considers the idea that Twitter has become so important that our national security requires nationalizing its technical infrastructure. In a two-part discussion about an open mesh / cross-service dashboard mashup and the role of Twitter as a sort of fundamental glue, the question surfaced of breaking up the centralized Twitter monopoly. You can hear the conversation here:
The conversation was provoked by some ongoing thoughts by Dave Winer around decentralizing Twitter. Initially the issues addressed were:
- Backup of Twitter user data in the event of utter failure of the service
- An alternative venue for the moments when Twitter is indisposed.
- Improving reliability through distributing the infrastructure to multiple players
- Redistributing Twitter’s monopoly power to multiple players for the common good
Discussion revolved around the general principle of open source standards and how Twitter should be re-created as a standard like ethernet, SMPT, POP, IMAP, XMPP, HTTP, etc. This would allow multiple vendors to compete with products using the same base protocols. For example, many vendors compete using a common standard for email, like 666casino.com/sv for having chat support to answers their online player’s concerns, give a quick solution, and sends promotions or rewards. Standards create a very useful interoperability in the case of email and web sites. Instant messenger has multiple protocols and requires debabelization services to enable conversation between platforms.
Chris Saad put forth an interesting proposal around the idea of publish/subscribe and Twitter literally as micro-blogging. His idea is to move Twitter to a model similar to that of blogging and RSS. Through a micro-blogging authoring tool, something like WordPress, an individual would publish Tweets. A group of followers who had indicated interest in receiving messages would be pushed a payload immediately on publication. A Tweet reader would be used to subscribe to the streams of various publishers.
On the Gillmor Gang call there was some confusion about the roll of RSS in Saad’s proposal. Because XMPP can be difficult to program against, Saad suggested authoring tools that output the RSS format into a gateway that would transform it into XMPP for immediate transport. The idea is to use RSS as XML, a simple transport markup that most blog authoring tools already know how to output. However this was confused with the common usage of RSS as a polling-based publish/subscribe blog syndication methodology.
In looking at decentralizing Twitter, the focus was on two aspects of the service, replicating the unique social graph Twitter creates through the ideas of following and being open to being followed; and the immediate stream of 140 character hypertext that is generated through that matrix of connections. These two elements of the service have created a rich fabric of relationship and information flow that satisfies and intrigues 80% of the users.
The stream of information can be followed in a number of ways. Most people use the Twitter web site which offers a stream through a periodic refresh and redraw of the screen. A number of Twitter clients have been created to automate that process based on a web/RSS model of updating and publication. This streaming model is the equivalent of 15 minute delayed stock quotes. The stream flows based on the polling intervals of the reader, not on the actual publication events.
Steve Gillmor has been championing the instant messenger model of Twitter consumption. In this method an instant messenger client like Google’s Gtalk or iChat is used to talk to Twitter through an XMPP server that relays the Tweets it receives as quickly as it can on the publication event. This also works on a teleputer via SMS, or as those devices are sometimes called these days telephones. This model doesn’t scale particularly well. Users like Robert Scoble and Jason Calacanis have well over 20,000 people they follow.
The consumption strategy that makes the instant messaging model of Twitter work is to follow a core group and then track keywords of interest. Tracking keywords adds people you don’t follow into your stream and provides a proper level of noise and negative feedback into the information ecosystem. This can also be accomplished through a diversified approach to following. In modern portfolio theory this is called covariance.
It’s tracking that makes a decentralized Twitter nearly impossible. Think of a 140 character Tweet as a series of space separated tags to which you can subscribe. In this model, you’re following everyone, or at least everyone who uses that particular tag. This feature radically changes the shape of the social graph underlying the information stream. Since you don’t know who might use a tag you’re tracking, the regular RSS style contract around publication and subscription doesn’t work. Track is not commonly used today, but it’s one of the more interesting features of the service.
The idea of building competitors to Twitter on the same platform, or redistributing Twitter to multiple players reminds me of the idea that New York City should be rebuilt in Ohio because it would be cheaper. Or perhaps we could distribute a little of New York City in every state of the Union. New York City is what it is because of the people who live and visit there. Building another New York City in Las Vegas doesn’t result in the phenomenon that is New York City. In a very important sense, Twitter is decentralized at its core, it is rhizomatic rather than arborescent.
Excellent photo. I remember when it came out. It was mesmerizing and dangerous.
The Jerry Rubin picture is a photograph made by me in 1966 or 67. LIFE magazine ran it as a full page to show protesting the War in Viet Nam and the Hippie movement.
Nice information. Its usefulness and significance is overwhelming the way you covered all the basic necessary information is really impressive, good work keep it up.
Nice information. Its usefulness and significance is overwhelming the way you covered all the basic necessary information is really impressive, good work keep it up.
Nice information. Its usefulness and significance is overwhelming the way you covered all the basic necessary information is really impressive, good work keep it up.
Nice information. Its usefulness and significance is overwhelming the way you covered all the basic necessary information is really impressive, good work keep it up.
[…] Decentralizing Twitter is unnecessary, if not impractical. Dave Winer was right the first time, when he intuitively grasped the power of Twitter was not in what it was designed to be but in what it could be used for. By building on top of it, Winer signaled that instinct that he marshaled into RSS, the gesture of respect, the idea that in Steve Stills’ words, “Somethings happening here, What it is ain’t exactly clear…”Twitter ain’t broke, and we don’t need to fix it. […]
OK, if you check daily. you can get more like this.
OK, if you check daily. you can get more like this.
[…] criticism is a reprise of the open source silo meme. The drumbeat for the nationalization of Twitter has re-emerged. The capitalist pigs at Twitter have chosen to build a business rather than […]
What Are We Missing? Or Who’s Missing Us?…
Robert Scoble asks, What are the tech bloggers missing? Your business!
The post hit a nerve with me. Robert is absolutely right!
As I read my RSS feeds on a daily basis, I’m tired of the same crap over and over again. Are Microsoft and Yahoo! t…
[…] Echovar on decentralizing Twitter There are roughly three New Yorks. There is, first, the New York of the man or woman who was born here, who takes the city for granted and accepts its size and its turbulence as natural and inevitable. Second, there is the New York of the commuter—the city that is devoured by locusts each day and spat out each night. Third, there is the New York of the person who was born somewhere else and came to New York in quest of something. […] Commuters give the city its tidal restlessness; natives give it solidity and continuity; but the settlers give it passion.* […]
[…] echovar » Blog Archive » A Venezuelan Moment: The Gillmor Gang considers nationalizing Twitter breaking up the centralized Twitter monopoly (tags: twitter social internet socialmedia socialnetworks technology) […]
Try “Track” using Twitter with an XMPP client. If you have a Gmail acct use Gtalk. iChat also works on the Mac. The client needs to speak Jabber. There are a growing number of people who use track to create an optimal flow of information.
Regarding ink and paper vs. words and ideas — I suggest looking at Derrida's “Writing and Difference” or “Speech and Phenomena.” Writing and thinking are very closely linked.
[…] You seriously discuss the nationalization of Twitter. […]
Hi Cliff,
While the graph of tweets (where followers can be followed) is not arborescent, the fundamental structure of twitter is star, with them at the centre and us at the edge.
When decentralising Twitter is discussed, I think IRC. In fact Twitter is a meagre echo of IRC, with a lot of additional limitations but the big plus, for some, of mass mobility thanks to SMS.
Yes its simplicity is its strength, allowing a lot of building on.
Back to IRC, a distributed mesh of servers.
Your call to “Think of a 140 character Tweet as a series of space separated tags to which you can subscribe.” is fanciful at best. No-one subscribes to a tweet, but to an end-point, the IP address of Twitter's name space, e.g. cgerrish.
Sure, twitter search (over what subset of tweets) might allow you to subscribe to a “tag” in a tweet, fine, but it hardly defines the fundamental service.
Keep in mind neither the ink nor the paper are the words or ideas. You can't attribute every outcome to its components.
This is a similar problem to that of real time quotes and the creation of ticker plants to provide “real-time” transaction data to equity traders. There are many vendors in the real-time quote space, but there's only one consolidated tape of transaction data.
Twitter is the consolidated tape. GTalk, or Twhirl when it implements XMPP, will be able to provide track — as long as Twitter provides follows and track keywords per identity.
This is where there may be a significant difference in the clients — Twhirl, as a desktop app, may have a problem filtering the full stream. It could certainly filter the follows. Google, and GTalk, would have a better chance of filtering the full tick-by-tick stream on the server side and passing that through to a web-based (or Google Gears) app.
[…] You seriously discuss the nationalization of Twitter. […]
Oh, absolutely; I anxiously await the day that we can apply “track” across, well, the internet. Right now google and technorati et al have no hope to build something like Twitter & track. I hope that federation will open up the web to enabling rich conversations around all sorts of media.
[…] A Venezuelan Moment: The Gillmor Gang considers nationalizing Twitter The idea of building competitors to Twitter on the same platform, or redistributing Twitter to multiple players reminds me of the idea that New York City should be rebuilt in Ohio because it would be cheaper. (tags: twitter social web) […]
If I fire up a Limewire client I can see this stream of search queries from other people on the P2P network. Couldn't decentralized Twitter servers be using something like that to exchange keywords and implement tracking?
Decentralized Twitter isn’t hard….
Cliff Gerrish writes on his Echovar blog: The consumption strategy that makes the instant messaging model of Twitter work is to follow a core group and then track keywords of interest. Tracking keywords adds people you don’t follow into your…
Blaine Cook wrote: “NNTP is a good conceptual representation, except that it's topic-based, rather than user-based.”
If you just consider each user to be a “topic” then NNTP matches very well. “Following” in Twitter is just topic-based publish/subscribe. Tracking, which is content-based publish/subscribe is the harder challenge — not implemented commonly by NNTP systems. Based on experience, I can assure you that reasonable systems (i.e. small number of machines) could provide full content-based tracking for tweets at rates of thousands of tweets per second…
[…] has become so important that uptime be damned, should be decentralised or even nationalised, or simply is not broken, the breaking news of a 7.8 earthquake in Sichaun, China is provides a crude reminder of how […]
[…] Cliff Gerrish has a great explanation for why Twitter cannot be decentralized: It’s tracking that makes a decentralized Twitter nearly impossible. Think of a 140 character Tweet as a series of space separated tags to which you can subscribe. In this model, you’re following everyone, or at least everyone who uses that particular tag. This feature radically changes the shape of the social graph underlying the information stream. Since you don’t know who might use a tag you’re tracking, the regular RSS style contract around publication and subscription doesn’t work. Track is not commonly used today, but it’s one of the more interesting features of the service. […]
Twitter's always-down problem was quite obviously system administration problem, you just need to read (the attitude of) their ISP migration announcements to tell. It was kinda of like, love-me-love-my-dog mentality. Whoever heading that team should be immediately fired, that's all.
Yes, this is similar to Dave's coral reef analogy. I don't consider Twitter separate from the things that emerge from it. This is part of the meaning of rhizomatic. Twitter is a piece, not a whole. It can be a piece of many things and so it is not yet what it will become.
Blaine
i am with you that federation has significant advantages but neither google nor technorati nor any trackback service does anything close to what twitter does in terms of capturing and facilitating real time discussion.
i run track on my phone (thanks for building that) and i have better discussions ten times a day than i get blog to blog once a month.
fred
maybe twitter is already everything it will become. maybe it's the dumb social net in the same way that the internet is a dumb network. and because of that amazing things emerge on top of it.
“in a venezuelan moment”. Sad, but true. But then again, Venezuela is packed with entrepreneurial spirit. check this out:
http://www.gemconsortium.org/about.aspx?page=pu…
cheers,
twitter: jgheller
I think both email and the web stand as counter-examples to the idea that localized virtual spaces are somehow more valuable than distributed ones. YouTube is a great example as well — I suspect a healthy portion of YouTube's views occur off-site, in embedded players.
Regarding the operational investment, it's a matter of scale — currently Twitter sends on the order of 10 updates per second over the public feed (you can independently verify this; I'm not sure what the actual number is), which requires a tiny investment to handle, doing whatever operations you'd care to do (search, textual analysis, whatever).
However, if we consider the number of blog posts that are made per second around the world, the investment is going to be much higher. Google does a pretty good job of keeping up with blogs, but the crawling, indexing, and notification infrastructure are non-trivial to build and sustain. As micro-blogging becomes larger in scope, a similar problem arises; a vendor providing track-like functionality is going to have to aggregate all the real-time streams, apply filters to identify relevant information, and then send notifications based on matches.
NNTP is a good conceptual representation, except that it's topic-based, rather than user-based.
Did anybody thought of Amazon S3 as a solution for the crashes suffered by twitter?
[…] some sense from the company that created it — has been going on for awhile now, and recently reared its head again on the Gillmor Gang, the podcast run by ZDNet blogger and tech guru Steve Gillmor. As described by […]
[…] Decentralizing Twitter is unnecessary, if not impractical. Dave Winer was right the first time, when he intuitively grasped the power of Twitter was not in what it was designed to be but in what it could be used for. By building on top of it, Winer signaled that instinct that he marshaled into RSS, the gesture of respect, the idea that in Steve Stills’ words, “Somethings happening here, What it is ain’t exactly clear…” Twitter ain’t broke, and we don’t need to fix it. […]
I got what Chris was saying, and Robert too. Notice that Blaine uses the term “RSS-like” on the show site comment. When someone demonstrates that capability, I'll be glad to start using it. The issue of decentralizing – or federating as Winer suggests today – is separate, and political, not technical in nature as Blaine's Google pitch would suggest. Remember: it's just an opinion I happen to share with Cliff. Office is dead.
Hi Robert,
Yes, I could hear that you got what Chris Saad was saying. But Steve, as proxy for the general user, wasn't hearing that piece of it. He was focused on polling versus “real-time” transport of messages.
Check out Blaine Cook's comments here and on the Gillmor Gang blog. Apparently the full real-time stream is available at the API level. Twitter's surprising level of openness continues to tantalize.
This the problem of “Celebrity.” Clay Shirky describes it very nicely in his new book. Fame changes a normally two-way communications mode into a broadcast mode. Paris Hilton will write to the steam, but will be unable to read her info-stream because it will be too dense. Just as celebrities can't answer every fan letter. It's likely that another Twitter identity would be created that was protected and distributed to a smaller group.
Thanks for the comment Blaine. I wrote the post to try and explain and expose the landscape based on the threads flowing through my info-stream.
I agree that the internet isn't a place in an ordinary sense of that word. But human interaction and history does seem to accrue into specific virtual spaces. Twitter, MySpace and FaceBook are all examples of that. We may be on the verge of a new idea personal virtual space with Marc Canter's Open Mesh, Ray Ozzie's Live Mesh and even the Evernote product.
You've exposed a key point in your comment — track is filtering the full global stream. As you noted on your comment on GillmorGang site, Twitter makes the full global stream available on an API basis. This takes the conversation beyond replicating pub/sub models for micro-blogging.
Perhaps you could expand on the “significant operational investment” required to filter the full global stream, and why you think a player like Google would be required.
Are you suggesting that the full stream would need to be replicated in real time? (Something like the way UseNet works) Or that queries would need to be joined across multiple full global streams?
Hi Dave, it's Cliff Gerrish – echovar is my blog, @cgerrish on Twitter. Thanks for the link from Scripting News.
It's an interesting conversation that resonates on many levels. I share your concerns about reliability, although I admire Twitter giving itself the space to fail now and then. 100% uptime is often achieved by locking things down, consolidating product gains and stopping the process of innovation. I think that Twitter is far from what it will eventually become.
[…] has into this question, but the most searching critique I’ve come across comes from Echovar. It’s worth reading the entire post. Here are a couple of excerpts: It must be an odd […]
Good post, Cliff.
I certainly wasn't confused about the RSS part; just that Chris said that the blogging Subscribe/Unsubscribe contract is all we need for a Twitter-like service. I think this is due to his mistaken view that Twitter is just micro-blogging. Wrapping RSS in XMPP may simplify implementation for developers, but architecturally it is neither here not there.
I particularly like your analogies in the last paragraph.
Cheers,
Robert
I agree with Blaine. I'm constantly surprised at how over-stated the functionality of Twitter is. When you boil it down to its essentials, I think it's clear that a distributed replica is quite viable.
I've suggested a few times that tracking could be implemented much the way SixApart implemented TrackBack back in the early days of blogging. All micro-blogging tools/sites would implement a commmon API for receiving “follow” pings from their readership.
How will these suggestions address the issue raised on TWIT recently about the system handling a Britney Spears or Paris Hilton with potentially tens of thousands of followers when it can't seem to support several people like Scoble or Rose or LaPorte who are just busting 5 figures of listeners?
[…] every yin there’s a yang. Here’s a brilliant counterpoint to what I’ve been writing here about decentralizing Twitter. I’ve excerpted the last paragraph […]
I have to strongly disagree — a similar argument could have been made about email ca. 1993, when AOL offered an email service that was qualitatively different than CompuServe, BIX, etc.
The internet isn't a “place”; it's distributed, and it's the individuals who use the internet that create a sense of space themselves — their desktops, their web browsers, the sites they visit all contribute to their experience.
Moreover, as the person who *built* the track feature (in 12 hours, no less), I'm comfortable saying that you could federate similar services and still retain that functionality. It would be harder for individual services to filter the full global stream without having a potentially significant operational investment, but that's where someone like Google comes in.
Thanks…Sums up the Gillmor Gang discussion very well. As well as the issues.
[…] a brilliant counterpoint to what I’ve been writing here about decentralizing Twitter. I’ve excerpted the last […]
Who wrote this? Great post esp the last paragraph.
Only one problem. As wonderful Twitter is in all the things it means to every person (every one views it differently) it unifies everyone when it disappears — then it sucks!
Then we're all on the same page –> FIX IT!
I loved the analogy about New York. 🙂
Right on.