Technology behind Twitter

Twitter

I was interested in the actual Web Technologies used to develop some of the big social media sites. So, I looked at Twitter and found some interesting information. It was built using Ruby on Rails and did face certain scaling issues which led to beliefs that Twitter would move off RoR or Rails in mid 2008. There was much controversy surrounding blame on Ruby for Twitter’s scaling problems and if you are interested in nerdy details of how Twitter evolved (which I found extreeemely interesting), check this:

Update 5: Twitter on Scala. A Conversation with Steve Jenson, Alex Payne, and Robey Pointer by Bill Venners. A fascinating discussion of why Twitter moved to the Java JVM for their server infrastructure (long lived processes) and why they moved to Scala to program against it (high level language, static typing, functional). Ruby is used on the front-end but wasn’t performant or reliable enough for the back-end.

Update 4: Improving Running Components at Twitter by Evan Weaver. Tells how Twitter changed their infrastructure to go from handling 3 requests to 139 requests a second. They moved to a messaging model, asynchronous process, 3 levels of cache, and moved their middleware to a mixture C and Scala/JVM.

I’ve heard about RoR many a times as an extremely fast web development framework but never had a chance to use it directly. The WordPress and user facing interface of Facebook is built using PHP. I am very little familiar with other platforms such as Coldfusion etc.

To gain some insight into how RoR was selected for Twitter, I found this post from AJ at IBM quite interesting:

JACK DORSEY: For example the use of Rails made the application accessible to people withing 2 weeks! The fact that people saw the application and reacted to it with such a short time was very important. Florian Weber , One of the Rails core committers was in the team (with close ties to DHH) and that also was enticing in choosing the framework.
The mistake we did was putting it all out there. We did not follow a controlled expose methodology and that is what came to bite us.
As for the technology perspective I would not have changed anything! We would be cautious and expect explosive growth but the platform would have been the same.

AJ: These days I work with Rails everyday and given the complexity of the application suite we (Max, Roy and me) are building, if we did not have Ruby on Rails, we wouldn’t have done even half the stuff ! Its amazingly productive and given a good rails tool set (I use Eclipse Aptana studio – free version of course) it becomes a really good competitor to any Web development framework. And Jack just polished the chrome making rails shine more

Hm, I am tempted to try a hand at RoR soon. I think the real question is between choosing something that helps you launch faster vs something that is robust and scalable in long term. In the world of Internet, ‘launching fast’ is a big advantage and since not all the services explode like Twitter, it gives you more time to redesign as your user base expands.

My interest was purely technical but here is where you can read how the idea behind Twitter was born.