Part of this whole “beta” thing that we are in is to see what our limits are, when things are going to break, and how they break when they do. We’ve got more data on what was and wasn’t working, and how it was working (when it did) than we could have hoped for – and the sum of that data? We couldn’t scale to where we want to go with our current setup.

Recently we’ve had to make a lot of changes to our systems to handle the load. Those changes included rolling out Redis (a NoSQL key/value store and (sort of) database) and migrating from MySQL where it made sense. Other than one subsystem that gets used in our reporting systems, we’ve pretty much completed this migration and our load has been reduced considerably.

Sorry, I’m about to get all geeky, but if you have the time, take a look at Redis’ new “hash” types, they are perfect for some of the stuff we do. We also quite heavily use the Sorted Sets (zsets) and the regular Set types. I want to commend Salvatore Sanfilippo (@antirez) for all of his hard work on Redis, it’s truly one of the best key/value databases available.

So, back to the point of this post, lately we’ve been forced to scale, scale, scale and now that we’ve done enough to give us some headroom, we’ve put together a plan that’ll hopefully let us scale to the point that we can open things up the rest of the beta users that are still waiting to get in and eventually to the public as a whole.

If you care what those changes will be: we’re planning on continuing to use Redis for the long term, both as a cache and as data store, but also adding a cluster of servers that will use Cassandra for storing our tweets and while we’re still evaluating search options (spoiler alert!), likely ElasticSearch will be used to allow you guys to better search through your tweets. We’ll still use MySQL for managing some of the account stuff, like your watchlist. I know I am glossing over a lot of details, but this is the main stack we intend on using. As more details finalize and we start rolling these out, expect updates on how it’s going.

Thank you to all of the beta testers in there breaking things, and thank you for your patience as we work through our growing pains. For those of you still waiting to get in, we’re trying our best to get you in as soon as possible. Thank you for your patience as well.

P.S. Oh, and Salvatore, if you read this, when Redis Cluster is ready for testing, we want to be part of it. We are definitely going to need it.