The scalability of Socialping hasn’t been looking too good lately. We’ve got hundreds of users in our beta and thousands of items being watched, collectively bringing in hundreds of thousands of tweets each and every day. These aren’t little terms that only get a few tweets an hour either, many of them are terms like Twitter, Facebook, Google, etc – you know, really popular things that really stress our systems. And we love you guys for it.
Part of this whole “beta” thing that we are in is to see what our limits are, when things are going to break, and how they break when they do. We’ve got more data on what was and wasn’t working, and how it was working (when it did) than we could have hoped for – and the sum of that data? We couldn’t scale to where we want to go with our current setup.
Recently we’ve had to make a lot of changes to our systems to handle the load. Those changes included rolling out Redis (a NoSQL key/value store and (sort of) database) and migrating from MySQL where it made sense. Other than one subsystem that gets used in our reporting systems, we’ve pretty much completed this migration and our load has been reduced considerably.
Sorry, I’m about to get all geeky, but if you have the time, take a look at Redis’ new “hash” types, they are perfect for some of the stuff we do. We also quite heavily use the Sorted Sets (zsets) and the regular Set types. I want to commend Salvatore Sanfilippo (@antirez) for all of his hard work on Redis, it’s truly one of the best key/value databases available.
So, back to the point of this post, lately we’ve been forced to scale, scale, scale and now that we’ve done enough to give us some headroom, we’ve put together a plan that’ll hopefully let us scale to the point that we can open things up the rest of the beta users that are still waiting to get in and eventually to the public as a whole.
If you care what those changes will be: we’re planning on continuing to use Redis for the long term, both as a cache and as data store, but also adding a cluster of servers that will use Cassandra for storing our tweets and while we’re still evaluating search options (spoiler alert!), likely ElasticSearch will be used to allow you guys to better search through your tweets. We’ll still use MySQL for managing some of the account stuff, like your watchlist. I know I am glossing over a lot of details, but this is the main stack we intend on using. As more details finalize and we start rolling these out, expect updates on how it’s going.
Thank you to all of the beta testers in there breaking things, and thank you for your patience as we work through our growing pains. For those of you still waiting to get in, we’re trying our best to get you in as soon as possible. Thank you for your patience as well.
P.S. Oh, and Salvatore, if you read this, when Redis Cluster is ready for testing, we want to be part of it. We are definitely going to need it.
4 Comments
Would be great if you can explain how are you using hashes in redis ?
Hi Sunil,
We’re using the hash type primarily for stats. Since everything we do is based on tags, we can create a hash for a tag with fields for each of the stats we track for that tag, then in real-time we can do HINCRBY for the items that we want to adjust the stats on. When we want to get the stats, it’s just one HGETALL call (via a function that turns it into a native array) and we can have all of the stats.
The beauty of this is twofold, first since redis is so fast, there doesn’t appear to be any additional overhead vs a standard string GET, and second, we’ve reduced latency in the interface because in just one call we’ve got all of the stats.
Could you explain more about your redis setup? From what I understand about redis, you have to have the entire data set in the memory.. I’d like to hear how you’ve clustered or plan to cluster the redis servers.
Hi Min,
Yes, you do need to have the entire data set in memory. Fortunately, for our needs this is easily possible, we are only at 1.5 GB worth of data (about 3.5 million keys).
We do not currently use clustering or sharding. We’ve got one server for Redis, and we’ll be bringing up a slave this week as a backup. Because we were fighting high loads and needed to get a solution up ASAP, we opted to bring just a master up at the time and then the slave later on.