This mentality is wrong, for most people.

Today we are in the middle of a massive Amazon outage that has been going on for many hours already, and no concrete time frame is available for when it will come online again, which brings me to my first point: using a service like AWS means you don’t get the one-on-one customer support you need when running a business that has critical needs, like 100% uptime.

Sure there are ways to load balance between available zones, but as today proves, even that doesn’t always work. In order to be truly available, you need multiple datacenters and those datacenters must from from multiple providers and be far apart from each other, like one on the east coast and the other on the west coast.

My other point is related to billing. Cloud servers ARE NOT CHEAPER, when you run them 24/7. Many people have cited cost savings as the main reason to use AWS. I don’t know about you, but every time I have tested this theory, it always worked out that it was cheaper to use dedicated servers than it did using a virtualized server. Additionally, with virtualized servers, you typically also get reduced disk I/O, network latency and more which make your service run slower.

How many services need to scale up and down their infrastructure daily? Far less than you might think. Today, Foursquare, Quora, CoTweet, Hootsuite, and I’m sure dozen’s more of popular services are offline because of the AWS outage. The only one I see that is likely benefiting from scaling up and down servers is probably Foursquare, the rest probably have their servers running 24/7.

Until recently we’ve always hosted our servers at Steadfast Networks. In the recent months, we switched to using ServerBeach as our primary provider. With both of them, we’ve used multiple dedicated servers in a load balanced, high availability manner. We also use replication and a few other techniques to make sure that our customers data is safe and our service is online all the time. We learned this the hard way in the past.

I know that this post is different from most of our posts on our blog, but we have a lot of experience working with these cloud services, and in that time have determined, they just are not worth it.

We know many people feel strongly about using them.  Feel free to let us know why in the comments below.

So, here’s an update: By now we’ve finished most of the API and we’ve got a new back-end in testing that is going to hopefully support us much better than the old setup did (for the nerds reading this, we’re using MongoDB for our tweets, Redis for a few things that need really fast access times, and MySQL for the account info).

At this point we’re still planning on rolling out a retrofitted version of the site using our new API instead of it having direct access, I think we are going to be doing more changes to the code than I anticipated, and thus there are likely going to be a few things that in fact, totally new.

I’m starting to get a bit of “can’t wait to release it” excitement because finally a number of the pieces are starting to talk to each other.  Just a few more talking pieces and we’ll have our MVP out and we’ll get to start iterating on it while you guys are trying to break it. Exciting stuff.

Want to help us break it when it comes out? Register for the beta here.

Part of this whole “beta” thing that we are in is to see what our limits are, when things are going to break, and how they break when they do. We’ve got more data on what was and wasn’t working, and how it was working (when it did) than we could have hoped for – and the sum of that data? We couldn’t scale to where we want to go with our current setup.

Recently we’ve had to make a lot of changes to our systems to handle the load. Those changes included rolling out Redis (a NoSQL key/value store and (sort of) database) and migrating from MySQL where it made sense. Other than one subsystem that gets used in our reporting systems, we’ve pretty much completed this migration and our load has been reduced considerably.

Sorry, I’m about to get all geeky, but if you have the time, take a look at Redis’ new “hash” types, they are perfect for some of the stuff we do. We also quite heavily use the Sorted Sets (zsets) and the regular Set types. I want to commend Salvatore Sanfilippo (@antirez) for all of his hard work on Redis, it’s truly one of the best key/value databases available.

So, back to the point of this post, lately we’ve been forced to scale, scale, scale and now that we’ve done enough to give us some headroom, we’ve put together a plan that’ll hopefully let us scale to the point that we can open things up the rest of the beta users that are still waiting to get in and eventually to the public as a whole.

If you care what those changes will be: we’re planning on continuing to use Redis for the long term, both as a cache and as data store, but also adding a cluster of servers that will use Cassandra for storing our tweets and while we’re still evaluating search options (spoiler alert!), likely ElasticSearch will be used to allow you guys to better search through your tweets. We’ll still use MySQL for managing some of the account stuff, like your watchlist. I know I am glossing over a lot of details, but this is the main stack we intend on using. As more details finalize and we start rolling these out, expect updates on how it’s going.

Thank you to all of the beta testers in there breaking things, and thank you for your patience as we work through our growing pains. For those of you still waiting to get in, we’re trying our best to get you in as soon as possible. Thank you for your patience as well.

P.S. Oh, and Salvatore, if you read this, when Redis Cluster is ready for testing, we want to be part of it. We are definitely going to need it.