HTMList.com, A Web Development Blog by Synapse Studios

Continuing their trend of releasing substantial features and additional services in their web services portfolio on a regular basis, Amazon announced this week the availability of versioning and multi-factor authentication across their Simple Storage Service (S3) property.

How S3 Versioning Works
Versioning is a critical feature many developers had requested as data stored on S3, while maintained in triplicate across the S3 file-system automatically, is still vulnerable to sweeping delete operations by developers, errant scripts, or other causes. Moreover, developers had to manually version changing files if they wanted to preserve the ability to roll-back to an earlier revision or undo a “delete”. In any event, a lot of custom code had to be created to replicate these behaviors, and most solutions weren’t particularly graceful.

Read More »

Tagged with: amazon, amazon aws, amazon s3, deletions, mfa, versioning

Posted in: Cool Stuff, Development

I first read about Facebook having lost some users’ notification settings on TechCrunch four days ago. This was worrisome to me, but I got sick over the weekend and didn’t have a chance to write about it. Then I got my very own email from Facebook telling me the same: they’ve lost my notification settings and if I’d be so kind as to reset them, and that they apologized for the inconvenience.

Facebook needs to publish a public post-mortem on this, as soon as humanly possible. When any data disappears from the cloud, no matter how innocuous, it calls into consideration serious questions of trust and competence. I’ve trusted Facebook for a long time. The engineers who have built it have done an amazing job at making sure things scale brilliantly, at cobbling together various pieces of technology and contributing their own back to the community to make the site highly available and without many of the horrible growing pains MySpace experienced, when Tom would send a message telling everyone bulletins will be down and to please not email him.

Read More »

Tagged with: amazon, amazon s3, cloud, data, data loss, facebook, fail, notification settings, post-mortem, trust

Posted in: Rants

Amazon has released a rather comprehensive write-up on their post-mortem analysis of why Amazon S3 went down last week. The S3 servers use a gossiping protocol to determine system states, including what servers are available and the status of the nodes across the network.

A single bit corrupted in several of these gossips such that they were still intelligible but reflecting inaccurate data about the system state. These propagated through the network (much like a virus, really) and caused most of the servers to spend most of their time gossiping or failing to complete the gossip; if the gossip doesn’t complete, the server can’t/won’t send its data.

While Amazon MD5 checksums data in containers to ensure its integrity as its being transmitted, they weren’t doing this on their gossips. They’ve since established several new practices to attempt to ensure that a problem like this won’t cause a failure across the entire system, including better failure handling with gossips and faster restoration when nodes do go down.

They end their missive simply enough, owning up in a way I give them credit for:

Though we’re proud of our operational performance in operating Amazon S3 for almost 2.5 years, we know that any downtime is unacceptable and we won’t be satisfied until performance is statistically indistinguishable from perfect.

“Statistically indistinguishable from perfect” is a rather poetic phrase, and I’d like to think we strive for that over at Synapse Studios. But my stats-masters programmer would just mock me.

Read their full statement here.

Tagged with: amazon, amazon aws, amazon outage, amazon s3, failboat, gossip, post-mortem, s3, statistics

Posted in: Tech News

Yesterday, we were Slashdotted. We had turned off caching the night before to test something with our sidebar, and didn’t figure /. would pick up Edgar’s piece, so we left it off. Naturally, the traffic crush hit and HTMList became unavailable at around noon, PST.

I’ll go into some more detail on what we did to bring things back up and keep the site available in another post, but suffice to say, we moved our background image, subscribe/RSS feed image and Synapse Studios logo over to S3 to help take some of the load off. And of course, S3 went down today.

Read More »

Tagged with: amazon aws, amazon s3, amazon sqs, amazon web services, cloud computing, cloud problems, scalability, sla

Posted in: Tech News