HTMList.com, A Web Development Blog by Synapse Studios

Continuing their trend of releasing substantial features and additional services in their web services portfolio on a regular basis, Amazon announced this week the availability of versioning and multi-factor authentication across their Simple Storage Service (S3) property.

How S3 Versioning Works
Versioning is a critical feature many developers had requested as data stored on S3, while maintained in triplicate across the S3 file-system automatically, is still vulnerable to sweeping delete operations by developers, errant scripts, or other causes. Moreover, developers had to manually version changing files if they wanted to preserve the ability to roll-back to an earlier revision or undo a “delete”. In any event, a lot of custom code had to be created to replicate these behaviors, and most solutions weren’t particularly graceful.

Versioning is easy to enable and brilliant for Amazon: make a simple API call to enable versioning for an entire bucket and all changes to the files within that bucket will preserve versions. Moreover, deletions of files through the typical DELETE API call will simply create a delete marker, making the file unavailable for all intents and purposes, but preserving its presence and all of its versions in the S3 filesystem, until its permanent deletion is called for. It’s brilliant for Amazon because you pay the additional storage fees for each version and for each “deleted” file that hasn’t been permanently destroyed.

This means that versioning can be easily enabled on any existing S3 implementation with out breaking compatibility with existing code. Naturally, developers will need to rewrite their delete calls to make use of the version delete options, but for an application with little-to-no deletions, this is a feature implemented with a single API call.

Amazon also introduced Multi-Factor Authentication. MFA works with security code-generating key fobs that act as a physical token one must have in order to perform an action. In this case, if you enable MFA, you need the token to log in and manage your AWS account, and you can also optionally require the token to commit a permanent deletion of a file version. For the uninitiated, it’s considered multi-factor as you need two types of authentication to login or make deletes on your account. (Your AWS username and password for account management and your AWS public/private key for API calls each make up the other half of the multi-factor equation with the MFA code.)

The S3 Trash Bin
Using MFA, it becomes possible to create a highly robust and secure safety net that prevents the wholesale deletion of an S3 account’s contents without first enforcing an additional review step. After enabling versioning and MFA-Delete mode for a bucket, a developer’s application will continue to function to the end-user as expected. This means very little rewriting of code needs to be performed to get these features up and running. A simple trash bin script could then be created which would identify all files whose current version is a delete marker, and present them to an admin. Since permanent deletions require the MFA code to be appended to the API call, only the individual with physical access to the code (which recycles every 30 seconds) would be able to commit the delete operations necessary. After reviewing the files queued for permanent deletion, they’d enter the code and the script would be sent off to the races.

Naturally, this still leaves some room for error: the script that commits the permanent deletes would need to be scrutinized to ensure that it itself is bulletproof, but beyond that, this should protect against nearly all accidental deletions, and establish a command and control hierarchy for an organization. It’s important to reduce the sheer number of individuals who have complete deletion privileges for myriad reasons, and this helps accomplish that. The MFA key fob is available for all of $13 and you’re off to the races. You simply input your serial number on the AWS site, and Amazon’s servers knows what the code it’s generated should be based on the time and a seed that’s private to them.

More on Amazon AWS Multi-Factor Authentication
S3 Versioning | Amazon Developer’s Guide

Popularity: 2% [?]

Tagged with: amazon, amazon aws, amazon s3, deletions, mfa, versioning

Posted in: Cool Stuff, Development

Amazon has released a rather comprehensive write-up on their post-mortem analysis of why Amazon S3 went down last week. The S3 servers use a gossiping protocol to determine system states, including what servers are available and the status of the nodes across the network.

A single bit corrupted in several of these gossips such that they were still intelligible but reflecting inaccurate data about the system state. These propagated through the network (much like a virus, really) and caused most of the servers to spend most of their time gossiping or failing to complete the gossip; if the gossip doesn’t complete, the server can’t/won’t send its data.

While Amazon MD5 checksums data in containers to ensure its integrity as its being transmitted, they weren’t doing this on their gossips. They’ve since established several new practices to attempt to ensure that a problem like this won’t cause a failure across the entire system, including better failure handling with gossips and faster restoration when nodes do go down.

They end their missive simply enough, owning up in a way I give them credit for:

Though we’re proud of our operational performance in operating Amazon S3 for almost 2.5 years, we know that any downtime is unacceptable and we won’t be satisfied until performance is statistically indistinguishable from perfect.

“Statistically indistinguishable from perfect” is a rather poetic phrase, and I’d like to think we strive for that over at Synapse Studios. But my stats-masters programmer would just mock me.

Read their full statement here.

Popularity: 7% [?]

Tagged with: amazon, amazon aws, amazon outage, amazon s3, failboat, gossip, post-mortem, s3, statistics

Posted in: Tech News

Yesterday, we were Slashdotted. We had turned off caching the night before to test something with our sidebar, and didn’t figure /. would pick up Edgar’s piece, so we left it off. Naturally, the traffic crush hit and HTMList became unavailable at around noon, PST.

I’ll go into some more detail on what we did to bring things back up and keep the site available in another post, but suffice to say, we moved our background image, subscribe/RSS feed image and Synapse Studios logo over to S3 to help take some of the load off. And of course, S3 went down today.

Read More »

Popularity: 5% [?]

Tagged with: amazon aws, amazon s3, amazon sqs, amazon web services, cloud computing, cloud problems, scalability, sla

Posted in: Tech News