By Chris Cardinal
On December 3rd, 2008
I first read about Facebook having lost some users’ notification settings on TechCrunch four days ago. This was worrisome to me, but I got sick over the weekend and didn’t have a chance to write about it. Then I got my very own email from Facebook telling me the same: they’ve lost my notification settings and if I’d be so kind as to reset them, and that they apologized for the inconvenience.
Facebook needs to publish a public post-mortem on this, as soon as humanly possible. When any data disappears from the cloud, no matter how innocuous, it calls into consideration serious questions of trust and competence. I’ve trusted Facebook for a long time. The engineers who have built it have done an amazing job at making sure things scale brilliantly, at cobbling together various pieces of technology and contributing their own back to the community to make the site highly available and without many of the horrible growing pains MySpace experienced, when Tom would send a message telling everyone bulletins will be down and to please not email him.
But as any developer knows, data doesn’t just disappear. Something happened. A commit was made, a query was executed, redundancies weren’t properly established, or deltas weren’t present—a series of wrongs likely had to occur for Facebook to lose this data forever. Now, Facebook is free. I don’t pay them for their services, and they don’t owe me an SLA. That issue becomes irrelevant the more Facebook asks me to share with them, and the more they attempt to assure me my privacy is intact and of the utmost importance to them. If my notification settings can be completely lost and reset, what’s to say it couldn’t have been my privacy options? Further, do privacy settings regress safely, to the most secure levels? Or do they regress, in the absence of all other data, as one may expect, to an open season on my profile, my pictures, my notes and everything else Facebook has asked me to share with them, with the knowledge that I could wall some of that data off?
And conventional wisdom is simple enough: Don’t post to Facebook that which you wouldn’t feel comfortable sharing with just about anyone. But Facebook is asking for us to trust them more than that. And in doing so, they’re making the application considerably more valuable to me. I can enjoy the site and share certain things with my friends on Facebook that I might not want readily apparent when a client Googles me. This isn’t to say that it’s not a risk, or that I don’t still temper myself to a point. But when I’m given the functionality to wall things off, I expect them to work. And if those settings are maintained just one table over, so to speak, from my notification settings… well then, we dodged a bullet this time, didn’t we? In fact, of all the data stored on Facebook, this was probably the most innocuous and least impactful data that could be lost.
Several months ago, Amazon’s distributed file storage system, S3, suffered a severe outage that lasted for hours. Now, the situation is a bit different: Entire businesses rely on Amazon S3 to be functioning for their livelihoods. Outages mean lost income and lost trust. So Amazon did what Facebook absolutely must do: they issued a full post-mortem that explained their engineers’ findings and failings, their root-cause analysis, what caused the problem to cascade across their network, and most importantly, the measures they’ve taken to ensure that, to the best of their ability, this would never happen again.
This is the only way to get users back on the road to trusting you, and Facebook needs to deliver more information to us than just “we apologize for the inconvenience.” They need to let me know exactly what was the cause of the problem, what failed, and what they’ve done to make sure that, say, next time, all of my messages won’t be converted to open-wall posts on my account. Because this time, it was notification settings.
They dodged a bullet. Most users won’t really think twice. But those of us who know what it takes for a system like this to fail in such a manner… we’re worried. Facebook’s not perfect, the cloud grows more ethereal, and we’re left to hope beyond hope that what data we’ve chosen to share stays within the walled gardens we’ve established for it. Help build back some of that trust, Facebook, and let us know what went wrong.
Posted in: Rants