By Bob Eagan
On June 25th, 2008
Recently, we needed to implement a rating system for a client’s application. The decision came after the client agreed that the results would effectively be useless due to the potential for manipulation by users but the end goal of giving warm fuzzies to visitors is what was more important. Unfortunately, since transactions aren’t processed or recorded through the site, we can’t limit who posts the feedback to actual, relevant purchasers. (Imagine if Craigslist provided user feedback profiles.)
The unfortunate “solution” we landed on involved having registered users fire an email to their customers. Their customers would follow a link with a unique hash that allowed them to leave feedback about their transaction with the user. Clearly, this has plenty of room for abuse. Savvy users would simply create several throw-away email addresses and send themselves links. Less savvy users would email their friends and have them give glowing reviews. Either way, it would take an honest user to send to actual clients and even then, no user is going to send a review request to someone they just pissed off. And again, since the transaction isn’t completed on our side, we can’t simply trigger a feedback request. So you’re looking at about a 0% usefulness factor, either way you slice it. We’ll get into some of the pitfalls with ratings systems and a great piece on this by Boxes and Arrows, after the jump.
One of the sites that I just recently added to my feed for regular reading is Boxes and Arrows. Per their site’s about page:
Boxes and Arrows is devoted to the practice, innovation, and discussion of design; including graphic design, interaction design, information architecture and the design of business.
Normal blog-type readers will find themselves overwhelmed with information; they present themselves more as a journal than a blog. I assure you though that it is well worth your time to read through some of their very well written articles as they’re written by some very talented people.
The article in particular that caught my eye last week had to do with the exact problem of ratings and reputation systems that we are struggling with currently.
The authors clearly define the reasons behind creating reputation and ratings systems and also explain that how you structure your rating criteria will effectively mold the behavior of your user-base (assuming you haven’t left it wide open with holes for abuse).
They walk through some real examples from popular and not so popular sites to showcase the good, the bad and the ugly with some of the systems currently in use. It gives a great look at some of the common problems that need to be addressed and overcome in order to have a truly effective rating system. For example:
- Reliability of people giving ratings—How do I know that I can trust that the review by User01487129871 is accurate and unbiased or that he and I have similar interests when it comes to that particular product or service? Some sites like Digg or NetFlix address this by allowing for linking to other users explicitly as “friends”. There is also the potential for implicitly linking users based on their history within the site (e.g. rating products/services similarly).
- Ambiguous ratings—Some sites ask for ratings just to say that they have ratings. I have no idea what the difference between giving this product 3 stars versus 4 stars is and neither do the people looking at the reviews if there is no option to provide additional feedback on what I rated as I did. But as the article points out, products and services have many different characteristics and to simply group them all together removes the usefulness from the rating.
- Inappropriate criteria and consistency—On sites with a wide range of products or services, it is difficult to come up with rating criteria that applies universally to all of them. In these cases it seems as though reviewers are trying to fit square pegs in round holes and end up dirtying the results of the review. Additionally, where one person may review a product based on its price another may be looking at a specific feature that was lacking.
These are just a few of the issues addressed in the journal article; they also provide some great visuals and more in-depth discussion of those topics (so go read it!). We’d also love some feedback on your own difficulties or successes you have had with rating and review systems on your applications or systems you have used.