The "5 Star" rating system is pretty standard when it comes to rating most products and services on most sites. The sites I frequent the most and look for these ratings are Amazon.com, Apple's App Store/iTunes and Netflix. However, ratings alone don't tell the whole story.
Don't judge an App by its rating alone
I recommended an app to a friend and later found out that he didn't get it because it only had a 3 star rating. After we talked and I explained to him that you can't go by the rating alone. What I told him was that you have to consider the fact that not everyone uses the same criteria when rating something. For example, the App Store's rating system was severely flawed at first because back when it first started you could rate an App that you had never actually installed or tried. In many cases people were rating Apps "1 Star" because they didn't like the price or it wasn't compatible with the touch or for some other reason that really didn't tell you anything about the App itself. Apple did fix that and now you can't rate an App unless you have at least downloaded it. However, the App Store's rating system is still far from perfect. I'd give it a 3 star rating (just kidding). There are a couple of potential problems that still exist. There are ratings and there are reviews and ratings. If you look at an App that you downloaded in the App Store on your iPhone or iPod touch you can "Review it" and of course give it a rating. If you go to your computer and look at that same App in iTunes on the iTunes Store, you can EITHER Rate it or Review and Rate it. So this means that you can give an App a 1 Star rating with no explanation what so ever.
Let's go back to your iPhone or iPod touch for a moment. If you delete an App right on the device itself, a dialog pops up and gives you the chance to rate the App you've just deleted. I'm going to go out on a limb here and guess that Apps that are rated during a delete process probably rarely receive a 5 star rating. How often are you deleting Apps that work perfectly that you absolutely love? So this again only tells one side of the story. Also there is no opportunity here to say why you're giving it the rating you're giving it or why you're deleting it? The people that have apps installed that love them aren't prompted at any point with a "hey we notice that you use this app a lot, would you like to rate it?" message. People that are happy don't tend to seek out a way to tell anyone. However, people that are unhappy tend to want to tell EVERYONE! For this "Rate upon deletion" feature to be fair, their should be a parallel feature that pops up the same dialog box say after you've used an app 25 times. "We noticed that you use this app a lot. Would you like to rate it?" The user of course would have the option to turn the feature off, but turning it off would turn off BOTH popups. No longer would you see the dialog when you delete an app either.
Reviews on Amazon.com
Usually before I buy something that I don't know much about, I turn to Amazon.com to see what others are saying about it? Once again the ratings don't tell me everything. I'm more interested in the reviews. Also I usually go for the negative reviews first and here's why: I'm already prepared to like the item. If I've done any research at all, I've usually figured out that this is something that I'm going to want. So I'm really just trying to see if there are any major problems with it. Does it have a high failure rate? Does it not work as advertised? I tend to ignore rants about pricing, delivery times, etc. because again the people that didn't feel like it costs too much or got it in a timely manner aren't usually posting reviews to say that. I look at what people "don't like about it" and decide for myself if those things are of any concern to me or not? So even if something on Amazon has a 2.5" star rating that doesn't necessarily mean that I'm going to not buy it. However, if a ton of people say the same thing about it breaking easily or falling apart or some other bad thing and it seems consistent, then I will really think twice (or three times) before buying.
Unfortunately, people can rate things on Amazon without actually owning them. Even my on iPhone Book 3rd Edition got a 1 star rating a while back from a guy who admitted that he didn't actually own the book and had not seen a single page of it. He rated the book 1 star solely because he didn't like the sample chapter that is automatically generated for Kindle users to download for free. A sample that the author (me) has no control over. He didn't like it because the sample didn't actually contain any of the content of the book. It was things like the acknowledgements, introduction, etc. However, someone looking to buy the book and judging it by the rating alone might think twice before buying it.
Another example here of a 2 star rating and review by someone who did NOT buy Joe McNally's "The Hot Shoe Diaries: Big Light from Small Flashes" because he didn't like the cover.
It's OK that he didn't like the cover and therefore exercised his right in not buying it, but to be able to rate and review something that you admittedly don't even own is just not right! By the way, Joe's book has an average rating of 4.5 stars with 110 5 star ratings!
Ratings on Netflix
In the case of a movie, I'm usually the opposite. That means that I could really care less about the reviews and honestly I take the rating a little more into account. Netflix seems to do a better job in comparing ratings of people with similar movie tastes. So chances are if I see a movie with a 3 star rating or less, I'm not going to like it. Sometimes I've ignored the 3 star rating and added the movie to my queue anyway only to be disappointed and kick myself after watching it. So unless it's something that I KNOW that I'm really going to like, I'll typically avoid movies with low ratings on Netflix.
In case you were wondering, yes I'd give Angels & Demons a 4!
How do you rate?
So that brings me back to the whole star rating thing and I'm curious how do you rate? Before you sound off in the comments below, let me give my formula for each category:
5 Stars – It's as close to perfect as it can be. Nothing is perfect and no matter how much I love a product, I can usually and pretty easily give you a list of things that I would like to see it do or have or change. But If I give it 5 stars that is the ultimate compliment and I'm quite happy with it. In the case of a movie, it's the kind of movie that I could immediately watch again. I liked it that much! It's also a candidate for one I'd want to buy on Blu-ray.
4 Stars – It's a really good product and works as advertised (this will be the norm). However, it's missing one or more things that would really take it to the next level. Or perhaps if it had a slight interface tweak it would earn the 5th star. Nonetheless I like it. In the case of a movie, I really enjoyed it. I was entertained. I'd recommend it to a friend, but it's probably a movie that I won't watch again. I'm good!
3 Stars – It's OK! It kinda works. There are several things that I would like to see added or changed, but it does work. In the case of a movie, it was barely OK. I'm not going to recommend it again and I wish it had ended sooner.
2 Stars – This product has several problems. I wouldn't review it here (unless I was asked to by the manufacturer) because I wouldn't be able to say very many good things about it and I wouldn't want to waste your time. I'm probably looking to return it. The only reason I would keep it is because I need to do what it does and there was no other alternative. In the case of a movie, it sucked! I barely got through it and I was checking my email several times while it was playing.
1 Star – The product should have never been shipped in its current state. The book should have never been written/printed. It's got MAJOR flaws and bugs and is pretty much unusable. It does NOT work as advertised. I'm demanding my money back. In the case of a movie, I walked out or turned it off. My time was more valuable to me than finishing it. I'm sorry that I paid to watch it or rented it. The Hitchhiker's Guide to the Galaxy comes to mind 🙂 I can probably count on one hand the number of times that I've rated something 1 Star.
The Bottom Line
While I've shared my way of rating with you, the problem is not everyone rates the same way. So you have to go beyond ratings and dig a little deeper. I've seen some of the silliest reasons for giving a product a 1 star or even a 5 star rating. When it comes to the App store, the 3 star rating has become the new 4 star rating as far as I'm concerned. So how do you rate?
Great points, Terry.
At a place like amazon, it also seems if people get a defect unit somehow, the ratings and reviews of that product gets very negative, instead of replacing the product (Amazon has a great return policy) and review a working unit.
There are examples of people not understanding how to use the product they just purchased, or didnt take the time to read the manual, and of course to them its crap, and they give it a low rating. DSLR’s comes to mind. I like to read reviews, and i too start with the negative ratings. They are on occasion a really fun read, but usually not very informative or useful.
I agree. I’ve seen a lot of these types of reviews for inkjet printers as well. But I have to say, getting defective units is exactly the kind of feedback I’m looking for. If I see three posts of people who all had to return their units, that’s valuable information. Additionally, if it’s not intuitive and requires reading a manual (…so last millennium), I’m interested in knowing as well. Maybe I’m willing to have a more sophisticated product that will take more of my time learning how to use it, but it’s nice to know if people found it unintuitive and difficult to use without reading the manual.
Reminds me of the App store’s (iTunes) ratings for the Sirius app. The ratings were flooded with very negative ratings due to the fact that Sirius wasn’t streaming the Howard Stern channels to the app.
Certainly, programming seems like a valid consideration, but it was obvious that most of the “reviewers” in this case gave the lowest rating possible based on this criteria alone, which offered readers absolutely no insight in how the app actually worked.
Exactly!
Sorry, but I can only give Angels and Demons a one or two star – it was so predictable.
I remember reading a post, think it was netflix, about how most users only really use the 1 star or 5 star rating. People pissed off or that have problems are highly likely to come back and rate the app/item. People who are extremely happy with the app/item will come back and rate. But most people either tolerate the problems or enjoy the app/item don’t spend the time to rate it. This results in a lot of 1 and 5 star ratings and very little 2-4 star ratings.
A thumbs up or thumbs down rating system seems to fit better, I like it or I don’t. Users don’t have to think about what a 3 star really means to them.
And let’s not forget the iPhone app raters who are totally clueless. How many times have you seen an app with a 1-star rating with a review that says “This is the most awesomest app!!!!1!!1!”? 🙂
Amazon’s reviews don’t seem to have this.
I started reading your blog recently after the nice comments you got from Joe McNally on his blog. Your blog is now on my list of “must read” blogs that I check daily. Thanks for doing it!
Your take on the subject of ratings is dead on. I had a side business at one point helping clients with customer feedback. Getting meaningful feedback is brutally hard and as you clearly pointed out the number rating system can be very misleading.
Really all you can hope to get out of the ratings numbers are a general idea as to what people think and by and large that works OK, assuming there is enough data. If there are 1000 ratings for something and the average rating is 2.5 stars then beware. However if there are 1000 ratings on two comparable items and one has 4.5 stars and the other has 4.3 then the star ratings don’t help you with the comparison and you just hope the rating system has associated comments to help you.
You ask about how people rate and my system is much like yours. However, just asking that question points out that other people think differently. For some people 5 stars is reserved for perfection (which never happens). Again, all you can hope for is that you get enough ratings that the outliers (like the perfectionists) are averaged out.
The democratization of reviews had really made it tough to find informed and insightful opinions on why or why I shouldn’t purchase something.
You see the same thing in the iTunes store. Uninformed rants by people that may or may not have bought an album or TV show about the pricing, the inability to purchase a specific track, or simply nonsense “look at me!” posts. Apple really needs to start limiting reviews to those that actually bought the item. Perhaps to encourage professional reviews, they could open unlimited reviews to anyone that has posted more than 25 that are rated “useful” by other users.
Oh, and don’t even get started on the practice of putting “click yes if you agree” in a nonsense review in an attempt to get to the top of the pile. There are certain phrases that should immediately send a review to a human for a review of its own.
One thing I tend to do is look at the 1 star reviews to see if the average is being pulled down for the people reviewing something that they didn’t buy or use. I will also look for things like if it breaks, etc., to see if there really is a problem.
For items with few reviews, I will also look at the reviewer’s other reviews. I’ve seen people review consistently low or high, and if I can find something that I’ve used and see that they rated it too high or low, I can figure how far off they may be on other reviews.
I enjoyed your article. You make a lot of good points. You’re exactly right about looking for bad reviews for products I’m thinking of buying (or vendors I’m thinking of buying from), and good reviews for movies.
You’re slightly more generous with your Netflix stars than I am. You seem to average out between 3 and 4 stars for movies you “liked.” I go with a more strict interpretation, which is why I appreciate that they have a pop-up tooltip to tell you what the stars mean (Hated, Didn’t Like, Liked, Really Liked, Loved). If I see a movie that was good, I enjoyed it for the most part, but I’m not likely to recommend it to anyone, that’s a 3 star movie in my book (a.k.a. “I liked it”). Your definition of a 3-star movie (“barely OK … not going to recommend it and I wish it had ended sooner”) sounds more like a “didn’t like it” to me, which Netflix defines as 2 stars. I’ve rated 2,952 movies. No point for saying that, just demonstrating how few “better things to do” I have. 🙂
P.S. I’m guilty of sometimes rating a product higher than I normally would, if I liked it and want to offset a bunch of “invalid” 1-star reviews (for things like “it didn’t arrive in time” or “it’s too expensive”.
I wish Netflix had an “it’s ok” rating between “didn’t like it” and “liked it”. Some movies are just ok. I don’t want to commit to saying I liked it, but then again I wasn’t offended enough to say I didn’t like it.
Donna, I totally agree!
I meant to mention that I like it when it displays the frequency distribution, and not just the average. iTunes and Amazon both do this. If I see something that has a deeply divided response, with lots of 1-star and lots of 5-stars, even though it has an average rating, I will usually check it out. I know that I’m likely to either love or hate it. Average ratings can be deceiving.
I find I rent a lot of “average” Netflix movies that people are deeply divided on (even though I have to comb through the reviews to get a sense of that, since Netflix doesn’t show distributions).
I’ve found most ratings and quality surveys are mostly a waste of everyone’s time and energy. They’re mostly focussed on how or why something doesn’t work. It doesn’t really offer any guidance to improve. You just know that it’s wrong and you have to try and figure out what to do to improve.
I came across a great rating system that’s productive and simple for everyone. I use it in my business and with everything else in my life. It’s been really successful.
Step 1:
Ask your customers, wife, kids, friends, etc to rate (what ever it is) out of 10 (or 5 doesn’t really matter)
Step 2: The most important step!
Then ask… If it’s not a 10, what would make it a 10?
You’d be surprised by the replies. The answers are more straight to the point and more focussed towards a solution. Try it out…