I came across a great rating system that’s productive and simple for everyone. I use it in my business and with everything else in my life. It’s been really successful.

Step 1:
Ask your customers, wife, kids, friends, etc to rate (what ever it is) out of 10 (or 5 doesn’t really matter)

Step 2: The most important step!
Then ask… If it’s not a 10, what would make it a 10?

You’d be surprised by the replies. The answers are more straight to the point and more focussed towards a solution. Try it out…

I find I rent a lot of “average” Netflix movies that people are deeply divided on (even though I have to comb through the reviews to get a sense of that, since Netflix doesn’t show distributions).

You’re slightly more generous with your Netflix stars than I am. You seem to average out between 3 and 4 stars for movies you “liked.” I go with a more strict interpretation, which is why I appreciate that they have a pop-up tooltip to tell you what the stars mean (Hated, Didn’t Like, Liked, Really Liked, Loved). If I see a movie that was good, I enjoyed it for the most part, but I’m not likely to recommend it to anyone, that’s a 3 star movie in my book (a.k.a. “I liked it”). Your definition of a 3-star movie (“barely OK … not going to recommend it and I wish it had ended sooner”) sounds more like a “didn’t like it” to me, which Netflix defines as 2 stars. I’ve rated 2,952 movies. No point for saying that, just demonstrating how few “better things to do” I have.

P.S. I’m guilty of sometimes rating a product higher than I normally would, if I liked it and want to offset a bunch of “invalid” 1-star reviews (for things like “it didn’t arrive in time” or “it’s too expensive”.

For items with few reviews, I will also look at the reviewer’s other reviews. I’ve seen people review consistently low or high, and if I can find something that I’ve used and see that they rated it too high or low, I can figure how far off they may be on other reviews.

You see the same thing in the iTunes store. Uninformed rants by people that may or may not have bought an album or TV show about the pricing, the inability to purchase a specific track, or simply nonsense “look at me!” posts. Apple really needs to start limiting reviews to those that actually bought the item. Perhaps to encourage professional reviews, they could open unlimited reviews to anyone that has posted more than 25 that are rated “useful” by other users.

Your take on the subject of ratings is dead on. I had a side business at one point helping clients with customer feedback. Getting meaningful feedback is brutally hard and as you clearly pointed out the number rating system can be very misleading.

Really all you can hope to get out of the ratings numbers are a general idea as to what people think and by and large that works OK, assuming there is enough data. If there are 1000 ratings for something and the average rating is 2.5 stars then beware. However if there are 1000 ratings on two comparable items and one has 4.5 stars and the other has 4.3 then the star ratings don’t help you with the comparison and you just hope the rating system has associated comments to help you.

You ask about how people rate and my system is much like yours. However, just asking that question points out that other people think differently. For some people 5 stars is reserved for perfection (which never happens). Again, all you can hope for is that you get enough ratings that the outliers (like the perfectionists) are averaged out.

Amazon’s reviews don’t seem to have this.

A thumbs up or thumbs down rating system seems to fit better, I like it or I don’t. Users don’t have to think about what a 3 star really means to them.

Certainly, programming seems like a valid consideration, but it was obvious that most of the “reviewers” in this case gave the lowest rating possible based on this criteria alone, which offered readers absolutely no insight in how the app actually worked.