Are reviews broken?
A mobile app developer was frustrated recently by receiving a one star review for their application.
The reason it frustrated them was that the review said the app had been perfect until the latest release. The developer was left asking why didn’t the user post a five star review while it was working well and only posted a one star review when things went wrong?
To “recover” the impact of the fours stars that were lost they needed to get a significant number of high scoring reviews, depending on the number of reviews they already had, to offset the effect on the average caused by that one review.
Like most people, when I look to purchase something on-line I often rely on the reviews to help make my choice. This means that reviews can have a huge impact on the success and the sales of items.
This in turn means that there is a clear incentive to skew the reviews in favour of your product and against those of your competitor.
On Amazon I check the average score and the number of reviews an item has so I can narrow down the items I am considering. I then read a few of the “most recent” reviews - not the “top reviews” which is the default sort order. I also check independent reviews where possible on other sites.
However, maybe using an alternative average could be a better approach?
Using the mode (the score with the highest number of votes) or the median (the middle score of all the votes) could be a more reliable approach. Or perhaps a combination of them in some way?
Amazon provide a break down of the number of ratings of each score and you could calculate these yourself, however it does give a good indication of the general ratings.
I’m sure there are ways to “game” both of these however in both cases it would potentially require a large number of votes to shift the average and the impact of outlying scores would be reduced.
The current approach to scoring reviews has it’s issues and, with the growing impact of AI, it looks like it will become a battleground to gain attention, sales and market share - similar to what we have seen with adwords and alike - and the potential rewards could be substantial.