Report Finds That Amazon's Product Star Ratings Are Warped by Incentivized Reviews

Reviews with incentivized ratings cannot be separated from the other star ratings, skewing results

Report Finds That Amazon's Product Star Ratings Are Warped by Incentivized Reviews
Image: Pixabay
September 20, 2016

A new report has found that people who receive products in exchange for a review on Amazon are skewing results and artificially increasing the products' star ratings.

To comply with Amazon's guidelines, these reviews have to have disclaimers that generally say something along the lines of the following: "I received this [product] in exchange for an honest review."

Prospective buyers can choose to question the reliability of the review when they see the disclaimer. However, the star ratings on these reviews cannot be separated from the overall star ratings for the product, which skews results.

"Consumers have growing distrust and even disdain for incentivised [sic] reviews, especially when it seems every single one is a glowing five-star review," said ReviewMeta, the company that issued the report. It analyzed 7 million product reviews on Amazon to figure out the extent of the bias caused by such reviews.

"We found that reviews containing language that would indicate the reviewer received the item for free or at a discount in exchange for a review (incentivised reviews) on average rate the product .38 stars higher than reviews that did not contain this disclosure (non-incentivised reviews)."

The company isolated products that had at least 10 incentivized reviews and at least 10 non-incentivized reviews. This produced a sample of 609,766 reviews and a positive bias of 0.29 stars on average.

It found that those reviewers who provided a review in exchange for the product were 12 times less likely to provide a one-star review and were four times less likely to provide a review that was in any way critical of the product. More and more incentivized reviews are cropping up, in addition, comprising the majority of new reviews posted on Amazon since February.

Although the increase of a star point by 0.38 may seem like small potatoes, ReviewMeta's data indicates that even an increase of that size could be enough to take a product from the 54th percentile to the 94th percentile.

"Considering that the average product on Amazon is rated around 4.4 stars, a boost from 4.36 to 4.74 stars can mean the difference between a mediocre product and a top rated product," it said.

Many of the people who provide incentivized reviews are members of "review clubs," which put together and facilitate exchanges of products. Generally, they seem to review several different kinds of products, providing very high or five-star ratings to each.

"Incentivised [sic] reviews are a new and rapidly growing breed of review," said ReviewMeta. "While some people label them as 'fake', incentivised [sic] reviews don't technically deserve it because they are likely from real people who had a real experience with the actual product. However, the data still shows they still have a bias – despite their claims to the contrary."

The company did not analyze the similar Vine reviewing system also provided by Amazon. The online retailer did not immediately comment on the findings of the report.