Are App Store ratings trustworthy?

Blog posted by Ste Pickford on Wed, 03 Aug 2011
Subject: Magnetic Billiards: Blueprint

Earlier this week we put out a press release announcing that we'd received 100 5-star ratings, and nearly 70 glowing reviews, for Magnetic Billiards: Blueprint on the UK App Store.

John and have been absolutely blown away by this response - both the universally positive reaction to the game and the effort that such a large proportion of our players have gone to in rating and reviewing the game. Our downloads have been OK, but not amazing yet, so the number of App Store reviews we've got really stands out.

Predictably we've been comparing our ratings to other games released around the same time, and been sneakily delighted by how much higher our ratings have been. Only games that have been out for a much longer time than us seem to have more ratings, which seems fair enough.

It was while basking the warmth of a positive critical response, and wondering how exactly to turn ratings into dollars, that my good mood was punctured by an email from somebody offering their services to "improve our visibility" on the App Store.

I was suspicious, but curious.

Curious because I've been asking pretty much every iOS developer I know for tips and advice (and every one of them, to a man, has been brilliantly forthcoming and helpful), so I'm always ready to listen to anyone who might be able to help us reach a wider audience for our game.

Suspicious because since releasing the game and firing out press releases to every review site I can find, I've soon learned that pretty much the only review sites who ever reply to emails are the ones who come back with a price list for the different reviews they offer. (Yes, really! I'd pay for straight advertising for a game, but I'd never pay for a review.)

Well, the "App Store visibility" guy emailed me straight back with his price list:

$100 for 100 App Store ratings and 20 written reviews

$200 for 200 App Store ratings and 45 written reviews

$300 for 300 App Store ratings and 70 written reviews

Woah!

Is this the kind of thing that goes on?

We spent months and months polishing our game, beta testing it, tweaking it, fixing bugs, responding to criticism and generally doing everything we could to make the game as good as possible, and our reward was a very hard-earned 100 5-star ratings. Someone else can throw any old app out there, then just drop $100 to get the same thing?

If App Store ratings and reviews can be paid for, can they be trusted?

Needless to say I didn't take this person up on their generous offer. And looking around the App Store, there don't seem to be many games with 100 or more ratings that don't look like they've been around long enough to deserve them. Maybe nobody has ever taken this guy up on his offer? And to be fair, he didn't promise *good* ratings or *good* reviews either.

It would be nice to think that Apple detects this kind of thing, and weeds it out. Maybe they do, and maybe apps that generate 100 5-star ratings overnight are pulled from the store.

Yikes, I hope our phenomenal critical response doesn't mean that people think we paid for those ratings!

» Magnetic Billiards: Blueprint - Now Available on the App Store!

Tags: indie, publishing, cheating, reviewing

Follow up post: Spong news story

Predecessor post: Top of the world (a small bit of it)

View the softography page for Magnetic Billiards: Blueprint »

Permalink to this post: http://www.zee-3.com/pickfordbros/blog/view.php?post=554

It's a horrid practice but pretty easy to come across everywhere else (in digital anyway). It was only a matter of time before it shifted to app reviews.

There are offshore companies that you can hire who will watch You Tube videos over and over, play your Flash games and sign up as Facebook fans for cash.

This means that you can go back to your boss/client with fantastic reviews and results to show that your work performed.

Comment by Guest rafrasilecs, added Wed, 03 Aug 2011 09:56:21 GMT

>It would be nice to think that Apple detects this kind of thing, and weeds it out.< Have you contacted Apple to see if they do, and if not why not? After all this is a business with money involved.

Comment by Guest Joyce Gill, added Wed, 03 Aug 2011 11:09:31 GMT

If you spend a bit of time on the 'coder for hire' boards, you'll see endless requests for code to create fake accounts/post reviews for various services.

The XBLIG scandal from earlier this year, where it was 'proven' that certain developers were paying to have their competition 'rated down', didn't even result in a comment from Microsoft, that I'm aware of.

I'm sure some people will say "it's the cost of doing business" tho and it could be worse - over half of my (and lots of other people's) reviews on the Android Market disappeared last week - finally reappearing sometime yesterday with no explanation or apparent cause!!

Comment by Guest John Peat, added Wed, 03 Aug 2011 12:14:01 GMT

Ste, you must be kidding. You've never seen this go on? I see it all the time. Look carefully for; 1. Really crappy grammar. e.g. "this is really addicting" 2. Any mention whatsoever of being comparable to Angry Birds.

Comment by Guest RobHayden, added Wed, 03 Aug 2011 12:43:54 GMT

I'm not surprised, but cheating on the popularity charts would be more useful as they're the only thing that actually matters.

Do you know how many downloads you need in a day to be #1 in Norway? About 10 should do it.

Also look out for the 'Oh, yeah, we know the guy who chooses the staff recommended apps' scam, a modern version of rainmaking.

There's no effective way to market iPod software. Unless things have changed recently, advertising is pointless and the only way to get more sales is to be featured on some kind of chart.

Publishers are always looking for a way to use money to give them a larger chance of success, so it's not surprising that people provided this sort of thing.

Comment by Guest CdrJameson, added Wed, 03 Aug 2011 13:19:06 GMT

Nobody is ever surprised after the fact!

We were surprised. We're new to the app store - I'm not sure how were were meant to have known about it.

We've got a few reviews comparing our game to Angry Birds and being really addicting but we haven't paid anyone or even asked anyone to review the game.

Comment by JPickford, added Wed, 03 Aug 2011 13:27:02 GMT

As a possible example of how these shenanigans might work in practice, check out the reviews for "8.0 MPX Digital Camera Simulator". Not only does the app have quite a lot of oddly similar-sounding 5-star reviews, those 5-star reviews have all been flagged as helpful so that they rise above the many 1-star reviews trying to warn shoppers that the app is a "scam", a "con", etc. When the warnings become too conspicuous, an update is published, the slate is cleaned, and a fresh batch of enthusiastic 5-star reviews appears. Neato!

Comment by Guest Colman, added Wed, 03 Aug 2011 14:30:20 GMT

I know its horrid Ste that it goes on and it goes on everywhere in the internet as whilst there is money to make on upping a games profile it will happen. This is why I wish there was more legitimate sites out there like ign and games radar with easier focus on the iOS games... even a You tube channel with unbiased reviews dedicated to digital content games where to buy and the like. Have you ever considered going down Xbox live and PSN, GOG.com route as this is more traditional ...

As for advertising your game for free I do think You Tube is a great tool for advertising your games..but you do need to respond and check it more regularly and become more engaged in the You Tube community to build this important communication avenue properly as well. I personally find Vlogs as they call it far more engaging than Blogs as it allows video and speech to give a lot more interest and so exposure. This is the truth unfortunately that writing a game is only half the battle you then have to place considerable effort in maintaining these different digital streams to garner interest in your products as it is such a saturated market without it even a great game like yours will be lost in the sea of titles out there. Hence why many sadly cheat and throw money at making their game stand out.

Comment by TheDrisk, added Mon, 22 Aug 2011 22:37:01 GMT