On “no scores.” It’s been tried, in print (CGW in the US, I think) and online (somewhere) and in both cases readership dropped. When people were used to scores, they moaned when they were removed. Mathematically proven. Sorry.

Nope, as convincing a mathematical proof as “somewhere, I think” is, I’m still not buying it. “Sorry.”

You say that CGW tried dropping scores, fans complained, and sales dropped. First, what don’t videogame fans complain about? That’s what we do. Second, was there ever a period over the last decade of CGW’s existence that sales didn’t drop? The far more likely scenario is that it was much easier for the publishers to say “bringing back scores will bring back sales” than to say “our business model is doomed, because print media about digital entertainment is becoming increasingly irrelevant.”

But they aren’t mainstream. Mainstream punters have probably never heard of those sites because they don’t care about how games are made, they just want to buy a good one. That means they want a score, and they’ll turn to whoever gives them one.

Claiming that Joystiq and Kotaku aren’t “mainstream” is ludicrous. In the US, at least. Especially if you’re claiming that NeoGAF is a better representation — I work in the industry, and I hadn’t even heard of that forum until a year ago. People are much more likely to have seen Something Awful and even Penny Arcade’s forums than NeoGAF.

It is true that Joystiq & Kotaku are only tangentially “review” sites, and they definitely didn’t build their audiences that way. And the big sites are still 1Up, Gamespot, and IGN. But I say you’ve got the cause and effect reversed: those aren’t popular because of their scores, they’re popular because they’ve always been there. The scores are something that people are just used to.

Publishers and reviewers are much more attached to scores than fans are. If they want to keep using them, that’s their prerogative. I just wish they’d be up-front about it, instead of claiming that fans insist on them. If you’re a reviewer and you really don’t like scores as much as you claim, then stop using them. Fans will adjust.

Joystiq and Kotaku are red herrings. The cynic in me suggests they don’t do scores because they don’t want to have the inevitable rows/threats/blacklisting that comes when a publisher is unhappy with a score. More charitably, they didn’t built their traffic on review scores (unlike a great many sites and print magazines)and a big chunk of their readers are the “no scores!” brigade in any case.

Good for them. But they aren’t mainstream. Mainstream punters have probably never heard of those sites because they don’t care about how games are made, they just want to buy a good one. That means they want a score, and they’ll turn to whoever gives them one. And they won’t document this process on NeoGAF because they don’t know it exists.

Do sales or subscriptions or site hits dramatically drop when you don’t use them, and can you attribute that directly to review scores and not any of the other 1000 capricious things that game fans will latch onto or reject?

Are there dozens of e-mails or forum posts complaining about the lack of a score?

From what I understand, this is exactly what happens whenever anyone tries it – they get inundated with letters from people who don’t want to read the review and just want the score.

Look at 1UP, as soon as it went to letter grades – NeoGAF insists on trying to convert the letter grades to numbers so it could compare it to the other sites that still use numbers, despite pretty much every reviewer on that site responding to what an A- is equivalent to with “it’s better than a B+ but not good enough to get a straight A.”

And I think NeoGAF is a good illustration that the drooling fanboy demographic is alive and well.

I have no idea how the blogs are getting away with it – maybe because they’re blogs, and so whenever anyone complains they can respond with a hearty “fuck you and the horse you rode in on” which would be the first recorded instance of the blogs doing the enthusiast press a favour.

You missed the main reason for scores: readers like them. The kind who posts on dedicated forums often denies this, but pretty much every attempt to do without them has failed. You can see it in the responses online – when a review comes out, it’s not the individual lines and criticisms that get passed around, unless it’s a fan-forum doing a line-by-line number, it’s the number at the end.
[...]
(I’m not a fan of scores in general, TBH [...] My favourite system was probably the old Daily Radar one, with games rated as Direct Hit, Hit, Miss or Dud [...] Sadly, it didn’t take off.)

You know, I keep hearing this, and I’ve been hearing it for years, and I’m still no closer to getting an idea of why we’re supposed to accept it, exactly. I’ve never come across anyone — in person, online, or in print — who really liked review scores — but people still insist that life doesn’t work without them. This is probably going to make it sound like I’m skeptical, but: I’m extremely skeptical. When I hear everyone describe something as a necessary evil, my first question is why is it necessary?

Basically, my question is the same as it is to Sessler: who’s making the call, and what is it based on? “Readers like them.” How do you know? What, exactly, made it not work for Games Radar? Do sales or subscriptions or site hits dramatically drop when you don’t use them, and can you attribute that directly to review scores and not any of the other 1000 capricious things that game fans will latch onto or reject?

Are there dozens of e-mails or forum posts complaining about the lack of a score? If so, why do they get more weight than the hundreds of e-mails and forum posts complaining OMG INTERNET BIAS! because Fallout 3 got a 98 and GTA4 got a 99? Sure, people on forums will list the score of a review instead of choosing quotes from it: that’s the effect, though, not the cause. If you put a number on a review, people (including me), are going to use it. If you don’t, people are actually going to say “positive” or “negative” or start pulling quotes from it.

It sounds to me like there’s this assumption of a highly lucrative drooling fanboy demographic that nobody particularly likes having to cater to, but “Hey, what are you gonna do?” And again, I’m skeptical. Joystiq and Kotaku have started doing reviews without scores, and although they’re not known for their reviews, they’re about as mass-market as you can get in videogames. Hell, movies have a much much larger and more varied audience than videogames, and Siskel and Ebert’s popularity didn’t suffer from their thumbs-up/thumbs-down scale.

That said, I absolutely agree with you on Metacritic scores. It’s a great aggregator for finding reviews, but it’s always baffled me that people use the aggregate score for real decisions. [...] you don’t get much of a feel for anything except the prevailing mood and what each source considers pithy.

I’d say you don’t even get that, because “each source” doesn’t enter into it. There is absolutely no room for nuance with Metacritic. You get everybody jumping onto the Latest Big Thing, with dozens of sites giving GTA4 or Fallout3 a perfect score, so the heavily-hyped games get even more hype. And everything else just settles into a yellow-green morass, just like any other bell curve.

Plus — and I’m mentioning it again just because it really bugs me both as a game developer and as someone who has a basic understanding of math: they weight sites like Gamespot, 1Up, IGN, etc. more heavily than the others, without indicating which ones are weighted, how much weight they’re given or their criteria for choosing.

People will stay in a 2 star hotel, or eat in a 3 star restaurant. Getting a 40% or a 60% on a test, however, is either failing or hanging on by a thread. Saying “its math, idiot” is missing the way that stars, percents, and x/10 scoring are traditionally used in different contexts.

I understand perfectly well what Sessler was trying to say. I’m still saying that it’s bullshit. He — and anyone else who says that the big problem with Metacritic is that it takes scores out of context — is claiming that it’s perfectly fine when a reviewer takes his thoughts and impressions of a game and reduces it to a number, but it’s horrible when Metacritic does the exact same thing. If you want all the subtle nuance of your review to be preserved, then it’s simple: don’t slap a number on the end of it. If you’re deeply offended that someone would respond “it’s math, idiot,” then it’s simple: don’t use math, idiot.

If you want your 2/5 stars to mean the same as the letter grade “D,” then use the letter grade “D.” If your gifts at critiquing a game are so finely-honed that you feel the need to start using D- or B+ or 2.5 stars or a 1-10 scale, then don’t go to San Francisco and bitch that Metacritic expects people to understand the difference between a 73 and a 74. And if you have to put a written description of what the score means every time you put up a 7.5 or an 84% or “Awesome!” or “Editor’s Choice!” then get a fucking clue and realize your score numbers aren’t working.

Comparing it to a hotel or restaurant is flawed, because price doesn’t enter into the equation: I’ll stay at a 2-star hotel because I know that it’s much cheaper than a 3-star, but still livable. But I have to pay the same $60 for F.E.A.R. 2 as for Skate 2, so 84 vs 77 doesn’t mean that much to me. And comparing it to a letter grade is flawed, too, because the person giving the grade has to establish a somewhat objective standard that is the same for every student being graded, and list what you got right and wrong.

Reviewers have incorrectly conflated their role as consumer advisor with that of critic for so long, that everybody just accepts it. And when you point out that it’s bogus, people give the knee-jerk “you just don’t appreciate the context, man!” But as a consumer advisor, you’re just giving one piece of advice: buy or don’t buy. (Or if you insist: buy, read the review first, or don’t buy). If you’re freaked out that the notion that 2/5 is a “buy” on your scale, but it’s a 40% “don’t buy” on somebody else’s, then how about this: say “buy” or “don’t buy.”

When you’re critiquing the game, that’s where the nuance and context comes in. And that’s what your writing is for, not some number. Why is this 74 and not 73? Read the review. How come this is 2 stars and not 3? Read the review. Why is Fallout 3 an A- but GTA4 an A? First: what does that even mean? Second: read the review.

I have to disagree with you on sales being a good way for measuring quality

My first thought was “I never said that,” but after re-reading, I can see that it sounds like I did. So I rewrote it.

My point was that yes, I would like to see publishers go back to treating sales as their only objective measuring stick, because sales are the only objective measuring stick. You’re trying to sell a game, you count how many people bought your game. Done.

Quality is inherently subjective. Aggregating scores from a group of reviewers is not objective, but it’s disguised to make it look like it is. People get nervous at the idea that sales are directly equal to quality, and it’s good that they do. But people regularly treat Metacritic scores as both 1) directly related to quality, and 2) objective. But it’s neither.

I will say this, though: the thing that makes Metacritic so highly-regarded in videogames — that deeply incestuous relationship between publishers and review sites — is the same thing that makes “critical darlings” more rare in games than other media. Reviewers drive sales of games so much more than movies or TV, because people rarely buy games without consulting a review site or a forum first. I don’t have any numbers to back it up, but I’d bet you anything that the highest-reviewed games of the past 5 years closely if not directly correspond with the best-selling ones. In movies, it’s usually the opposite.

I’ve always thought that Metacritic should divide its scores up. The letters go into one place, the percentages into another, the 10s into a third and the 5s into a last place.

That could potentially be more “fair,” but it would obviate the Metacritic altogether. The appeal of the Rotten Tomatoes method, to me, is that you can still get at the separated data, but it does boil everything down to one number. The difference is that it doesn’t present that number as a master score; it always depicts it as an aggregation.

Your restaurant or hotel doesn’t isn’t deemed a failure if you are rated 2.5/5 stars. It might mean you’re not the fanciest one around, but you’re probably still serviceable and will still do just fine if the actual text of your review and word of mouth isn’t unkind.

Metacritic lumping those together ignores the fact that, while they are mathematically equal, they are being read by human beings who use things beyond raw numbers when they determine what your score means.

This was a very thoughtful post and I’ll have to mull it over before I say anything, but I have a couple of kneejerk reactions:

- I have to disagree with you on sales being a good way for measuring quality. I know you don’t mean first weekend sales but eventual ones too (long tail, etc.), but everything considered there’s a lot of great quality stuff out there that no one’s ever heard of. Jack Ketchum’s work, for example, which up until two years ago was totally out-of-print, even though he is probably the world’s greatest horror writer. The works of Anthony Burgess. Films like Heart of the Earth. Yeah, these things do happen where people pick-up on quality later on – Psychonauts, Ico, Beyond Good and Evil – but it’s struck me as being…Basically, if Kurt Cobain had never talked about The Vaselines as often as he had, no-one would ever have known about them. As it is, he did and they have a cult following now, but they would have just faded into obscurity otherwise.

- I’ve always thought that Metacritic should divide its scores up. The letters go into one place, the percentages into another, the 10s into a third and the 5s into a last place. At least that way you can count down on the number of errors, and have numbers that may represent different schools of thought. I know for a fact that assigning a game a score out of a 100 is totally different than out of 5.

Anyways I have a bit more to say but I need to churn it around a little. Good post.

That said, I absolutely agree with you on Metacritic scores. It’s a great aggregator for finding reviews, but it’s always baffled me that people use the aggregate score for real decisions. If the whole page is bright red or something, sure, that tells you something, although games in that camp are usually dead-men-walking anyway. Other than that, you don’t get much of a feel for anything except the prevailing mood and what each source considers pithy.

(I’m not a fan of scores in general, TBH – especially percentages, which try to make it look like there’s some great scientific process behind the whole thing. My favourite system was probably the old Daily Radar one, with games rated as Direct Hit, Hit, Miss or Dud – effectively thumbs up or thumbs down, with an extra modifier on either side. Specific enough to end the review, vague enough to mean you had to read the text to find out the details, and open enough that the highest score didn’t have to translate to something like “Everyone, no matter what they play, needs to play this game.” Sadly, it didn’t take off.)