A Relic of a System: Looking at the inherent flawed nature of the Numerical Rating Systems

Why must we adhere to a relic of a system to this day, willful of its flaws and unreliability?

For quite some time now, the norm of reviews has included some sort of numerical rating system. These systems have sprung up on many video game related websites such as IGN and Metacritic. Although the prospect of having a numerical rating system ideally seems safe and reliable, that cannot be further from the truth. The prospect of using a numerical rating system stems from the idea that consumers wish for a quick and easy means of gauging the viability of a purchase. This mindset is understandable; money is important and one does not wish to waste it on a purchase that ends up being regretted and could have been prevented if only a system was in place. I personally would love to have seen a system in place that would have adequately represented the worth of a video game, but that is simply not the case.

One of the key factors of using a numerical rating system or any rating system in general is trust in the merit and validity of the result. For a system to be used by a consumer there must be some form of trust in the rating system accurately representing the worth of the video game. A rating system without trust is inherently worthless and a waste of time. Because of that belief, and the fact that numerical rating systems are still widely used, it should not be too far-fetched to conclude that there is sufficient faith in the system. This realization I find quite unsettling due to the rather corruptible and inherently flawed nature of such a system.

Now, it is understandable to question why I have such a rather bleak view on the systems in place. This view stems from various variables that influence and alter how the numerical rating systems are used. These variables range from basic societal reactions, the limitations of the ratings, and the possibility of abuseandcorruption. Once these variables are understood and used in conjunction with the numerical rating systems, it becomes blatant as to how unreliable and inherently flawed they can be. Afterwards, perhaps as a community, there can be a push toward a more reliable and useful system and not reliant on outdated and relic of a system.

Societal Norms and Reactions

In gauging the inherent flawed nature of the numerical rating system, we must look to how as a society, we react and indulge in rating systems. With the age of the internet, society has become accustomed to gathering information both vast in quantity and at increasing speed. The age of hearing news and having to wait before reacting has long passed. When there is something that catches our attention, there is a desire to want to do something about it immediately. This mindset leads to kneejerk reactions and can have very negative repercussions. I do not believe that as a community, let alone society, we have the restraint to react to something without overreacting.

Often enough, it is the case that when major news comes out, not all the information has been released. One noticeable flaw in how as a society we enjoy reacting without the whole story. This can be devastating due to how as humans; we tend to act in unison. An example of this would be with debacle that was Guise of the Wolf. TotalBiscuit, known as the Cynical Brit released a reactionary video known as WTF Is... Guise of the Wolf. To put it bluntly, the video did not make the game out to be very good at all. The video was later flagged and copyright strikes were added to TotalBiscuit's Youtube Account. This did not sit well with him or the community so the masses flocked to the games Steam page. Following the unprofessional responses via Email and twitter by FUNCreators, the community began picking up their pitchforks and torches. They flocked to Metacritic and without all information being known (and this occurring in a matter of hours) which resulted in the score of the game dropping to a now 0.6. You can read the compilation of the story on Kotaku here.

Now do not get me wrong, standing up for what you believe is good but people are easily swayed by emotions and following the crowd. Consequences are almost never taken into account and the desire to put down grows to be too strong. This is the result of the inability to properly place how they feel toward the company, the game, or how to get their opinions across. The current numerical rating systems are inherently limited and how one can express their opinion on the game. The systems in place do not adequately represent the game resulting in a seemingly broken system. Take the Metacritic system as an example and how one rates a game.The previously mentioned event did an excellent job of illustrating one of the major flaws of the numerical rating system and how it can easily be broken. The reviews given are not that of gamers who tried the game and were disappointed, they were instead angry members of a community that felt the need to voice their opinion by giving the game an incredibly low score. Whether or not the game deserves the 0.6 is an entirely different discussion, but it does show how easily a single event can damage a score permanently. While others in their frustration move on, this score will forever be low and will not represent the games worth. It also shows how easily a community, especially large faithful communities can move in unison just like a flock of birds. There is something empowering about having hundreds if not thousands of people all focused on a single entity and feeling the same. It is difficult not to want to join which can prove to be incredibly bad in the long run.

Inadequacy due to Apparent Limitations

When you are to review a video game, you have a box where you can write your review and a scoring system ranging between 1 and 10. Now imagine if you were reviewing a game and you loved the story but the combat was horrible, what score would you give it? Some people may care more about the combat over that of the story so the scoring can end up completely different. How can one accurately look at a review and gauge the important aspects of the game to the consumer? This is an inherent problem when it comes to using numerical systems in general. You are somehow expected to have an objective understanding of how to rate the video game while at the same time, needing to be subjective and giving your opinion. This simply does not work; your interpretation of what a score of 7 reflects may and most likely will differ from that of another reviewer.

An example of this would be the rating system used by the popular video game critic, Angry Joe. When reviewing a video game, he often reiterates how his rating system works since he realizes what we may perceive as bad is average for him. For example a score of 4 which would be terrible to some people. According to Angry Joe’s rating, that means the game is mediocre. Even a score of 6 (which based in academia would be near fail) would be seen as exceptional. The rating is viewed differently by each person which makes having a system based on a numerical rating to be unreliable.

Now, one may argue the focus of the reviews is not the score but what follows. That would be true if emphasis on the written review was more prominent and focused. If you go to the major websites for video games, what do they all do? They have highlighted, in a very large manner, the score for the game. If you go to Metacritic, you primarily see the rating and shown cutoff versions of the reviews (so either expand it further or view more of them. And in the end, it still falls for the same limitations of having a vague scoring system with inadequate means of rating particulars (music, aesthetics, game play, story, etc.). This ultimately diminishes any faith in the accuracy and representation of the system leaving it irrelevant and pointless. In furthering the issue with how the systems are developed, we end up with the more controversial problem of potential abuse and corruption of the system.

Personal Interest and Abuse

It is by no means a stretch of the imagination to consider people with a financial agenda trying to either mislead or alter the scores based on their own interest. The idea of people manipulating rating systems or influencing them is nothing new. It occurs in many different areas, even presidential races. In the end, there will be some that try to overstep the boundaries for their personal interest, often financially driven. We do not have to go too far to see the seed of doubt at work. In particular, we have websites such as IGN which many view as unreliable and biased. This is due to the belief that there are reviews that are slanted due to personal bias or some alternative interest.

This particularly is true when it comes to video game journalism and journalistic integrity (or lack of). Trust seems to be lacking when it comes to journalist and giving unbiased reviews. Perhaps one of the more famous and hallmark representation of this occurring is with Geoff Keighley and his infamous Halo interview. Here we have Geoff Keighley sitting in front of a camera with a conveniently placed a bag of Doritos and Mountain Dew next to him. This notorious interview peeved many gamers because the second that was shown, the idea of trusting his opinion as being unbiased vanished in thin air. When the idea of someone essentially being paid off becomes planted, doubt will emerge and becomes incredibly difficult to get rid of.

Although not representative of all reviewers or journalist, it does give some credit to those that believe some cannot be trusted. Or at minimum, there is potential for abuse within the system. There have been cases where there have been some sketchy actions taken by some. In one such occurrence, there have been employees caught reviewing their own game. This brings into question the ethics of whether or not one should be reviewing their own work. An example of something like this happening is with Telltale and their Jurassic Park video game. As purported in Escapistmagazine, “… a series of four positive user reviews cropped up in quick succession, each giving the game a solid ten and lathering on the praise”.

“In true Telltale form this company decided to take on the task of adding new stories onto a beloved franchise that first came out almost 20 years ago," said one review. "I'm happy to say, I'm glad they did! The best way I can describe this game is if Steven Spielberg decided to direct Heavy Rain."

After some research, it was found that these reviews were written by employees of Telltale Games; a user interface artist and a cinematic artist. This was defended by the company as refraining from censoring their employees but does nothing to address the ethics of their actions. Luckily Telltale did respond by stating that anyone who does post will admit to being employees (which is at least a step in the right direction). Such an event is not alone for it has also occurred with a BioWare employee for Dragon Age 2 and coincidentally so too did they give it a 10/10 without any mention of being an employee.

Another prime example of underhanded actions taken by employees was with the release of the highly anticipated (and high disappointing) Star Trek video game. At its release, there were many issues with bugs and in particular, with co-op. During the rather poor spectacle of a release, there was a comment made which seemed quite odd to some. Since the online fiasco was widespread, it did seem peculiar that a user on steam by the name of Kenneth Lindenbaum stated, “Works fine for me” and mentioned, “Game uses Steam servers”. Now, when there is a major problem, pretty much well known and spreading like a virus, some might question the validity or honesty of a comment such as that. After some research, it was found that the user had already accumulated over 150 hours at launch and obtained a verity of achievements. This was a red flag and when further looked into (via LinkedIn), it was found that he was in fact the senior producer of the game.

Now this is where I was going to add one last example of the potential abuse and corruption of the numerical rating system but something has come up and changed my original plans. I was planning on discussing Rambo the Video Game and pointing out some questionable reviews giving the game 10 and on more than one account, using the same wording (both with the exact two games and with the exact same score). I went to the Rambo the Video Game Metacritic webpage to get the quotes but to my surprise, all the sudden the reviews with 10 were missing. What is even stranger is that in place of the User Rating, is now a box stating the game will not be out for another 54 days. Mind you, this was for the Pc version which had been released on Steam on the 21^st of February already. I promptly went to the Amazon website and the release dates are for March 25, 2014 which is 20 days from now, not 54.

I do find it fascinating and suspicious how the game which already had a user score now seemingly vanished and will be released on another day. It is as if instead of showing the score, someone decided it would have the game showing its release date for later in the year. As you can see with the picture, it also shows 11 positive reviews but only one review will show up if you click on it. If you click the “See all 42 User Reviews, the Positive reviews jumps from 11 to 26. Only after clicking further do you see the User Score of 2.3. Although I cannot say with certainty that trickery is afoot here but this is far from the norm.

Closing Remarks

The current systems in place, and in particular, the numerical rating systems are inherently flawed and open to abuse. We should strive for the removal of the current systems and push for a far more coherent and reliable system. For one, I do not believe ratings should be available upon release. The rating system should be delayed so to detour kneejerk reactions and give the developers some time to work out the kinks. For example, Diablo III was released with major connection issues (due to massive volume of players). Without even being able to play, hundreds flocks to Amazon and Metacritic to give terrible reviews. Yes, many were disappointed with the game but at the same time, we should avoid reaction reviews based on highly emotional state of minds.

Finally, I do believe we need to strive for more comprehensive and elaborate systems. These systems should allow the examination of variables such as game play, music, and story. Furthermore, the focus of the review ought not to be fixated on a simplistic summary of the game (such as, 2 out of 10) but rather; the focus should be on the content of the review itself. Break up the variables and have users review on those such as game play. That way, when one looks at a review, they can skip to whatever variable he or she deems most relevant to them personally.

I didn't read all that.

But there's a reason why numerical ratings systems are so popular: because numerical ratings systems are total lowest-common-denominator bullshit that any idiot can understand. And that's precisely why they should stay in place: because most consumers either are lowest-common-denominator idiots, or BUY as if they're lowest-common-denominator idiots. And this doesn't just apply to games. The fact is that for a vast majority of people, movies/games/albums are just TOYS. This is why mindless summer blockbuster movies rake in billions of dollars while cerebral movies that make people think generate far lower revenues. It isn't that most people are stupid, it's that most people work their asses off at stuff that is ACTUALLY important, so they don't want to put in that much WORK to enjoy something that they just consider to be a toy.

Really, ANY rating sytem is gonna be lowest common denominator trash for the stupid and/or lazy, because real appreciation of ANY art form (be it movies, games, or music, or something else) is HARD and takes a lot of WORK. The real value is in DISCUSSION of the art and EXPOSURE to the art, and exposure to various well-thought-out analyses of the art where there's an intelligent and informed back-and-forth discussion going on in which everyone (whether they agree or not) gain a better appreciation and understanding. But that's not really conducive to getting people to buy or not buy a game, because that requires WORK. And most people just plain aren't gonna dive that deeply on everything that they're considering buying. No one has time for that shit, aside from the people who treat the art as their livelihood. And most people DON'T think of this shit as their livelihood, they just want to have fun for a little while without feeling like they wasted their time. Rating systems seem to accomplish that pretty well.

I think that instead of just accepting the standard, we should try to improve upon it. Consumers are far too complacent in their 'research' and rely on overly simplistic means of looking at a products viability. This has trained consumers into the way that they are because there is no push or desire to improve upon the current systems. Instead of writing it off as mere actions of the ignorant and naive, we ought to look for ways of determining why they are and try to counter it. Having such simplistic numerical rating systems in place that are open to abuse, corruption and unreliability is essentially say, "I give up, no point in even trying".

To further my points, one of the issues I brought up was the limitations of them and how they should me more inclusive and not exclusive in their variables. For example, the addition of bringing up the game play, the music, the story, etc. and have those account and not simply, 1 to 10 and use an arbitrary system. Hell, I would even argue that the inability for people to read something of this length (which is actually not even long) illustrates how lazy consumers have become. Of course many could simply not read this due to the topic, the disregard for its validity, or thinking it is pointless (or boring). In the end, we shouldn't throw the baby out with the bathwater. Even if it is gradual, there should be a desire to improve and not become complacent because when it does, we all suffer.

Well, you sort of answered your own question. You can say that the problem is "laziness", but it's equally valid to state that as "different priorities". The fact is that many people simply DO NOT CARE about this stuff enough to approach it in anything other than a "lazy" manner. And honestly, why SHOULD they care that much to not be lazy about it?

Don't get me wrong, I love my movies and music and video games. But the vast majority of this stuff is not high art for the elite, the vast majority of this stuff is pop culture for the general masses who don't really give a shit. You can think of this as bad or dumbing down if you want to, but the fact of the matter is that the stupid pop culture aspect fuels the serious artistic aspect. Where would photography be as an art form if it had been relegated to the educated artists instead of being entertainment for dumbass tourists? It wouldn't have gotten anywhere, because it was initially rejected as a valid artform and dismissed as fluff for uneducated yokels. That pop culture mass market sensibility then led to more exposure and increased accessibility and ultimately led to serious art that never would have existed if not for the existence of the medium as lowest-common-denominator trash for the masses.

Or think about it this way: you complain about numerical scores, but most numerical scores are also accompanied by an ACTUAL WRITTEN REVIEW. It's not that consumers don't have other options, it's that they often choose the EASIEST option because they just don't care that much.

And this elitist attitude bugs me when dealing with pop culture garbage. Elitists act like the casuals are wrong for just not caring about, when it's the existence of the casuals that guarantees that the elitists have something to enjoy. Go over to system wars and see people make comments like, "if you aren't gonna take games seriously, then you should just give gaming up." That's some fucked up delusional thinking right there, to think that the industry is driven by the minority of consumers who take this shit super seriously, when it is in fact driven mostly by the consumers who really don't care all that much. Sure, you can say that the popularity of rating systems is based on "laziness" or people "not caring", but WHY SHOULD THEY CARE enough to put work into it when all they want out of it is a cheap and easy way to unwind for a little while? There's nothing wrong with YOU taking this stuff super seriously, but why should I take this stuff seriously when I just plain don't fucking care all that much? Can you give me an objective reason why I SHOULD care that much about games or movies or sculpture or photography to actually put effort into understanding the stuff on a meaningful lebel?

To be fair, my use of laziness was only due to you bringing it up. And it is correct, the priorities are what matter (although the inability to read more than 500 words I view as more of a social issue of laziness brought on by technology and other variables but that is for another topic). What I was getting at and I shall reiterate is that the current systems allow and encourage the use of unreliable and flawed systems. Whereas one, as a consumer, should be able to use a system that they can actually trust due to it being valid.

Now, with regard to your comment on my complaint of numerical ratings, you are simplifying the matter too much. It is not simply that I have issue with numerical system, it is that they are inherently flawed and borderline useless. Whether or not they are accompanied by a review holds little weight due to the focus is on the score. I even touched upon this in the beginning too--the reason why on websites the written reviews are not up front and the main focus is because they are just that, the afterthought. When you have a list of reviews with in large numbers, the score, and not even a hint of the review, that is wrong. They are designed in such a way, intentionally, to bring attention to the score and make that what matters. Which leads to the problem of well, the scores themselves are flat out broken.

So question of why one should care (since that seems to be the crutch of what you are getting at), it is quite simple. By accepting things the way they are, you are accepting that the "casuals" use a broken and unreliable system. Sure, people can care less and not be bothered and use a broken system. I personally would rather have the "casuals" use a more reliable system to get a better grasp or even make a choice leading to perhaps a better experience. In the end, the "casual" person will not have to be bothered of how damaging the current systems can be to businesses, careers, since ignorance is bliss. But I will finish with the last point because I assume you never finished reading or missed some of the points.

The assumption being made is that (to label and put into easy groups) we require the "elitist" role to be taken on my the consumer and the one that reviews. When in reality, there can be a "casual" friendly system where behind the scenes, it was provided in a "elite" sort of manner. For example, let us use a numerical system. Although I personally would not like this type but it illustrates the point. Since "casuals" tend to not be the ones to review (since they can't be bothered to put in the work to look into the matter apparently), it ends up being placed on the "elites". Now, since "elites" are differentiated by virtue of their willingness to put in the work, we can assume that added word into reviewing would be tolerable. So instead of having a basic, rate 1-10 and that is it; we can have a system that scores individually the game play, music, aesthetics, etc.. Now, it can go either one or two ways; you can either have the eye-catcher rating (the one in large numbers on the front page) have the average of all combined or show each one individually (guess you can have both, showing all but that may be too much information). You see, issues like this can be tackled in a variety of ways and can even keep them "simple" for those that do not care while at the same time, creating a more reliable and complex system.

In the end, I'd rather have some progress compared to accepting, borderline promoting the use of unreliable systems to those that know no better. Call what I said snarky if you will but you know what, I'd rather come off as "elitist" and wish to better the current norm than accept and be tolerant of people using a broken system to begin with.

P.S. See this quote form your post, "It's not that consumers don't have other options, it's that they often choose the EASIEST option because they just don't care that much." that view right there shows why I am not a fan of focusing purely on numerical ratings. Because the standard is so low that the numerical is the easiest instead of raising the standard but that is more of an issue of society.

Edit: @Iszdope, I am sure you might get to it at some point :) I have faith in you! Just gotta sit for two minutes or so and get through it. Afterwards, feel satisfied that you finished it and realize that you have forgotten most of which you read leading to annoyance and then giving up.

TC is correct you guys need to do more research

ANY rating system is going to be unreliable and flawed, because you can't boil down valid criticism into a freaking rating. Even if you break it down into individual categories like music and aesthetics, you're still just applying a number to something that cannot be understood with a number.

wrong forum

I think it should be acceptable. It's not as if games are the only things that receive ratings, the exact same thing applies to movies, books, music, etc. He might be specifically referring to games, but this is actually about the validity of rating systems and I think that's broad enough to be allowed.

That is pretty much why I chose this forum due to the core of the topic having to do with rating systems and not specific to video games. I would rather have had it on a forum that that was video gamecentric but this ended up being more fitting. And I do agree, all systems will be flawed and unreliable in some way. But there are some that are more or less reliable than others.

Also as I mentioned, I am not a fan of using numbers and my was just an example. But, I do believe that breaking them up would be better, especially for ones not caring to look into the matter further. One variable may be more important compared to another. Alas, I have a feeling general consensus either do not care or disagree so might let the topic die off.

You can say that this or that rating system is better than others, but then you're just splitting hairs. Whether it's numerical or letter grade based or fresh/rotten or thumbs-up/thumbs-down, the same thing applies. It's Idiot Criticism that does little to nothing to promote an actual understanding of the work in question. You could use a 10 point scale or a five point scale, use decimal or round to whole numbers, split it up into subcategories or lump everything into a single rating. The result is the same. It says nothing substantial about the work.

Oh, gee. I see that this game got an 8 for story and a 7 for graphics. Doesn't tell me shit. Was the story cliched? Were there plot holes? Why the score for the graphics? Were the textures muddy? Too much pop-in in the background? A 5 for sound? What does that even mean? Was the audio clarity poor? Was the soundtrack repetitive and annoying? No way to tell from the score (even if you divide it into subcategories or use a non-numerical based system).

This is an inherent flaw with ALL ratings. Sure, some rating systems are better than others, but it's not that big of a deal since ratings are inherently shit anyway. If shitty and uninformative and unreliable rating systems are THAT bad, then there's only one solution:do away with ratings entirely, and then force people to read just the review. The only problem is that under that system, most people WON'T read the freaking reviews or even click on the link, which is why most reviewers use ratings in addition to the review.