On rating quests

According to ZC's quest rating system, 3 stars is "good", 4 stars is "great" and 5 stars is "excellent." I've noticed that many of the newer quests disproportionately have many, many five star ratings compared to quests rated in years gone by. I don't know why this is so, but it bugs me. I am of the opinion that the coveted five-star ratings should be reserved for quests that really stand out in a crowd. Obviously some of the classics from ZC's heyday by the likes of Mr. Z, DFW, Freedom, shoelace, etc., THOSE are five-star quests. I'm not meaning to single out any quests for criticism, as some of the quests on the first page under "newest" are certainly fun, playable efforts, but many of them are, frankly, NOT games that belong in the same discussion with the all-timers, and yet the relative "grade inflation" we see today has them listed there if someone were to sort by highest ranking. Five-star quests should either have superior level design, eye-popping graphics, a truly epic feeling, or innovative features that give them a unique quality. Frankly, in my subjective opinion, of the all-new quests released in the last few months, only "the Hero's Memory" by Aevin comes close to belonging in the same discussion as quests that would have been rated five-stars back when it seemed to mean something.

Thoughts? Agree? Disagree? As someone who plays a lot of custom quests, I think the rating system is more useful if people are little more critical and selective.

I agree. It seems that most people rate a quest based on the way the tileset was used. It's as if anyone can build a quest using the pure tileset and throw little layer 4 trees everywhere, resulting in an automatic 3-4 stars.

To be honest though, at times I rate quests highly simply so that the author won't get discouraged. I've only put up one quest on this place so far and it got 4/5, which I was happy with. Not everyone takes rating quests seriously.

I think the problem here is that people try and rate quests too high/too low to move the average vote closer to where they think it should be as opposed to honestly voting what they think about it. I'm pretty sure the new site has a system for combating this, so this should hopefully be a thing of the past. I agree though, rank inflation is an issue if one is trying to compare an older quest to a newer one.

How would that work?

I know the imdb (internet movie database) uses an algorithm that makes your votes count less towards the total rating if you regularly give extreme ratings. Is it something like this?

Taking the median of all the ratings to determine the "final" rating would be an effective method of combating that type of rating style. Again, I'm not sure what's possible with the new site, so LtM is free to hop in here and call me a liar

I agree with the sentiment, on a whole. It seems like the majority of the userbase just goes "Oh this was fun, 5/5." It's incredibly annoying, but hey, what can you do about it? Nothing, really.

Indeed. Though I've only voted on two quests in my entire life, I've been around to see and hear enough of the ranking system and all that other gossip that came with it. I didn't know of a new ranking system possibly on the new site, that'll be interesting.


Question: Will some older quests get a default rating to start on, or are all ratings being reset for all quests on the new site?

First of all, let me say to John ... I'm really flattered that you enjoyed my quest so much. I'm putting in a lot of work, taking all the reviews into account and trying to improve it all the time. (I hope the staff won't hate me for submitting frequent updates).

Hmm ... I've been thinking a lot about the rating system lately, in part because, to be honest, I'm a little neurotic about my own. A few things I've noticed ...

1. Comedic quests tend to rate higher on average.
2. Almost nobody rates lower than a 3.
3. Yes, newer quests tend to rate higher.

It could be that with time, more votes come that average out the quests, whereas the newer quests are more likely to suffer from extremes. That is, if a quest "should" be rated 4 stars on average, but one person rates it a five, it's more likely to jump up to 4.7 and round up if there's fewer four star ratings to balance it out.

I also find that when rating, many people comment about the difficulty, the graphics, glitches and item locations, but very few comment on story or atmosphere.

Also, this. A rather lengthy article describing the tendency for five-star rating systems to lean toward the extremes of one star and five star.

Edited by Aevin, 02 April 2013 - 03:00 PM.

I hope not, because that would almost guarantee the kind of vote Evan was talking about (voting extremely to influence how you think the quest should be rated in the database, as opposed to voting honestly).

An algorithm to make people's votes count more or less based on how diverse their ratings are would be the best solution IMO.

Another oddity I've noticed is things like this:

"A pretty good quest, but it has some major flaws. Four stars." And then, "This quest is virtually perfect in every way and everybody EVER should play it!!! Four stars!"

(Not actual ratings, but I've seen some like them out there.)

Edited by Aevin, 02 April 2013 - 03:05 PM.

Looks like we should adapt Youtube's Like and Dislike system =P

I'm not very good at rating certain ratings. If I really like it, I usually just give it a 5 :/

We could scale the ratings based on the average rating of all quests within a 6-8 month period. If the average was 3 stars for 2012, and then 4 stars for 2013 so far, 4 star quests would still be four stars (+1 from 3) in 2012, but whacky 5 start 2013 quests would get bumped back to 4 stars again after adjustment.

When i rate a quest i do it the same way i rate a movie on netflix. If i want to continue playing meaning i did not get enough i give it 5 stars. If i was satisfied and wouldnt mind playing something similar i give it 4 stars. If i kind of dont care but it was entertaining enough i give it 3 stars. If i would rather not see something like it again but it wouldnt be the end of the world seeing it again i give it 2 stars. If i absolutely do not want to try something like it again because i got bored as hell i give it 1 star only. 0 stars i only give for things that had no effort whatsoever and the same problem as 1 star.

I think rating would be easier if those descriptions were to be added in a short form under the rating because otherwise people assign their own meaning to the star system. Force a meaning and people will be more likely to think about fitting it to the meaning.

So i other words put clear description under the stars that tell what each star means. 1 star is god awful. 5 is very enjoyable i want more. Etc.

Edited by aaa2, 02 April 2013 - 03:35 PM.

Firstly, not to fear, ratings won't be wiped and there will in fact be an algorithm that determines the score submissions get based on more factors than just the average of all the ratings.

To the topic at hand, from a general standpoint, indeed, people seem to rate too high or too low. I chalk that up to there being very few members of PureZC who are actually professional reviewers.

To elaborate, I feel like maybe the majority of raters don't know how to properly review, and that's fine, it won't hurt them in life. Unless they make a bad recommendation to a friend and that friend beats them up for their bad taste. Er, well, I do feel like people tend to rate quests too highly just because they like them, and while that does annoy me, that's fine, that's none of my business. I try to rate fairly when I do rate, and not based on the rest of the votes.

It was noted that joke quests tend to score higher, and honestly I have mixed opinions about that. Usually people tend to make joke ratings in response that praise it, or even joke reviews that are negative. But I think the feeling is that few people want to be seen as a buzzkill and so don't rate honestly. It's kind of annoying to see joke quests listed highly in the top-rated list, unless they're actually pretty solid gameplay as well, like The Darknut Within. There are others that are just awful, and while I enjoy the spirit of the mood with the quests, seeing endless praise of it does get kind of annoying. What other solution would I have? Make a joke comment that says you're giving it five stars but actually rating it a 3, or something like that, I dunno.

At the end of the day, though, it's honestly none of our business. Would it be nice if everyone rated things thoughtfully rather than "I like this, five stars"? Yes. But again, few (if any) of us are professional reviewers, and few of that number will think it worth the time to improve their feedback. We can try reaching out, like in this thread, in hopes that people will think about their review styles and changing them, but that's about all we can do. We can't force it.

While I like a good story, and it can definitely enhance an already great game, it's honestly the very least of my concerns when it comes to rating games/quests. Maybe it's because I grew up with the more simplistic games of the NES and SNES, but story has never been an important factor in judging the quality of a game for me. If the story is awful but the gameplay is amazing, I forgive the bad story because I want to keep playing anyway. Atmosphere I guess is similar, but possibly more important to me than story, since that can affect how you feel playing the game, and if it's a poor atmosphere, I'm hesitant to continue.

There's something else few people seem to rate on: Use of music. But again, no one here is a professional so they probably just think "Eh, the music was okay/great/subpar, but whatever," and leave that out of their ratings. I guess if the music choice is okay or good, people will just be like "That's how it goes" and talk about other aspects of the quest. If it's great or awful, I suppose that's where the exception lies, because people will want to praise your music choice or berate you for choosing awful music for places. Or outright having horrible quality MIDIs.

Blah blah I'm rambling here so done now.

I think a lot of people think like that. And I don't find anything wrong with it. Given that many of the people here are programmers and veteran game designers/rom hackers, it's not surprising that the interests of the majority are in the technical stuff behind game making. I, on the other hand, consider myself a storyteller who just happens to make games, so I do sometimes wish people valued story more. I should also clarify that I don't believe a good story needs to be very complex.

Edited by Aevin, 02 April 2013 - 03:57 PM.