Jump to content

Photo

Reviews and Ratings Rule Addition


  • This topic is locked This topic is locked
97 replies to this topic

#91 Sheik

Sheik

    Deified

  • Members

Posted 29 October 2016 - 04:40 AM

If outliers would be considered (which are in fact based on distance from median, and not distance from mean), the most fair way would need to factor in percentiles.  Even in the subject of Statistics, there is no universal rule that states where exactly the border is between an outlier and a non-outlier, but if you did, percentiles would need to be involved.  Even without a universal rule, it's standard practice (but decided arbitrarily) for outliers to be outside of 1.5 times the interquartile range.

That much be given and I don't think that's where the issue lies. I think the other problem is the way the ratings are being used. I did of course not ask every member how they factor in the ratings when browsing the database, but from a statistial point of view they are averaged in order to be comparable to each other. However, I cannot really compare an average from a sample of five reviews with the average from a sample of twenty reviews. They are different things and I cannot say that quest A is better or worse than quest B based on these two averages. At about N=30 we can assume that they are normally distributed (after having taken care of outliers) and then we can compare them with each other in a meaningful way. There's no getting around this porblem because the community is not big enough to gather so much data on all quests at which point I am wondering whether having the rating system in place is a useful thing at all. It looks like it was saying something useful, but it doesn't. So in a way it is kind of deceiving even.


Edited by Sheik, 29 October 2016 - 04:44 AM.

  • Mani Kanina likes this

#92 TheLegend_njf

TheLegend_njf

    Deified

  • Members
  • Real Name:Grant

Posted 29 October 2016 - 10:54 AM

Can I just add that this in some ways this seems to be an issue of 2.50 bias in a rating system that seems to encourage the belief that Top 10 or Top 20 matters?

 

We have a rating system where it seems people are terrified to see their favorite 2.50 quests go below 4.5 and that to me tells me enough why suddenly low ratings have become a big issue. With a bias such as this, I can see any personal rating below 3 in a solid well loved 2.50 quest to be met with suspicion, considered "negative", and possibly even met with disciplinary actions under the notion that the member is "trying to manipulate the rating". Again, many people I believe still feel that this topic is encouraging people to keep their votes 3 and above to ensure that the community continues on in a peaceful way.

 

Are we really that afraid to see some of our beloved 2.50 quests go anywhere past page 3 or 4 in the top rated quests section? Don't get me wrong, I love 2.50 quests over the older stuff. When done right, I believe most 2.50 quests outshine 2.10 quests by a mile! I also believe perhaps our general view over 2.10 quests today are pretty poorer than what they once were. But as long as our favorite 2.50 quests are rated just as fairly as the old quests that were once great, but probably considered dated now, we're just going to have a lot of people hold this bias.

 

*** I believe 2.10 quests and 2.50 quests should be rated under two completely different categories due to the huge evolution zc and quests in general had between these two generations. I feel 2.50 is of a new ZC generation and all 2.50 quests should be treated as such and thus have their own category for independent ratings. Treat it like a new gen console in that way. You can't compare NES games with SNES games for this reason. Yes, you can slightly, but it wouldn't be a very helpful or accurate ratings because of the evolution and the change in rules that these consoles lived by***

 

Quests like Promised Lands, Forbidden Ascent, Umbral Cloud, and MAYBE Isle of Rebirth (I say maybe because IoR is still rated fairly high) comes to mind when I see great 2.50 quests who now struggle to compete against some older and probably considered inferior quests because of a few bad ratings. They had much higher ambitions, but clearly none of them could remain top 10 in this system, and this is where I feel the fears lie. There's a lot of 2.1 quests that we probably now consider average. Quests like Hero of Dreams, Origin, Power of Geduina will almost always have an advantage because of their age old impact on the database I guess during a time when PureZC itself was a bigger community where now a few bad ratings probably doesn't affect these quests a whole lot.

 

But still, a bias is a bias and I'm sorry, any attempts to control the ratings will probably not be met favorably by public opinion. We have a very old database, it's almost expected that most great 2.5 quests are doomed to this exact same situation as the quests above are in. 


Edited by NewJourneysFire, 29 October 2016 - 11:05 AM.

  • Anthus and Deedee like this

#93 Shoelace

Shoelace

    The Shaman of Sexy!

  • Members
  • Real Name:Michael
  • Pronouns:He / Him
  • Location:Arizona

Posted 29 October 2016 - 12:36 PM

My Goodness!  This is quite a thread.  I haven't read everything but here is my opinion about the situation:

 

These are fan quests.  There are not 15 member teams working on the game 40 hours a week whom get paid.  These are never going to be perfect games.  I am going to use NES as an example, if I rated Super Mario Bros. 3, I would give it a 5 star because of the quality, balance, and fun factor that it provides.  Say the same game was released for the first time on the Nintendo Switch.  The game would probably be rated a 2 or 3 because it isn't up to the quality of the new Marios, it has aged graphics, compared to other games it is no where near them.  However, when people say, what is my favorite Mario game I say Super Mario Bros. 3.  Why is that?  Because if you base it being played on the NES and the surrounding games, it was the cream of the crop.  It deserves a 5 star.

 

Same thing with Zelda Classic, these are not PS4/Wii U games.  They are fan games.  When I played Lost Isle and Isle of Rebirth, I mentioned that there were many things I didn't like about the games.  What can be worked on, what can be improved, pointers and such.  However, I gave them 5 stars.  Why did I do that you ask?  Because the amount of quality and hard work those quests provided were amazing.  You can tell that love and passion was poured into the games and to me they are the cream of the crop.  

 

I think this is where the frustration lies, when you get a one star on a game, you slaved over for 3 years.  One star to me, means there is not an ounce of quality, not a single thread of passion, just put together without thinking.  That's where I look at those at those ratings, and just scratch my head and say, wow this person has no respect, they are literally just doing it to do it.  Worst of all, when doing it to quests that don't have the number of ratings to even hope of returning to the front, it is just disgusting.  The only way this system would possibly work is if the said persons rates EVERY single quest in the database.  Again to me it's just plain rude when you see those low of scores to a game that has quality.

 

Now, can the rating system be better? Yes, but either way, people are going to take advantage of it.  The only way to do it in my opinion is to split up the ratings:

  • All Quest Types
  • 1.92 or lower quests
  • 2.10 quests
  • 2.5

This way, you have the separation, much like NES, Gameboy, SNES, N64.  Because again, of course a 1.92 isn't going to have the fancy greatness of 2.5.  So they shouldn't be considered the same thing.  And people that rate quests should now and respect that.


  • Nathaniel, Air Luigi, Shane and 6 others like this

#94 Eddard McHorn Van-Schnuder

Eddard McHorn Van-Schnuder

    smash the bye button

  • Members
  • Real Name:Ronny Wiltersen

Posted 29 October 2016 - 02:10 PM

I think this is where the frustration lies, when you get a one star on a game, you slaved over for 3 years.  One star to me, means there is not an ounce of quality, not a single thread of passion, just put together without thinking.  That's where I look at those at those ratings, and just scratch my head and say, wow this person has no respect, they are literally just doing it to do it.  Worst of all, when doing it to quests that don't have the number of ratings to even hope of returning to the front, it is just disgusting.  The only way this system would possibly work is if the said persons rates EVERY single quest in the database.  Again to me it's just plain rude when you see those low of scores to a game that has quality.

I think you've got some interesting points, but this part stands out to me. If I rate a quest lower than the average because I didn't like it, who are you, or anyone else to say that they shouldn't rate it? I mean, you use words like disgusting and rude when describing low ratings, but can you explain exactly why it is rude? That is to say, I don't think it is, and I think you're approaching it from the wrong angle.

 

See, I agree that there are ways to improve the current database situation, but let's not pretend that reviews aren't completely subjective. True objectivity when it comes to something like reviews doesn't exist, we all have different standards and we all have different expectations. I find it incredibly disgusting (to use your own words) that there are opinions in this thread that literally say that other people's opinion are somehow disrespectful or bad simply because they don't share your view. You could even call it rude, but I think you get my point. You can't dictate what other people think about something, and claiming that they're being rude because of it... that's just petty.

 

And while I've said this before I'll say it again because I have a feeling not too many people have bothered to read my long posts about this: reviews does not exist to please the creator. They exist to help players find something that will suit their tastes. The little ratings next to the written reviews can be a helpful way to quickly get an idea, but it's the written word that holds any kind of value. And again, that value is to the player, not the creator. That said I understand that people like to get comments, and of course it's always nice when those comments are positive. But this isn't some sort of magical fantasy land where everybody loves everything you've ever done, and yes it is possible to dislike something so much that you rate it 0 stars without that having to mean that you hold some sort of grudge, or like you implied, that they are being rude in regards to the rating.

 

As someone with absolutely zero authority here all I can do is ask, but I am begging you, and everybody else that have been doing this, to look inside yourself. You're preaching about acceptance but ask yourself: are you being accepting of these so called 'disgusting' opinions? I don't think you are, and I'm having a real hard time not to be a little offended by that as someone who used to have somewhat a reputation of being a little harsh from time to time. It's like saying that our opinions doesn't matter or are somehow worth less - and why? Because some creator might not be able to say that his quest has a full score? I mean come on...


Edited by Robin, 29 October 2016 - 02:32 PM.

  • Sheik, Jared and coolgamer012345 like this

#95 Saffith

Saffith

    IPv7 user

  • ZC Developers

Posted 29 October 2016 - 02:55 PM

There's a fundamental question here that everyone seems to have a different position on. What is the purpose of the review system? More precisely, to what degree is it each of the following:
  • A forum for expressing one's opinions
  • A feedback mechanism
  • A recommendation system
These are at odds with each other, to a degree. A recommendation system benefits from some sort of guidelines on ratings and possibly some way of dealing with unusual ratings (outliers and users who tend to rate higher or lower than others). Content and style rules befit a feedback mechanism. But in a forum of opinion, rules and guidelines are effectively censorship.

The system could serve all three purposes adequately with more users, but failing that, the rules need to be tailored to its goals. The staff, I think, are largely in agreement that the review system is primarily for recommendation and feedback.
  • ShadowTiger, Rambly, nicklegends and 1 other like this

#96 pixcalibur

pixcalibur

    Warrior

  • Members

Posted 29 October 2016 - 02:55 PM

Honestly, I just look at the ratings for a quick glance at the quest.  I mean, if the screenshots look decent and there aren't multiple complaints on the same kinds of problems (permastuck scenarios, obvious broken designs, unfair luck-based difficulty requiring TAS-like play just to survive, etc), I may take a quick try on the quest, and if it's not to my taste, then I skip it and move on.

 

I feel ratings and reviews are mainly for a quick overview of the quest.  Really only by playing it yourself that you'll get a better picture of the quest and if it fits your style or not.  Plenty of 3-star quests I've played through and found them solid enough.  They're not perfect by any means, but they're at least not some broken mess that's unwinnable.  On the flipside, many quests 4.5 and above I may not go through because the quest styles didn't fit me, not because the quests were bad in any way.  Usually 0, 1, and 2 star quests have some blatant problems in them, but again, could have some biased ratings bringing the scores down.  So again, it's up to the player if they want to test drive the quest to see if such ratings were warranted or not.  I don't tend to review quests given my biases and the things I can drag down ratings on that may be unreasonable expectations.  If I get far enough to actually record an LP, I'll find some way to get around those sticky points in the end.

 

Everyone has their own grading scales, with some harsher than others.  It's all opinion, and really should be taken with a grain of salt and not to be taken that personally.  No quest is perfect out there.  Honestly, a number of these would easily beat out the older console games.


  • Eddard McHorn Van-Schnuder and Jared like this

#97 Mani Kanina

Mani Kanina

    Rabbits!

  • Members

Posted 29 October 2016 - 04:07 PM

People in this thread have expressed their ideas on what the various quest ratings mean, but that's all just individual opinions (for the most part) from their own perspectives. I have in some manner also expressed how I handled my own view of ratings. However, I do think it would be a dangerous road to walk down to pick one viewpoint and then enforce any and all people to abide by it. For example, if a quest that have been in product for 3 years isn't allowed to get any ratings less than a 3 or a 2 then the whole ratings scale is not actually used. Furthermore, development time is not necessarily relevant to the quality of the product nor the experience the player who plays it gets.

Anyway, I'll leave here some questions that don't directly need to be answered, but can be worth to think about:

If anything under a 5 is considered a downvote, and is a negative rating, then that might that not signify that the rating system is used/looked at incorrectly? Since given how the ratings are listed currently, a 3 and a 4 are both positive.

If a reviewer is made to change how they write reviews and do ratings to better fit someone else's judgement, can you really say it's representative of their own view anymore?

Is it reasonable to put a creator in a position where they can judge the validity of someone's opinion of their work to the extent that they can, or urge to, remove it?

Can a statement that judges the work alone really be considered an insult to the creator?
  • Rambly, Eddard McHorn Van-Schnuder, coolgamer012345 and 1 other like this

#98 Evan20000

Evan20000

    P͏҉ę͟w͜� ̢͝!

  • Members
  • Real Name:B̵̴̡̕a҉̵̷ņ̢͘͢͜n̷̷ę́͢d̢̨͟͞
  • Location:B̕҉̶͘͝a̶̵҉͝ǹ̵̛͘n̵e̸͜͜͢d҉̶

Posted 29 October 2016 - 04:29 PM

Alright, this has run its course and everyone with something to say about separating the reviews from the ratings seems to have weighed in. At this point, it's just going further back to the topic of a single person's reviews which is not the point of this thread. If anyone has any major suggestions to LtM's propositions, feel free to leave them in the dropbox. In the coming weeks, expect the changes he highlighted in this thread. :D

 

 




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users