Jump to content

Photo

Reviews and Ratings Rule Addition


  • This topic is locked This topic is locked
97 replies to this topic

#76 Moosh

Moosh

    Tiny Little Questmaker

  • ZC Developers

Posted 28 October 2016 - 02:08 PM

I'm sorry, I don't understand this at all? Are you saying that only 0 and 5 stars matter? You're concerned about a 5 being a 4, but not a 4 being a 3? You only care about fairness for the quests over 4 stars?
I'm not trying to be provocative here - I really don't understand.

In a sense, because of the way people have reviewed for the past 10+ years, yes. 5 star ratings are the only ones that impact scores positively for the upper third of the database. For anything with over a 4 star rating a 5 can be looked at an upvote and anything less than that is basically varying degrees of downvotes. The feedback itself can all be positive but the impact  is still overall negative. And once a quest sinks below the first few pages it starts its journey on the path to eventual death. Nobody sees it. Nobody plays it.
 
That's not to say I think everybody should boost their positive ratings to a 5. We haven't switched to an upvote/downvote system yet, guys. But it bugs me when I feel like I cannot satisfy the conditions to get a 5 star review, when I could address every single complaint the reviewer has and they still won't go higher than a 4. Because nothing can be perfect. Because nothing can live up to the standards they set. If your rating scale is made up entirely of different shades of downvote, I just don't feel that's fair. There has to be at least that faint glimmer of hope. The one quest that they'd call close enough to perfect, that I could hope to compete with. Unfortunately, criticism itself can prevent me from receiving further criticism. And that's not what I want, I want my work to be picked apart so I can improve.
 

As for "what objectively gets a 3", that's completely not what I meant. Of course it's super subjective, but so are a 0 and a 5. However, just because it's impossible to objectively define doesn't mean you should completely ignore it.

0 and 5 themselves are subjective, but the existence of 0 and 5 shouldn't be. All I'm saying is I believe every reviewer should have a quest that sets the 0 standard and a quest that sets the 5 standard for their reviews. As for everything in between...It baffles my mind thinking of putting it to words. Maybe someone can do it, but it won't be me. 2spooky m8.


  • Air Luigi likes this

#77 Sheik

Sheik

    Deified

  • Members

Posted 28 October 2016 - 02:28 PM

0 and 5 themselves are subjective, but the existence of 0 and 5 shouldn't be. All I'm saying is I believe every reviewer should have a quest that sets the 0 standard and a quest that sets the 5 standard for their reviews.

Yeah but if you think about it: 0 is a weird place to start the counting for the ratings with 0 apparently meaning the presenec of something bad. However, 0 usually implies the absence of whatever it is that is being rated... So let's say that I want to rate a quest solely on its originality. If it is hella original I rate it with 5, fair enough. But what do I do if it's not just not creative but a blatant rip-off? Do I rate it 0? Should I not actually rate it with a negative value (-5?)? Or what if I rated only the music: a wonderful soundtrack would get 5, a boring soundtrack is 0 but what if the music is so bad it makes my ears bleed? Would that not warrant a negative rating, too, to denote that not only a nice soundtrack is lacking but there, too, is a terrible soundtrack in place? How do I rate something that is in the quest as bad on a 0-5 scale where 0 seems to imply absence of a given quality? So the scale should either go -3, -2, -1, 1, 2, 3 or the points on the scale should be given names of what they represent. In this latter case (which is the system in place right now) they should be truely dimensional (very good, good, somewhat good, somewhat bad, bad, very bad -- for instance) instead of 6 distinct predicates (excellent, poor, horrible... ...). The entire scale in place right now is a nightmare from the point of view of the proper construction of a scale.


Edited by Sheik, 28 October 2016 - 02:30 PM.


#78 Air Luigi

Air Luigi

    Tio guachi

  • Members
  • Location:Spain

Posted 28 October 2016 - 03:12 PM

You have to rate it for the overall impression, I suppose xd You can give the details in the review..A quest can deserve a 5 even if it's not perfect. I don't know, I think is obvious when people put a ton of effort in the quest, but obviously that don't guarantee nothing. But in doubt, I think is better to be positive than negative, this is a hobby.


I understand what Moosh said... It was hilarious when he overhauled LQFTH2, dedicated a lot of new content with a ton of heart to a certain member... and the member still thinks that 5/5 was too much xd It feels like trolling

#79 Evan20000

Evan20000

    P͏҉ę͟w͜� ̢͝!

  • Members
  • Real Name:B̵̴̡̕a҉̵̷ņ̢͘͢͜n̷̷ę́͢d̢̨͟͞
  • Location:B̕҉̶͘͝a̶̵҉͝ǹ̵̛͘n̵e̸͜͜͢d҉̶

Posted 28 October 2016 - 03:14 PM

Remember, this isn't to discuss the merits of one person's review, but of the overall system and proposed compromise LtM is willing to make.


  • Rambly, Mani Kanina, Moosh and 1 other like this

#80 Nathaniel

Nathaniel

    Deified

  • Members

Posted 28 October 2016 - 03:25 PM

I do agree with those saying that the one word descriptions of the star ratings needs to be changed.  It was already mentioned about the use of "horrible" for the 0 rating.  I have no doubt that this should use a different word.  But what also bothers me is how 3 is "good", while 2 is "bad".  Where is the middle ground?  Where is the "fair" rating?  Or even any word that is remotely similar to that?  But while I agree that these should be reworded, there should also be some more detailed description available, and that definitely can be worked on, and I think sharing some ideas on that would be good.  I think such descriptions need to look at it holistically, since it can easily get quite complex if we started categorizing.  Some items will no doubt be better in some aspects than others.  I think such descriptions for quests should primarily focus on how one felt about the overall playing experience.  As for tilesets, loose tiles, music, and scripts, those would certainly need other descriptions.

 

As for the one word (or a few words) descriptions, I think the following would at least be better than what is currently there, but this is only my opinion, not necessarily reflective of what any other staff or members in general might think.  In parentheses are some alternative words I thought of.

5 - Excellent (Superior Quality)

4 - Great (High Quality)

3 - Good (Good Quality)

2 - Fair (Acceptable Quality)

1 - Poor (Unacceptable Quality)

0 - Bad (Inferior Quality)

 

This certainly poses questions about the wording for sure, but again, it's just brainstorming for me at the moment.
 

EDIT:  My scale may actually reflect something different than how it currently is considered, since it would probably change what it means to be a "2", so I'll need to mull over this more.  But in such a case, 3 could serve the fair or acceptable instead, if we consider how the scale should go.


  • Sheik likes this

#81 Sheik

Sheik

    Deified

  • Members

Posted 28 October 2016 - 03:42 PM

Yeah. The current scale seems to represent something more like:

 

(5) --- (3)

(4) --- (2)

(3) --- (1)

(2) --- (0)

(1) --- (-1)

(0) --- (-2)

And that's also somehow what I meant to say above but made confusing, twisting sentences instead. :shrug:


  • Nathaniel likes this

#82 Jamian

Jamian

    ZC enthusiast

  • Members

Posted 28 October 2016 - 03:43 PM

How about:

5 - Superior Quality

4 - High Quality

3 - Good Quality

2 - Poor Quality

1 - Inferior Quality

0 - Unacceptable Quality

 

I like Nathaniel's idea, but 2 out of 5 is not even a passing grade so I think the wording "fair/acceptable" is too positive compared to the rating. I also think "unacceptable" is harsher than "inferior". And I feel a 0/5 rating should indeed mean unacceptable, for example if the quest is completely broken or not even playable.


  • KingPridenia likes this

#83 Sheik

Sheik

    Deified

  • Members

Posted 28 October 2016 - 03:45 PM

-snip- but 2 out of 5 -snip-

But it is actually 2 out of 6 because 0 is included.



#84 Nathaniel

Nathaniel

    Deified

  • Members

Posted 28 October 2016 - 03:46 PM

Yeah, that's what makes this difficult.  Some of us will have different ideas of what is good or better, and what is bad or worse.  I'll admit I had my head scratching over that.



#85 Jamian

Jamian

    ZC enthusiast

  • Members

Posted 28 October 2016 - 04:08 PM

We could brainstorm on ideas when it comes to the specific word being used. But there needs to be an agreement on what the stars mean, and I'm not sure everybody has the same idea when it comes to that. Maybe that is part of the issue.

 

The way I personally see it, it's like: 

 

Would recommend:

5 - This quest has truly impressed me, it's one of my favourites in the database
4 - This quest was great, there are others I like better, but I still really enjoyed it
3 - This quest was quite fun to play, nothing too special, but I had a fairly good time

 

Would not recommend:

2 - This quest had issues, and my overall impression was negative, altough I've found a couple of things to like about it
1 - I really did not like this quest and felt like playing it was just a waste of my time, but at least it's playable
0 - This quest is broken / unplayable / offensive / I'm ashamed I've even played it


Edited by Jamian, 28 October 2016 - 04:20 PM.

  • Air Luigi, Sheik, Avaro and 1 other like this

#86 Mani Kanina

Mani Kanina

    Rabbits!

  • Members

Posted 28 October 2016 - 04:31 PM

But it bugs me when I feel like I cannot satisfy the conditions to get a 5 star review, when I could address every single complaint the reviewer has and they still won't go higher than a 4. Because nothing can be perfect. Because nothing can live up to the standards they set. If your rating scale is made up entirely of different shades of downvote, I just don't feel that's fair. There has to be at least that faint glimmer of hope. The one quest that they'd call close enough to perfect, that I could hope to compete with. Unfortunately, criticism itself can prevent me from receiving further criticism. And that's not what I want, I want my work to be picked apart so I can improve.

The problem with this train of logic is that not all problems that a user have are even problems in the first place. You can have a game that is entire flawless (impossible, but this is in theory), and it would still be flawed for people who simply don't have the same tastes as presented in the work. You can't patch away taste differences, that's not how it works, at least not for any sensical developer who cares about their own vision. One can play an RPG and be left unhappy with it because it wasn't a shoot em up, but that does not mean it's a reasonable claim to bring up, nor one that the author should reasonable address. But that does not change the fact that the player was not satisfied with what they played. This is an extreme example, and you could make the argument that the player went in with unrealistic expectations in it. But in a real given example it's a lot more muddy.

While it's been stated in this thread that the rules should take the focus, but it's hard to discuss something without examples, especially when they are relevant. And it's not like others are too subtle in regards to what they talk about.

I played hookshot 2 because I had seen Moosh's other works and it was one of the highest rated quests in the database. I'll admit, maybe that made me have very lofty expectations going in, (I honestly don't remember anymore). It was a fairly decent quest for the most part, but the few dips in quality kinda made me less happy about the overall experience. Since I was at the end of the day not happy I gave it a (2)(bad(?)) instead of a (3)(good), and that's something I stand by today. I was not expecting Moosh to try and go in with the aim to address all my concerns, that's kind of an absurd thing and generally don't happen.

Due to the reasoning behind my update, I decided that it's only fair that I give it a spin again. (I also did this in regards to The Flow of Time, which also got an update in response to my critique. In fact, it's getting yet another update which I aim to play/stream when it comes out). I played the quest, and I for the most part like it. And while I'm well aware that some of the things were patched yet again after that playthrough, I don't think reviewers are obligated to change their ratings because of it. Indeed, that's not happening anywhere else on Pure. Anyway, while I liked the quest for the most part I did still have problems that I found to be a hindrance when I played it. One major thing annoyance was the overlay that blocked line of sight, the other was that I found a lot of the older dungeon designs to be rather middling.

While in my first playthrough of the quest I found those to be the best part of it (or maybe that was the writing), but they were so in contrast to the other things. I still did notice and appreciate some of the subtle (and not so subtle, *cough*level 8*cough*) changes that were made that made these dungeons better. In fact, I still stand by that Level 8 in hookshot 2 is one of the best ZC dungeons I have played, what with the new changes.

Even though the rest is amazing does that mean that my thoughts on level 1 through 7 (and the mirror shield place), shouldn't also be accounted for when I make my judgement? Because a lot of those dungeons were kind of forgettable, and I have played through them twice. Does that make them bad and mean that you should throw them for my sake? No, that's stupid, I was actually fairly surprised that a bunch of dungeons were thrown out seemingly only because I happened to not like them. After all, there are people that DO like them, quite a fair bit actually if the reception for the quest is to be looked at. And given that's impossible to please everyone....


The main thing seems to be that my standards are unreasonable because I don't hand out 5s. But in the same sense, it's apparently not unreasonable that there are people who only hand out 5s? But I get it, trying to please someone who seem impossible and never hands out 5s, except for when I did, is pretty high on the problematics counter. Though I guess it's fine to disregard my review I left of The Hero's Memory back in the day because it's old. I'll admit, I'd probably change my rating if I replayed it today..., but I'm also not. And it was also brought up earlier in the thread that people have different opinions in the past, because there was less things to compare too.

But the idea that I don't give out 5s simply because I have never done so is also an assumption. It's an assumption that aims to paint me in the light of an unreasonable individual that can never be pleased. And that's fine on a personal level, I can't really say I care if people believe that of me. But it becomes something else when I'm made increase all my ratings simply because I'm somehow incapable of using the system properly. Although, by that account I can also never hand out a "0" anymore by design. My belief is that I hand out 5s to games that have flaws which do not impact my experience much at all, (because a flawless game is a pipedream), while it on the same time appeals to my tastes. (A flawless fighting game, for example, would be rather irrelevant to put in front of me). I of course have more specific preferences in regards to my tastes, and I could sit here all day if I were to list them all (but I don't think that'd beneficial to anyone). Another quest that got really close to getting a 5/5 from me was the recent Randomizer Returns. Funny thing there is that it's the type of thing I have been wanting for a long time. But it had some things I didn't like which made me not give it a 5, and I brought those up in my review. But those complaints weren't even things I even considered would be problems before I sat down to play this product.

People rate things differently, everyone has their own scale. Is it very hard to get a 5 from me? Absolutely, I have what many would consider absurd standards when it comes to fan games. But I do have a scale, and it is one all things I play gets put on. Just because someone decides that they want to try and please me does not mean that they are entitled to getting the highest mark. And if I may, I'd even go as far as to suggest that I put more thought into my review scale and most people on this site, I even go into my list of reviews whenever I write a review and place them exactly where I believe they fall on aggregated quality in contrast to everything else I have reviewed. The list reads from quest I consider the best to quest that I consider the worst.


*sigh*, this got a bit long winded. I didn't mean to jump on you Moosh, or anything. (I don't really feel I should need to say this, but people have their concerns). I just think of statements more as a platform for discussion.
 

Pointing out negatives is very fair, most people do that. I encourage to do that. But being negative overall is a different thing. All people have their favorites and proper tastes, that's expected. Also, I can understand being negative about quests that you don't like (I did that sometimes). The thing is, a member gives the impression to people that he(sic) hates practically the entire database (almost). And that's a bigger deal, because the member is consistently harsh. Obviously the staff noticed this.

I understand what Moosh said... It was hilarious when he overhauled LQFTH2, dedicated a lot of new content with a ton of heart to a certain member... and the member still thinks that 5/5 was too much xd It feels like trolling

If you have a problem with me you can at least take the decency to address me by name, rather than using implications that you're speaking more broadly than you are.
  • thepsynergist and Matthew like this

#87 Air Luigi

Air Luigi

    Tio guachi

  • Members
  • Location:Spain

Posted 28 October 2016 - 04:57 PM

Well, the staff don't want to target users. I don't have many options, honestly...

#88 Evan20000

Evan20000

    P͏҉ę͟w͜� ̢͝!

  • Members
  • Real Name:B̵̴̡̕a҉̵̷ņ̢͘͢͜n̷̷ę́͢d̢̨͟͞
  • Location:B̕҉̶͘͝a̶̵҉͝ǹ̵̛͘n̵e̸͜͜͢d҉̶

Posted 28 October 2016 - 05:07 PM

Remember, this isn't to discuss the merits of one person's review, but of the overall system and proposed compromise LtM is willing to make.

I was worried this got deleted or something with how readily people ignored it. Seriously, stay on topic. We're leaving this topic open because there's still merit to it, but if it continues to drift off then it will have to be closed.



#89 CDi-Fails

CDi-Fails

    Initiate

  • Members

Posted 28 October 2016 - 06:42 PM

Rating systems are at their core going to be subjective; it'd be far too hard to get everyone to agree on what qualities in a quest calls for a certain rating. If you want to narrow down the choices, go for a like or dislike system. Otherwise, maybe find the median rating instead of average so that the score isn't dragged down by outliers?



#90 Nathaniel

Nathaniel

    Deified

  • Members

Posted 28 October 2016 - 08:48 PM

LinktheMaster has already established that he will not change the current 0 to 5 star rating system, so a like/dislike system will not be in the works, CDi-Fails.  As for the second thing you said, I have actually in the past proposed the idea of median being a factor in some way.  You can't rely on median alone though, since the median would almost always be an integer value, leaving nearly all database entries within many-way ties.

 

If outliers would be considered (which are in fact based on distance from median, and not distance from mean), the most fair way would need to factor in percentiles.  Even in the subject of Statistics, there is no universal rule that states where exactly the border is between an outlier and a non-outlier, but if you did, percentiles would need to be involved.  Even without a universal rule, it's standard practice (but decided arbitrarily) for outliers to be outside of 1.5 times the interquartile range.  Now if that sounds too technical for anybody, in simpler terms, the interquartile range includes the middle 50% of all data (from 25th to 75th percentile).  1.5 times that range would certainly extend to well beyond 50% of the data, which in the case of small amounts of reviews, would almost definitely include all of the ratings.  Outliers should be more likely to be considered as such in cases of very different ratings when there are more ratings overall.  Well, I tried to explain it.  I did teach a course in Statistics before, so I know about this stuff.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users