97 replies to this topic

[Sheik]

I'm not sold on this. I don't quite understand why the impact of outliers should be lowered. Isn't this like saying "You think different so your opinion doesn't matter"?

The question is: Why would you want to have a descriptive statistic such as the mean of something anyways? The idea of an average is that it is meant to be representative of whatever it is the average of. So the average of ratings should be representative of the ratings as a whole. This is not granted when outliers bias the average. It is an issue of statistical analysis that is being taken care of by excluding or weighing down the impact outliers have on the average rating. That way statistics are more useful given their purpose is to give representative data.

 

The biggest problem is figuring out how far off something has to be for it to be consider an outlier.

Not really. That can also be mathematically decided based on the sample that is being examined (though here still parameters have to be decided on first, true).

Edited by Sheik, 28 October 2016 - 09:07 AM.

[LinktheMaster]

While I certainly agree that this wasn't a good way to irk for a change, I would like to point out that it's not like there were any other options. A discussion thread for this rule proposal was not given in advance and the rule itself was brought forward and presented with a large sense of finality.

I was more talking in general about the overall tone of the thread rather than discussing the changes entirely. Discussion over things like this are expected, but I feel that the general hatefulness in this thread was rather unnecessary. I'm not necessarily calling you out on this, nor even one side of the argument. I just think we can have civil conversations about stuff like this without resorting to some of the behavior in this thread.
 

By this extent I assume that the actions taken against me will still stand? Including both the changes I have done to past ratings, any future ones, as well as how I write any new reviews.

At this point, yes. I'll admit I wasn't involved in the original decision, but since you decided to include private conversations...

This decision was made was for various different reasons. It's not only the fact that you've rated multiple quests that had high average ratings with a score of 1. You have also used rather hateful language in many of your reviews, such as calling things or even the quest itself as garbage, crap, or horrible. After lowering a quests ranking, you even specifically called out that you did so as if boasting about removing it from its ranking. Combine all that with the fact that you have previously discussed your dislike of the rating system, it seemed like some of these ratings were meant to make a point by being unnecessarily low rather than offer a true and fair assessment of your opinion of things.

Any further discussion is outside the purpose of this topic, though.

The biggest problem is figuring out how far off something has to be for it to be consider an outlier. If a quest only gets 5s, 4s, and 3s, does that mean that a new review with a 2 is an outlier? What about a 1? Given that a 0-5 scale don't have that many choices care will have to be taken I'd say. That being said though, I don't think this will be a change that will do a whole lot. Quests with less than 20 ratings probably can't accurately determine outliers, even more so if the quest has 10 or less. Most quests in the database don't get huge amounts of reviews/ratings though. Maybe that will change when you don't have to write reviews though? Hmmmm.

Yeah, figuring out the exact algorithm will be a bit of work. One thing to keep in mind that all outliers won't be affected, nor do I think I'll remove outliers entirely. The fewer the reviews, the less I'll affect outliers as well. I'll probably play around with outliers starting out around 5 reviews, but if it seems too volatile, I'll bump it up to 10 or even 20. Honestly, I just don't quite know what I'll do yet fully. I'll have to play around with the numbers and see what seems to sorta work the best.

possible remove them from her reviews should she want to, assuming that's how this would work?

All ratings will be split from reviews automatically, and all previous ratings (and reviews) will be able to be deleted (or altered).

[Sheik]

You have also used rather hateful language in many of your reviews, such as calling things or even the quest itself as garbage, crap, or horrible.

The zero star rating explicitly states that the qualtiy being rated is given the predicate "horrible". So calling something horrible in this context seems to be a perfectly agreeable use of the terminology of the rating system itself.

[LinktheMaster]

The zero star rating explicitly states that the qualtiy being rated is given the predicate "horrible". So calling something horrible in this context seems to be a perfectly agreeable use of the terminology of the rating system itself.

I'll definitely concede that you got me on that one in particular, but my general point still remains that it was a decision of a multitude of factors, not a single one. Either way, I don't think this is really the best place to discuss the particulars of the actions of the staff on an individual member.

 

(Honestly those labels are kind of just still around from days long gone, and the exact labels should possibly be revisited perhaps.)

[Moosh]

The zero star rating explicitly states that the qualtiy being rated is given the predicate "horrible". So calling something horrible in this context seems to be a perfectly agreeable use of the terminology of the rating system itself.

I just did a ctrl+f for "Horrible" looking through Lunaria's reviews. The only one I found where it came up in a negative connotation was Banana Blood God, a four star review. Top kek. It's the context that matters, I think. If you call a horrible quest horrible (you can only call quality subjective so far, broken is broken and there can be no denial) is that a problem? Not really. Call a spade a spade. When a review of something less egregiously bad focuses mainly on the negative...Your mileage may vary. I don't mind reviews that get straight to the point, but some members take it more personally. I didn't find the BBG review that bad though, personally.

[Air Luigi]

Lunaria is negative even when she rated a quest high... that's the thing. Maybe she is just that way, but there are weird tips that tell me that she enjoyed the harassment. She shared privated conversations here, and she tried to twist Aevin words all the time to provoke harassment of the community against the staff :/ I was in Purezc since 2005, and I don't remember that the staff force a member to change all his ratings. This is just a special and weird case..

[Sheik]

I just did a ctrl+f for "Horrible" looking through Lunaria's reviews. The only one I found where it came up in a negative connotation was Banana Blood God, a four star review. Top kek. It's the context that matters, I think. If you call a horrible quest horrible (you can only call quality subjective so far, broken is broken and there can be no denial) is that a problem? Not really. Call a spade a spade. When a review of something less egregiously bad focuses mainly on the negative...Your mileage may vary. I don't mind reviews that get straight to the point, but some members take it more personally. I didn't find the BBG review that bad though, personally.

Then it was a wrong use of the word in the given context. The rating system itself explicates that horrible=0. A review based on this system should use the terms as suggest by the system, obviously. In principle at least, since it doesn't make sense any other way.

[grayswandir]

(Honestly those labels are kind of just still around from days long gone, and the exact labels should possibly be revisited perhaps.)

Actually, I wanted to comment on this. I think a lot of the problem here is that different people have a different idea of not only what kind of quest gets what kind of rating, but also what the rating themselves actually mean. Horrible/Good/Whatever is really rather vague.

 

What I'd like to see is a set of "official" guidelines on what constitutes a given star rating. I'm thinking something like a short paragraph or several bullet points for each rating. I don't want these to be actionable or hard limits on what kind of quest gets what kind of rating. Instead, I just want a decent way to calibrate what I think is 3 or 4 stars versus what everyone else does. That would help to reduce one of the two sources of variance, which is rather helpful for the low number of reviews we get, statistically.

 

So, basically, if you're going to be mandating that people rate a certain way, I'd like a bit more information on what that way actually is.

[LinktheMaster]

Alright, alright. This is getting really off topic, and it doesn't benefit anyone or this conversation to keep targeting members. Let's get back on the subject of reviews and the rating system.

 

(Thanks, grayswandir. You posted that before I made my reply. Your post is fine.)

[Moosh]

Alright, alright. This is getting really off topic, and it doesn't benefit anyone or this conversation to keep targeting members. Let's get back on the subject of reviews and the rating system.

Sorry about that. I fail at reading today. Lack of sleep probably. My intent wasn't to target anyone.

 

Actually, I wanted to comment on this. I think a lot of the problem here is that different people have a different idea of not only what kind of quest gets what kind of rating, but also what the rating themselves actually mean. Horrible/Good/Whatever is really rather vague.

 

What I'd like to see is a set of "official" guidelines on what constitutes a given star rating. I'm thinking something like a short paragraph or several bullet points for each rating. I don't want these to be actionable or hard limits on what kind of quest gets what kind of rating. Instead, I just want a decent way to calibrate what I think is 3 or 4 stars versus what everyone else does. That would help to reduce one of the two sources of variance, which is rather helpful for the low number of reviews we get, statistically.

 

So, basically, if you're going to be mandating that people rate a certain way, I'd like a bit more information on what that way actually is.

As for this...How? What matters ultimately is that you believe a 5 star quest exists and that you believe a 0 star quest exists (that's a lot harder to prove because of positive review bias). Everything in between that is super subjective. There's no way to say "this is what objectively gets a 3" because the standards are constantly changing. Our concern here is with those that might judge using a four star rating scale when the fifth star exists for a reason. It's not fair to the 1/3 of the entire quest database with a score of over 4 stars.

[Sheik]

Actually, I wanted to comment on this. I think a lot of the problem here is that different people have a different idea of not only what kind of quest gets what kind of rating, but also what the rating themselves actually mean. Horrible/Good/Whatever is really rather vague.

 

What I'd like to see is a set of "official" guidelines on what constitutes a given star rating. I'm thinking something like a short paragraph or several bullet points for each rating. I don't want these to be actionable or hard limits on what kind of quest gets what kind of rating. Instead, I just want a decent way to calibrate what I think is 3 or 4 stars versus what everyone else does. That would help to reduce one of the two sources of variance, which is rather helpful for the low number of reviews we get, statistically.

 

So, basically, if you're going to be mandating that people rate a certain way, I'd like a bit more information on what that way actually is.

 

Yeah, that's what I am trying to say, too. This wouldn't even have to be in the rating system itself, it could be like a post that everybody is expected to read, somewhat like the first post in the Quest Screenshot Thread.

Edited by Sheik, 28 October 2016 - 12:10 PM.

[Nimono]

Lunaria is negative even when she rated a quest high... that's the thing. Maybe she is just that way, but there are weird tips that tell me that she enjoyed the harassment. She shared privated conversations here, and she tried to twist Aevin words all the time to provoke harassment of the community against the staff :/ I was in Purezc since 2005, and I don't remember that the staff force a member to change all his ratings. This is just a special and weird case..

Well, pointing out a negative isn't inherently a bad thing. As I stated to Dimentio earlier in the topic, pointing out the negatives AND the positives is integral in offering constructive criticism. The creator has to be told of what they did wrong, after all, so it can be avoided/fixed in the future. For all we know right now, the context of Lunaria's review could be "so and so segment/implementation was horrible, and here's why", and that's valid criticism. It's something that shows the creator what went wrong, and why they didn't like it, leading to some experimentation to try to find a good solution.

[Avaro]

Well, I'm pretty much liking the proposed changes. I don't know about the new rule, but splitting reviews and ratings is a great idea. I hope these changes will happen

[grayswandir]

As for this...How? What matters ultimately is that you believe a 5 star quest exists and that you believe a 0 star quest exists (that's a lot harder to prove because of positive review bias). Everything in between that is super subjective. There's no way to say "this is what objectively gets a 3" because the standards are constantly changing. Our concern here is with those that might judge using a four star rating scale when the fifth star exists for a reason. It's not fair to the 1/3 of the entire quest database with a score of over 4 stars.

I'm sorry, I don't understand this at all? Are you saying that only 0 and 5 stars matter? You're concerned about a 5 being a 4, but not a 4 being a 3? You only care about fairness for the quests over 4 stars?

I'm not trying to be provocative here - I really don't understand.

 

As for "what objectively gets a 3", that's completely not what I meant. Of course it's super subjective, but so are a 0 and a 5. However, just because it's impossible to objectively define doesn't mean you should completely ignore it.

[Air Luigi]

Nimono:

Pointing out negatives is very fair, most people do that. I encourage to do that. But being negative overall is a different thing. All people have their favorites and proper tastes, that's expected. Also, I can understand being negative about quests that you don't like (I did that sometimes). The thing is, a member gives the impression to people that he hates practically the entire database (almost). And that's a bigger deal, because the member is consistently harsh. Obviously the staff noticed this.

Edited by Air Luigi, 28 October 2016 - 01:35 PM.