134 replies to this topic

[LinktheMaster]

For those interested, here's the new top 10 list:

1. The Hero of Dreams
2. Origin
3. The Hidden Duality: Director's Cut
4. Power of Geduina
5. Stranded
6. How To: Over-Under Bridges
7. Zelda's Butt
8. MegaMan: Dr.Wily's Revenge - Director's Cut
9. Link's Birthday Deluxe
10. Bikdip's Adventure 2: Electric Boogaloo

It doesn't greatly change the list, but some things are shuffled around a bit.

But, like I said, no system is prefect. Length sounds good at first thought, but I'd rather not encourage people to spam up their reviews just to make them count more. Like/dislikes only really work with large numbers of reviews and could make "hidden gems" even more prone to being lost if they don't happen to get reviewed much at all. I'm happy to consider a better system in the future if everyone likes it, but you're never going to find that perfect system that fits every scenario.

[Moosh]

I wouldn't want a rating with 1024 characters to count that many times and one that's just "Great!" to only count 6. I'd just want ratings that exceed say 500 characters to hold some more weight than ones that don't. If someone does something rubbish and substanceless people can always just report it just like when somebody makes a substanceless review currently.

[SofaKing]

Bayesian average would actually make some sense. And the more reviews a quest has, the less it would impact the score. Nate Silver discusses Bayesian theory a lot in his book "The Signal and the Noise." Anyone interested in statistics, and wants them explained in plain english, should give it a read. Another option is throwing out the highest and lowest ranking for any quest with more than a given number of ratings. That helps with outlying scores. With most quests there's always the five given by someone who liked the first level (seriously, I can't understand rating a game you've played 1/9 of), or the one given by someone who got frustrated too quickly.

[Jamian]

I don't like the "push people to write longer reviews" idea, writing more words doesn't mean the review actually has more substance. And for moderators to decide which posts don't have enough substance to stay would be hard to judge, unless it's really very obvious.

Edited by Jamian, 03 April 2013 - 11:31 AM.

[Aevin]

I think the five star scale with the average system introduced above would be fine ... I do think it's a little frustrating how virtually any system except the existing one disadvantages newer quests. The longer it's been here, the more people play it, so with two quests of equal strength, the one that's here the longest will still be rated higher. It's quite a hill for newcomers to climb, but I guess there's nothing for it.

I actually like Robin's 100-point system better for more accurate reviews, but still with quests starting with a 50% rating. It sucks that, as is about to happen to my quest, a 4.5 shows up as a five star, but once it drops down to 4.49, it becomes a 4 star. If it drops .01, it appears to drop a full star all at once. A percentage system would allow for more accurate comparisons and require a little more thought.

Edited by Aevin, 03 April 2013 - 02:50 PM.

[Koh]

A percentage system is fine.

As for the Bayesian average, the problem is something can never be 5/5 (or 10/10) or 0/5 (or 0/10) because of that random 3 (or 5) rating.

[nicklegends]

I do think it's a little frustrating how virtually any system except the existing one disadvantages newer quests. The longer it's been here, the more people play it, so with two quests of equal strength, the one that's here the longest will still be rated higher. It's quite a hill for newcomers to climb, but I guess there's nothing for it.

In one sense you are correct, but you don't acknowledge the opposite side of the story. Newbies will tend to release lower-rated quests at first. If their average rating is below a 3 (with our implementation), then the Bayesian average actually helps them out in the short term, and the more ratings the quest gets, the lower the score will be.

It sucks that, as is about to happen to my quest, a 4.5 shows up as a five star, but once it drops down to 4.49, it becomes a 4 star. If it drops .01, it appears to drop a full star all at once. A percentage system would allow for more accurate comparisons and require a little more thought.

This is true in the graphical sense, but you can always hover over your star rating to see the extended decimal. Not a full replacement, I know.

As for the Bayesian average, the problem is something can never be 5/5 (or 10/10) or 0/5 (or 0/10) because of that random 3 (or 5) rating.

It's true that you can never reach exactly 5 stars or 0 stars with a finite number of ratings, but I don't see it as a problem. It has the nice property that if two quests get exclusively 5-star ratings, the one with more 5-star ratings will be ranked higher. Inversely, if two quests get exclusively 0-star ratings, then the one with more 0-star ratings will be ranked lower. (The interesting case is, when a quest has an average review of 3 stars it will absolutely always have exactly a 3-star rating as well.)

[aaa2]

Why are you choosing 1 for the constant in the Bayesian average why not a higher number. That would keep ratings sticking near a 3 for a longer time and thus would be good. In the limit of large numbers it reaches the usual average anyways so it shouldnt matter.

[The Satellite]

... the constant is 3.

[Omega]

Well, I'll go ahead and jump in to say that the new site has a more advanced algorithm to come up with ratings. It's called a Bayesian Average. It basically helps weed out outliers by adding in an extra rating of a 3.

So, let's say a new quest comes in and has 2 ratings of 5. Instead of being rated 5, it would be rated like this:

(5+5+3)/3 = 4.33

Likewise, if someone just comes and gives a horrible first review, it helps solve that too.

(0+3)/2 = 1.5

I think this is the best system since it just drifts everything slightly toward the center. The less reviews, the more it pushes the rating toward an basic 3. The problem is, no system is perfect. Personally, I'm not sure a like/dislike system would work well on PureZC. That really seems to shine when tons of people post ratings.

TS is correct, as posted here.

[Air Luigi]

I find it sad that a certain group of people (who we all know) have been giving such ratings to Linked to the Past.

Guys, you are pathetic. I find it pitiful that your little group has gotten together to give awful ratings to such an outstanding quest. This game deserves at least 4 stars. If you don't like that the game follows Z1 style it's understandable, but not seeing the insane amount of fun that there is behind this quest is just sad. The jealously behind some people never ceases to amaze me. If I were you, I would actually start asking myself why more people play Jamian quest than yours... Seriously, think about it for a minute. That Russ, out of all people, came here to teach game design classes is simply hilarious.

[strike]

Fight!
Fight!
Fight!
Fight!




-Strike

[The Satellite]

Stop right there.

Air Luigi, this thread was for the discussion of how quests are rated, which then led to a discussion of the current rating system and the one coming with the new site. This is not an opening for you to call out other members who have legitimate opinions of a quest that differ from yours. Having watched at least two of them stream it, I honestly have to agree with their opinions. The quest is not for them, and not for me; that's none of your business. Neither is it ours if you or others happen to like it. The fact that you're also making personal attacks against these people is completely unacceptable. It's in our rules: "Do not harass other members."

If you have a problem with this, you may take it up with me or another staff member via PM, or use the dropbox. Otherwise, let's stay on topic here, people.

[Air Luigi]

They didn't even have the intelligence of rating the quest little by little, but instead they all just went in to rate it at the same time. At least make an attempt to hide it...

It's clear that "the special group" got together on this, that's the sad part, even more so than their opinion and rating. But if denouncing an injustice is prohibited in this forum, then I won't insist anymore.

Edited by Air Luigi, 03 April 2013 - 06:50 PM.

[Omega]

A percentage system is fine.

As for the Bayesian average, the problem is something can never be 5/5 (or 10/10) or 0/5 (or 0/10) because of that random 3 (or 5) rating.

Kinda reminds me of zeldaclassic.com You rarely see a 5 star quest there. The most I've seen over the years are 4 1/2 stars max. Maybe one 5 star, and that was sorta recent I'm thinking. I remember checking out the rating system years back when searching for top quests, and I noticed it then as well.