134 replies to this topic

[Jamian]

Hi Air Luigi, I appreciate that you are defending my quest but I really don't want it to cause arguments, that makes me uncomfortable. Obviously it is for some people but not for everybody At least some people are really enjoying it and that's all that matters to me. Now back to the topic at hand...

Edited by Jamian, 03 April 2013 - 07:06 PM.

[Omega]

Should (4. Power of Geduina) be rated that high? Maybe if it were translated over to English?

[Aevin]

Ouch ... I think it's really dangerous to bring up specific quests. Even if you don't think they're that good, I wince thinking of the people who put effort into them. Individual reviews are one thing, but personally I'd be crushed if someone came out and said "your quest just doesn't deserve the ratings it gets." It's not up to single individuals to decide where a quest belongs in the database.

I'm just interested in seeing a rating system that stratifies the quests as appropriately and clearly as possible.

Edited by Aevin, 03 April 2013 - 07:06 PM.

[Russ]

They didn't even have the intelligence of rating the quest little by little, but instead they all just went in to rate it at the same time. At least make an attempt to hide it...

It's clear that "the special group" got together on this, that's the sad part, even more so than their opinion and rating. But if denouncing an injustice is prohibited in this forum, then I won't insist anymore.

Stop right there. For starters, the ratings came over three days, not all at once. By that logic, I guess I oughta accuse arieltap, Mister Snooze, and Artistic of ganging up to give it a high rating. Do you see how ridiculous that sounds? Just because multiple people dislike a quest doesn't mean there's some conspiracy going on. Yes, all the ratings came within three days. But it's not because we all ganged up; the quest became very popular very quickly, so we all played it to see what the hype was all about.

Basically, different people have different opinions. To take a close-to-home example, you didn't like my quest. Fair enough, I can see the reasoning behind not liking it. But I'm not gonna start accusing you of ganging up with people to rate it down. And I certainly wouldn't do something like tell Jamian he had no place rating my quest if he were to give it a bad rating, simply because I didn't like his own quest. That kind of attitude's not okay; you can't start calling out people like that. But furthermore...

If you have a problem with this, you may take it up with me or another staff member via PM, or use the dropbox. Otherwise, let's stay on topic here, people.

If the staff say stay on topic, please, stay on topic. If you want to respond to me, please do so, but do it via PM, not here. This thread's been derailed enough.

[Omega]

For those interested, here's the new top 10 list:

1. The Hero of Dreams
2. Origin
3. The Hidden Duality: Director's Cut
4. Power of Geduina
5. Stranded
6. How To: Over-Under Bridges
7. Zelda's Butt
8. MegaMan: Dr.Wily's Revenge - Director's Cut
9. Link's Birthday Deluxe
10. Bikdip's Adventure 2: Electric Boogaloo

It doesn't greatly change the list, but some things are shuffled around a bit.

But, like I said, no system is prefect. Length sounds good at first thought, but I'd rather not encourage people to spam up their reviews just to make them count more. Like/dislikes only really work with large numbers of reviews and could make "hidden gems" even more prone to being lost if they don't happen to get reviewed much at all. I'm happy to consider a better system in the future if everyone likes it, but you're never going to find that perfect system that fits every scenario.

What's Zeldaclassic system based on, or is it relative to PureZCs current other then 5 star max? Also no lost Isle there? Gasp! I thought that quest had mostly 5 star ratings.

[Koh]

What's Zeldaclassic system based on, or is it relative to PureZCs current other then 5 star max? Also no lost Isle there? Gasp! I thought that quest had mostly 5 star ratings.

I believe it wasn't there because LI was made during the betas, and has a very specific build to be used. But otherwise, maybe ZC.com uses the Bayesian system. Well I can definitely say whatever system is used, the icon should definitely be more accurate to the decimal. If it's still 5 stars, we should be able to see 4.75, 4.5 and 4.25 stars (quarters should at least be the minimum.). The rounding of whole stars has always been, and always will be one of the strangest things.

[Dawnlight]

I personally think the rating system should be more through. Other than just one score, I think there should be other scores based on Design, Gameplay, Presentation, etc. I was thinking of something like RatemyProfessors.com.

Then again, like several people stated, some reviewers don't go as far as being through when reviewing and some could make detailed paragraphs describing all the nooks and crannies about a quest. In other words, no matter how simple and advanced a rating system is, you are bound to have flaws because people have many different ways to express their opinions about things.

Just go to Metacritic and see how different the Reader's Average is compared to a game's Metacritic score.

[LinktheMaster]

A percentage system is fine.

As for the Bayesian average, the problem is something can never be 5/5 (or 10/10) or 0/5 (or 0/10) because of that random 3 (or 5) rating.

... So? Honestly, right now, when do you see database entries with an actual 5/5 that doesn't have less than 4-5 reviews? Even PureZC's most popular quests have low ratings, so they're not going to have a 5/5 anyway. Now, if you're talking about having 5 stars, then quests will still have that. For images, everything is rounded to the nearest star. As far as doing quarter stars... honestly I'd rather not because I just don't think visually that looks good. Maybe half stars? But both on the old and new site, if you hover over a star, you can see the full rating.

... the constant is 3.

I think what he meant was that it has more "dummy ratings". So for a submission with 3 five ratings, you would maybe have something like this:

Average: (5+5+5)/3 = 5
Base 1 Bayesian Average: (5+5+5+3)/4 = 4.5
Base 3 Bayesian Average: (5+5+5+3+3+3)/6 = 4

While this sounds good, I have to sort of hark back to what I said about likes/dislikes. I don't think PZC is big enough for this. With PZC getting as few ratings on quests as it does (most regular quests have 10 or less ratings I believe), I think having more dummy ratings my skew things a little too much toward the center. I think it would start shoving normal submissions way too far into the center to the point that I think they would sort of blend together near 3 too much.

What's Zeldaclassic system based on, or is it relative to PureZCs current other then 5 star max? Also no lost Isle there? Gasp! I thought that quest had mostly 5 star ratings.

Unfortunately, I can't say. They don't seem to show any detail about the number of ratings a submission go or it's actual value.

I will say that from my research, Bayesian Average is pretty much deemed the best formula for general averages based off of normal scores.

[Peteo]

Also no lost Isle there? Gasp! I thought that quest had mostly 5 star ratings.

Haha, weird, isn't it. Even my wastly inferior MMDWR quest is above Lost Isle.
Have you looked at some of the ratings there in the Lost Isle page?

"I can't find level 1" - 2 stars

"This quest is ok... The only fun level was level 4. I thought this quest would be better but it wasn't." - 1 star


And it's just not Lost Isle that suffers from these kinds of troll ratings, here's one from Dr Wilys Revenge

"Sorry to give it a 2, but this quest clearly doesn't belong in this database." - 2 stars


And from Link to the Heavens, which seemed like a good quest

" It's not a terrible quests, but is it really worth 5 stars? Seriously." - 2 stars



I don't know about the rest of you guys, but I think it would be great if people were required to post more than a few lines of text when giving a 1 or a 2 star rating to a quest. I mean I'm not upset about bad ratings, I used to be a pretty harsh critic myself, but the low ratings would mean much more if people gave reasons why they dislike the quest so much. Just saying "boring" or "oh no, classic graphics, yuck!" shouldn't be allowed in my opinion. Heck, even one liner 5 star ratings are annoying. I'm sure everyone appreciates a 5 star rating, but if it only says "excellent!" or "good quest!" why write anything at all?

Sorry if this is a little off topic from the rating system discussion, but at least were talking about ratings so I had to bring that up.

[Omega]

Haha, weird, isn't it. Even my wastly inferior MMDWR quest is above Lost Isle.
Have you looked at some of the ratings there in the Lost Isle page?

"I can't find level 1" - 2 stars

"This quest is ok... The only fun level was level 4. I thought this quest would be better but it wasn't." - 1 star
And it's just not Lost Isle that suffers from these kinds of troll ratings, here's one from Dr Wilys Revenge

"Sorry to give it a 2, but this quest clearly doesn't belong in this database." - 2 stars
And from Link to the Heavens, which seemed like a good quest

" It's not a terrible quests, but is it really worth 5 stars? Seriously." - 2 stars
I don't know about the rest of you guys, but I think it would be great if people were required to post more than a few lines of text when giving a 1 or a 2 star rating to a quest. I mean I'm not upset about bad ratings, I used to be a pretty harsh critic myself, but the low ratings would mean much more if people gave reasons why they dislike the quest so much. Just saying "boring" or "oh no, classic graphics, yuck!" shouldn't be allowed in my opinion. Heck, even one liner 5 star ratings are annoying. I'm sure everyone appreciates a 5 star rating, but if it only says "excellent!" or "good quest!" why write anything at all?

Sorry if this is a little off topic from the rating system discussion, but at least were talking about ratings so I had to bring that up.

Some of those ratings there you mentioned are pretty atrocious. SMWcentral.net requires you to write a review with a higher character amount then PureZC. And they also have staff who make sure trolls don't post crap comments and wrongfully rate a hack. I personally don't rate hacks or quests, even if I dislike them. I only rate if I enjoy them. Same with Youtube, I've not thumbs down ONE video in my life. Even when I was raging about Lost Isle I never went and rated it bad. Same goes for any quest. You either get a five or you get no rating at all from me.
I agree though, this idea for more feedback for lower votes sounds good. Higher star votes seem to effect you less. You're in the list of higher rated quests rather then the lower list which gets less views I'm sure.

[Koh]

As far as doing quarter stars... honestly I'd rather not because I just don't think visually that looks good. Maybe half stars? But both on the old and new site, if you hover over a star, you can see the full rating.

I bet you were thinking of it Zelda style, and seeing the stars fill up by the corners...and that's not what I meant XD. I just mean filling horizontally, so a quarter fill would look as filled as halfway horizontally as half a star. Think ZC.com with it's half stars, but more accurate! Why hover over whole stars when the stars themselves should be accurate enough to depict the true rating?

[Avaro]

... So? Honestly, right now, when do you see database entries with an actual 5/5 that doesn't have less than 4-5 reviews? Even PureZC's most popular quests have low ratings, so they're not going to have a 5/5 anyway. Now, if you're talking about having 5 stars, then quests will still have that. For images, everything is rounded to the nearest star. As far as doing quarter stars... honestly I'd rather not because I just don't think visually that looks good. Maybe half stars? But both on the old and new site, if you hover over a star, you can see the full rating.

I think what he meant was that it has more "dummy ratings". So for a submission with 3 five ratings, you would maybe have something like this:

Average: (5+5+5)/3 = 5
Base 1 Bayesian Average: (5+5+5+3)/4 = 4.5
Base 3 Bayesian Average: (5+5+5+3+3+3)/6 = 4

While this sounds good, I have to sort of hark back to what I said about likes/dislikes. I don't think PZC is big enough for this. With PZC getting as few ratings on quests as it does (most regular quests have 10 or less ratings I believe), I think having more dummy ratings my skew things a little too much toward the center. I think it would start shoving normal submissions way too far into the center to the point that I think they would sort of blend together near 3 too much.

Unfortunately, I can't say. They don't seem to show any detail about the number of ratings a submission go or it's actual value.

I will say that from my research, Bayesian Average is pretty much deemed the best formula for general averages based off of normal scores.

Now that's pushing it a bit.. It's completely nonsense! Why should the quest be rated 4,5 or even just 4 when it's got 5 star reviews only?

Edited by Avataro, 04 April 2013 - 05:47 AM.

[LinktheMaster]

I know it sounds silly at face value, but like me give you an example.

Hero of Dreams, the highest rated quest on PureZC currently, has a rating of 4.85 with 102 ratings. That's a really really solid review set, so you know it's going to be a fairly accurate depiction of the quest. Now let's say Joe Schmoe uploads a quest and immediately get a couple of his friends to rate it. Say his 3 PZC friends all rate it 5.

Which rating would you trust more? Hero of Dreams, right? Currently, which one is rated higher? Joe Schmoe's quest, because it has the higher average.

Let's take a look at the opposite end of the spectrum. Mario Quest currently has a rating of .07 with 29 ratings. That's another solid rating that's pretty trustworthy. Now Alex Schmalex submits a quest that's... okay, but not very good. He unfortunately gets targeted by a very harsh reviewer who doesn't like the quest and rates it a 0.

Which rating is going to be more accurate? The one on Mario Quest, obviously. However, Alex's quest would be rated lower than Mario Quest. Other people may skip over it later seeing it have 0 stars.

Bayesian average fixes this by lowering the impact of single ratings. It would take a 13 5-star reviews to trump Hero of Dreams. If you compare that with the amount of reviews it has, I don't think that's particularly unfair.

[Hoff123]

I know it sounds silly at face value, but like me give you an example.

Hero of Dreams, the highest rated quest on PureZC currently, has a rating of 4.85 with 102 ratings. That's a really really solid review set, so you know it's going to be a fairly accurate depiction of the quest. Now let's say Joe Schmoe uploads a quest and immediately get a couple of his friends to rate it. Say his 3 PZC friends all rate it 5.

Which rating would you trust more? Hero of Dreams, right? Currently, which one is rated higher? Joe Schmoe's quest, because it has the higher average.

Let's take a look at the opposite end of the spectrum. Mario Quest currently has a rating of .07 with 29 ratings. That's another solid rating that's pretty trustworthy. Now Alex Schmalex submits a quest that's... okay, but not very good. He unfortunately gets targeted by a very harsh reviewer who doesn't like the quest and rates it a 0.

Which rating is going to be more accurate? The one on Mario Quest, obviously. However, Alex's quest would be rated lower than Mario Quest. Other people may skip over it later seeing it have 0 stars.

Bayesian average fixes this by lowering the impact of single ratings. It would take a 13 5-star reviews to trump Hero of Dreams. If you compare that with the amount of reviews it has, I don't think that's particularly unfair.

Sounds great. I would like this kind of system .

[Moosh]

The problem I see with that is very few quests get that many ratings. I could release a quest with no flaws at all and it wouldn't be able to stand up to Hero of Dreams with this new method because the quests that get more than a small handful of ratings are the ones a the top and bottom of the top rated quest list.

Edit: Also occasionally someone like Jamian will get a decent number of reviews despite being at neither extreme and I really wish I knew what their secret was...

Edit 2: Do you think maybe we could have the site pick a random quest from the database every day and have a Featured Quest of the Day on the front page? I feel like that might help get more attention to the gems in the middle of the quest database that have been forgotten?

Edited by Moosh, 04 April 2013 - 07:47 AM.