134 replies to this topic

[LinktheMaster]

The problem I see with that is very few quests get that many ratings. I could release a quest with no flaws at all and it wouldn't be able to stand up to Hero of Dreams with this new method because the quests that get more than a small handful of ratings are the ones a the top and bottom of the top rated quest list.

Edit: Also occasionally someone like Jamian will get a decent number of reviews despite being at neither extreme and I really wish I knew what their secret was...

Edit 2: Do you think maybe we could have the site pick a random quest from the database every day and have a Featured Quest of the Day on the front page? I feel like that might help get more attention to the gems in the middle of the quest database that have been forgotten?

As far as your first comment, I think you're just getting a bit bummed out over some quests not being higher, to be honest. I think most good quests get close to that number. The Darknut Within, for example, has 12 reviews. I was actually wrong in my last post (I was factoring the current rating of HoD, not the new rating, which is 4.83). It would take 12 reviews to be even with HoD. The Darknut Within, had it gotten perfect 5s, would have already tied it after only being on the database for a month. I think that's pretty fair.

I'll admit there are probably some hidden gems, though. But I do think that most really good quests do get around and get plenty of reviews from word-of-mouth.

As far as your second edit, the new site actually already does that, only it's slightly different from what you mentioned. People can feature quests and quest projects themselves. When they get featured, they'll be shown on the site for 3 weeks (3 are shown at a time with each quest getting 1 week as the first quest in the list). Obviously this amount of time is tentative and may be changed, but I picked a week to start out. Right now, the site will randomly pick a quest/quest project only if no quests have been featured by people. However, I have been thinking about adding a bit of a random chance of the site just going ahead and picking a random one regardless. I just haven't gotten around to that yet.

[Avaro]

I know it sounds silly at face value, but like me give you an example.

Hero of Dreams, the highest rated quest on PureZC currently, has a rating of 4.85 with 102 ratings. That's a really really solid review set, so you know it's going to be a fairly accurate depiction of the quest. Now let's say Joe Schmoe uploads a quest and immediately get a couple of his friends to rate it. Say his 3 PZC friends all rate it 5.

Which rating would you trust more? Hero of Dreams, right? Currently, which one is rated higher? Joe Schmoe's quest, because it has the higher average.

Let's take a look at the opposite end of the spectrum. Mario Quest currently has a rating of .07 with 29 ratings. That's another solid rating that's pretty trustworthy. Now Alex Schmalex submits a quest that's... okay, but not very good. He unfortunately gets targeted by a very harsh reviewer who doesn't like the quest and rates it a 0.

Which rating is going to be more accurate? The one on Mario Quest, obviously. However, Alex's quest would be rated lower than Mario Quest. Other people may skip over it later seeing it have 0 stars.

Bayesian average fixes this by lowering the impact of single ratings. It would take a 13 5-star reviews to trump Hero of Dreams. If you compare that with the amount of reviews it has, I don't think that's particularly unfair.

Ok thank you for the explanation, now I get how that kind of rating system would be good. I'm for it now! The "impact of single ratings" has to lowered by averaging it to the center!

[Legen Dary]

I understand the complains, but they seem a little bit childish if you ask me.

Think about it: it's like if you go, for example, to www.rateyourmusic.com or www.imdb.com and you see some movies you consider masterpieces rated too low, or the opposite in this case. It is all about people's opinion, and that's fine.

Trying to change the rating system of a site like PureZC only because you think that there are people rating too high some quests is kind of selfish, in my opinion, though I understand your point of view (and, to be fair, I'm more on your side).

The present rating system is OK and doesn't need to be changed. It is simple and IT WORKS. Complicating things won't make any good to the database or to the people, because the idea is that anyone who plays a quest can register and rate it with the only requirement of writting a short comment. The more complex the rating system is, the less people will participate, and that would turn into a less active community. In fact, if the rating system had to be changed, I would remove the need to leave a comment when rating. That's the key to more participation: simplifying things.


And people will dislike, overrate and troll whatever you do, forever and ever. Peace.

[Eddard McHorn Van-Schnuder]

I understand the complains, but they seem a little bit childish if you ask me.

Think about it: it's like if you go, for example, to www.rateyourmusic.com or www.imdb.com and you see some movies you consider masterpieces rated too low, or the opposite in this case. It is all about people's opinion, and that's fine.

Trying to change the rating system of a site like PureZC only because you think that there are people rating too high some quests is kind of selfish, in my opinion, though I understand your point of view (and, to be fair, I'm more on your side).

The present rating system is OK and doesn't need to be changed. It is simple and IT WORKS. Complicating things won't make any good to the database or to the people, because the idea is that anyone who plays a quest can register and rate it with the only requirement of writting a short comment. The more complex the rating system is, the less people will participate, and that would turn into a less active community. In fact, if the rating system had to be changed, I would remove the need to leave a comment when rating. That's the key to more participation: simplifying things.
And people will dislike, overrate and troll whatever you do, forever and ever. Peace.

I think the flaws that have been brought up about it in this threat proves that the current system does not, in fact, work.

This isn't about changing the system for the benefit of some quests, it's about improving the system in general. I think what the staff has got planned here is a great change, and while it's easy to say how one might do things a little differently, the whole average-thing seems like a really smart addition.

[Omega]

I think the flaws that have been brought up about it in this threat proves that the current system does not, in fact, work.

This isn't about changing the system for the benefit of some quests, it's about improving the system in general. I think what the staff has got planned here is a great change, and while it's easy to say how one might do things a little differently, the whole average-thing seems like a really smart addition.

Robin's correct. It's not simply for one fact. Wasn't this rating system in progress on the new site before this thread was even created? The rating system needs to be changed, that much we got here. Then we branched off into the Like or dislike options and the other suggestions that came with it. Though LTM already has plans for the rating system so ideas here would just have replaced that system possibly if it were good enough. Let's give that system LTM explained a try and if things don't work good for the community later, a new topic on the new site can be created.

[Aevin]

I agree. I can think of other options I personally like better, and some I like a whole lot worse. But I think rounding to half stars and having a center-weighted average sounds like a huge improvement and a good compromise. I like the theory behind it -- assuming a quest is "good" unless people play it and find otherwise (in either direction). Or maybe having an "unrated" graphic for those with no reviews?

Edited by Aevin, 04 April 2013 - 04:25 PM.

[nicklegends]

Trying to change the rating system of a site like PureZC only because you think that there are people rating too high some quests is kind of selfish, in my opinion

The new rating system won't stop this at all, but it's not designed to either. The idea of the Beyesian average is that content has to establish a position via consistently high or low ratings before it ends up at the top or bottom of the list. With the current system, a quest with 999 five-star ratings and 1 four-star rating will be placed below a brand new quest with 1 five-star rating. That's just silly.

[LinktheMaster]

What's Zeldaclassic system based on, or is it relative to PureZCs current other then 5 star max? Also no lost Isle there? Gasp! I thought that quest had mostly 5 star ratings.

Okay. I was able to find out. ZC.com just uses a regular average like PZC does currently.

[Omega]

Okay. I was able to find out. ZC.com just uses a regular average like PZC does currently.

That's interesting. I've seen FAR more 5 star quests here over the years then over at Zeldaclassic.com

[LinktheMaster]

That's interesting. I've seen FAR more 5 star quests here over the years then over at Zeldaclassic.com

Ah. This seems to be because of they round down, so everything looks lower than it really is. For example, anything with a 4 stars is a rating of 4 to 4.49. 4.5 stars is 4.5-4.99. 5 stars is a rating of 5. So it's basically impossible to get a 5 star rating over there unless you have a low number of high ratings.

PureZC rounds its star ratings. So 4 stars is 3.5-4.49. Just as a better comparison, if we had half stars like ZC.com, 3.75-4.24 would be 4 stars. 4.25-4.74 would be 4.5 stars. 4.75 and above would be 5 stars. This is just overall a better system in my eyes because it makes it possible to get a five star rating.

[Shane]

I feel as if there are some ratings that are solely just against another's rating. I can feel it strongly when five star raters point out one or two star rater's names and question their opinion. I saw in especially in a practically quest submission which I would not like to point out. Those are the ratings I'd like to question. But they don't bother that much. In fact, I don't care about the ratings, just what I think of the quest/tileset/tiles/music/script I'm rating.

A more important thing to accept that not everyone will give out five star ratings. While Lost Isle pulled a miracle and got most likely 25+ five star ratings there are some people who gave three or two. And I thought LI was pretty much perfection.

Edited by Shane, 05 April 2013 - 02:53 AM.

[aaa2]

Why not calculate the standard deviation of each rating and give the average rating plus minus standard deviation. The average rating then of course only needs to be as exact as the standard deviation. Say the standard deviation is 1.02 the exact average rating 4.1067 then the rating can be given as 4+/-1 as an example. Say we have a standard deviation of 0.02 and an exact average rating of 3.687 then the rating displayed can be given as 3.69+/-0.02. This way it will be easy to read off how far peoples opinions are apart and what the real average rating is and what chances there are that you will like or dislike a game also.

It is straight forward to combine this approch with Bayesian averages so i wont detail that.

Edited by aaa2, 05 April 2013 - 09:14 AM.

[Avaro]

I feel as if there are some ratings that are solely just against another's rating. I can feel it strongly when five star raters point out one or two star rater's names and question their opinion. I saw in especially in a practically quest submission which I would not like to point out. Those are the ratings I'd like to question. But they don't bother that much. In fact, I don't care about the ratings, just what I think of the quest/tileset/tiles/music/script I'm rating.

Maybe just hide the ratings and comments for the user until he has rated the submission himself, so his rating is not affected by other peoples opinions? I don't know, I'm just mind storming.

[Eddard McHorn Van-Schnuder]

Maybe just hide the ratings and comments for the user until he has rated the submission himself, so his rating is not affected by other peoples opinions? I don't know, I'm just mind storming.

Hehe, it's a novel idea, but it happens to have the unfortunate drawback of rendering the ratings unusable. Remember, their point is to allow prospective players to get an idea of the level of quality they can expect from the quest. If they can't see the rating itself until they've rated it (something they can't do without having played the quest first, according to the rules), then... well, you get the point. xD

[Omega]

Again Robin is correct. Maybe the reviews are hidden but the actual ratings are not? That would at least prevent others from copying other ratings to seem like they know what the deal really is before playing thoroughly. It's hard to figure out what someone meant simply by a score.

Eh?

"You must give a review before you can see other reviews"? Still think that's kinda stretching it but again, eh!