So, after some consideration, I've come up with a compromise that I am hoping to put into place in the coming weeks. This issue and related problems such as people using reviews as a method of revenge have come up multiple times in the past. It's an issue that has divided the community for a long, long time, and I absolutely hate seeing something fracture everyone like this.
Despite everyone's disagreements, we all need to remember that we have the common bond that we're all here because of Zelda Classic, and the reason we're all passionate about this is because we enjoy making and playing quests. The reason we're all here and discussing this is because we all honestly want an environment that is, to the best of our abilities, best for everyone. While there are obviously differences in the thoughts on what that is, I think the string that ties us all together fundamentally brings us closer than the tears which things like this create. There's really no reason for any vile behavior in regards to all this. We can all discuss this calmly without accusations of bad intent, putting words in each others mouths, or acting like the other party is stupid for their beliefs. None of that stuff helps either make a decision or bring the community together.
Everyone lashing out and piling on just one or two people isn't the right way to request change, either. I know the rating system is dear to everyone's hearts, but I don't think we need multiple people grouping up to pick apart a single person's statements. Friendly discussion and disagreements about things like this are fine (and somewhat expected since I think it's foolish to expect everyone to like something), but I really don't think the way this topic has run its course was the proper way to do this. I think it's much better for all of us to come together civilly to come up with a solution that most can agree on.
Which comes back to my proposal of a compromise.
1. Reviews and ratings will be split
Basically, people will be able to rate quests without writing a reviews, and people will be able to write a review without assigning a rating. My hope is with this change, it will promote a higher number of ratings as they'll be easier to make while also raising the quality of the average review as people who will be writing them won't be writing them just so that they can make a rating.
2. Ratings will stay as a 5 star system
I still think this is the best middle ground. We've had some people in the past ask for a base 10 system, or even a base 100 system. So, I don't think everyone necessarily disagrees with the base 5 system in the same way. Any conversion of the old rankings would be wonky, and with it not being a clear near unanimous decision it being better, I don't think it's worth the sacrifice of correctness of the old reviews.
3. Average ratings won't be affected by outliers as much
To compensate for us using the 5 star system, I'll try to come up with a new algorithm that will minimize the impact of outliers. For quests with large number of reviews, this means that some ratings will not be factored into the average rating. For quests with fewer reviews, it'll probably mean that outliers will be less impactful on the rating.
4. A new sorting category will be added (at least for quests)
To either supplement or replace the database sorting on ratings, I'm going to come up with a system that will instead sort entries on popularity. This will include several different data points, not just ratings. Not that ratings will be ignored, but they won't be the sole deciding factor in the ranking. It will be similar to the trending quests, but factor in all historical data rather than simply current numbers.
5. The new rule will stay in place
While I do understand the concern regarding the rule, we have had people in the past use ratings as a means to an agenda. Be it as revenge for a completely unrelated comment, as an statement against the system in general, or because they want to have control over the list of top rated quests. We've also seen people who act vile and bash quests with little to absolutely no constructive criticism. You may think it's absurd that this would happen, but we have unfortunately seen these sort of things multiple times. These things simply hurt the community as a whole. they remove any value of ratings and they can dismay quest makers who see bad reviews but don't know how to improve themselves. I don't think it's unreasonable to have a rules in place to prevent any sort of abuse of ratings, even if such a thing almost never happens.
I know that this won't please everyone. Heck, I'm not sure if it'll please anyone completely. But, I do think it's a bit of middle ground with not necessarily scrapping the entire current system, but addressing some of the biggest concerns with it.
I'm hoping to have the changes completed in a week or two.