A Discussion on Personal Rating Standards

[Strolen]

A conversation taken from The Philadelphia Shipyards


((After a couple thoughts of editing the posts to show the actual author I am taking the easy way out and posting quoting it. Apologies for not making it cleaner.))

Too much real world familiarity and players begin to know what to expect, lessening the uncertainty that comes from the unknown. (If you want feel free to add in some decommissioned warship suggestions, there's plenty of room for'em in the shipyard.)Edit: On a side ntoe I welcome any other suggestions you have to enhance the sub and raise it sub above your initial 4.0 rating

Actually, what it is is that I'm not an easy grader. A 5.0, in my book, is "Perfect, cannot be bettered" or a rare "Hammered with so freaking much information that I can forgive a lapse or two." I've given, I believe, only six 5.0s to date.

I must inquire, why set the standard so high? Most here aren't professional writers (nor wish to be) and contribute subs that are designed to be immediately usable by the majority of visitors rather then a mini source book.
(There are some exceptions but those often take a month or more of solid writing and teaming up with others)

With your standards of quality so high, perhaps you would find it more satisfying to provide reviews on drive-thru rpg for products that offer the amount of material you consider worthy of a high vote?

(On a related side note, Am I correct in assuming you expect your own works to be held to the same high standards you holds others?)

What I expect is for people to apply to my works the same standards they apply to anyone else's. That includes their prejudices, their preferences and, of course, their voting patterns. I don't anticipate that my works are going to be everyone's cup of tea, any more than anyone else's is..

That being said, is there some reason my voting pattern is a problem? I wouldn't have figured that a vote which (according to the drop down menu) equates to a "sweet idea with some artistic flair or increased usefulness which really sets it apart" would prove upsetting, enough to provoke a suggestion that I take my business somewhere else. I'm sure that no one here feels failure or attacked if they don't nab coast-to-coast 5 votes, right?   

It's more when the bar is set so high that the average Strolenite is unable to achieve the mark without spending a month on a sub, it becomes "out of reach" for the majority of participants. This makes the "5" of little value when so few can attain it by the rather unusually high standard you assigned to it.(I think perhaps what you consider a "5" others would rate a "Golden sub?")


As for taking your "business" elsewhere, I wasn't aware anyone here paid you for your feedback.


In all seriousness I was merely suggesting if you find the level of content and quality here not up to expectations you may find it more personally fulfilling to provide critiques on RPG material written by those who are in the literal business of getting as close to a "5" on their products as possible.


It's not about anyone feeling they "failed" or are "attacked" by a vote that's not a 4.5+, but more are left wondering "why is this sub not worth a higher vote by people?" Such questions by authors are usually answered by an in depth reply from the commentee about what they liked, didn't like, and want to see more of.


When an voter doesn't provide such feedback, well it's only natural for those seeking to improve their subs occasionally to ask "so how do you think I can improve it?" It's when such a question is asked and the answer is "more content" an author finds it somewhat of an empty response that of little use to them on their future works. (More so if it's on a work an author spent over a week and a half on.)



To put it plainly, if the only way by your standard, for one to earn a "5" is to write a mini source book sub, then perhaps you may wish to consider that your bar is set too high for a site whose main participants contribute as a hobby rather then a means of making a living.

Well, to put it plainly in return, I never said that I required nothing short of "mini source book subs," nor did I say I find the quality of submissions here lacking, nor have I demanded that Strolenites spend a month preparing subs. Would you mind ceasing to infer that I do?

You asked why I didn't give you a higher vote - beyond the "here's a missing element" I already gave - and if you don't like the answer, I can't help you.

What's the goal here - to share gaming material with one another and the gaming public, to get ideas ourselves, and to improve with feedback ... or to chase after perfect scores, and get bent out of shape when we don't receive them? Honestly, I'm dismayed; this is pretty unseemly.

Actually your comment where you stated: Actually, what it is is that I'm not an easy grader. A 5.0, in my book, is "Perfect, cannot be bettered" or a rare "Hammered with so freaking much information that I can forgive a lapse or two." I've given, I believe, only six 5.0s to date.


Implies rather clearly one would need to write a mini source book to fulfill that criteria. (Rather hard to fit "hammered with so freaking much information" into anything short of a sub of that length, or to reach that level of "perfect cannot be bettered".)


As you said, the goal is to get ideas and improve with feedback, and when the feedback I receive isn't sufficient to help me understand how in that commentors opinion I can improve I continue to inquire. (Most view a subs vote score as a eyeball of its "rpg idea value/usage" and naturally want to present and improve their presented ideas when/where practical.)

As for being dismayed by an author questioning your comments, (and the bar you set for voting) and asking for more feedback to improve their current and future subs, well, that's your prerogative I suppose. *shrugs* 

I think this quite a good sub - it has a lot of potential for a modern RPG.


As for my own voting method, I am also quite stingy with a full '5' rating.  For me, a 5 needs to set off alarm bells in some manner - a holy crap moment .  Sometimes it is due to the magnitude and effort, sometimes due to a unique aspect that makes me think.
For the record, there are subs which have the Golden designation that have 4.5 ratings from me.

I don't know _how_ to change this sub to become a 5 - but I need to reserve that score for those subs that do connect with me at that level.   

[EchoMirage]

Well, since a certain not-too-distant incident, I came to think some people apply different standards to what they expect of others and what others may dare to expect of them.

As for my voting preference, I do give 5s rarely. I do not give them solely to long subs, but to those that make me go "whoa, I wish I had thought of that, I gott a use it!", to those subs that spark a dozen new ideas in my head.

So, I think writing a 5-scoring sub should be like getting an A test score - it does not have to be perfect, but the perfection has to be in the nineties.

For the exceptional subs, there's the HoH, and the Golden Vote. I will HoH a sub which I deem exceptional despite not giving it a 5, but I will vote Gold only on subs which I think should be 5's and beyond.

[Dozus]

Way I see it, these aren't legal documents or mathematically-calculated scores, they're opinions. I can name people who practically (maybe literally) never give 5s, and others who give 5s every day. While the Citadel provides suggestions on how to score, it's hard to mandate a standard that will match each individual Citadellian's personal tastes.

For me, I have no problem giving a sub a 5 if I think it's really inspiring and well-written. And I have no problem giving a sub a 3 if it's just okay. I rarely give below a 3, however, because it seems that most of the subs that deserve such a low score are done by first-timers who might need more guidance.

And I think finally, scores shouldn't be taken (nor given) too personally. If I submit a sub, I know it's out there for all to see and judge. Some people will love it. Some won't. If I get a critical score, I just take it on the chin and keep it in mind for future editing and subs.

[Pieh]

I agree with Dozus; his words are very wise. They say a picture is worth a thousand words. A number (a vote) is, to me, worth much less. I have always preferred a detailed comment, especially one that makes me rethink the possibilities of my own submission, over a good vote with very few words attached. There are parts of each Silv's and RGT's arguments that I agree with. It's nice to get good votes, it sucks to work on something you think is great and get a s**tty vote. I would say, for any vote, please sum up what you like and don't like about the submission. However, sometimes it isn't that easy. Not every idea can be a 5er. For as well-written, fleshed-out, and solid your submission might be, there will always be ones that just don't sit well with someone else. For reasons from prose or style, to content and the actual ideas mentioned.

Anyone have a good moral for this story to end with? I'm sick of the voting arguments. Live and let live, vote and let vote. Maybe?

[Silveressa]

Indeed, one should always vote how they feel the piece warrants, but when voting lower then 4.5- 5 I've found it helps to provide a bit of extra feedback to a sub, what specifically is liked, and what can be done to make it closer to a 5 in my eyes.

The extra attention helps the author make future subs better, and perhaps improve one that failed to meet expectations with less of a "where do i go from here?" feeling.

It would also be handy if when a sub is updated those who have voted on it before get a prompt to review the sub and possibly re-vote. (I've seen some subs on here that have gone from iffy 2.5ish to outstanding 4+ after the author took some of the comments advice and reworked the sub, only to have so few re-votes on it, the sub still averages a 2.87.

[MysticMoon]

I think the essential part of the exchange on Shipyards (at least as I read it) was a request from the author to turn the rating into useful feedback. We all have our systems for voting and it is very subjective. I like the voting system, despite its imperfections, but I think the important thing here is that a vote is just a number without feedback (positive and/or negative.)

[Dozus]

I'll amend my comment with what was said above. Comments and votes should always be specific and constructive. "I gave it a 4 because I liked X, but thought Y could be improved," etc. Otherwise, we're just throwing numbers at each other.

[RGTraynor]

Indeed, one should always vote how they feel the piece warrants, but when voting lower then 4.5- 5 I've found it helps to provide a bit of extra feedback to a sub, what specifically is liked, and what can be done to make it closer to a 5 in my eyes.

I can see that.  You did just that when you went back four days ago to the only submission of mine that had had a "5" rating - one I'd posted over two years ago, and other than the two current quest subs, the only submission of mine on which you've voted in at least a year - and gave it a "4," with the main complaint being it was too comprehensive and indepth for your liking.  Gosh, can't please everyone, I suppose.

That being said, this sort of flap is exactly what happens with such a voting system - all manner of angst ensues over "You didn't give me a high enough score!" "How come you gave the other poster a better score than mine?" "You deliberately lowballed my subs because I gave you a low score!" "You hit that sub with a low score because of the competition!"  "It's not that the sub sucked, it's just that you hate (milieu/style/genre)!" and other such predictable permutations.  I assume that people understand that to be the price of doing business here.  (d**n, I used that word again.  Here comes the cracks about "getting paid" once more.)

I editorialized on the subject just the once, and had my head torn off.  Being in a bad place in my life right now and not having been prepared to get angst over a godd**n vote, I don't see much point in belaboring the subject.  See you folks in another half-year or so, if the bad taste leaches out of my mouth.

[Silveressa]

Actually I attempted to provide a quality reply and give you a clear example of what I mean by "constructive feedback".

The amount of information was one of the strong points (which I mentioned in my commentary as a "good thing") What I did say was it felt a little rigid and lacked any plot hooks or adventure concepts and would be improved by such an addition, as well as some tips on how to work the faith into an already established fantasy setting. (I guess you don't want ideas on how to improve a sub and make it more useful to the community?)

As for voting on your stuff, I hadn't come across any of your material until the quest subs, (which I voted and commented on I might add) and so in an attempt to provide a better example for you and give you some useful feedback on your subs at the same time I went to your profile and started with the first sub on your list.  I get the impression no matter what sub of yours I commented and voted on I would of gotten this same response.

Would it have been better if I had only voted without any comment? Or ignored your subs entirely? 

If I wanted to point out your hypocrisy I could mention you've done something vaugely similar on The Temple of the Man sub, ( http://strolen.com/viewing/803#77375 ) giving it a less then five. (different then 19 other "5" votes I might add) yet with a snarky comment that was of no benefit to the author what so ever or explained why it was worth less then 5 to you.

To summarize, in my honest opinion the number one votes doesn't matter what matters is one should generally try to:

1: Vote true to themselves and what they feel the sub value is to them.

2: Provide feedback when voting lower then 4.5 on what can be done to improve the sub in their eyes, as well as point out what they do like within the sub and their over all impression of it.


Not too hard. Granted #2 can be time consuming and take effort, so we (myself included) don't always do a solid job there, which is no big deal in the grand scheme of things.

It's when the author asks a commentor for more feedback on how to improve and that feedback is unhelpful or provocative that difficulties arise.

[Ancient Gamer]

RGT: I quit The Citadel for similar reasons.

I know The feeling.

You seem like an emotional guy, attached to your subs. So am I, and many others as well.

You also seem like a Stern critic. I am as well. When people give me feedback that sticks in mu throat, I usually try to accept it, learn from it and Move on.

What Silveressa has done here has probably been a "crime of passion". The way you two lunge at each other confirms that.

Problem is: We don't live in a perfect world. Your criticism may be true and well meant, but talk to a musician. Everybody hates the critic.

My guess is that Silveressa got hurt, and like women Are wont to do, she stabbed you in the kidney, where it hurts The most.

Ria Did The same to me.

There is just no honor with 'em womenfolk.

And your feedback, as mine, is cruel and harsh, and tough to swallow.
We Are sadistic and vain and egotistical.

So be it.

Stay. Contribute. Learn to deal with other people, Even though they Are insufferable

[Silveressa]

My guess is that Silveressa got hurt, and like women Are wont to do, she stabbed you in the kidney, where it hurts The most.

If I was going to stab him, I'd go for the jugular, incapacitating wounds don't do much but enrage a target, a clean kill is the surest way to victory. 

There is just no honor with 'em womenfolk.

What can I say? We learn from all the honorless examples men like you regularly put forth. 

(tbh AG your post was better suited for a PM to Traynor as it works as a classic piece of flame bait for the community and will only serve to drag this discussion further off topic.)

Edit:

On second thought I'm locking this thread to prevent just such a flame war from erupting and creating further discord within the citadel.

[Ancient Gamer]

First thing:
Silveressa and RGTraynor: Don't vote or comment on each other for a while, okay?

As for me:
Locking it is fine, but I think another admin than you should have done it. You are a part in this conflict, and I don't think either of you are innocent.
It is a typical hurt feelings scenario.

Seen 'em a million times in here.

The sexist commentary was flame bait for sure. I like going to extremes to make a point.
Actually I love women, and most of my best friends are female.

I am an honorless male for sure.
Worst thing about it is that I am proud of it! Love every second, and gonna continue being honorless to the day I die! (In bed with a filthy woman, if I get my way :p)

RGTraynor: You and Silveressa dish this out via pm. If I got my way you'd both stay active.

[Strolen]

Sigh

Just when you think it is safe to go back into the water. I unlocked the thread because I really see no reason to stop any conversation on the Citadel. I think old quests are the only locked things on it...

Let me try and clarify a few quick things. And I only breezed through the thread and didn't read it in depth because, frankly, there is no reason to, the Strolenati has lived it all before.

I know people take their stuff pretty seriously and there is a lot of pride in certain subs but I never can understand how these kind of things get out of hand. Although I am an INTP and don't let many things bother me so that is my personality.  It always seems to be some of the best authors with awesome, mature and critical minds that, for some reason, start knocking heads. I think the main problem is the fact that they are awesome, mature and critical. It seems harder when two folks like that get together and try and understand each other. Moonhunter and I still don't understand each other sometimes when we are saying the exact same thing using different words. Usually takes a week for Moon to explain it enough for me to go "Ah ha" and comprehend that we were saying the same thing the entire time.

In some cases though, we need the stupidest cliche: Agree to disagree.

There are probably half a dozen people reading this that have had this issue. The disagreement on how to vote, the perceived revenge voting that is claimed as innocent (and it usually is innocent EXCEPT the timing behind it belies the intent of the vote. When there is a heated disagreement it is bad form to do any more voting on the other party till the debate is settled as it will either be revenge voting when low or sympathy vote when high. Either way it won't be appreciated. Even after a seize fire has been called I suggest a honeymoon period of no votes till a long, comfortable truce is attained.)

These kind of issues have been discussed since the beginning (for instance: http://strolen.com/guild/index.php/topic,1834.msg25606.html#msg25606) and there are more threads out there.

There are multiple submission on it too:
http://strolen.com/viewing/Whats_in_a_Vote
http://strolen.com/viewing/Voting_Practices

Voting has just as many negative issues as it does benefits. Honestly, debating on your opinion on how votes/comments/whatever in relation to your own opinion probably won't get that far with anybody. Sure it would be interesting to understand why they voted that way but chances are you aren't going to agree or fully understand it and trying will just frustrate everybody. Anyway, as soon as you understand one voter and get what they want there will be another that doesn't agree with that and you are back where you started with no real gain. A feedback loop that will drive you batty.

There is no fix to this and there are no answers. It boils down to the personalities and their ability to put aside their confusion and frustration with each other and soldier on.

Bottom line with the voting: You don't have to agree or understand the system the person uses.  As long as the person is basically consistent unto themselves with their voting/upvoting/HoH/Golden/whatever, that is the best thing we could ask for!

I really do hope to see you both stick around. It would be a horrible hit to the Citadel if we lost you over such a matter as submission voting.

[Silveressa]

It will take drastically more then a verbal disagreement over voting ethics to make me leave the Citadel. (My former 2-3 year absence was due to real life circumstances far beyond my control and an unfortunate series of events that followed.)

More then anything the Citadel is home, and its members family. For better or worse not everyone in a family gets along due to conflicting view points, beliefs, or personalities.

Taking a step back and look at the discussions about voting, I can see part of a trend where people who don't understand others voting habits can respond poorly to what they feel is an overly critical response to their sub. (When in reality person X is critical of all subs they vote/comment on so this is a "normal" response form them.)

Perhaps if there was a way to see at a glance a fellow Strolenite's voting style and how critical they are when voting it would help prevent such reactions?

It might even be interesting to add an (optional) aura rating to peoples profiles, that lets people understand at a glance a low(er) vote from this person isn't necessarily a bad thing, it's just they are tougher on subs/more critical then some of their fellow Strolenites.

Something sort of like this:

Aura Ratings

A user's aura would representative of their voting disposition. There would be three types of auras, none either bad or good. (perhaps their voting trends could shape their auras color or the user could choose it themselves on their profile, or hide the aura rating entirely if they wished.)

Neutral: A Strolenite with a neutral aura commonly casts votes that are close to the average score for that particular sub They give each sub a fair chance and their opinion doesn't sway too far from the group.

Light: A user with a light aura tends to give subs high marks, including subs that everyone else might not. Perhaps this Strolenite is easily pleased by subs of a specific genre and/or only votes on subs they like. Whatever the case, this Strolenite tends to be more supportive and likes to make people feel good about their work.

Dark: A Strolenite with a dark aura has a tendency to vote lower on subs. Keep in mind, they don't have a dark aura because they vote low on low quality subs - they have a dark aura because they vote lower on subs that most other people vote higher on. Strolenites like this are usually more critical of subs, and have set a high standard for excellence. Nearly every sub in the Citadel could use enhancement and improvement so just because someone has a dark aura doesn't necessarily mean they are unreasonably harsh. They might simply see the potential of subs and trying to nudge the authors to raising these subs to the level of quality they feel it is capable of.

Just a rough idea that may work well; thought I'd toss it into the discussion as another enhancement possibility. 

[valadaar]

I think it is easier then that.

1. Make no assumptions about votes.  Be happy that there are any at all.

  Vote only? Good. Sub's existence was acknowledged.
  Vote + Small comment?   Great! Had an impact of some type of kind.
  Vote + Constructive comments?  Awesome.

I have many subs here that got essentially a "meh" response. I've learned not to blame the audience. If it was a 'meh' response, there was probably a reason.  It was overlooked? There were too many subs or the Summary did not sell the piece. It was panned? Probably for cause. Review and rewrite if you feel it necessary. These of course are generalities, but generalities can save a lot of time.

Questioning votes rarely has a positive outcome from what I've seen. Those that question mine tend to see me vote less frequently - if at all - on their subs, as for me, time has become such a precious commodity that I have to carefully ration it.  I have to assume it is for others, so even a minor comment on makes my day.

I also think it may be time to drop XP penalties for low-rated subs.

[MysticMoon]

I also think it may be time to drop XP penalties for low-rated subs.

I second that.

[Ancient Gamer]

Val is our voice of reason

[dark_dragon]

Neutral: A Strolenite with a neutral aura commonly casts votes that are close to the average score for that particular sub They give each sub a fair chance and their opinion doesn't sway too far from the group.

Light: A user with a light aura tends to give subs high marks, including subs that everyone else might not. Perhaps this Strolenite is easily pleased by subs of a specific genre and/or only votes on subs they like. Whatever the case, this Strolenite tends to be more supportive and likes to make people feel good about their work.

Dark: A Strolenite with a dark aura has a tendency to vote lower on subs. Keep in mind, they don't have a dark aura because they vote low on low quality subs - they have a dark aura because they vote lower on subs that most other people vote higher on. Strolenites like this are usually more critical of subs, and have set a high standard for excellence. Nearly every sub in the Citadel could use enhancement and improvement so just because someone has a dark aura doesn't necessarily mean they are unreasonably harsh. They might simply see the potential of subs and trying to nudge the authors to raising these subs to the level of quality they feel it is capable of.

That's an interesting idea... I wonder how much statistics on a person's voting history you could use. Creating a distribution of votes for a particular person would be really interesting (vote on the x axis vs frequency on the y) and then tie that to the deviation of a person's mean vote rank against the complete citadel average.

You know Strolen, you actually have access to a very interesting dataset. There's probably a bunch of interesting research that could be done on approval, influence and peer review systems.

As for the voting issue, it reminds me of Fight Club (the movie):

You are not your job. You are not how much money you have in the bank. You are not the shoes you wear. You are not the contents of your wallet.

We are not our subs. We are not the rating of our subs. We are not the view that others hold of our subs. We are not the views that others hold of us.

Honest - Even if it sometimes feels that way.

PS: Oh, and hello everyone, long time no see.

[Dozus]

Welcome back, d_d!

I agree with you to an extent. The trouble with the Internet is, you're putting a very small piece of yourself onto a site, and setting it up as a symbol of yourself. Even if you're not the sum of your subs, that's all people are able to see on the Citadel - a collection of ASCII that you wrote and is, as far as anyone can see, you.

So I totally get it when comments are subs are taken personally. I've done that plenty of times myself. Even if the sub isn't the person, I think they should garner a level of respect, a level that demonstrates you're aware there's someone on the other side of the Interwebs that wrote the item and wants to know what you think about it. We should abide by a certain unwritten code which says, as someone on the Tubes so delicately put, "Be a f***ing human."

Hope that made some semblence of logic.

[dark_dragon]

a level that demonstrates you're aware there's someone on the other side of the Interwebs that wrote the item and wants to know what you think about it. We should abide by a certain unwritten code which says, as someone on the Tubes so delicately put, "Be a f***ing human."

Oh totally. I know that rule as Wheaton's Law where Wil says: Don't be a dick! 

For those with a spare hour:

PAX2007 Keynote

[Scrasamax]

Wow, unexpected drama

[Silveressa]

Wow, unexpected drama

It's summer, all the soaps are on reruns. 

Seriously though I think the drama died after the commercial break and the subject has moved onto DD's return to the citadel, 

speaking of which:, Welcome Back Dark Dragon! 

[dark_dragon]

speaking of which:, Welcome Back Dark Dragon! 

Thankyou Silveressa!

I probably didn't pick the right thread to make my return

I have been kept pretty busy (2 moves and a new job in the past four months), so I haven't been able to keep up with the goings on of the citadel at all. (Or keep my resolutions and promises.)

Hopefully, now that I'm settled in my new job, I'll have more time to spend in here! I missed this place

[Ancient Gamer]

D_d!!!!!!!

[RGTraynor]

I really do hope to see you both stick around. It would be a horrible hit to the Citadel if we lost you over such a matter as submission voting.

Mm, somewhat prescient on my part, that I said I'd see folks in half a year.  (Not that I have any intention of returning, what with the sour taste still in my mouth, but this thread turned up in a Google search of my handle.  Go figure.)  In any event, I'm sure the Citadel has gone on just fine without me.