Judging: Next Phase

AuthorTopic: Judging: Next Phase
The Establishment
Member # 6
Profile #0
All scores are now in, so begins the discussion phase. Before I go ahead and start this, I want to put something forth. I looked through the scores without doing any rigorous analyis and it appears that almost every scenario stands a good shot of advancing to the finals. Now, I don't care if this be the case as if every scenario is very good, all should advance. However, my fear is that there may be a shortage of judges.

So, before I post scores, I want to get feedback of the judges and scenario authors. We have a few options:

1) The status quo. Most scenarios will probably advance, we accept this, and whatever judges we can find, we accept their scores. Again, this may lead to few judges making decisions that do not reflect a consensus view.

2) We raise the cutoff to 90% of the top score in the category. This will ensure less scenarios advance, but it would involve changing the rules which is not desirable, but may be helpful.

3) In our discussion phase, the judges agree to try to separate out the scores a little better. Foreseeing this, I proposed that a score of 5 be an average, yet it seems people are still following the old CSR scale. This would require no rule changes, but would require the cooperation of the judges.

4) Some idea I have not thought of that neither significantly changes the rules nor requires judges to refine their scores in a large way.

As judges and authors, what are your thoughts on this?

UPDATE:

Regarding option 2, I did, crunch the numbers (I figure they will be out soon enough anyway) and here are the general resultsfor scenarios advancing versus cutoff assuming I got everything right:

80% -- 10/11 scenarios advance
90% -- 7/11 scenarios advance
95% -- 5/11 scenarios advance

I'd lean toward a 95% cutoff or option 3 outlined above.

[ Sunday, December 17, 2006 08:54: Message edited by: *i ]

--------------------
Your flower power is no match for my glower power!
Posts: 3726 | Registered: Tuesday, September 18 2001 07:00
Infiltrator
Member # 5132
Profile Homepage #1
I'd say #2. 3 sounds a little sloppy to me. Of course, I'd like 90%, but that's just because of the selfish reason that I'd have a better chance of advancing. :P

Do what you want.

Question: is the discussion over who wins preliminary, or who wins final?

--------------------
"Let us see what the new day brings." - Temas, Areni.

Visit my realm!

Rate My Scenarios!
Fort Emerald Robbery
The Nephils' Defense
The Final Spire
The Fifth Tower of Magi
The Portal
Posts: 626 | Registered: Monday, October 25 2004 07:00
Off With Their Heads
Member # 4045
Profile Homepage #2
I'm in favor of 2 and 3. Seven scenarios in the finals is about right, regardless of the number of entries. Five would be on the small side, and if they really are pretty near to each other, it might be unfair to eliminate some that are on the brink.

Here's the numbers of scenarios in the finals in previous contests:

3rd: 6
4th: 7
5th: 8
6th: 5
7th: 8

With this in mind, 7 seems pretty reasonable.

Also, there's no reason we should be abiding by CSR ratings here. These scenarios really should score about the same on CSR, but that makes for crappy contest judging (and I realized this as I was sending in my scores).

[ Sunday, December 17, 2006 09:09: Message edited by: Kelandon ]

--------------------
Arancaytar: Every time you ask people to compare TM and Kel, you endanger the poor, fluffy kittens.
Smoo: Get ready to face the walls!
Ephesos: In conclusion, yarr.

Kelandon's Pink and Pretty Page!!: the authorized location for all things by me
The Archive of all released BoE scenarios ever
Posts: 7968 | Registered: Saturday, February 28 2004 08:00
The Establishment
Member # 6
Profile #3
As for the question about who wins what, this relates to the initial round. Of course, this decides who advances to the final round, so it affects who gets to participate in that. I doubt it would change the winner, to be honest, but it would change the number of scenarios judges need to play in the finals.

Either the 90% or the 95% does a decent job of cutting things off, nobody, as the scores stand now, is really close and cut. Again, the numbers I put are subject to change based on how judges reallocate their scores.

[ Sunday, December 17, 2006 09:23: Message edited by: *i ]

--------------------
Your flower power is no match for my glower power!
Posts: 3726 | Registered: Tuesday, September 18 2001 07:00
Law Bringer
Member # 4153
Profile Homepage #4
I'd prefer #3, actually... I'm beginning to realize just how inflated some of the scores I've given stuff on CSR really are, and that carried over a bit into my scores here.

--------------------
Gamble with Gaea, and she eats your dice.

I hate undead. I really, really, really, really hate undead. With a passion.
Posts: 4130 | Registered: Friday, March 26 2004 08:00
Law Bringer
Member # 6489
Profile Homepage #5
#2 with 90% sounds good.

--------------------
"You're drinking liquor because you're thirsty? How nasty is your freaking water?" —Lazarus
Spiderweb Chat Room
Avernum RPSummariesOoCRoster
Shadow Vale - My site, home of the Spiderweb Chat Database, BoA Scenario Database, & the A1 Quest List, among other things.
Posts: 1556 | Registered: Sunday, November 20 2005 08:00
Infiltrator
Member # 4784
Profile Homepage #6
#2 and 90%

I had already decided to be harder on the scenarios in the contest then I would normally do for regular rating. I tried to be tough but fair.

--------------------
Forever Always on Past the End

TrueSite for Blades - Blades Walkthroughs
Pixle Profusion - BoE Graphics Archive
Posts: 563 | Registered: Tuesday, July 27 2004 07:00
The Establishment
Member # 6
Profile #7
Unfortunately, others were a bit too forgiving, in my opinion, leading to an overly generous distribution of scores. I've decided to go with the 90% tentatively, and I've also layed out a basic rubric for what score range means what. Hopefully these will be a little more consistent.

--------------------
Your flower power is no match for my glower power!
Posts: 3726 | Registered: Tuesday, September 18 2001 07:00
The Establishment
Member # 6
Profile #8
Either way, everything should now be set with the private forum. Let me know if there is anything wrong.

--------------------
Your flower power is no match for my glower power!
Posts: 3726 | Registered: Tuesday, September 18 2001 07:00
Off With Their Heads
Member # 4045
Profile Homepage #9
IMAGE(http://home.sanbrunocable.com/~tommywatts03/tm.png)
This is just a great image.

[ Sunday, December 17, 2006 20:20: Message edited by: Kelandon ]

--------------------
Arancaytar: Every time you ask people to compare TM and Kel, you endanger the poor, fluffy kittens.
Smoo: Get ready to face the walls!
Ephesos: In conclusion, yarr.

Kelandon's Pink and Pretty Page!!: the authorized location for all things by me
The Archive of all released BoE scenarios ever
Posts: 7968 | Registered: Saturday, February 28 2004 08:00
The Establishment
Member # 6
Profile #10
I created that account myself. It will be gone before long.

--------------------
Your flower power is no match for my glower power!
Posts: 3726 | Registered: Tuesday, September 18 2001 07:00
Infiltrator
Member # 5132
Profile Homepage #11
Darn, I thought we finally found out TM's birthday. :P

--------------------
"Let us see what the new day brings." - Temas, Areni.

Visit my realm!

Rate My Scenarios!
Fort Emerald Robbery
The Nephils' Defense
The Final Spire
The Fifth Tower of Magi
The Portal
Posts: 626 | Registered: Monday, October 25 2004 07:00
Warrior
Member # 7000
Profile Homepage #12
quote:
Originally written by BainIhrno:

Darn, I thought we finally found out TM's birthday. :P
7/7/87 :P

--------------------
"It's sort of like Star Wars. Except Jeff didn't make Erika shoot first in Avernum 3, nor did he introduce annoying computer-animated aliens." —Arancaytar
The Spiderweb Chat Room
Shadow Vale - My site, home of the Spiderweb Chat Database, BoA Scenario Database, & the A1 Quest List, among other things.
Posts: 178 | Registered: Thursday, April 6 2006 07:00