Analysis of Scenario Ratings

AuthorTopic: Analysis of Scenario Ratings
Infiltrator
Member # 5576
Profile Homepage #0
Inspired by TM's Scenarios-by-Level chart, I've finally gotten around to doing something I'd meant to do a long time ago: making a chart comparing CSR ratings of scenarios.

Disclaimer: The categorizations of scenarios, while subjective, are my estimates of community opinions, not my own opinions. As described below, I attempted to organize categories only according to patterns I observed in the data.

The chart to which I will be referring is here .

When I initially made this chart, it was simply in alphabetical order. However, I noticed immediately several distinct groups of scenarios: those with a small spread in ratings and a high average, a small spread and a medium average, and a small spread with a low average. Then, there were scenarios with high, medium, and low averages whose spreads were much larger. Having seen this, I began to sort the scenarios into the categories in the current chart. After I'd sorted all of the ones that seemed by eye to obviously belong to each group, I wrote a set of rules describing each group. Using made up numbers, (because I don't remember all of the real inputs to the calculation) supposing that the lowest average of a good scenario was 8 and the highest average of a medium scenario was 7, I averaged these and declared that the dividing line was at an average of 7.5. This is the list of actual rules that I derived:

  • ---'Controversial': Spread >3.0
  • 'Medium': Average >= 5.7
  • ---'Controversial': Spread >2.2
  • 'Poor': Average >= 3.7
  • 'Bad': Average <3.7
Note that 'Controversial-Good' scenarios should not be considered poorer than 'Good' scenarios for which there is 'Consensus', there is merely a greater difference among people's opinions, but the averages lie in the same ranges. Also the distinction between 'Poor' and 'Bad' was mostly for aesthetic reasons, but the division was placed in the relatively large gap between 9Var and UV.
In conclusion: I have no idea what this tells us, but I think it looks pretty.

A few final notes:
The source code of the program I wrote to draw the chart is available. The data I used was yesterday's data on Aran's CSR summary page . The program is very bare-bones, it reads in a plain text file on each line of which is a scenario name in quotes, then separated by spaces the minimum, average, and maximum ratings the scenario received. I'm hoping to either re-write it to parse Aran's page (or get him to port it to php and have it run directly from his data :P ).

I also tested the chart in every web-browser I have; it renders correctly in Safari 3 and Firefox 2, slightly less well in Camino, Opera, etc. Hopefully at worst only the rulers get screwed up, so everyone can see it more or less correctly.

--------------------
Überraschung des Dosenöffners!
"On guard, you musty sofa!"
Posts: 627 | Registered: Monday, March 7 2005 08:00
Shake Before Using
Member # 75
Profile #1
You probably shouldn't include the 0-1 bracket if possible, since sub-1 scores are not legal on CSR.
Posts: 3234 | Registered: Thursday, October 4 2001 07:00
Apprentice
Member # 6009
Profile #2
Pretty neat.
Something like standard deviation or variance might be a much more useful indicator of how much controversy is associated with any given scenario's rating... I guess it would take a bit of effort in terms of data collection though.
Posts: 18 | Registered: Friday, June 24 2005 07:00
Law Bringer
Member # 2984
Profile Homepage #3
I like this chart a lot. It also makes me wish there was more data - more dimensions as well as more votes...
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Law Bringer
Member # 2984
Profile Homepage #4
(Sorry, still no edit button.)

Also, observation: The masterpieces of TM and Kel are noticably more controversially rated than those masterpieces designed by others. Also controversial (but medium on average) are the works of Jeff Vogel.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Agent
Member # 8030
Profile Homepage #5
This is indeed a very nice chart. Great work

--------------------
Decca Records - "We don't like their sound. Groups of guitars are on the way out."
Posts: 1384 | Registered: Tuesday, February 6 2007 08:00
Raven v. Writing Desk
Member # 261
Profile Homepage #6
ASR isn't controversial though.

Great chart. One comment: isn't 'poor' typically a worse gradation than 'bad'? I always remember excellent, good, fair, bad, poor from school (and Angband pseudo-ID).

--------------------
Slarty vs. DeskDesk vs. SlartyTimeline of ErmarianG4 Strategy Central
"Slartucker is going to have a cow when he hears about this," Synergy said.
Posts: 3560 | Registered: Wednesday, November 7 2001 08:00
...b10010b...
Member # 869
Profile Homepage #7
quote:
Originally written by Restoration Action Figure:

ASR isn't controversial though.

Great chart. One comment: isn't 'poor' typically a worse gradation than 'bad'? I always remember excellent, good, fair, bad, poor from school (and Angband pseudo-ID).

Apparently descriptive linguistics does rot your brain. :P

I've always considered "poor" to be less bad than "bad". And the worst Angband pseudo-ID is "terrible"; as far as I'm aware, there's no "poor" pseudo-ID.

--------------------
The Empire Always Loses: This Time For Sure!
Posts: 9973 | Registered: Saturday, March 30 2002 08:00
Raven v. Writing Desk
Member # 261
Profile Homepage #8
You're right about Angband. I went to look for examples and I found... not much. Most of the rating scales or systems I can find through a quick google search that use bad or poor, do not use both. This is accomplished in various ways -- having only one negative rating, having a more clearly horrendous word for the worse option, "very bad", etc.

--------------------
Slarty vs. DeskDesk vs. SlartyTimeline of ErmarianG4 Strategy Central
"Slartucker is going to have a cow when he hears about this," Synergy said.
Posts: 3560 | Registered: Wednesday, November 7 2001 08:00
Law Bringer
Member # 2984
Profile Homepage #9
I haven't seen "bad" in any grading system at school; the worst was always "poor" or "insufficient". I have seen it in a rubric for marking the condition of a library book, though - the scale went "New", "Excellent", "Fair", "Poor", "Bad". Apparently saying that a student is bad would be rude, but the library book doesn't mind.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00