RPGDot Network    
   

 
 
Al Emmo and the Lost Dutchman's Mine
Display full image
Pic of the moment
More
pics from the gallery
 
 
Site Navigation

Main
   News
   Forums

Games
   Games Database
   Top 100
   Release List
   Support Files

Features
   Reviews
   Previews
   Interviews
   Editorials
   Diaries
   Misc

Download
   Gallery
   Music
   Screenshots
   Videos

Miscellaneous
   Staff Members
   Privacy Statement

FAQ
Members
Usergroups
The skewed Top 100 - What can we do?
  View previous topic :: View next topic
RPGDot Forums > RPGDot

Author Thread
Dhruin
Stranger In A Strange Land
Stranger In A Strange Land




Joined: 20 May 2002
Posts: 1825
Location: Sydney, Australia
   

We shouldn't be surprised the list is top-heavy. You can pre-suppose that visitors to our site are RPG fans. Most players will read some reviews or do some basic research before purchasing a game all of which is going to result in more high scores than low ones. I know I haven't played a game I'd rank under "5" for a long time.
_________________
Editor @ RPGDot
Post Wed Jan 21, 2004 9:06 pm
 View user's profile
Lintra
Elf Friend
Elf Friend




Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES
   

First off, sorry about the double post, but the info is very different.

Folks, we have our work cut out for us. Myrthos kindly gave me an excel spread sheet with a dump of the game name, reviewer, rating, and comments. My first pass was to rank the reviews by rating, and a measure of the number of words ... I measured the length of the comment feild as a proxy and divided by 15. The result for the 1 and 10 bracket are as follows:

code:
   Rating   
Length     1     10
0        -       431
1-14     289     872
15-29    164     700
30-44    103     465
45-59    40      328
60+      272     2,109




So this means that over 1/2 of the 10 ratings have comment fields of over 60 characters in length!!! and NONE, that right NONE of the one rated games has a comment feild of 0 length (there was one review that was over 32k of 's**t' over and over).

I read through the comments for the 10 ratings that where 1-14 and the majority were things like Great game, or Excellent. Hard to fault those.

My next cut at the data is going to be by user to see if there are any patterns.

Any other suggestions folks?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless=
Post Wed Jan 21, 2004 9:10 pm
 View user's profile
Gorath
Mostly Harmless
Mostly Harmless




Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany
   

quote:
Originally posted by Myrthos
We can also do it in stars, ranging from a half to 5 stars, with steps of a half star.
10 = 5 stars
9.5 = 4.5 stars
9 = 4 stars
8.5 = 3.5 stars
8 = 3 stars
7.5 = 2.5 stars
7 = 2 stars
6.5 = 1.5 stars
6 = 1 star
Everything below that is 0.5 star

Most games should be above a 6 anyway to be even remotely considered to be worth playing, so adding a few steps could make things easier.

I donīt like this idea. We should use dragons instead of stars!



Sorting the game list:
1. 1/2 stars
2. average rating --> invisible for the reader
3. number of reviews ---> more=better
(4. release date) --> newer=better (because we tend to forget that games are aging)
5. alphabet
_________________
Webmaster GothicDot
Post Wed Jan 21, 2004 9:16 pm
 View user's profile
dteowner
Shoegazer
Shoegazer




Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia
   

This is why I tossed out the idea of starting over. I'm still not saying that's the right answer, but it's looking tougher and tougher to accomplish what we're wanting to achieve and still maintain the integrity of the data.

We could resort to a post-by-post review as Val has done in the past, but that seems to be incredibly labor intensive (Val has implied so, and I certainly believe it), subject to the arbitrary whims of individual team members, and lacking direction for users that would take time to do a review in the future.

Since Lintra has the data and is a number cruncher extrordinaire, he might find a string we can pull to untangle the whole ball. If not, I don't think we should automatically rule out a fresh start. It's entirely possible that we're trying to fly with a handful of lead balloons. I don't see any shame in saying "We've tried everything we can think of, but this just ain't gonna work." You grab a fresh sheet of paper and start a new blueprint.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys!
Post Wed Jan 21, 2004 9:33 pm
 View user's profile
Lintra
Elf Friend
Elf Friend




Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES
   

The next biggest issue is number of reviews done by the reviewer. Almost half of the 0's and 10's come from people who have reviewed only one game. And the 2 review guys are AMOST, but not quite as bad. As the number of reveiws increased, the distribution became much less skewed ... but I am very suspect of those guys who reviewed 10 games and gave ALL of them either a 1 or a 10!

I also found another good one, this guy has 24 reviews, 20 1's, 1 8, 3 10's. Guess he's a hot or cold kinda guy, eh?

I have also found a bunch of names that should just be deleted, like "Falloutsucks" or "FalloutFanBoy" for example.

So what is the next step? Do I start to make a list of all the reviews I think should be deleted, or checked? To whom do I send it? Who is on the team? AGGGGG so many questions.

@Myrthos - how privilaged is the database? Can I give out to just anyone who asks, or only team members?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless=
Post Wed Jan 21, 2004 9:39 pm
 View user's profile
Toaster
Bread Alert
Bread Alert




Joined: 27 Jan 2003
Posts: 5475
Location: Sweden
   

But if we use stars, a game which today have 7 in rating, which should mean a good game, get only two stars, which is interpreted as bad. I think keeping the 1-10 scale is good, but doing as goshuto suggested with it. As said, whatever is chosen, new reviews has to be checked so people doesn't have to do this all over again.
_________________

Tabbrowser Extensions
DictionarySearch
Post Wed Jan 21, 2004 9:40 pm
 View user's profile
dteowner
Shoegazer
Shoegazer




Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia
   

I don't think the non-existent (Corwin taught me how to spell!) committee has a non-existent leader yet. Given recent events, I think Val and I should be taken off the list of leadership candidates if not reduced all the way to "consultant" status. We probably need to make the team official, with a list of who's on it and who's in charge. That duty probably falls to Myrthos.

If wiping out the single-review reviewers eases our problems. I'd do it. Anyone that truly wants their opinion heard on just a single game only has to re-enter a single review and it will be checked prior to becoming part of the database (We seem to have unanimous support for that change). That seems like a lot of "bang" for little disruption and very little effort.

@Lintra- Are there many instances of a single user giving out a bunch (>5) of 10's? This has been my big issue, and if it turns out there are very few instances of that happening, I'll shut my yap about it and move on to the next issue.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys!
Post Wed Jan 21, 2004 10:11 pm
 View user's profile
Lintra
Elf Friend
Elf Friend




Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES
   

quote:
Originally posted by dteowner
...
@Lintra- Are there many instances of a single user giving out a bunch (>5) of 10's? This has been my big issue, and if it turns out there are very few instances of that happening, I'll shut my yap about it and move on to the next issue.


Deftly sidestepping the leadership issue ... Lintra the data cruncher replies: I have to head outta here now dte ... wish I'd seen this 5 minutes ago (once an excel spread sheet is properly built questions like that fall out in seconds!) ... I'll get you an answer in the AM. Mrs Lintra and I have a tennis date!

Tata all 'till tomorrow!
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless=
Post Wed Jan 21, 2004 10:16 pm
 View user's profile
sauron38
Rara Avis
Rara Avis




Joined: 14 Jan 2002
Posts: 4396
Location: Winnipeg's Sanctum Sanctorum
   

Whereas A)
if there are 13 contributors to the thread

Whereas B)
4 people working very hard under Myrthos will be needed to complete this

Whereas C)
I just read a swath of the internet about math combinatorics.
_____

code:
n = 12
r = 4
nCr =

(n! ) / (r! (n - r)!)

12! / 4! (12 - 4)!
12! / 4! (8!)
479001600 / 967680
= 495




There are thusly 495 combinations of possible arrangements of 4 positions from a pool of 12, all under Myrthos.

Hope this helped.
_________________
Make good choices.
Post Wed Jan 21, 2004 10:54 pm
 View user's profile
dteowner
Shoegazer
Shoegazer




Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia
   

quote:
Originally posted by Lintra
I also found another good one, this guy has 24 reviews, 20 1's, 1 8, 3 10's. Guess he's a hot or cold kinda guy, eh?
This is the sort of thing that troubles me, although it's the opposite opinion I've been griping about. Soooo, I did some thinking. After they put out the fire, here's what I've got. One of the tricks of stats is to "normalize" the data between respondants, right? How about this?

We seem to agree on a few things. Most games released are "good". Some are "great" and some are "awful". That means that we should get a slightly skewed bell curve centered somewhere between 7 and 8, right? We also should be able to determine a "reasonable and expected" standard deviation for any given user with a few reviews. We could use those two numbers to either 1) determine "score extremists" that get a little carried away with their 10's and 1's and fix or eliminate them 2) develop a transform that would "correct" all our existing data to match what we expect to see.

Being an engineer, I'm always willing to make liberal use of the UHC (Universal Homework Coefficient- take your answer, multiply by 0, and add the answer in the back of the book) to bake the numbers, so the second choice above looks easier and more effective. It's also ethically questionable and pretty well destroys the integrity of the data.

I think it will probably be most effective to do any analysis based on user, but I'm not the stat guru.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys!
Post Wed Jan 21, 2004 11:03 pm
 View user's profile
Val
Risen From Ashes
Risen From Ashes




Joined: 18 Feb 2002
Posts: 14724
Location: Utah, USA
   

Putting things into practice, an example:

Criteria for elimination:
10 words minimum
Reviewer must have 3 reviews minimum

Game: Fallout
Of the 488 "reviews", 380 were eliminated by those two simple criteria. Needless to say, a lot of 1s and 10s got thrown out. More 10s than 1s actually.
Of the 108 left, here's the breakdown:
code:

Score          Number of entries
10                67
9                 29
8                 1
7                 5
6                 0
5                 1
4                 1
3                 3
2                 0
1                 1



There may be a margin of error since I did that all by hand.

The game averages out to a 91.9%.
I'd say that's fairly accurate for a game that gets listed in top 10s so often.

Observations from reviewing reviews by hand.
1. It was made a heck of a lot easier because of the word count.
2. More would be eliminated because I noticed that some of the reviews that got a user their 3 reviews minimum would be eliminated by the 10 word limit.

@dteowner: I did not say that we can't throw out anything. I said that we shouldn't throw out data that'll fit in the new system.

Once the 10 word limit is applied to the existing data, it's not as daunting a task to review the reviews by hand as it would seem. Especially if you divide the workload. More reviews probably would have been eliminated if I was looking for them to be actual reviews. Some of it was just smack talk between fanboys.
_________________
Freeeeeeedom! Thank heavens it's summer!
What do I have to show for my hard work? A piece of paper! Wee!
=Guardian, Moderator, UltimaDot Newshound=
Post Wed Jan 21, 2004 11:23 pm
 View user's profile
Myrthos
Spoiler of All Fun
Spoiler of All Fun




Joined: 07 Jul 2001
Posts: 1926
Location: Holland
   

I would like some people who will stay on the 'team' to keep a watch over this. I'll modify the interface to things so that you can select the listed entries by whtaever criteria and delete those that are selected, similar to the moderators panel (those who are moderator should know what I mean).
Then there are some other options like: "select all reviews by user x" or whatever you can come up with that makes sense.
Before I start on that I would like to ask you to continue debating this and come up with a list of requirements, like the 10 word count, the minimum numeber of required reviews and if a review has to be accepted by someone from the team and how you will synchronise this. Suppose there are 5 teammembers then they all get an e-mail when a new review is entered, who will take care of it? Whoever does it first? Or some flagging system like we know have for moderators?

In short I can try to fit the interface for the team to your needs when I know what it is you need
_________________
Kewl quotes:
I often have an odd sense of humor - Roach
Why quote somebody else, think of something yourself. - XeroX
...you won't have to unbookmark this site, we'll unbookmark you. - Val
Reports Myrthos for making me scared and humbled at the mere sight of his name - kayla
Post Thu Jan 22, 2004 12:00 am
 View user's profile
dteowner
Shoegazer
Shoegazer




Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia
   

A few people have noted that games that receive a score in the 7 range may not merit an extended text review. For a "7" game, is something like "Fun to play, but nothing special" really out of line and cause for deletion? "10"'s and "1"'s probably need extended text to justify the extreme score, but average games may not.

I'm not a big fan of the word count criteria for that reason. Of course, I'm just one voice among many, and I'm usually off-key.

@Myrthos- I'd probably assign the workload by the game's title. Put the pending reviews into a single queue (the game title would need to be easy to see) and let the appropriate person pick out what needs doing. Send a daily summary email to the appropriate person when a review comes in. Reviewchecker A would be responsible for games that start with A thru F. Reviewchecker B gets the G's (that's gonna be plenty of work), and so forth. Not sure about the exact breakdown, but we could figure that out. That evens out the workload, eliminates overlapping responsibilities, and still allows someone to ask for help if they get buried in reviews (like mods can ask for backup if they go on vacation). Plus, Lintra could put "Proud Uber-reviewer of the Q's, R's and S's" in his sig.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys!
Post Thu Jan 22, 2004 12:02 am
 View user's profile
Lord Chambers
Eager Tradesman
Eager Tradesman




Joined: 22 Dec 2003
Posts: 29
   

I think you guys are going toward a good system with good criteria, and I'm not trying to discourage movement in that direction, but a thought did occur to me that I don't think will hurt to toss out.

Auditing. It keeps people honest on their taxes, and could be used to try and keep people honest in their reviews. It could be random auditing of any review in the database, where people who have been found to be submitting unreasonable scores are somehow punished (perhaps by being IP banned from the site?), or audits of indidividual users. If a user is found to have an unreasonable review, their punishment could be to have all their reviews deleted.

I'm not saying it's a good idea, or one that I favor over current developments, I just thought I'd toss it out, so someone can set it aflame, stomp it out, pee on it, and then dispose of it.
Post Thu Jan 22, 2004 12:47 am
 View user's profile
Gorath
Mostly Harmless
Mostly Harmless




Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany
   

quote:
10 words minimum
Reviewer must have 3 reviews minimum


These criteria are unsuitable IMO. We lose (a)all reviews with less than 10 words, for example 'A nice game. Recommended. - 7 points', and (b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.

I donīt see a reason to stick to single criteria, applied one after the other. We can combine them to eliminate a smaller subset. For example
[Keep review if (length >=10 OR # of reviews >=3)]
would only filter out short reviews by people with less than 3 reviews but keep their long reviews.

Iīm not sure something like 'length' should be used at all when dealing with the old reviews.
_________________
Webmaster GothicDot
Post Thu Jan 22, 2004 10:18 am
 View user's profile


Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9  Next
All times are GMT.
The time now is Thu Apr 11, 2019 9:52 am



Powered by phpBB © 2001 phpBB Group
 
 
 
All original content of this site is copyrighted by RPGWatch. Copying or reproducing of any part of this site is strictly prohibited. Taking anything from this site without authorisation will be considered stealing and we'll be forced to visit you and jump on your legs until you give it back.