| |
Site Navigation
Main
News
Forums
Games
Games Database
Top 100
Release List
Support Files
Features
Reviews
Previews
Interviews
Editorials
Diaries
Misc
Download
Gallery
Music
Screenshots
Videos
Miscellaneous
Staff Members
Privacy Statement
|
|
|
|
Dhruin
Stranger In A Strange Land
Joined: 20 May 2002
Posts: 1825
Location: Sydney, Australia |
We shouldn't be surprised the list is top-heavy. You can pre-suppose that visitors to our site are RPG fans. Most players will read some reviews or do some basic research before purchasing a game all of which is going to result in more high scores than low ones. I know I haven't played a game I'd rank under "5" for a long time.
_________________
Editor @ RPGDot |
Wed Jan 21, 2004 9:06 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
First off, sorry about the double post, but the info is very different.
Folks, we have our work cut out for us. Myrthos kindly gave me an excel spread sheet with a dump of the game name, reviewer, rating, and comments. My first pass was to rank the reviews by rating, and a measure of the number of words ... I measured the length of the comment feild as a proxy and divided by 15. The result for the 1 and 10 bracket are as follows:
code:
Rating
Length 1 10
0 - 431
1-14 289 872
15-29 164 700
30-44 103 465
45-59 40 328
60+ 272 2,109
So this means that over 1/2 of the 10 ratings have comment fields of over 60 characters in length!!! and NONE, that right NONE of the one rated games has a comment feild of 0 length (there was one review that was over 32k of 's**t' over and over).
I read through the comments for the 10 ratings that where 1-14 and the majority were things like Great game, or Excellent. Hard to fault those.
My next cut at the data is going to be by user to see if there are any patterns.
Any other suggestions folks?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Wed Jan 21, 2004 9:10 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Myrthos
We can also do it in stars, ranging from a half to 5 stars, with steps of a half star.
10 = 5 stars
9.5 = 4.5 stars
9 = 4 stars
8.5 = 3.5 stars
8 = 3 stars
7.5 = 2.5 stars
7 = 2 stars
6.5 = 1.5 stars
6 = 1 star
Everything below that is 0.5 star
Most games should be above a 6 anyway to be even remotely considered to be worth playing, so adding a few steps could make things easier.
I donīt like this idea. We should use dragons instead of stars!  
Sorting the game list:
1. 1/2 stars
2. average rating --> invisible for the reader
3. number of reviews ---> more=better
(4. release date) --> newer=better (because we tend to forget that games are aging)
5. alphabet
_________________
Webmaster GothicDot |
|
Wed Jan 21, 2004 9:16 pm |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
This is why I tossed out the idea of starting over. I'm still not saying that's the right answer, but it's looking tougher and tougher to accomplish what we're wanting to achieve and still maintain the integrity of the data.
We could resort to a post-by-post review as Val has done in the past, but that seems to be incredibly labor intensive (Val has implied so, and I certainly believe it), subject to the arbitrary whims of individual team members, and lacking direction for users that would take time to do a review in the future.
Since Lintra has the data and is a number cruncher extrordinaire, he might find a string we can pull to untangle the whole ball. If not, I don't think we should automatically rule out a fresh start. It's entirely possible that we're trying to fly with a handful of lead balloons. I don't see any shame in saying "We've tried everything we can think of, but this just ain't gonna work." You grab a fresh sheet of paper and start a new blueprint.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Wed Jan 21, 2004 9:33 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
The next biggest issue is number of reviews done by the reviewer. Almost half of the 0's and 10's come from people who have reviewed only one game. And the 2 review guys are AMOST, but not quite as bad. As the number of reveiws increased, the distribution became much less skewed ... but I am very suspect of those guys who reviewed 10 games and gave ALL of them either a 1 or a 10!
I also found another good one, this guy has 24 reviews, 20 1's, 1 8, 3 10's. Guess he's a hot or cold kinda guy, eh?
I have also found a bunch of names that should just be deleted, like "Falloutsucks" or "FalloutFanBoy" for example.
So what is the next step? Do I start to make a list of all the reviews I think should be deleted, or checked? To whom do I send it? Who is on the team? AGGGGG so many questions.
@Myrthos - how privilaged is the database? Can I give out to just anyone who asks, or only team members?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Wed Jan 21, 2004 9:39 pm |
|
|
Toaster
Bread Alert
Joined: 27 Jan 2003
Posts: 5475
Location: Sweden |
But if we use stars, a game which today have 7 in rating, which should mean a good game, get only two stars, which is interpreted as bad. I think keeping the 1-10 scale is good, but doing as goshuto suggested with it. As said, whatever is chosen, new reviews has to be checked so people doesn't have to do this all over again.
_________________
Tabbrowser Extensions
DictionarySearch |
|
Wed Jan 21, 2004 9:40 pm |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
I don't think the non-existent (Corwin taught me how to spell!) committee has a non-existent leader yet. Given recent events, I think Val and I should be taken off the list of leadership candidates if not reduced all the way to "consultant" status. We probably need to make the team official, with a list of who's on it and who's in charge. That duty probably falls to Myrthos.
If wiping out the single-review reviewers eases our problems. I'd do it. Anyone that truly wants their opinion heard on just a single game only has to re-enter a single review and it will be checked prior to becoming part of the database (We seem to have unanimous support for that change). That seems like a lot of "bang" for little disruption and very little effort.
@Lintra- Are there many instances of a single user giving out a bunch (>5) of 10's? This has been my big issue, and if it turns out there are very few instances of that happening, I'll shut my yap about it and move on to the next issue.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Wed Jan 21, 2004 10:11 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
quote:
Originally posted by dteowner
...
@Lintra- Are there many instances of a single user giving out a bunch (>5) of 10's? This has been my big issue, and if it turns out there are very few instances of that happening, I'll shut my yap about it and move on to the next issue.
Deftly sidestepping the leadership issue ... Lintra the data cruncher replies: I have to head outta here now dte ... wish I'd seen this 5 minutes ago (once an excel spread sheet is properly built questions like that fall out in seconds!) ... I'll get you an answer in the AM. Mrs Lintra and I have a tennis date!
Tata all 'till tomorrow!
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Wed Jan 21, 2004 10:16 pm |
|
|
sauron38
Rara Avis
Joined: 14 Jan 2002
Posts: 4396
Location: Winnipeg's Sanctum Sanctorum |
Whereas A)
if there are 13 contributors to the thread
Whereas B)
4 people working very hard under Myrthos will be needed to complete this
Whereas C)
I just read a swath of the internet about math combinatorics.
_____
code:
n = 12
r = 4
nCr =
(n! ) / (r! (n - r)!)
12! / 4! (12 - 4)!
12! / 4! (8!)
479001600 / 967680
= 495
There are thusly 495 combinations of possible arrangements of 4 positions from a pool of 12, all under Myrthos. 
Hope this helped.
_________________
Make good choices. |
|
Wed Jan 21, 2004 10:54 pm |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
quote:
Originally posted by Lintra
I also found another good one, this guy has 24 reviews, 20 1's, 1 8, 3 10's. Guess he's a hot or cold kinda guy, eh?
This is the sort of thing that troubles me, although it's the opposite opinion I've been griping about. Soooo, I did some thinking. After they put out the fire, here's what I've got. One of the tricks of stats is to "normalize" the data between respondants, right? How about this?
We seem to agree on a few things. Most games released are "good". Some are "great" and some are "awful". That means that we should get a slightly skewed bell curve centered somewhere between 7 and 8, right? We also should be able to determine a "reasonable and expected" standard deviation for any given user with a few reviews. We could use those two numbers to either 1) determine "score extremists" that get a little carried away with their 10's and 1's and fix or eliminate them 2) develop a transform that would "correct" all our existing data to match what we expect to see.
Being an engineer, I'm always willing to make liberal use of the UHC (Universal Homework Coefficient- take your answer, multiply by 0, and add the answer in the back of the book) to bake the numbers, so the second choice above looks easier and more effective. It's also ethically questionable and pretty well destroys the integrity of the data.
I think it will probably be most effective to do any analysis based on user, but I'm not the stat guru.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Wed Jan 21, 2004 11:03 pm |
|
|
Val
Risen From Ashes
Joined: 18 Feb 2002
Posts: 14724
Location: Utah, USA |
Putting things into practice, an example:
Criteria for elimination:
10 words minimum
Reviewer must have 3 reviews minimum
Game: Fallout
Of the 488 "reviews", 380 were eliminated by those two simple criteria. Needless to say, a lot of 1s and 10s got thrown out. More 10s than 1s actually.
Of the 108 left, here's the breakdown:
code:
Score Number of entries
10 67
9 29
8 1
7 5
6 0
5 1
4 1
3 3
2 0
1 1
There may be a margin of error since I did that all by hand.
The game averages out to a 91.9%.
I'd say that's fairly accurate for a game that gets listed in top 10s so often.
Observations from reviewing reviews by hand.
1. It was made a heck of a lot easier because of the word count.
2. More would be eliminated because I noticed that some of the reviews that got a user their 3 reviews minimum would be eliminated by the 10 word limit.
@dteowner: I did not say that we can't throw out anything. I said that we shouldn't throw out data that'll fit in the new system.
Once the 10 word limit is applied to the existing data, it's not as daunting a task to review the reviews by hand as it would seem. Especially if you divide the workload. More reviews probably would have been eliminated if I was looking for them to be actual reviews. Some of it was just smack talk between fanboys.
_________________
Freeeeeeedom! Thank heavens it's summer!
What do I have to show for my hard work? A piece of paper! Wee!
=Guardian, Moderator, UltimaDot Newshound=  |
|
Wed Jan 21, 2004 11:23 pm |
|
|
Myrthos
Spoiler of All Fun
Joined: 07 Jul 2001
Posts: 1926
Location: Holland |
I would like some people who will stay on the 'team' to keep a watch over this. I'll modify the interface to things so that you can select the listed entries by whtaever criteria and delete those that are selected, similar to the moderators panel (those who are moderator should know what I mean).
Then there are some other options like: "select all reviews by user x" or whatever you can come up with that makes sense.
Before I start on that I would like to ask you to continue debating this and come up with a list of requirements, like the 10 word count, the minimum numeber of required reviews and if a review has to be accepted by someone from the team and how you will synchronise this. Suppose there are 5 teammembers then they all get an e-mail when a new review is entered, who will take care of it? Whoever does it first? Or some flagging system like we know have for moderators?
In short I can try to fit the interface for the team to your needs when I know what it is you need
_________________
Kewl quotes:
I often have an odd sense of humor - Roach
Why quote somebody else, think of something yourself. - XeroX
...you won't have to unbookmark this site, we'll unbookmark you. - Val
Reports Myrthos for making me scared and humbled at the mere sight of his name - kayla |
|
Thu Jan 22, 2004 12:00 am |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
A few people have noted that games that receive a score in the 7 range may not merit an extended text review. For a "7" game, is something like "Fun to play, but nothing special" really out of line and cause for deletion? "10"'s and "1"'s probably need extended text to justify the extreme score, but average games may not.
I'm not a big fan of the word count criteria for that reason. Of course, I'm just one voice among many, and I'm usually off-key.
@Myrthos- I'd probably assign the workload by the game's title. Put the pending reviews into a single queue (the game title would need to be easy to see) and let the appropriate person pick out what needs doing. Send a daily summary email to the appropriate person when a review comes in. Reviewchecker A would be responsible for games that start with A thru F. Reviewchecker B gets the G's (that's gonna be plenty of work), and so forth. Not sure about the exact breakdown, but we could figure that out. That evens out the workload, eliminates overlapping responsibilities, and still allows someone to ask for help if they get buried in reviews (like mods can ask for backup if they go on vacation). Plus, Lintra could put "Proud Uber-reviewer of the Q's, R's and S's" in his sig. 
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Thu Jan 22, 2004 12:02 am |
|
|
Lord Chambers
Eager Tradesman
Joined: 22 Dec 2003
Posts: 29
|
I think you guys are going toward a good system with good criteria, and I'm not trying to discourage movement in that direction, but a thought did occur to me that I don't think will hurt to toss out.
Auditing. It keeps people honest on their taxes, and could be used to try and keep people honest in their reviews. It could be random auditing of any review in the database, where people who have been found to be submitting unreasonable scores are somehow punished (perhaps by being IP banned from the site?), or audits of indidividual users. If a user is found to have an unreasonable review, their punishment could be to have all their reviews deleted.
I'm not saying it's a good idea, or one that I favor over current developments, I just thought I'd toss it out, so someone can set it aflame, stomp it out, pee on it, and then dispose of it. |
|
Thu Jan 22, 2004 12:47 am |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
10 words minimum
Reviewer must have 3 reviews minimum
These criteria are unsuitable IMO. We lose (a)all reviews with less than 10 words, for example 'A nice game. Recommended. - 7 points', and (b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.
I donīt see a reason to stick to single criteria, applied one after the other. We can combine them to eliminate a smaller subset. For example
[Keep review if (length >=10 OR # of reviews >=3)]
would only filter out short reviews by people with less than 3 reviews but keep their long reviews.
Iīm not sure something like 'length' should be used at all when dealing with the old reviews.
_________________
Webmaster GothicDot |
|
Thu Jan 22, 2004 10:18 am |
|
|
|
Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9 Next
All times are GMT. The time now is Thu Apr 11, 2019 9:52 am
|
|
|
|
|
|
|
