| |
Site Navigation
Main
News
Forums
Games
Games Database
Top 100
Release List
Support Files
Features
Reviews
Previews
Interviews
Editorials
Diaries
Misc
Download
Gallery
Music
Screenshots
Videos
Miscellaneous
Staff Members
Privacy Statement
|
|
|
|
Myrthos
Spoiler of All Fun
Joined: 07 Jul 2001
Posts: 1926
Location: Holland |
Based on what criteria should we count the number of reviews? We have the username, e-mail address and IP number.
The same user can have reviews in there where these three are not the same for all reviews.
Or should we restrict voting to those with an account only? In that case we would have to check the username against a registered username and throw away the rest.
_________________
Kewl quotes:
I often have an odd sense of humor - Roach
Why quote somebody else, think of something yourself. - XeroX
...you won't have to unbookmark this site, we'll unbookmark you. - Val
Reports Myrthos for making me scared and humbled at the mere sight of his name - kayla |
Thu Jan 22, 2004 12:28 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
quote:
Originally posted by Myrthos
Based on what criteria should we count the number of reviews? We have the username, e-mail address and IP number.
The same user can have reviews in there where these three are not the same for all reviews.
Or should we restrict voting to those with an account only? In that case we would have to check the username against a registered username and throw away the rest.
I did some thinking about this yesterday ... once the number of reviews gets up past 3 or 4 a proper bell curve of begins to develop .. but we would lose 4,732 reviews if we tossed all those by people w/ few than 4. In addition, this rule would have to be a clean up only rule, since new reviewers HAVE to start with their first review.
@dte - In answer to your earlier question, 2,808 10's are from people who gave out ONLY 10's the distribution is as follows:
code:
1 2,013
2 197
3 63
4 21
5 8
6 3
7 2
8 -
9 2
10 1
11 -
12 -
13 1
14 -
15 1
I would suggest that the 18 reviewers that gave out 5 or more 10's be looked at. Hmm I will try to post the total distribution. Nope .. it does not display well ...
code:
# Rvws 1 2 3 4 5 6 7 8 9 10
1 328 72 39 36 52 47 67 143 366 2,013
2 12 6 5 5 49 24 40 31 105 197
3 8 3 1 11 7 12 27 28 47 63
4 1 - 1 2 4 6 34 26 28 21
5 - 3 2 1 2 7 15 22 22 8
6 - 1 - 1 6 9 8 14 8 3
7 1 - 2 1 - 6 13 7 10 2
8 - - - 2 3 4 7 6 6 -
9 1 3 - 2 1 3 7 5 9 2
10 1 - - 1 - 7 7 7 7 1
11 - - - - - 1 3 5 3 -
12 - 1 - - 2 1 6 5 1 -
13 - - - - - 2 2 2 2 1
14 - - - - 1 1 4 2 5 -
15 - - - - - 2 6 3 4 1
16 - 1 - 2 - - 2 1 1 -
17 - - - - 1 2 3 1 5 -
18 - - - - 1 1 2 1 1 -
19 - - - - - 1 2 2 - -
20 - - - - - 2 1 1 2 -
21 - - - - 1 1 - 4 1 -
22 - - - - - - - 1 - -
23 - - - - - - 2 1 - -
24 - 1 - - - - 4 2 - -
25 - - - - 2 - 1 - - -
26 - - - - - - 2 - - -
27 - - - - - 1 1 - 1 -
28 - - - - - 2 - 1 - -
30 - - - - - 1 - - - -
34 - - - - 1 - - - - -
35 - - - - - - 1 - - -
36 - - - - 1 - - 1 - -
38 - - - - - - 2 - - -
39 - - - - - 1 - - - -
41 - - - - - 1 - 1 - -
46 - - - - - - 1 - - -
47 - - - - - - 1 - - -
51 - - - - - 1 - - - -
61 - - - - - 1 - - - -
75 - - - - 1 - - - - -
78 - - - - - 1 - - - -
98 - - - - - - - 1 - -
128 - - - - - - 1 - - -
@Anybody, can I embed a pdf file of the output in a post? If so, how?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Thu Jan 22, 2004 1:19 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
Okay, try 2
code:
# Rvw 1 2 3 4 5 6 7 8 9 10
1 328 72 39 36 52 47 67 143 366 2,013
2 12 6 5 5 49 24 40 31 105 197
3 8 3 1 11 7 12 27 28 47 63
4 1 - 1 2 4 6 34 26 28 21
5 - 3 2 1 2 7 15 22 22 8
6 - 1 - 1 6 9 8 14 8 3
7 1 - 2 1 - 6 13 7 10 2
8 - - - 2 3 4 7 6 6 -
9 1 3 - 2 1 3 7 5 9 2
10 1 - - 1 - 7 7 7 7 1
11 - - - - - 1 3 5 3 -
12 - 1 - - 2 1 6 5 1 -
13 - - - - - 2 2 2 2 1
14 - - - - 1 1 4 2 5 -
15 - - - - - 2 6 3 4 1
16 - 1 - 2 - - 2 1 1 -
17 - - - - 1 2 3 1 5 -
18 - - - - 1 1 2 1 1 -
19 - - - - - 1 2 2 - -
20 - - - - - 2 1 1 2 -
21 - - - - 1 1 - 4 1 -
22 - - - - - - - 1 - -
23 - - - - - - 2 1 - -
24 - 1 - - - - 4 2 - -
25 - - - - 2 - 1 - - -
26 - - - - - - 2 - - -
27 - - - - - 1 1 - 1 -
28 - - - - - 2 - 1 - -
30 - - - - - 1 - - - -
34 - - - - 1 - - - - -
35 - - - - - - 1 - - -
36 - - - - 1 - - 1 - -
38 - - - - - - 2 - - -
39 - - - - - 1 - - - -
41 - - - - - 1 - 1 - -
46 - - - - - - 1 - - -
47 - - - - - - 1 - - -
51 - - - - - 1 - - - -
61 - - - - - 1 - - - -
75 - - - - 1 - - - - -
78 - - - - - 1 - - - -
98 - - - - - - - 1 - -
128 - - - - - - 1 - - -
If I copy the excel table to a seperate sheet, then save as a space delimited text file (*.prn type) then open w/ note pad copy and paste here it is properly formatted ... not an easy process!
Any way, as you can see, the bell curve begins to show up once the number of reviews is greater than 3 or 4.
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Thu Jan 22, 2004 1:32 pm |
|
|
Myrthos
Spoiler of All Fun
Joined: 07 Jul 2001
Posts: 1926
Location: Holland |
What's a Bell curve?
_________________
Kewl quotes:
I often have an odd sense of humor - Roach
Why quote somebody else, think of something yourself. - XeroX
...you won't have to unbookmark this site, we'll unbookmark you. - Val
Reports Myrthos for making me scared and humbled at the mere sight of his name - kayla |
|
Thu Jan 22, 2004 2:23 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
quote:
Originally posted by Myrthos
What's a Bell curve?
Sometimes refered to as a Gaussian ... it is the normal distribution. It is the distribution any large set of random numbers will regress to (the Central Limit Theorum). I is defined having the form:
f(x) = Exp(-x^2)
And is shaped like a ... ready for this ... a bell. Low at the extremes coming to a soft peak at the mean. f(x) above is the simpest form and is centered around f(x) = 0 as the mean. A more rigorous form is:
g(x) = k * exp{ [(x - mean)^2]/(std dev)^2 }
where k = sqr root(2*pi*std dev)
Anyway the important thing is that we see evidence of the bell curve so this would mean that the reviewers with more reviews under their belt are behaving in a statistically proper way ... so there is no evidence of tampering.
Edited: Sorry about that, I just reread this ... but that is what you get for asking an actuary about a distribution!! 
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless=
Last edited by Lintra on Thu Jan 22, 2004 6:20 pm; edited 1 time in total |
|
Thu Jan 22, 2004 5:40 pm |
|
|
Val
Risen From Ashes
Joined: 18 Feb 2002
Posts: 14724
Location: Utah, USA |
quote:
Originally posted by Gorath
(b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.
No, we don't throw those out. The way I'd like to see it is if they have a review that is over the 10 word limit, then it is kept. However, if the user has not posted at least three reviews, then their score that they gave should not effect the overall score of the game.
So their review is still there, it's just that the score which they gave is ignored until they have submitted at least 3 reviews that fit in the 10 word minimum limit.
_________________
Freeeeeeedom! Thank heavens it's summer!
What do I have to show for my hard work? A piece of paper! Wee!
=Guardian, Moderator, UltimaDot Newshound=  |
|
Thu Jan 22, 2004 6:01 pm |
|
|
goshuto
Wanderer
Joined: 29 May 2002
Posts: 1142
|
That bell curve is centered at 7-8. If we use the new text ratings in which "average" is 5, won't that skew the whole thing in the future?
edit: Or maybe not, seeing how people tend to give above average scores anyway....
_________________
"Tree stuck in cat. Firemen baffled."--Simcity 3K
"Life can only be understood backwards; but it must be lived forwards."--Soren Aabye Kierkegaard |
|
Thu Jan 22, 2004 6:03 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Val
quote:
Originally posted by Gorath
(b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.
No, we don't throw those out. The way I'd like to see it is if they have a review that is over the 10 word limit, then it is kept. However, if the user has not posted at least three reviews, then their score that they gave should not effect the overall score of the game.
So their review is still there, it's just that the score which they gave is ignored until they have submitted at least 3 reviews that fit in the 10 word minimum limit.
I thought you wanted to remove them.
So what makes number 3 more valuable than number 2 if the reviewer has at lest 3 reviews, and both more valuable than number 1?
1. Best game I ever played! - 10
2. NNNN is the best game Iīve played all my live. - 10 (-->10 words, single review)
3. NNNN is the best game Iīve played all my live. - 10 (-->10 words, >=3 reviews)
_________________
Webmaster GothicDot |
|
Thu Jan 22, 2004 7:01 pm |
|
|
balconygolf_ThE_bRiDe
Alien Dwarf
Joined: 26 Dec 2003
Posts: 705
Location: Greece/Athens or *BrOtHeRhOoD* |
If someone will write something like this:
quote:
Originally posted by Gorath
[2. NNNN is the best game Iīve played all my live. - 10 (-->10 words, single review)
)
then we should never consider him as a serius reviewer....the fellow man can't even type in English.It should be more like this:
"NNNN is the best game I' ve ever played in my hole life"
_________________
 I come from space.From outer space! |
|
Thu Jan 22, 2004 8:03 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
quote:
Originally posted by Gorath
quote:
Originally posted by Val
quote:
Originally posted by Gorath
(b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.
No, we don't throw those out. The way I'd like to see it is if they have a review that is over the 10 word limit, then it is kept. However, if the user has not posted at least three reviews, then their score that they gave should not effect the overall score of the game.
So their review is still there, it's just that the score which they gave is ignored until they have submitted at least 3 reviews that fit in the 10 word minimum limit.
I thought you wanted to remove them.
So what makes number 3 more valuable than number 2 if the reviewer has at lest 3 reviews, and both more valuable than number 1?
1. Best game I ever played! - 10
2. NNNN is the best game Iīve played all my live. - 10 (-->10 words, single review)
3. NNNN is the best game Iīve played all my live. - 10 (-->10 words, >=3 reviews)
Gorath, I think that what Val is getting at, is...on a going forward basis we will be screening the reviews so if they currently have only two reviews and stay that way then they have labled themselves as a 'fan/anitfan boy' reviewer. Once they breach the barrier of three games we will screen the latest review, and if it is of low quality it gets canned ... thusly keeping them at two reviews.
Past behavior indicates that those reviewers with 3 or more reviews are actually more careful. Now, you are correct to some extent, MW and Gothic fanboy may post review #3 for Gothic 3, but we would hope that either the queue would catch the obvious fan boy post, or we can hope that given the historic short attention span of such reviews that they won't review Gothic 3 on this site. In either case, it is better than what we currently have.
@balconygolf_ThE_bRiDe - That should be "whole" life
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Thu Jan 22, 2004 8:22 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Lintra
Gorath, I think that what Val is getting at, is...on a going forward basis we will be screening the reviews
So far I agree! 
quote:
so if they currently have only two reviews and stay that way then they have labled themselves as a 'fan/anitfan boy' reviewer.
This is a weird concept. You simply donīt know if the reviewer is a fan/antifan. There are certain other hints which could support such an impression, but the number of reviews alone doesnīt provide enough info to draw such a conclusion.
Only one example: Somebody posts two 5 sentence reviews about FAllout and NWN, 1 hour before the freeze for the reorganisation. He is obviously not just a fan.
quote:
Once they breach the barrier of three games we will screen the latest review, and if it is of low quality it gets canned ... thusly keeping them at two reviews.
Why 3 reviews? Why not 5 or 10?
quote:
In either case, it is better than what we currently have.
Correct. But the top100 sucks for more than 6 months. No need to hurry. We should optimize the new system as far as possible before action is taken.
I have the impression some of you are way too fast when it comes to removing reviews or rendering them irrelevant for the score. Let me remind you, weīre not talking about faceless data here. Every review has been written by one of our readers. He invested his time, he made up his mind about one or more RPG(s) and contributed to the project. How much time he spend shouldnīt matter. If he wrote a 'good' review ('good' meaning a review we want; this has to be defined) it should survive the cleaning AND be relevant for the score, without a second criterion like the number of reviews. How do you want to communicate to our readers that their opinion is valuable to us, but only after the 3rd review?
The 'fan' argument doesnīt count. A 'fan' would hardly register on a nameless site to influence a ranking list. This means the RPGDot Top 100 has some importance, how low it may be. Itīs worth being manipulated. We did find a way to prevent this. Now we have to find a way to eliminate fan & antifan reviews, and only them.
Whether or not the Gauss bell doesnīt look like it should doesnīt really matter. Itīs only a tool to investigate the data.
IMHO itīs not acceptable to delete anywhere near 50% of the reviews. How do you want to explain our readers that we deleted thousands of reviews without manipulating the new top 100?
If it adds up to more than about 10-20% we should seriously consider if we delete the whole thing or do only minimum cleaning plus a system conversion and hope that the DB will heal itself.
_________________
Webmaster GothicDot |
|
Thu Jan 22, 2004 9:44 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
*Lintra chuckles some*
Gorath, you have the bad luck of making a post when I really need a break from work *grin*
I think the idea of not allowing reviews in until a certain number of reviews has made is a valid idea ... once the review has made his/her third or maybe fourth (looking at the data the 3 reviews are also pretty skewed) review all their data would enter the data base.
While it is certainly true that a lot of the one or two review reviewers are not out to skew the results, the data indicates that a lot of them are. So what to do? As I see it there are many sol'ns but here are two to get kicked around:
1. The 'team' could review all 3,000 or so reviews written by 1-3 reviewers that give only 1 or 10 as a score. While a lot, it is not un-doable over a period of weeks. The negative here is how to tell a garbage review from a thoughtful one? I gave M&M6 a 10 w/o any verbiage at all, but most of my other 20 or so reviews do have verbiage, including my only other 10 score. I have made enough reviews for an independent person to make an informed decision as to whether or not I am just junking up the system.
2. We can remove all the reviews done by those that have done less than 3 or 4 until they post their third (or fourth) review and at *that* time pass judgement on the prior ones (since now we would have more than one or two to judge from).
What do you (the potential team) think?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Thu Jan 22, 2004 10:08 pm |
|
|
goshuto
Wanderer
Joined: 29 May 2002
Posts: 1142
|
@Lintra: I don't know if I'm in the "potential team" or not, but allow me to make a suggestion. The problem here seems to be, for the most part, those darn 1 and 10 ratings. But, as Gorath points out, it's not really fair for the average user (1-2 reviews) to have his review ignored because others could potentially abuse the system. So why not consider ratings of 2-9 (3-9 in the scale I proposed) for everyone, and only consider the 1's and 10's for those who we suppose are serious reviewers (that is, those with >3 reviews) or those who provide a review "the team" feels support the rating?
As for your first point, I really don't know yet how to "guarantee the integrity" of the current ratings without going over all of them, one by one. 
_________________
"Tree stuck in cat. Firemen baffled."--Simcity 3K
"Life can only be understood backwards; but it must be lived forwards."--Soren Aabye Kierkegaard |
|
Thu Jan 22, 2004 10:34 pm |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
I'm not sure what Gorath wants to do. Can you be a little more detailed for thickheaded folks like me?
I think perhaps we've lost sight of our goal here. What are we trying to accomplish?
1) get rid of current fan/antifan reviews
2) above, plus install system to prevent it in the future
3) complete overhaul of the review system so the data has more meaning and scores show a more proper distribution
Once we figure that out, then we can pick a method to do it.
1) "clean up" of existing data
2) implementation of new system, wipe out old data for a clean start
3) implementation of new system, applies only to future entries
4) implementation of new system, edit old data into conformance
We're working on the second question, when I'm not certain we've got agreement on the first yet.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Thu Jan 22, 2004 11:03 pm |
|
|
dteowner
Shoegazer
Joined: 21 Mar 2002
Posts: 7570
Location: Third Hero of Erathia |
quote:
Originally posted by goshuto
As for your first point, I really don't know yet how to "guarantee the integrity" of the current ratings without going over all of them, one by one.
And that approach is only going to work if the group decides what we're looking for and what we want to do when we find it.
Bottom line, IMO, is that if we're going to be serious about the integrity of the data, we either have to accept ALL the data we have, warts included, or trash it all, gems included. Any middle ground is going to skew the remaining data into whatever vision we choose. Not picking any of those three options at this point (my opinion has been expressed in the past), but that's pretty much what we've got to pick from.
_________________
=Proud Member of the Non-Flamers Guild=
=Benevolent Dictator, X2/X3 and Morrowind/Oblivion Forums=
Sorry. No pearls of wisdom in this oyster.
RIP Red Wings How 'Bout Dem Cowboys! |
|
Thu Jan 22, 2004 11:10 pm |
|
|
|
Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9 Next
All times are GMT. The time now is Tue Apr 09, 2019 8:46 pm
|
|
|
|
|
|
|
