| |
Site Navigation
Main
News
Forums
Games
Games Database
Top 100
Release List
Support Files
Features
Reviews
Previews
Interviews
Editorials
Diaries
Misc
Download
Gallery
Music
Screenshots
Videos
Miscellaneous
Staff Members
Privacy Statement
|
|
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
quote:
Originally posted by goshuto
@Lintra: I don't know if I'm in the "potential team" or not, but allow me to make a suggestion. The problem here seems to be, for the most part, those darn 1 and 10 ratings. But, as Gorath points out, it's not really fair for the average user (1-2 reviews) to have his review ignored because others could potentially abuse the system. So why not consider ratings of 2-9 (3-9 in the scale I proposed) for everyone, and only consider the 1's and 10's for those who we suppose are serious reviewers (that is, those with >3 reviews) or those who provide a review "the team" feels support the rating?
As for your first point, I really don't know yet how to "guarantee the integrity" of the current ratings without going over all of them, one by one. 
I HOPE you're on the "team" ... and you make a good first point, but I am not sure how to procede at this point. So let me ask:
1. Are we willing to go through all the 10 and 1 ratings given out by the 1-3 review reviewers? Then go through the few hundred or so other lop sided reviewers all told about 4,000 reviews would have to be scanned through.
2. Would this be enough to 'clean' up the data to a level acceptable to every one?
To get to dte's points:
That takes care of #1 in the first set ... if we can agree to it.
The rest have been discussed before ... I am not sure we are all in 100% agreement, but I seem to recall the following:
-Change the review page to spell out what we are looking for and give guidence.
-Set up the queue system.
At that point, we have an operational system that should help to ensure the future integrety (sp?) of the new data, and have cleaned up the old data to some degree ... so we then roll it out with the usual fanfare.
Have I missed anything? More importantly I think a new thread might be in order to keep track of agreed upon changes (so we stop rehashing the old stuff). Most importantly we need to agree upon a clean up method. I am currently leaning towards #1 above ... and I am willing to help with it - which is more than I intended when I first stuck my toe in these troubled waters!!
EDIT: Dang it all dte, you posted while I was typing!! I think if you read through the reviews you'd find quite a few that should be tossed. I've given examples before, but 32k of "S**T" should be tossed, a review by "FalloutSux" should be tossed, etc! On the whole I am not willing to toss a review due to lack of words .. but I AM willing to not count it until such time as the reviewer has reviewed enough to decide if he/she is serious or not ... but I am DIGRESSING ... AGAIN!
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
Thu Jan 22, 2004 11:18 pm |
|
|
goshuto
Wanderer
Joined: 29 May 2002
Posts: 1142
|
quote:
Originally posted by dteowner
Bottom line, IMO, is that if we're going to be serious about the integrity of the data, we either have to accept ALL the data we have, warts included, or trash it all, gems included. Any middle ground is going to skew the remaining data into whatever vision we choose.
I agree. But the fact is, the data already is skewed, as Lintra's chart show. The question we have to ask ourselves is, how much more are we willing to interfere with the data in order to get rid of "unacceptable" reviews? This leads, basically, to one single question which, if answered, will probably lead us to the methodology we should apply to current reviews: what is an unacceptable review?
I'm not sure I know the answer yet.
_________________
"Tree stuck in cat. Firemen baffled."--Simcity 3K
"Life can only be understood backwards; but it must be lived forwards."--Soren Aabye Kierkegaard |
|
Thu Jan 22, 2004 11:33 pm |
|
|
Myrthos
Spoiler of All Fun
Joined: 07 Jul 2001
Posts: 1926
Location: Holland |
I knew what a Gaussian was, never heard of the Bell curve 
As the posts are a bit to lengthy for my taste, I'm counting on someone making a summary once you reached a concensus.
_________________
Kewl quotes:
I often have an odd sense of humor - Roach
Why quote somebody else, think of something yourself. - XeroX
...you won't have to unbookmark this site, we'll unbookmark you. - Val
Reports Myrthos for making me scared and humbled at the mere sight of his name - kayla |
|
Thu Jan 22, 2004 11:58 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Lintra
More importantly I think a new thread might be in order to keep track of agreed upon changes (so we stop rehashing the old stuff). Most importantly we need to agree upon a clean up method.
Absolutely. I would even say we open 2 new threads. This makes 3 to discuss the following aspects:
1. Cleaning of the existing data.
2. The new review system.
3. Transformation of the cleaned data.
1 + 2 are independent, no.3 has to be discussed when the others are fixed.
_________________
Webmaster GothicDot |
|
Fri Jan 23, 2004 3:59 am |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Lintra
I think the idea of not allowing reviews in until a certain number of reviews has made is a valid idea ... once the review has made his/her third or maybe fourth (looking at the data the 3 reviews are also pretty skewed) review all their data would enter the data base.
Of course itīs a valid idea, although a bad one. 
a) Once a pipeline is implemented _every single new review is evaluated_. So why should somebody with 1 review who submits his 2nd be ignored for the score until his 3rd arrives? If his 2nd review sucks it should be deleted, not posted and marked passive.
b) You didnīt answer my question. Iīll rephrase it. Playing with open cards is part of RPGDotīs attitude towards the readers. How do you sell such a system? Do you say 'Sorry guys, 20% of your reviews sucked, weīve deleted them. Another 40% were posted by noobs. We put them on ignore. But thanks for your help anyway!' ?
Every explanation will sound arrogant, or manipalating at best.
c) If we do not trust our readers we shouldnīt have a Top 100 list at all.
quote:
While it is certainly true that a lot of the one or two review reviewers are not out to skew the results, the data indicates that a lot of them are.
This is too easy. The task is removing as many of the bad reviews as possible while maintaining all good ones. When a dead end is reached we have to decide if the result is sufficient. Then the three options are (1)keep the remaining data, (2) delete the whole thing or (3) try another concept.
quote:
Originally posted by dteowner
I'm not sure what Gorath wants to do. Can you be a little more detailed for thickheaded folks like me?
Iīm not sure what we should do. I only said some of the suggestions donīt work IMO.
quote:
I think perhaps we've lost sight of our goal here. What are we trying to accomplish?
1) get rid of current fan/antifan reviews
2) above, plus install system to prevent it in the future
3) complete overhaul of the review system so the data has more meaning and scores show a more proper distribution
1.a remove the antifan reviews without hurting the good reviews
1.b remove the fan reviews without hurting the good reviews
2. decide on a new, seriously improved system
3. implement the system
4. convert the data
edit:
I just had a brief look at the reviews for G1, G2, MW, BG2 and FO1. I think before we decide anything we should do some preliminary cleaning. Maybe the data looks differently after weīve filtered out the most common unwanted comments, for example:
-comment
-crap
-crap!
-blabla
-[empty]
-.
This will kill hundreds of entries we definitely donīt need.
_________________
Webmaster GothicDot |
|
Fri Jan 23, 2004 4:49 am |
|
|
Jaz
Late Night Spook
Joined: 20 Jan 2002
Posts: 9708
Location: RPGDot |
Yes, the latter sounds very sensible.
_________________
Jaz |
|
Fri Jan 23, 2004 10:18 am |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
I don't have a lot of time right now, but I am willing to take a first pass at scrubbing out reviews for possible deletion, flagging them as such, and then passing the data base on to the next person.
Or ... we can split the data up into 3,4,5, or more equal chunks and each of us scrub our portion.
Hmmm, I'd best be very clear here. Myrthos sent me a dump of the data base, and I talking about marking records for possible deletion on THAT data base ... NOT the source data!
'Cuz Gorath is right. Before going any further we have to get rid of the obvious crud and see what is left. It might be that after some scrubbing the data looks pristine!
So thoughts on THIS approach?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Fri Jan 23, 2004 1:02 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
I guess itīs too early for your idea yet. Because ...
quote:
Originally posted by Lintra
'Cuz Gorath is right. Before going any further we have to get rid of the obvious crud and see what is left. It might be that after some scrubbing the data looks pristine!
So thoughts on THIS approach?
... I think filtering the obvious crap out of the original data is definitely an improvement and it gives us a clearer picture of the situation.
To do this we need:
1. a short list with the most reppeatedly used phrases we donīt want. Maybe we can find another 5 or 6.
2. somebody with the necessary user rights who filters the relevant entries out of the DB.
_________________
Webmaster GothicDot |
|
Fri Jan 23, 2004 6:23 pm |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
@Gorath - Why use the source data when we work in Excel offline? I will most likely have some down time at the office tomorrow (a Saturday &#^^$%@!), and so will be able to make a first pass through.
How's this for an idea, I will add two columns to my local data base called "Delete?" and "Reason" fill in the first with a 1 if I think if should be deleted, a 2 if I am up in the air about it, and then explain the 2 (or the 1 in some cases) in the column "Reason".
Feed back?
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Fri Jan 23, 2004 6:41 pm |
|
|
Gorath
Mostly Harmless
Joined: 03 Sep 2001
Posts: 6327
Location: NRW, Germany |
quote:
Originally posted by Lintra
@Gorath - Why use the source data when we work in Excel offline?
Because I want to avoid double work. At least the 6 examples I listed above, and probably more, will be deleted from the DB anyway because these reviews manipulate the rankings.
If this kills ~500 entries those no longer have to be read and evaluated by the team.
Regarding manual cleaning - the above is semi-automatic filtering - I think it would be an idea to concentrate on every single review only, without crosschecking what else the reviewer wrote. This save a lot of time because many users only wrote a singe review.
In this case an interface very similar to the actual list would be sufficient. Open two windows, check reviews for deletion, update the page once and wham, all the crap for this game is deleted.
quote:
I will most likely have some down time at the office tomorrow (a Saturday &#^^$%@!), and so will be able to make a first pass through.
How's this for an idea, I will add two columns to my local data base called "Delete?" and "Reason" fill in the first with a 1 if I think if should be deleted, a 2 if I am up in the air about it, and then explain the 2 (or the 1 in some cases) in the column "Reason".
Feed back?
edit: This could be useful to find out what we want.
_________________
Webmaster GothicDot |
|
Fri Jan 23, 2004 7:10 pm |
|
|
Val
Risen From Ashes
Joined: 18 Feb 2002
Posts: 14724
Location: Utah, USA |
Well, I have a free weekend Lintra if you want help with the initial cleaning.
_________________
Freeeeeeedom! Thank heavens it's summer!
What do I have to show for my hard work? A piece of paper! Wee!
=Guardian, Moderator, UltimaDot Newshound=  |
|
Fri Jan 23, 2004 9:18 pm |
|
|
mkreku
Keeper of the Gates
Joined: 22 Oct 2003
Posts: 112
Location: Uppsala, Sweden |
quote:
Originally posted by Gorath
quote:
10 words minimum
Reviewer must have 3 reviews minimum
These criteria are unsuitable IMO. We lose (a)all reviews with less than 10 words, for example 'A nice game. Recommended. - 7 points', and (b) we globally say reviews by people with less than 3 reviews are worthless and simply discard them all, no matter how good some of them might be.
I donīt see a reason to stick to single criteria, applied one after the other. We can combine them to eliminate a smaller subset. For example
[Keep review if (length >=10 OR # of reviews >=3)]
would only filter out short reviews by people with less than 3 reviews but keep their long reviews.
Iīm not sure something like 'length' should be used at all when dealing with the old reviews.
Well, the ten word review length wouldn't mean we "lose" any reviews as you say. I meant applying the new rules only for new reviewers, so the old ones stay.
And for the 3 reviews minimum: we wouldn't lose the people who vote once either. They would be stored in the database and added to the points race as soon as the reviewer reaches 3 reviews or more. Just look at how the ratings look today: 95% of the people who review ONE game vote 1/10 for the game they don't like and then never uses that same name again. Not many REAL reviewers vote 7/10 for one game, never to be seen again. And I think losing the 5% honest voters who only vote one time is worth it compared to how many "GothicSuXX0rz" we could rid the scores of.
I've seen this kind of system used before: in HotOrNot! (I was a moderator there once). They also use some complex mathematic formula to calculate the scores to "fit in" the current users profile, meaning someone who always vote 1 or 10 will get a calculated score of between 4 and 6, since that person isn't using the entire score table.
_________________
Swedes visit NordicGamers for the latest game reviews in swedish! |
|
Sat Jan 24, 2004 11:32 am |
|
|
Lord Chambers
Eager Tradesman
Joined: 22 Dec 2003
Posts: 29
|
It's almost a week later. Any news? |
|
Fri Jan 30, 2004 9:28 am |
|
|
Lintra
Elf Friend
Joined: 23 Apr 2002
Posts: 9448
Location: Bermuda, the triangle place with SANDY BEACHES |
Sorry, I just got snowed under with work. As I wrote somewhere else, it is now Friday morning ... the last time I saw my wife when she was awake was for a quick cup of coffee on Tuesday morning...or was it Monday?
I am too bleeping old for this!
Anyway, in my spare time (on the bus) I decided that Gorath (at least I think it was Gorath) was on the right track to some extent. I should look at and *scrub* the data for the most rated games first ... and hopefully find a pattern.
Killing the reviews with no verbiage is not going to make any difference in the outcome. When I looked at it (a while ago), it made no diffence to the distribution and killed votes by the like of myself, Michael C., Gareth, Vaticide, Ekim and others who I do not feel are guilty of gaming the ratings.
BUT! I still have approach 2 to try, all I need is a little spare time 
_________________
=Member of The Nonflamers' Guild=
=Just plain clueless= |
|
Fri Jan 30, 2004 1:23 pm |
|
|
HiddenX
The Elder Spy
Joined: 20 Jul 2001
Posts: 749
Location: NRW / Germany |
I like the top 100 -> my way of using it:
look for a reviewer, who has the same feeling/taste for rpgs and then i am hitting
the button "see more reviews from this reviewer".
suggestion:
If I could add more than one person in my "personal reviewers group" lets say
Corwin
Lintra
MichaelC
DTE
for example, and then view their average ratings -> I would have a perfect personal rating list.
_________________
=Member of The Nonflamers' Guild= |
|
Sat Feb 07, 2004 8:08 am |
|
|
|
Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9 Next
All times are GMT. The time now is Tue Apr 09, 2019 8:47 pm
|
|
|
|
|
|
|
