I agree with redelf. There's many scores I doubt, but the only score i plan to immediately challenge is the spinout on odyssey2. I think that one was probably a data entry error anyway and so i dont imagine any controversy.

This is good

Originally Posted by Jace Hall

The dispute always stays open in order to prevent another user from creating an exact duplicate dispute.

If a dispute does not have enough evidence to warrant score removal, and without a shred of doubt the score remains valid, closing that dispute and removing it opens up the possibility for another member (who is unaware of the previous dispute) to raise the exact same dispute all over again. This approach would be problematic.

By leaving the dispute record where it is, any new member can read the current status and see all the information and then know that there is no need to raise an already raised issue that has been solved.

The dispute system helps support the idea that it is optimal to put the best evidence packages together for a score submission, as having solid believable evidence is the best way to avoid a future dispute claim.

Two/three queries:

After TG makes a decision on a given dispute, are the votes reset or retained?
Is the CR of the voter included into whether the dispute reaches TG decision time?
If the CR is included in when the vote reaches TG decision time, is that CR calculation based on when the individual voted in the dispute or their current CR?

Otherwise, cautionarily welcomed. ;)

Originally Posted by Jace Hall

Your claim will remain open until it achieves enough voting power to close the vote and trigger a score removal recommendation.

The above "trigger mechanism" is open to abuse. eg.....the dig dug submission, if you voted NO on that submission, then don't vote AT ALL in the challenge thread, that way a "score removal recommendation" will NOT be triggered. The challenge stays in limbo indefinitely!
The above strategy is one that any member facing a cred loss on any challenge could follow. And will.


john

.

[QUOTE=Jace Hall;bt7373]A dispute once opened, can never be closed - unless the score is removed.

All that happens is this -

Dispute is opened
Voting takes place
Voting reaches a level where Admin is notified (at this point voting is stopped)
Admin makes a decision.
If Admin agrees, score is removed - dispute is over.
If Admin does not agree - [U]same dispute thread is REOPENED[/U] to gather more votes/evidence. Once reopened dispute voting reaches the "next level" - Voting is stopped, admin is notified to make a decision - and the cycle keeps repeating.

Make sense?[/QUOTE]


I know you like to go around it but Jace seriously the Archives and Private Messages need to be USED in these cases, it's the ONLY way to have a proper challenge!!!!!!!!!!


Besides Patrick

3-4 Others I know cheated

1 even admitted to it on the private archives

Originally Posted by Rudy

the Archives and Private Messages need to be USED in these cases,

What next, open up everyone's PM's to scrutiny?

Not going to happen, Rudy.


john

.

HugDD liked this post

Its true anyone who voted wrong on a submission is motivated to vote incorrectly again to protect themselves. I'm not sure if thats necessarily a flaw though, since who are you or anyone else to say their wrong ahead of the fact? if the majority stands by the earlier adjudication -- even if that means the majority is just standing by themselves, then you still have the majority agreeing with it. It should be hard to remove a score. The fact that the people who previoulsy voted one way will likely stand by their vote and keep the score status the same i'd see as a good thing. Bad scores do have to go, but the burden of proof has to be high.

Jace Hall liked this post

Originally Posted by redelf

I would think that limiting the amount of challenges a person can do at one time would be necessary.

This is not possible under the current system. Since dispute claims live forever until a score is actually removed, limiting the number of dispute claims would be problematic as quickly people would hit their limit and potentially never again be able to make a claim.

The way a dispute claim should be looked at is as a permanent log/discussion attached to a particular score forever that has the ability to potentially get that score removed - or potentially further solidify the score's validity. There is no need for a discussion/claim to ever be removed or ended UNLESS a score is removed - otherwise it just remains potentially useful information about the score and its history.

Originally Posted by redelf

I imagine that some people will go nuts and challenge almost every score on a track or challenge most of the scores of 1 player. If this allowed to happen we will have thousands of unfounded challenges clogging up the system. We have a little over 50 people that will be able to challenge a score. How many challenges should be allowed at one time. Once the system is open there will be a flood of challenges and we will be overwhelmed quickly. There's going to be a lot of scores that will be contested early on. I would hope that we spread them out over time. Should some system be put into place to limit that amount so that we can do a better and more thorough job.

There is no time limit on disputes. It does not matter how many there are. They all represent claim discussions, that's it. Either they eventually resolve to an admin action or they don't. If they don't they will just sit there until they do. A disputed score remains in the database and is considered valid until removal.

We believe that members with 4000+ CR at this point will behave responsibly in this area. We will have to see if that is true. If there gets to be a problem we will raise CR requirements for initial dispute claim submission which will narrow the opportunity. We will see how it goes.

Originally Posted by redelf

When a score is contested can we truncate the score or change the score. Can we just have it moved to another track. What options are going to be available to us. Can we have the option of moving, changing, removing, or adjusting some part of the score so that it can be corrected if needed and not just thrown away.

Data can not be altered. It can only be moved to another site area (rejected, different track, etc.)

Originally Posted by redelf

There also needs to be some sort of punishment for abuse of the system, like wasting adjudicators time on frivolous challenges. Once a persons challenge is rejected are we just going to allow that individual to keep challenging scores. There just has to be a limit on the amount of failed challenges that a person can have before they shouldn't be allowed to challenge again. Some time limit or some kind of restriction needs to be in place for those that don't use the system as designed.

Certainly TG admin will generally punish any overt abuse of the site's features/functions, but a specific punishment is not necessary here. All dispute claims by default are considered invalid until TG admin agrees with the claim evidence and takes action. There never is a "rejection" moment. There are no "failed" challenges. There is only an "Admin agrees" moment. As long as admin does not agree the dispute conversation is allowed to continue for as long as it takes. This could be forever in some cases.

Originally Posted by redelf

Guidelines should be spelled out in the beginning for what will make a successful challenge. Like video proof and screen shots. Evidence brought forward without these kinds of proof are just going to bring a lot of discussion and he said she said kinda thing.

Weak and unconvincing evidence simply will result in admin never agreeing to score removal. It also will likely make that dispute thread have lower activity since people will lose interest in any weak claim that makes no sense. Please keep in mind that unlike score submission adjudications, there is no CR reward for participation in disputes, so there is no imperative to participate in them if a member doesn't want to.

Originally Posted by redelf

I like the idea of really limiting the release of the system to those that are going to have very strong video evidence and a well though out reason and explanations as to why its not a good score. Allowing people to just throw out a challenge based on a rule or its just not possible without supporting evidence shouldn't be allowed. But then how and who decides what that should be. Maybe at the start of this challenge system there should be some strong penalties to dissuade people from just throwing up a random challenge and hope that the community goes along.

I get what you are saying here but the truth is that the system does not need to be that complex. If a score dispute claim is bunk, it just won't achieve a removal. It's that simple.

We can all see how it goes and consider looking at placing some kind of thing in place that limits but it could be very problematic since it creates additional issues.

One tool that is always available to the community is the TRIBUNAL system - where if everyone starts to feel that someone is abusing the dispute system deliberately, they can ban that person from the system. That of course should always be a last resort.

The community could just simply start by asking that person to stop abusing the system.

TG Admin will certainly ban anyone that operates so rapidly in the abuse that it would need immediate action.

Originally Posted by redelf

I guess what I'm asking is that we start out slow and only with those that we absolutely know will be an acceptable challenge. If you aren't sure then maybe you shouldn't challenge the score yet till you have more evidence and a stronger feeling that you can persuade the community.

I just don't want this to be a witch hunt and a negative experience for the community. It would be just great if we had a few challenges at a time to deal with and everyone could see the overwhelming evidence brought forward and come to an easy and quick decision.

Bring on the challenge system!

Let's all see how it goes. I know that this type of system many in the community have been waiting for - for years. There is a lot of backlogged desire to correct some things in the database and TG has virtually never offered a real opportunity to directly address them. What TG is doing now is unprecedented. The hope is that we all act responsibly.

I personally have not ever had an issue with score removal as a concept, it's just that I absolutely would never do it in an undocumented, non-transparent, non-reversible way.

The data in the TG database is valuable. Many people long before me contributed their time and effort to build it. Is it perfect? No - but what is? We all must be cautious and have care to not destroy one of the most unique things in the video game universe.

The responsibility is on all of us together to be fair and balanced in our quest for database integrity. It is very easy to doubt something, but rarely does doubt build anything useful.

Let's see what happens when this system comes online. It is in its final testing phase now...

Barthax, HugDD liked this post

Originally Posted by Barthax

Two/three queries:

After TG makes a decision on a given dispute, are the votes reset or retained?
Is the CR of the voter included into whether the dispute reaches TG decision time?
If the CR is included in when the vote reaches TG decision time, is that CR calculation based on when the individual voted in the dispute or their current CR?

Otherwise, cautionarily welcomed. ;)

After TG makes a decision, if the dispute claim is not agreed to, the vote reopens but is not reset. It simply continues on from where it left off.

A member's CR is what they vote with. Just like normal adjudication.

CR is accrued on a vote at the time of voting.

Barthax thanked this post

[QUOTE=Snowflake;bt7426]Its true anyone who voted wrong on a submission is motivated to vote incorrectly again to protect themselves. I'm not sure if thats necessarily a flaw though, since who are you or anyone else to say their wrong ahead of the fact? if the majority stands by the earlier adjudication -- even if that means the majority is just standing by themselves, then you still have the majority agreeing with it. It should be hard to remove a score. The fact that the people who previoulsy voted one way will likely stand by their vote and keep the score status the same i'd see as a good thing. Bad scores do have to go, but the burden of proof has to be high.[/QUOTE]

I have many mixed feelings on ur stance BUT I stand by what Jace should do!!!!!!!

Originally Posted by Jace Hall

CR is accrued on a vote at the time of voting.

So if someone wants to solidify their own vote, they can keep unvoting and re-voting as their CR increases?

I think what some of us are worried about is that someone is going to go rogue and dispute a bunch of scores out of spite and make a mess out of the system. We are just looking for some sort of control to make sure this doesn't happen. I like John's idea of limiting the number of disputes even if it's something as simple as one per day. Just don't allow someone to get mad and make a huge mess.

redelf, HugDD liked this post

Originally Posted by Barthax

So if someone wants to solidify their own vote, they can keep unvoting and re-voting as their CR increases?

Theoretically, yes.

Which ultimately all that does is move it closer to an admin decision point.

Barthax thanked this post

Originally Posted by D.B. Cooper

I think what some of us are worried about is that someone is going to go rogue and dispute a bunch of scores out of spite and make a mess out of the system. We are just looking for some sort of control to make sure this doesn't happen. I like John's idea of limiting the number of disputes even if it's something as simple as one per day. Just don't allow someone to get mad and make a huge mess.

If a person started behaving that way out of spite Admin will simply ban them. No joke.

We will look into that 1 per day idea however...

redelf, HugDD liked this post

Originally Posted by lexmark

The above "trigger mechanism" is open to abuse. eg.....the dig dug submission, if you voted NO on that submission, then don't vote AT ALL in the challenge thread, that way a "score removal recommendation" will NOT be triggered. The challenge stays in limbo indefinitely!
The above strategy is one that any member facing a cred loss on any challenge could follow. And will.


john

.

The trigger mechanism is the thing that forces admin to make a public decision about a dispute claim.

However, admin will likely be monitoring disputes in general and can decide on its own to agree to a documented claim without the trigger mechanism causing the decision.

So there will likely be cases where evidence is so compelling and immediate that admin will agree and take action regardless.

For instance - an accepted track for a game that somehow got accepted on PAL instead of NTSC. Admin may see a dispute claim like this and take immediate action.

The primary goal of the dispute system is to make sure TG has public documentation and rationale regarding any and all changes to the score database in the pursuit of database integrity.

Everything in the system is in service to that goal.

Most dispute cases should be clear-cut and non controversial. Solid evidence, no doubt, clean. These will be acted upon quickly.

JasonV91 liked this post

In the beginning of the challenge system and how I suggest it should be done.
Make if very expensive to create a challenge. Like a total of 10,000 submissions points donated like we do currently in a track creation format. These points will be given back upon a successful completion of the challenge. They will be lost only if the community rejects the challenge. If the community accepts the challenge but TG does not then the points will also be restored.


Why so expensive in the beginning? To weed out more difficult challenges and keep the challenges more on point. If we just allow anyone to create a challenge then people will be challenging some very minor details and such and just clutter up the challenge system. The reason for a high cost is so that it is very costly to have one rejected. Keep the challenges to a minimum in the beginning. Later these restrictions can be reduced and will therefore open up more difficult challenges. This also makes it so that the most obvious and easiest to prove will be brought forth in the beginning and the challenge system will gain confidence and strength as we move forward.


To keep the challenges down to a minimum at the beginning maybe also limit the number of challenges you can have limited to the amount of credibility you have. So limit the amount you can have open by each 1k, 2k or 10k or whatever amount you think reasonable. So if I have ** and for each 3k I have I would be able to have 2 open, 3 if I had 9k.


So lets say we set it up like the new track section. Cost should be nothing to start it up, just need 4k in credibility to be able to start one. The cost to make it active though should be very expensive, something like around 10k in the very beginning. Later after things have been running for a while the cost can go down and down as time goes on. Just make it very expensive and costly at the beginning so that we only get the best and easiest challenges to start off with. As the cost goes down the more difficult challenges will start to flow into the mix. By this time we will have a better understanding of how it works and wil have better community guidlines as to what is acceptable and what is not. Below is how I would try to do it.


1. You create a challenge on game Squidsearch.
2. You put into the description the reason why, your evidence, and outcome you want to hapen. Like moving it to the correct track, fixing the score or time, removing for whatever reason.
3. Then the donor section will be created. At this point it has cost you nothing except one of your open slots to create a challenge.
4. If you have enough submission points you can fill it and then it becomes active. Or you can wait for others to help fill it up and take some of the risk away. Remember you lose your submission points if you loose.
5. Once it is filled it becomes active and others can now start to vote on it.


By all means this isn't the only way to do it and I'm sure you have other ideas and thoughts along with the community on how to do it. I'm just suggesting that something other than allowing people to go nuts with the challenge system would be better. I even wish that some scores couldn't be challenged at the opening bell. The less contraversial things are the better they will be for the challenge system.


This will allow almost all members to contribute to the challenge system, but will limit who can create a challenge to start things off. Those with higher credibility have put more effort into adjudication and therefore seem to have more interest in what goes on the scoreboard. Those with higher credibility will probably also get messages from those that can't start a challenge. It will be up to them if they want to take the risk or not.


The reason I suggest such a high cost and a slow rate of start is to keep things simple and neat. If you allow some stupid challenge to be created and it fails or just makes the community fight then you lose the affect that the challenge system is supposed to have. Lets give an example that I think might come up. Robotron marathon top score end in 80. It's not possible blah blah blah. Lets say that the community decides that it should be romoved but final word comes from TG. TG says no and the score stays. Well there goes alot of faith from the community in the challenge system. If you start off with contraversy you will never gain the trust and faith of the community. So I suggest you make it hard to create a challenge and limit the challenges to those that will easily get confirmed. Why do I say easily, well the high cost to ones account will deter those from creating a challenge they don't have faith in themselves. There are lots of examples of easyily identified errors and such that will be picked by the community to be removed from the scoreboard. After some time these will become harder and harder to find and the cost to great to start the challenge. So at that time the cost and risk go down and then more difficult challenges start to come in.


We don't need 400 challenges going off at once. I'd be happy with 10 or so a week. I know that seems like a small number at this time. I'd rather that it gets done right and without upsetting or creating further rifts between the community. No one really knows what is going to happen, there could be a huge amount of interest from the community or very little. It might just be a few individuals that create the challenges, after all there are only around 50 that can do that. There seems to be a large group of members that talk but don't have the credibility to do anything yet. Keeping things slow will allow those members that want to articipate in creating the challenges time to grow there credibility and then start contributing with challenges.


The high cost will also make it so that very contraversial scores wont be challenged in the beginning and create negative feelings and lost faith in the system when they get tied up in discussion about this and that. It will also make it so that when the community votes yes and TG staff votes no, there won't be as much strife and upset.
TG staff will get alot of sour faces if they start off this early going with "we have decided to keep the score even though the community has stated otherwise". Scores like Robotron marathon, Atari 2600 Dragster, and others that have had open and ongoing discussions wont be in the opening onslaught. Why would someone try to take down a score that is hugely debated and risk all their time and effort on it and take the huge risk in the beginning just to maybe have it removed. They will go after the easier scores and save the harder ones for later. These scores have been up for a very long time and still haven't come off the scoreboard, they can wait till later. You will also have certain people create challenges of scores that have been recently accepted by the community through the adjudication process. They will try to state their case again, they still wont have new overwhelming evidence to provide and it will just get rejected and the accepted score will remain, while other more important and easy to remove scores will go in limbo while we waste our time on less important scores imho.


People are passionate about some of these scores and have equal concerns to have them removed. Lets just start with the easy ones, and leave the more contraversial ones to later in the process. People will learn what the community wants for evidence and reasons for the score to be removed. By making the cost and punishment high in the beginning will make people take more time and thought before they go off and challenge a score. If your not sure your challenge will succeed then you better wait till later when the cost and risk go down, do more research, gain more knowledge on the subject before you try.


I have a list of games that need to be removed from the scoreboard and in that list are some scores that I wan't removed for reasons that I can't support with evidence or complete knowlege that they aren't legit. I think that in the beginning we have to and should only concern ourselves with the ones that we know we can provide overwhelming evidence to contridict the score that is being challenged. Allowing feelings, he said she said, he saw she saw, it's just not possible, they had to have cheated kinda thing into these early challenges will only make things worse than they are now.


This is the most passionate I've been about a subject on this site. I hope this challenge system only leads to a better scoreboard with more integerity. Allows the community to grow and to heal from past events. Jace I think you should invest some more thought into the challenge system before it goes live. Get some more feedback from the community before you release the beast. Maybe think more about the risk and rewards of challenging a score.


By the way when will I ever be able to edit my post, the system currently won't allow me to, and I make all sorts of typos.

MyOwnWorstEnemy liked this post

Originally Posted by redelf

The reason for a high cost is so that it is very costly to have one rejected.

First of all - That's not good that you can't edit your own post! Everyone should be able to edit their own posts. The edit function is at the top of your individual post. Is it not there for you? I just tested this and it seems to work fine. Any additional info you can provide on your inability to edit post would be helpful. What browser are you using?

Ok - back on topic -

The primary issue with why some of what you suggest is problematic is that there is no "rejection" point for a dispute.

To be clear, in the system there can not be a specific rejection point with closure.

Having dispute closure based on rejection of the claim and then dismissal that dispute claim makes it possible for the same exact dispute claim to be filed by another incoming user that has no knowledge of the previous dispute.

The cycle would be circular. It must be avoided.

I do understand your concern about a flood of dispute claims, and your desire to make it hard to file a claim as a way to gate the amount of initial claims - however making it harder and harder to file a claim works to defeat the goal of the dispute system.

The goal is to work toward database integrity. To do that, people need to be able to contribute. The harder we make it for people to contribute, the less likely they will do so and that hurts the effort toward the goal.

We have thought through many different scenarios:

CR reward / punishments
SP cost to initiate
Higher CR requirement
etc.

In the matter of score disputes with our goal in mind, all of the above have disadvantages - big ones. TG wants participation. TG wants bad scores out of the database.

The fact of the matter is that there is a 20 year build up of issues around "bad scores" in the TG database and people wanting them out and proper action taken. There is just no way around that reality for TG. Having a system that can finally address the matter is going to be explosive. It just will be no matter what. So all we can do is the best we can with the issue. We don't want to create MORE frustration by making a dispute system and then making it so exclusive to use that it makes things worse.

One thing that we are looking at is limiting dispute claim filing to 1 every 24 hours per person. That might be a reasonable initial limitation on the system. We will see.

I do appreciate your passion on this issue. We are definitely taking it seriously.

Marcade liked this post

Originally Posted by redelf

ILets give an example that I think might come up. Robotron marathon top score end in 80. It's not possible blah blah blah. Lets say that the community decides that it should be romoved but final word comes from TG. TG says no and the score stays. Well there goes alot of faith from the community in the challenge system.

Using this scenario there are couple things to keep in mind:

1.) Even if TG does not agree to remove the score, the dispute remains open and more and more evidence and discussion can accrue which may lead to TG eventually agreeing to remove the score.

2.) The entire argument around this marathon score is now specifically tied to the actual score FOR ALL TIME. Now the contention and knowledge is easily accessed and referred to every time the score is accessed. This is a good thing. This is what does not exist today.

Unless you are a big Robotron player / community member you have no way of knowing this controversy about the score. All that changes with the dispute system. That is a very valuable thing and I dont see how the community loses faith in that as a minimum outcome.

Knowledge about scores will be permanently part of a score's life, instead of only living in a few people who happened to be there, or are in the know somehow.

redelf liked this post

will there be a leaderboard for who initiated the most successful challenges?

Thanks for the new challenge system, looks like a ton of work went into it.

I guess my biggest problem is really figuring out what kind of evidence is valid for overturning a video.

NES scores that don't show the system?
Never see the cart?
Don't show Bootup?
Don't show controller?

If the burden of proof is on the challenger but we don't have a set of standards that everything gets rejected on if it does not show, I think many things are going to be very hard to overturn.

In my eyes a lot of scores just slip on in because people don't want to clink no for lack of evidence. But we are just going to do challenges in the same way.

Not that I have any idea how such a thing would be fixed with the current systems in place, just something that I have a personal concern with on how I am going to vote on challenges.

HugDD liked this post