Preview of Twin Galaxies’ Score Submission and Adjudication Process

Good news! We are rapidly approaching the activation of Twin Galaxies’ new submission and adjudication system!

In anticipation of this new era of score tracking, we wanted to take a moment and provide a preview of what is to come and allow the community to discuss the new system in more detail.

This is not intended to be a comprehensive and detailed description of the new submission and adjudication process on a technical level. It is only intended to help users understand how our Submission and Adjudication system functionally operates at the highest levels and explain the general philosophy behind it.

In the simplest view, the TGSAP (Twin Galaxies Submission and Adjudication Process) is essentially a tightly controlled peer-review system. This means that you submit your score and all evidence you have to support your submission to Twin Galaxies, and Twin Galaxies’ community and organization will review the materials and determine validity. Simple.

Twin Galaxies takes the subject of validating scores very seriously. The foundation of Twin Galaxies is in record keeping and the records that it keeps must be extremely credible and without reproach – therefore TG has spent many months designing a comprehensive and totally unique adjudication system that maximizes the benefits of a peer-review methodology, while simultaneously eliminating many inherent flaws that a peer-review system can have which undermine credibility.

To demonstrate our understanding of the problems and concerns associated with a peer-review system, we will discuss some (but not all) of the items we have identified and thoroughly addressed:

1.) Collusion - This is where multiple users of the system secretly “team up” to deliberately manipulate results and outcomes. There are numerous variations on this theme.

2.) The value of experts vs. non-experts – This is where people with less knowledge about a subject are participating with people who have more knowledge about the subject and there is no consideration or measured differential in the participant’s contribution to the outcome. Since every ‘vote’ is equal, it becomes possible to come to incorrect conclusions if the uninformed vastly outnumber the informed in a peer-review process.

3.) Accountability. In traditional peer-review, participants are merely a part of a total group decision and are not individually tied to the outcome. This leads to less motivation for accuracy, personal investment, or interest in the process. Individuals have no vested interest in the outcome, regardless of their choice during the participation.

As mentioned above, these are just some of the pitfalls of a standard peer-review system. Many record keeping sites today suffer some of these problems and it undermines the credibility of the results that are produced there.

The TGSAP however, is not a standard peer-review system. It is a totally unique, one-of-a-kind system that sets a new standard in score adjudication.


SUBMISSION
Scores can only be submitted by VERIFIED users. VERIFIED users are users that have gone through the Twin Galaxies’ user verification process, and TG maintains contact information on each user. These users are real people, who have voluntarily submitted their identity and personal information to be able to be eligible to be awarded world records and other forms of public recognition for achievement. No new world records will ever be allowed into the TG database without full name and proper contact information. TG needs to know who its participating users are, and the participating users themselves need to know that they are competing with other real and serious participants. All user information is kept private and confidential in accordance with our Terms of Service.

When a VERIFIED user submits a score, they are presented with a form to fill out. This form specifies:

Most of these fields are self-explanatory and were common to the previous Twin Galaxies referee-based adjudication process, however the “Evidence Description” and “Video Upload” fields are new and will be explained below.

Historically, Twin Galaxies would adjudicate a score by having a person watch a score performance in person live, or by having a person review a recording of a score performance. These methods are very effective, however they also severely limited the scope of score performance presentation.

In an effort to accommodate the widest number of methods of score performance presentation, the TGSAP has moved to a format that allows user submission of all forms of evidence with a score claim, whatever that may be, with TG simply providing very strong recommended guidelines for submissions in general.

The “Evidence Description” field is the part of the submission form where you are given an opportunity to explain any details of your achievement that you feel supports your score claim. This field is driven by a fully functional WYSIWYG text editor that is also equipped to allow you to upload pictures, screenshots, and supporting documents. This is an opportunity to build your case of evidence, head off any skepticism, provide insight or simply brag about what you have accomplished. This is the opportunity to provide as much context as you feel you need to help make sure that anyone judging your claim is as informed as possible.


The “Video Upload” field is where you upload any recorded performance video that you may have of your score performance. This video is part of your overall “evidence package.”


Twin Galaxies will post recommended requirements for how a score performance should be recorded, and how evidence can best be presented. Twin Galaxies will post its preferred and recommended type of submission. This will help the peer-review community understand the kinds of things they may want to look for and potentially expect when trying to determine the validity of a score performance – however, TG is only making recommendations and it is not required to follow them explicitly to submit a score.

Whatever evidence you submit is what the peer-review process is going to consider. So it is really up to you to determine what you feel is adequate to present. It’s that simple.

While Twin Galaxies will recommend that you present a rock-solid case of indisputable visual evidence that leaves no doubt that your score performance is valid, the fact of the matter will be that if you do not want to (or simply can’t) submit a video of your score performance and instead only provide screenshots and other evidence, that is your option and the peer-review process will determine whether or not the case you have presented is compelling enough to be considered valid.

Once all of your evidence is entered/linked/uploaded, you will click the SUBMIT button and then the adjudication process begins.


ADJUDICATION
(Please note – The adjudication system is multi-layered in its complexity. In order to explain it, we are going to describe the user experience first, then discuss the mechanics behind it in the form of Q&A after. Please bear with us.)

The User Experience of Adjudication -
When you click the SUBMIT button, all of the evidence you have provided (including video) is compiled and transformed into a stand-alone special kind of forum thread that resides in a forum section called SUBMISSION REVIEW.

This forum can be viewed by anyone, but only VERIFIED users are able to participate in it.

The SUBMISSION REVIEW forum will contain all incoming submissions that need to be adjudicated. These submissions take the form of individual threads.

Your submission thread will contain all the evidence information you provided in a simple and organized view. Your score performance video (if one was provided) is front and center and easily viewable.

At the top of each submission is a poll mechanism that asks a simple question:

All individuals from the VERIFIED community can now review the evidence you presented and cast their vote.

If when reviewing the evidence, there are questions or concerns, a participating voter can utilize the discussion thread that is part of your submission thread and directly raise those questions or concerns with you and other community members. You are able to respond and participate in the discussion as much as desired. If more evidence is required to satisfy voter concerns, you will be able to add it to the submission thread (however you will not be able to alter the original submission materials) until the voting period closes.

When the voting period ends, the submission thread is closed and the final results are computed and the score is either ACCEPTED into the Twin Galaxies database, or it is REJECTED.

If the score is ACCEPTED, the now-permanently-closed-submission-thread is moved to the ADJUDICATION ARCHIVE, under the ACCEPTED category as a permanent record and visible reference of the adjudication – and the newly entered score in the database is hot-linked to this permanent record for all see. As the database fills up with new scores over time, all the scores in the Twin Galaxies database will become clickable, and link directly to the supporting evidence for anyone who is curious about a particular score.

If the score is REJECTED, the now-permanently-closed-submission-thread is moved to the ADJUDICATION ARCHIVE, under the REJECTED category as a permanent record and visible reference of the rejected adjudication result.

A submission, whether ACCEPTED or REJECTED, is permanently documented for all time.

That is the user experience. Very simple and straightforward.

The Adjudication Mechanism- (Question and Answer format):

Q:Wait a minute. How are you preventing collusion? If it is just a simple ACCEPT/REJECT vote question, people can just get all their friends to vote one way or the other and win every time!!

Collusion is prevented by a multi-layered stack of interacting thematic measures:


Asymmetrical Adjudication Qualifiers (AAQ) –
At the moment a score is submitted, the TGSAP looks at a wide range of relevant dynamic system parameters (for instance, total submission rate/site activity/adjudication response times/etc.) in order to decide what the specific computational measurement requirement will be for a successful adjudication of THAT submitted score.

This means that every adjudication is dynamic in terms of what it will specifically require from a voting participant calculation perspective.

For instance, just one of the many parameters considered by the TGSAP are how many total participant votes it will take to “close” a particular submission. This will vary from submission to submission. It will be impossible for anyone to know this information or predict it.

Although these computations are dynamic and produce variation, they are all based on a set of algorithms that are unilaterally applied to all submissions so it is fair and consistent. The intention of this computational system is to make sure that peer-review requirements are always in-line with the total activity of the Twin Galaxies website, so that there is always a systemic correlation between participation and adjudication requirements. The added benefit of this method is that it makes collusion extremely challenging, because it is no longer a function of just brute force group voting. Someone seeking to collude will have to develop a system to figure out the algorithms and patterns, and that requires direct experimentation on our system to see how it responds. This testing allows Twin Galaxies to easily detect user accounts trying to do this.

The bottom line is that it is extremely hard to manipulate what you cannot predict – so a potential cheater/colluder will need to put a ton of effort into figuring out how to predict this stuff.

And remember, this is just ONE part of the TGSAP. On to the next part…

Information Deprivation –
Due to the AAQ, every submission is different from a computational perspective. In order for someone to be sure that a collusion will be effective and manipulate the peer-review process, the parameters of the final computation must be known. For instance, it is pointless to get 10 of your friends together to vote in unison on something if the required amount of votes to achieve a result is 100.

The TGSAP is designed to obscure any/all computational information when it comes to the voting process. This means:

• Users never know how many total votes must be cast for any particular submission to “close” and be computed.
• Users never know or see what other users have voted.
• Users never see where the vote “currently stands” while the voting period is open.
• Users never know how long any particular submission will remain open for voting.
• All of this information and anything else that relates to it is obscured.

All a user ever sees, is the submission evidence, what THEIR vote currently is, and whether or not the vote is still open or if it has closed. That’s it.

When a vote finally does close, users never see who voted, how many votes it took to close the vote, OR what other users’ votes were.

Again, information deprivation adds to the difficulty of any collusion or manipulation – but this only the second mechanism that is at work… below is the next preventative measure…

Emergent vote weighting –
This mechanism not only further disrupts collusion, but it also addresses the inherent peer-review issue of the value of experts vs. non-experts – the aforementioned issue where people with less knowledge about a subject are participating with people who have more knowledge about the subject and there is no consideration or measured differential in the participant’s contribution to the outcome. Since every ‘vote’ is equal, it becomes possible to come to incorrect conclusions if the uninformed vastly outnumber the informed in the peer-review process…

Soon you will see that every VERIFIED TG USER ACCOUNT has a new parameter added to it. This parameter is titled CREDIBILITY.

The CREDIBILITY scale is from 0 (min) – 100,000 (max).

All new VERIFIED users will start with a CREDIBILITY of 10.

To cause differentiation and disrupt vote equality in a peer-review system, TGSAP will factor into its algorithm each participating VERIFIED users’ CREDIBILITY RATING (CR) during the adjudication process.


When you correctly adjudicate a score, the TGSAP will award you additional CREDIBILITY POINTS for being correct, which in turn raises your CR and makes your vote on future adjudications WORTH MORE.

However, if you incorrectly adjudicate a score, the TGSAP will deduct CREDIBILITY POINTS from you for being incorrect, which in turn lowers your CR and makes your vote on future adjudications WORTH LESS.

For clarity, here is a simple example:

· A user named JOHN SMITH submits a Ms. Pac Man score of 800,000 points. He provides whatever evidence of the score he has, screenshots, a description, and a video performance.

· You have a current CR of 50.

· You see JOHN SMITH’s submission and review the materials. Based on what you saw, you believe it to be a credible submission, so you vote ACCEPT, and then go about your business on the TG site (posting to forums, whatever.) The voting process remains open.

· A couple days later, you see that JOHN SMITH’s submission is still being voted on / adjudicated. You go back to the thread and see several comments below the submission that raise questions about the submitted recording. You think that the questions being raised might be valid, so you decide to change your vote to REJECT, and then go about your business on the TG site (posting to forums, whatever.) The voting process remains open.

· The next day you come back to the Twin Galaxies site, and see that JOHN SMITH’s submission is still under review and there are even more comments on the submission thread. You read them and think to yourself, I really believe that this score is valid – and so you change your vote BACK to ACCEPT, and then go about your business on the TG site (posting to forums, whatever.) The voting process remains open.

· A few hours later, JOHN SMITH’s adjudication process closes, and the votes are algorithmically computed, factoring in all of the participating users CR weightings.

· It turns out the score was ACCEPTED.

· You receive an e-mail saying “Congrats, you were on the correct side of the JOHN SMITH adjudication! You have received 5 more CREDIBILITY POINTS.”

· You now have a current CR of 55.

Generally speaking, the amount of points you can “win” by being correct, will always be smaller than the amount of points you stand to “lose” by being incorrect. So it is much harder to build CREDIBILITY within the system than it is to lose it.

For example: the CR points acquired from 10 correct adjudications could be theoretically lost by just 2 or 3 incorrect ones.

This part of the TGSAP is what will allow the experts (people that care) to separate from the non-experts (people that don’t care) over time, and simultaneously make those experts votes count MORE during adjudication than the non-experts.

Any user that does not put care and attention to their voting decisions so that they are on the correct side of most adjudications, will quickly see their CR drop to 0 and quite literally their vote will not matter within the adjudication computation.

This system will naturally balance itself and it removes one of the inherent problems of peer-review where all votes count the same. Within the TGSAP, all votes do not count the same, and anyone within the TGSAP that maintains a high CR is someone who has consistently been correct when reviewing submissions. This person has become an authority. Becoming an authority cannot happen by accident.

Q: But what about people who just take “easy” submissions and try to build up their CR, for the sole purpose of being powerful and then misusing that power to alter adjudications inappropriately??? Or what about people who just build up their CR, and just sit on it and don’t adjudicate anymore???

There are two active things that the TGSAP does to prevent these issues:

1.) The TGSAP is always considering the TOTAL VOTING POWER of all users collectively, and generating a system of bell-curves to compensate for a collective rise or fall in the total economy of CR. (INFLATION) So that means that if everyone is always adjudicating correctly, and everyone is getting more points, the relative vote weighting doesn’t change so no one’s vote is worth more than anyone else’s – even though the CR numbers on individual users are all elevated. The key to this system is that the CR considerations are always relative to everyone else’s CR. The “easy submission build up strategy” simply won’t work for the purposes of adjudication manipulation.

2.) The TGSAP will slowly and lightly tax all user’s CRs’ on a continuous basis. Much like the arcade game GAUNTLET, where the player’s health is always moving down and they need to continuously find food – VERIFIED users that are interested in maintaining their CR must continue to adjudicate.

Dynamic Incentivization/Penalization
The nature of the TGSAP encourages users to build and maintain their CR. The more you correctly adjudicate, the more points you receive, the more your voice matters in the process. You cannot buy, cheat or manipulate your way into a high CR at Twin Galaxies. You must earn it, and everyone in the community will know that.

Through this system, you will be able to demonstrably prove your knowledge and authority in regard to video games. Your CR will literally be backed up with documented proof of your track record and decision making. Your authority on the subject will be respected, and it will be deserved.
No other organization in the video game industry offers you this opportunity. No other organization is comprehensively recognizing achievement both on the player and adjudicator sides of the equation.

The scores going into the Twin Galaxies database will be the most heavily scrutinized in the world.

Beyond the respect, beyond the recognized authority, Twin Galaxies will, over time, additionally provide titles, certificates, badges, exclusive access to TG branded collectibles to those members of the community who develop and maintain a high CR. The possibilities are endless.

All of this opportunity works to incentivize the community to not risk the pitfalls of collusion or cheating of any kind.

Anyone caught or even suspected of cheating/colluding may be banned from Twin Galaxies for life.

Q: When CREDIBILITY POINTS are awarded or deducted, how is the amount decided?

There is always a standard minimum amount PLUS a bonus amount that revolves around a consideration of factors that directly relate to a user specifically, such as how much general participation they demonstrate, recent post count, number of received ‘thanks’ or ‘likes” on posts, etc.

Participation within Twin Galaxies of any kind is a good thing, and the system will always be looking at the total experience and activity of a user and considering it when determining awards of any kind.

Again, there is an algorithm and computation here.

Q: I DON’T CARE ABOUT ALL THIS ADJUDICATION STUFF. I JUST WANT TO SUBMIT MY SCORE. WHY SHOULD I PARTICIPATE IN ALL THIS?

The TGSAP requires that VERIFIED users that wish to use the system maintain a positive SUBMISSION TO ADJUDICATION RATIO (STAR).

This means for every score performance that you submit, you must have CORRECTLY adjudicated a certain number of other people’s submissions.

Please take note of the word CORRECTLY.

Simply going into the SUBMISSIONS forum and randomly voting in order to “get your vote count up” will not help your STAR. In fact, it will likely damage it.

The STAR calculation is only looking at adjudication results that you have CORRECTLY called.

Adjudications are not instant processes, and can take time – so it is recommended that you participate in adjudication as much as possible ahead of your submissions so that your current STAR is more than adequate to qualify you.

If you want to submit scores to Twin Galaxies and receive the recognition, benefits and status of having a world record entry into the database, then you must be willing to participate and contribute.

The STAR system is designed to prevent users from abusing the TGSAP for selfish purposes.

Again, this document is not intended to be a comprehensive detailed description of the new submission and adjudication process on a technical level. It is only intended to help users understand how our Submission and Adjudication system functionally operates at the highest levels and explain the general philosophy behind it.

Hopefully this has given you some insight, understanding and excitement about submitting your amazing score to Twin Galaxies and being recognized for your achievement!

SPECIAL NOTE – Jace Hall, Head Custodian of Twin Galaxies, is planning on holding a live candid round table discussion broadcast soon regarding Twin Galaxies’ past, present and future. He will be answering any/all questions regarding the discussion topic. The community is invited to join the broadcast at www.twinglaxieslive.com and directly raise any concern or issue and get straight and direct answers. Please stay tuned for specific time/date and get your questions ready!

-TG