Disputes
Anyone may dispute any figure. The process is public and its outcomes are published whichever way they go.
A dispute about our own system's row runs through the same steps and is marked in the log as a self-dispute, so a reader can count how many there have been and how they went. Nothing leaves the log: a dispute we lose stays visible, with the correction linked to it.
How to raise one
Open the "dispute a result" issue template in nagu-io/benchmarks, or write to hello@entailmentlabs.com.
A dispute must name the table, the row, the dataset and harness versions, and what is wrong: a defect, or a disagreement with a definition.
The log
| Raised | Table and row | Kind | Status | Outcome |
|---|---|---|---|---|
| No dispute has been raised. Nothing has been published to dispute: every results table reads "not run". | ||||
The process
Read from charter 1.0.0 sections 8.3 and 8.4 when this page was built.
8.3 Pre-publication notice. Every vendor whose system appears is sent its own rows, the reproduce command and the commit hash, placeholder working days before publication. The purpose is to catch a configuration error before it is published rather than after. Notice is not approval. A vendor cannot delay, edit or veto a publication, and silence is not consent to anything except the schedule.
8.4 Disputes process. Anyone may dispute any figure. The process is public and its outcomes are published whichever way they go.
| Step | What happens | Time |
|---|---|---|
| 1. Raise | Anyone opens a dispute through the "dispute a result" issue template in the public repository, or by writing to hello@entailmentlabs.com. It must name the table, the row, the dataset and harness versions, and what is wrong: a defect, meaning wrong ground truth, a harness bug, a wrong price, a wrong model version or a misconfigured adapter; or a disagreement with a definition | — |
| 2. Acknowledge | We acknowledge and publish the dispute in the disputes log with its status and the date | Within placeholder working days |
| 3. Re-run | We re-run the affected row from the stated commit. Where the dispute names a different configuration, we run both and publish both | Within placeholder working days |
| 4. Decide | Upheld: the table is corrected, the version is bumped under section 7, the changelog names the correction, and the original figure stays visible marked superseded. Not upheld: the reasoning and the evidence are published in the disputes log. Definition disagreement: the definition does not change for the current release, the objection is published beside the definition, and the change is considered for the next major charter version | Within placeholder working days |
| 5. Appeal | One appeal per dispute, decided by the named benchmark owner, published either way with its reasoning | Within placeholder working days |
8.4.1 Nothing leaves the disputes log. A dispute we lose stays visible, with the correction linked to it.
8.4.2 A dispute about our own system's row runs through the same steps and is marked in the log as a self-dispute, so that a reader can count how many there have been and how they went.
8.4.3 A dispute never pauses publication of the rest of a release. The disputed row is marked "under dispute" with a link to the log entry, and the rest stands.
The periods above read "placeholder" because they are a decision a person has not made yet. A placeholder in a published process is a defect, and it is listed as one: charter section 8.
Our interest in the outcome
We are not neutral parties. We sell systems in all four categories these suites measure. The neutrality rules are the discipline we accept in return for publishing at all, this page is the route to hold us to them, and neither removes the interest we have in the outcome.
Charter 5.10 and 9.10