Changelog
No release has been published, so there is no release changelog entry. Everything below is at its first version.
A superseded table stays published, marked superseded, with a link to the version that replaced it and a changelog entry saying what changed. Nothing is quietly deleted or overwritten; that is the whole point of publishing the previous number.
Cadence
Quarterly. Each release publishes, per suite: the leaderboards, the calibration report, the findings, the reproduce commands, the changelog, the public dataset sample and the disputes log. Between releases a correction may be published at any time, and it carries its own entry.
The first release date is not set. It waits on a run, which waits on the items in charter section 10.3.
Versions in force
| Component | Version | First published | Changes since |
|---|---|---|---|
| Charter and methodology | 1.0.0 | not published | none — first version |
| Contract clauses | 1.0.0 | not published | none — first version |
| Harness, entail-bench | 1.0.0 | not published | none — first version |
| Dataset: Messy Scan | not a dataset | not published | none — first version |
| Dataset: Honest Containment | not a dataset | not published | none — first version |
| Dataset: Exception Economics | not a dataset | not published | none — first version |
| Dataset: Day-60 | not a dataset | not published | none — first version |
Release entries
| Release | Date | Suites | What changed |
|---|---|---|---|
| No release has been published. The first release publishes when a suite has been run, and it will list every table it contains. | |||
What counts as a version change
7.1 Datasets, the harness and this charter each carry a semantic version.
| Change | Dataset | Harness | Charter |
|---|---|---|---|
| Major | An item changes or is removed; ground truth is corrected; a tier definition changes | A scoring rule changes in a way that can move a figure | A metric definition, denominator or exclusion changes |
| Minor | Items are added and every existing score stays valid | A capability is added without changing any existing score, such as a new adapter | A section is added; a clause is added to contract-clauses.md |
| Patch | A correction that changes no score, such as a datasheet typo or a deterministic generator fix producing identical output | A fix that changes no score | Wording that changes no definition |
7.2 A ground-truth correction is always a major dataset version. It changes scores by construction, so every affected table is re-run before it is published again.
7.3 No table mixes versions. Every results table states the dataset version and hash, the harness version and commit hash, the prompt set hash and the price list date in its header. A row produced under a different version sits in a different table.
7.4 Superseded results stay published, marked superseded, with a link to the version that replaced them and a changelog entry saying what changed. Nothing is quietly deleted or overwritten; that is the whole point of publishing the previous number.
7.5 Raw model responses are retained for placeholder months so that a disputed figure can be re-scored without re-running the model. The private split is retained and never published.
7.6 A charter major version triggers a re-run or an explicit supersession of every table that depended on the changed definition, and contract-clauses.md is re-issued with the same version number so that a signed clause can always be traced to the definition in force when it was signed.
<!-- Benchmark charter, part 5. Indexed in ../methodology.md. Sections 8 to 10. -->
Charter 1.0.0, sections 7 and 8
8.1 Cadence. Quarterly. Each release publishes, per suite: the leaderboards, the calibration report, the findings, the reproduce commands, the changelog, the public dataset sample, and the disputes log. Between releases, a correction may be published at any time under section 8.4, and it carries its own changelog entry.
8.2 Changelog. One file per release, listing dataset changes with their version bump, harness changes with their commit range, prompt changes, models and services added or removed with the reason, corrections made under the disputes process, tables marked superseded, and any run discarded under section 5.12.
8.3 Pre-publication notice. Every vendor whose system appears is sent its own rows, the reproduce command and the commit hash, placeholder working days before publication. The purpose is to catch a configuration error before it is published rather than after. Notice is not approval. A vendor cannot delay, edit or veto a publication, and silence is not consent to anything except the schedule.