Benchmark charter and methodology

The charter, in parts

This charter is published in parts so that a section can be linked, reviewed and cited on its own. The parts are the charter: nothing here is a summary and nothing is left out.

The two rules that govern every part

A figure with no run is written `not run` with the reason. It is never estimated, extrapolated, interpolated from a neighbouring tier, or replaced with a plausible-looking figure, in a table, a chart, a chart's sample data, prose or a code fixture. Sections 3.1.8 and 10.4.

The figures marked "arithmetic example" are arithmetic. They are invented numbers chosen to demonstrate a formula. They are not results, ours or anyone's, and quoting one as a result is a misuse of this document. Section 10.5.

Status

No suite has been run. The datasets, the harness and the Day-60 rubric are built and validated. No model interface key was available and the build environment could not reach a model interface. The part carrying section 10 has the status table and what a first run needs from a person.

<!-- Benchmark charter, part 1. Indexed in ../methodology.md. Sections 1 to 3.6. -->

1. Purpose

1.1 These benchmarks exist to measure the distance between the number in a vendor deck and the number a process produces in production, and to state both in definitions a BPO can put in a contract.

1.2 The distance is usually made of four things: a clean test set standing in for messy inputs, a denominator that has quietly dropped the hard items, a success definition that counts an abandoned contact as a win, and a cost figure taken at a token count that real documents do not reach. None of that requires anyone to lie. It only requires the definition to go unstated.

1.3 So the test of every metric in section 3 is not whether it is interesting. It is whether a BPO could paste the definition into a statement of work and hold a supplier to it. If a metric cannot survive that, it is not finished. Each metric therefore carries a contract sentence, and contract-clauses.md turns each one into clause language.

1.4 Who this is written for:

1.5 What these benchmarks are not for. They are not a ranking for its own sake, they are not sold or sponsored, and they do not replace a test on the partner's own material. The bake-off in ../bakeoff/ exists because a public benchmark on synthetic data can only narrow the field, not settle it.

1.6 Our conflict of interest, stated at the front rather than in a footnote. We design, build and operate systems in all four categories, and our own system is in the tables. That is a reason to distrust us. The neutrality rules in section 5, the reproduction requirement in section 5.5 and the disputes process in section 8.4 exist because of it. They are the only answer we have to it.

2. The four suites

2.1 Four suites, one per line of work in BRIEF.md section 3. Each scores a different unit, and the unit decides every denominator in section 3.

2.2 Messy Scan. A model or service reads synthetic documents that have been degraded on purpose, across five tiers and several formats and languages, and returns a structured record. It is scored on field-level accuracy, straight-through rate, exception rate, calibration, cost and latency, broken out per tier, per document type and per language. It deliberately does not measure integration effort, review-queue design or anything a person does after the output is produced.

2.3 Honest Containment. A simulated caller follows a hidden script; the agent under test talks to it; rule checks and a judge model score the outcome against the scenario's ground truth and policy pack. The suite exists because containment is the metric most often quoted and least often defined. Every agent is therefore scored under four common industry definitions and under ours, and the spread between them is published as a column, not as a footnote. It does not measure voice quality, speech recognition in isolation, or telephony reliability.

2.4 Exception Economics. Work items with known answers are run at three confidence thresholds. The suite scores automation rate, the cost of wrong automations in rework minutes, reviewer minutes per exception and net cost per item, using a labour model published with the dataset. Its point is that automation rate alone is not a result: a system that automates everything and is wrong on a tenth of it can be more expensive than one that automates half and is right. It does not measure the partner's own reviewer productivity, which varies more than any model in the table.

2.5 Day-60. Not a dataset. A rubric and a set of scripted exercises run against a live deployment, ours or anyone's, sixty days after go-live: inject a drift and time the detection and the notice; raise an incident and time the restoration and the communication; force a rollback and time it; audit the monthly report against a completeness checklist. It produces a 0 to 100 score against a published rubric, and it has a self-assessment version a BPO can run against its current supplier in an afternoon. It does not measure model quality at all, which is the point of running it alongside the other three.

3. Metric definitions

3.1 Rules that apply to every metric in this section

3.1.1 Every metric states its unit, its numerator, its denominator and its exclusions. A metric quoted without its denominator is not a metric, and no Entailment Labs document quotes one that way.

3.1.2 Nothing is silently dropped. An item that cannot be processed, a field with no ground truth, a contact that never connected, an incident that was withdrawn: each is excluded only where this section says so, and each exclusion is counted and reported beside the figure it was excluded from. The harness fails a run whose reported counts do not reconcile with the manifest.

3.1.3 Every figure is reported with its sample size, its dataset version, its harness version and its run date. A figure without a sample size is not evidence, in a benchmark report or in a partner report.

3.1.4 Three runs per model per suite at identical settings. Every published figure is the mean of three, with the standard deviation and the minimum and maximum beside it. A single-run figure is never published, and a difference smaller than the reported spread is not reported as a difference.

3.1.5 Rounding. Rates are reported to one decimal place, times to the precision the source clock supports, money to four significant figures. Rounding never moves a figure across a contractual floor or ceiling: a floor is met only if the unrounded figure meets it.

3.1.6 Percentage points and percent are not interchangeable. A change in a rate is stated in percentage points. A change in a cost or a time is stated in percent of the baseline.

3.1.7 Where a suite has tiers (section 4), every headline figure is accompanied by the per-tier table and the tier mix used. A headline figure moves when the mix moves, so the mix travels with it.

3.1.8 Where a figure has not been produced by a run, it is written not run with the reason. It is never estimated, extrapolated, interpolated from a neighbouring tier, or replaced with a plausible-looking figure, in a table, a chart, a code fixture or a sentence.

3.1.9 All ground truth is synthetic, and is labelled as synthetic wherever it appears. Section 6 governs the data.

3.1.10 Each metric below is set out in the same order: formula, numerator, denominator, exclusions, an arithmetic example, and the contract sentence. The arithmetic examples use small invented numbers to show the mechanics of the formula. They are marked, and they are not results.

3.2 Index of metrics

3.3 Field-level accuracy

3.3.1 Formula. Field-level accuracy = correct field instances ÷ field instances assessed.

3.3.2 Numerator. Field instances whose returned value matches the labelled value under the match rule for that field type. Match rules, which are the rules already used in 06-delivery/build-standards.md section 6.1: exact for identifiers and codes; normalised for dates (compared as ISO 8601 calendar dates), amounts (compared as a numeric value plus a currency code, with separators, symbols and whitespace removed), and names (case, punctuation, diacritics, honorifics and multiple spaces normalised); free-text fields scored only where a tolerance rule is stated, and reported separately from the exact and normalised fields. A field the ground truth marks absent is correct only if the system returns an explicit null or empty value for it.

3.3.3 Denominator. Every field instance in the ground truth for the documents scored, plus every field instance the system returned that has no counterpart in the ground truth. A returned field that should not exist counts as one assessed instance and is always incorrect. This is the point of the rule: a system cannot raise its accuracy by inventing fields, and it cannot raise it by returning fewer.

3.3.4 Excluded. Fields outside the published schema for that document type. Field instances whose ground truth is marked unreadable by the labeller, which are counted and reported as a separate line rather than scored. Documents that failed to process at all, which are excluded from this metric and counted under the processing-failure line, never dropped from the manifest.

3.3.5 Arithmetic example, not a result. Five documents, twelve ground-truth fields each, so sixty ground-truth instances. The system also returns two fields that do not exist in the ground truth, giving sixty-two assessed instances. Fifty-seven returned values match under the match rules. Field-level accuracy = 57 ÷ 62 = 91.9 percent. The invented numbers in this paragraph exist to show the arithmetic.

3.3.6 Reported as. Overall, per field, per document type, per tier and per language, with three-run mean and spread.

3.3.7 Contract sentence. "Field-level accuracy means the number of field instances whose returned value matches the labelled value under the match rules in Schedule [X], divided by the number of field instances assessed, where field instances assessed comprise every field instance in the labelled set for the documents scored plus every field returned by the System for which the labelled set holds no counterpart. The Provider shall achieve field-level accuracy of not less than placeholder percent, measured on the labelled set named in clause [X] at each measurement point."

3.4 Document-level straight-through-processing rate

3.4.1 Formula. Straight-through-processing rate = documents released with no human touch ÷ documents admitted to processing. Straight-through means zero human touch, not "little human touch", and not "approved quickly".

3.4.2 Numerator. Documents whose output was released to the downstream system without any human action of any kind: no review-queue entry, no field correction, no approval action, no reviewer opening the item. One exception, stated so that measurement does not corrupt the measure: a document drawn into the sampled-audit queue (AUDIT in 06-delivery/build-standards.md section 7.1) still counts as straight-through if the audit reviewer changed nothing, because the audit is a measurement device applied after release. If the audit review changed the output, the document is not straight-through and is counted as an exception under section 3.5.

3.4.3 Denominator. Documents admitted to processing in the window: documents received, less documents rejected before processing by a pre-processing rule named in the statement of work. Both counts are reported, so the effect of the admission rule on the rate is visible.

3.4.4 Excluded. Documents rejected before processing by a named rule, which are counted and reported separately. Documents still in flight at the end of the window, which are counted and reported as in flight; the window is chosen so that the in-flight share is small enough to state.

3.4.5 Arithmetic example, not a result. One thousand documents received, twenty rejected by the named pre-processing rule, so 980 admitted. Of the 980, 715 were released with no queue entry other than a sampled audit; three of those audits changed the output. Straight-through documents = 715 − 3 = 712. Straight-through-processing rate = 712 ÷ 980 = 72.7 percent. The invented numbers in this paragraph exist to show the arithmetic.

3.4.6 Reported as. Overall and per tier, per document type and per language, always beside field-level accuracy on the same row. Straight-through rate on its own is not a result: a system that releases everything untouched scores 100 percent and may be wrong on most of it.

3.4.7 Contract sentence. "Straight-through-processing rate means the number of Documents released to the Partner's downstream system with no human action of any kind, divided by the number of Documents admitted to processing in the measurement window, where a Document drawn into the sampled audit is treated as straight-through only if the audit did not change its output. The Provider shall achieve a straight-through-processing rate of not less than placeholder percent while maintaining field-level accuracy at or above the floor in clause [X], and neither measure is satisfied without the other."

3.5 Exception rate

3.5.1 Formula. Exception rate = documents that entered human review ÷ documents admitted to processing.

3.5.2 Numerator. Documents with at least one review-queue entry under the codes LOWCONF, VALFAIL, FLAG, FAIL or DRIFT, plus documents drawn into the sampled audit whose output the reviewer changed. A document counts once however many fields triggered it and however many times it re-entered the queue; the count of re-entries is reported separately.

3.5.3 Denominator. Documents admitted to processing, the same denominator as section 3.4. The harness asserts the identity: straight-through rate plus exception rate equals one. A supplier's figures that do not satisfy that identity are using two different denominators, and the report says so.

3.5.4 Excluded. Sampled-audit draws that confirmed the output unchanged, which are reported separately as the audit rate. Documents rejected before processing by a named rule. Queue entries created by the partner for training or calibration, which are counted separately.

3.5.5 Arithmetic example, not a result. Continuing the arithmetic of section 3.4: 980 documents admitted, 712 straight-through, so 268 entered review. Exception rate = 268 ÷ 980 = 27.3 percent, and 72.7 + 27.3 = 100. By entry code the same 268 might break down as 180 LOWCONF, 60 VALFAIL, 25 FAIL and 3 corrected audits. The invented numbers in this paragraph exist to show the arithmetic.

3.5.6 Reported as. Overall, by entry code, per tier and per confidence threshold, because the exception rate is a function of the threshold and moves the moment the threshold does.

3.5.7 Contract sentence. "Exception rate means the number of Documents that entered the human review queue under any entry code other than a sampled audit that resulted in no change, divided by the number of Documents admitted to processing in the measurement window. The Provider shall keep the exception rate at or below placeholder percent, measured over a calendar month at the confidence thresholds recorded in the configuration in force during that month, and shall state any threshold change and its date in the Monthly Report."

3.6 Confidence calibration, expected calibration error

3.6.1 Why it is here. A review queue is built on a confidence threshold. If the confidence is not calibrated, the threshold routes the wrong items, and both the accuracy floor and the exception ceiling become unreachable at the same time. Calibration is therefore a delivery metric, not a research one.

3.6.2 Formula. Expected calibration error = the sum over bins of (instances in the bin ÷ total instances) × the absolute difference between the accuracy of the bin and the mean reported confidence of the bin.

3.6.3 Numerator and denominator. The unit is one field instance for the document suites, one decision for the voice and back-office suites. For each bin: accuracy of the bin = correct instances in the bin ÷ instances in the bin; mean confidence of the bin = the mean of the reported confidences in the bin. The weight of each bin is instances in the bin ÷ total instances carrying a reported confidence. Ten equal-width bins on the interval zero to one are the default; any other binning is stated with the figure, because the bin count changes the number.

3.6.4 Excluded. Instances for which the system reports no confidence, which are counted and reported as unscored for calibration with their share of the total. A system that reports no confidence at all is reported as "no confidence output". It is never reported as an expected calibration error of zero. Instances excluded from accuracy under section 3.3.4 are excluded here too.

3.6.5 Reported with. Two figures always travel with the expected calibration error: the reliability diagram, and the high-confidence check, which is the accuracy of instances whose reported confidence is at or above 0.95. The high-confidence check is what a threshold actually depends on.

3.6.6 Arithmetic example, not a result. One thousand field instances carry a confidence. Three bins are occupied. Bin 0.5 to 0.6: 100 instances, mean confidence 0.55, accuracy 0.50, gap 0.05, weight 0.10, contribution 0.005. Bin 0.8 to 0.9: 300 instances, mean confidence 0.85, accuracy 0.79, gap 0.06, weight 0.30, contribution 0.018. Bin 0.9 to 1.0: 600 instances, mean confidence 0.97, accuracy 0.91, gap 0.06, weight 0.60, contribution 0.036. Expected calibration error = 0.005 + 0.018 + 0.036 = 0.059. The invented numbers in this paragraph exist to show the arithmetic.

3.6.7 Contract sentence. "Expected calibration error means the weighted mean absolute difference between the accuracy and the mean reported confidence of each of ten equal-width confidence bins, measured over field instances carrying a reported confidence. The Provider shall keep the expected calibration error at or below placeholder, and shall ensure that field instances returned with a confidence of 0.95 or above are correct in not less than placeholder percent of cases. Where either measure is missed, the Provider shall recalibrate the confidence thresholds at no charge and restate the resulting exception rate before the change is applied."

<!-- Benchmark charter, part 2. Indexed in ../methodology.md. Sections 3.7 to 3.13. -->

3.7 Cost per document

3.7.1 Formula. Cost per document = total run cost ÷ documents admitted to processing.

3.7.2 Numerator. Every charge incurred to process the documents in the run: input, output and any separately billed reasoning tokens; per-page or per-request charges for document services; the charges for retries and for calls that failed and were repeated; and, for a self-hosted model, the compute rate stated in harness/prices.yaml multiplied by the measured occupancy. Charges are taken from published list prices on a stated date. The price list date is printed with the figure.

3.7.3 Denominator. Documents admitted to processing, as in section 3.4. Cost is reported per document, and also per page and per thousand documents, because document size differs between suites and between partners.

3.7.4 Excluded. Human review labour, which is measured separately in section 3.16 and combined with this figure only in the net cost per item in the Exception Economics report. One-off build and integration cost. The cost of running the harness itself. Negotiated discounts, committed-use pricing and any private rate, all of which are excluded because they are not reproducible by a reader; a partner's own effective price is expected to differ, and section 9.4 says so plainly.

3.7.5 Arithmetic example, not a result. Fifty documents. Each consumes 3,000 input tokens and produces 400 output tokens, giving 150,000 input tokens and 20,000 output tokens for the run. At an illustrative rate of USD 2.00 per million input tokens and USD 8.00 per million output tokens, invented here for the arithmetic and not quoted from any price list, the run costs 0.15 × 2.00 + 0.02 × 8.00 = USD 0.46, and cost per document = 0.46 ÷ 50 = USD 0.0092. The invented numbers in this paragraph exist to show the arithmetic.

3.7.6 Contract sentence. "Cost per document means the total provider and compute charges incurred in processing Documents in the measurement window, at the list prices in force on the date stated in the Monthly Report, divided by the number of Documents admitted to processing. The Provider shall keep the cost per document at or below placeholder in the currency stated in clause [X], and shall notify the Partner within placeholder Business Days of any provider price change that would take the figure above that ceiling, together with the options to restore it."

3.8 Latency

3.8.1 Formula. Latency is a distribution over documents, not a ratio. Report the 50th, 95th and 99th percentiles by the nearest-rank method, with the mean, the maximum and the count.

3.8.2 What is timed. From the moment the document is admitted to processing, which the harness records immediately before the first provider call and which in production is the intake timestamp, to the moment the complete validated output is written, or the moment the document enters the review queue, whichever comes first. Both endpoints are recorded in the audit trail.

3.8.3 Numerator, denominator and population. Latency is a duration, not a ratio: each observation is one document's elapsed time, so there is no numerator to count. The denominator is the count of documents in the population, and it is what fixes the rank a percentile lands on. The population is documents that completed, in the window or in the scored split. The population size is printed with every percentile, because a percentile over a handful of documents is one document.

3.8.4 Excluded. Time waiting in the review queue, which is queue age and belongs to operations, not to the system. Documents that failed to process, whose times are reported separately as failure times. Time spent in rate-limit backoff, which is reported twice, once with backoff included and once without, because backoff is a property of the account, not of the model.

3.8.5 Arithmetic example, not a result. Ten documents complete in 2.1, 2.4, 2.6, 2.9, 3.0, 3.3, 3.8, 4.1, 5.2 and 9.4 seconds. By nearest rank, the 50th percentile is the ceiling of 0.5 × 10 = the 5th value = 3.0 seconds, and the 95th percentile is the ceiling of 0.95 × 10 = the 10th value = 9.4 seconds. The invented numbers in this paragraph exist to show the arithmetic and to show why the population size is printed.

3.8.6 Contract sentence. "Latency means the elapsed time from admission of a Document to processing until the complete validated output is written or the Document enters the human review queue, whichever occurs first, reported as the 50th, 95th and 99th percentiles by the nearest-rank method over Documents completed in the measurement window. The Provider shall keep the 95th percentile at or below placeholder seconds. Time spent in the human review queue is not included, and is reported separately as queue age."

3.9 Containment

3.9.1 Our definition. A contact is contained when all three of the following hold. First, the stated intent was resolved: the outcome recorded matches the ground-truth resolution for the scenario, or in production the transaction the caller asked for was completed. Second, no human was requested and none joined: the caller did not ask for a person at any point, and no person joined the contact, took a callback or completed work on it afterwards. Third, there was no repeat contact from the same person about the same intent within seven calendar days of the first contact ending.

3.9.2 Formula. Containment = contacts meeting all three conditions ÷ contacts admitted to the agent.

3.9.3 Numerator. Contacts meeting all three conditions in 3.9.1. Resolution is judged against the scenario's ground truth by rule checks on the recorded outcome, with a judge model where the outcome is expressed in language rather than in a field; the judge prompt is published and its agreement with human labels is reported (section 5.9). Repeat contact is taken from the hidden caller script in the benchmark, and from the contact record matched on caller identifier and intent class in production.

3.9.4 Denominator. Every contact routed to the agent, including contacts whose intent is outside the agent's configured scope. This is the rule most often varied. Narrowing the denominator to in-scope intents is the single easiest way to raise a containment figure, so the in-scope-only figure is also reported, in its own column, beside this one.

3.9.5 Excluded. Contacts that never reached the agent: abandoned before connect, or a telephony failure before the first agent turn; these are counted and reported separately. Test and monitoring contacts. Duplicate legs of the same contact: a transfer produces one contact, not two.

3.9.6 Not in the numerator, said explicitly. A contact that ended because the caller hung up. A contact where the agent asserted a resolution the ground truth does not support. A contact where the caller asked for a person and the agent declined or deflected, however the contact then ended. Ending a contact is not resolving it.

3.9.7 The four common definitions, scored alongside ours. Every agent is scored under all five and the spread is published.

3.9.8 Arithmetic example, not a result. Three hundred contacts are admitted. 231 ended with no human joining, which is definition B. Of those 231, 24 contained a request for a person that the agent deflected, leaving 207. Of those 207, 22 were followed by a repeat contact about the same intent inside seven days, leaving 185. Of those 185, nine recorded a resolution the ground truth does not support, leaving 176. Containment under 3.9.1 = 176 ÷ 300 = 58.7 percent, against 231 ÷ 300 = 77.0 percent under definition B. The invented numbers in this paragraph exist to show how the three conditions compose, and to show why the spread column exists.

3.9.9 Contract sentence. "Containment means the number of Contacts in which the caller's stated intent was resolved in accordance with the outcome definitions in Schedule [X], no person was requested by the caller and none joined or performed work on the Contact, and no further Contact was received from the same caller regarding the same intent within seven calendar days, divided by the number of Contacts routed to the Agent, including Contacts whose intent falls outside the Agent's configured scope. The Provider shall achieve containment of not less than placeholder percent. A Contact that ended without resolution is not contained, however it ended."

3.10 False containment

3.10.1 Definition. False containment is the share of contacts counted as contained under a named reference definition that are not contained under section 3.9.1. It is always reported against a named reference, because "false" only means anything relative to the claim being tested.

3.10.2 Formula. False containment = contacts contained under the reference definition but not under section 3.9.1 ÷ contacts contained under the reference definition.

3.10.3 Numerator. Contacts that pass the reference definition and fail at least one of the three conditions in 3.9.1, broken down by which condition failed: not resolved, person requested and not provided, repeat contact within seven days. The breakdown is published, not just the total, because the three failures have different costs to the BPO.

3.10.4 Denominator. Contacts counted as contained under the reference definition. A contact that the reference definition does not count as contained cannot be falsely contained, and is outside this metric.

3.10.5 Excluded. Contacts excluded from section 3.9 under 3.9.5. Contacts where the ground truth marks the intended outcome as ambiguous by design, which are reported as a separate ambiguous count.

3.10.6 Arithmetic example, not a result. Using the arithmetic of 3.9.8, the reference definition B counts 231 contacts as contained and our definition counts 176 of them. False containment against definition B = (231 − 176) ÷ 231 = 23.8 percent, made up of 24 refused requests for a person, 22 repeat contacts and nine unsupported resolutions. The invented numbers in this paragraph exist to show the arithmetic.

3.10.7 Contract sentence. "False containment means the number of Contacts counted as contained under the containment definition used in the Provider's or a third party's own reporting, which are not contained under clause [X], divided by the number of Contacts counted as contained under that definition. Where the Provider reports containment under any definition other than clause [X], the Provider shall report the false containment rate against that definition in the same report, with the breakdown by failing condition."

3.11 Escalation accuracy

3.11.1 What is measured. Three numbers, always reported together, because one alone can be gamed by escalating everything or nothing.

3.11.2 Numerator. A correct escalation is one where the ground truth requires escalation, the agent escalated within the turn budget the scenario states from the trigger appearing, and the escalation was not preceded by an out-of-policy commitment. Ground truth requires escalation where the scenario carries any of: an explicit request for a person, a distress cue on the published cue list, an out-of-policy request the agent must not fulfil, an identity that cannot be verified, or a policy trigger named in the scenario's policy pack.

3.11.3 Denominator. For recall, contacts whose ground truth requires escalation. For precision and quality, escalations the agent made. The combined figure, where a report gives one, is the share of all admitted contacts where the escalate or do-not-escalate decision matched the ground truth, and any table carrying it says which of the three it is.

3.11.4 Excluded. Scenarios labelled "either outcome acceptable", which are counted and reported separately and never scored as an error in either direction. Contacts that never reached the agent. Escalations caused by a platform failure rather than by a decision, which are counted as processing failures.

3.11.5 Arithmetic example, not a result. Three hundred contacts, of which 84 require escalation under the ground truth. The agent escalated in 96 contacts. Of those 96, 71 were required and 25 were not. Recall = 71 ÷ 84 = 84.5 percent. Precision = 71 ÷ 96 = 74.0 percent. Of the 71 correct escalations, 63 reached the right queue carrying every required context field, so quality = 63 ÷ 96 = 65.6 percent, or 88.7 percent if measured over correct escalations only, which is why the report states the denominator on the row. The invented numbers in this paragraph exist to show the arithmetic.

3.11.6 Contract sentence. "Escalation recall means the number of Contacts requiring escalation under Schedule [X] in which the Agent escalated within the turn budget stated for that trigger, divided by the number of Contacts requiring escalation. Escalation precision means correct escalations divided by all escalations made. The Provider shall achieve escalation recall of not less than placeholder percent and escalation precision of not less than placeholder percent, and shall ensure that not less than placeholder percent of escalations reach the destination and carry the context fields listed in Schedule [X]. A failure to escalate on an explicit request for a person, or on a distress cue on the published cue list, is a Severity 2 Incident under the Retainer Schedule clause 4 regardless of the measured rate."

3.12 Hallucinated-policy rate

3.12.1 Definition. The share of contacts in which the agent asserted at least one policy, price, entitlement, timeframe or procedure that the policy pack supplied to it for that scenario does not support.

3.12.2 Formula. Hallucinated-policy rate = contacts containing at least one unsupported assertion ÷ contacts containing at least one policy assertion. The rate over all admitted contacts is reported beside it, so that an agent that avoids the subject entirely is visible rather than flattered.

3.12.3 Numerator. Contacts with one or more unsupported assertions. Detection is a rule check first, on every quotable value in the policy pack, prices, fees, periods, thresholds, eligibility conditions, followed by a judge model for assertions expressed in language. Every judged contact enters the adjudication pool, and a human-labelled subset measures judge agreement (section 5.9). Assertions are also classified by consequence: financial or entitlement, regulated disclosure, procedural, and incidental. The financial and regulated counts are published separately and are never averaged into a single comfortable number.

3.12.4 Denominator. Contacts in which the agent made at least one policy assertion, as identified by the same rule and judge pass.

3.12.5 Excluded. Statements the agent hedged and offered to check, which are counted separately as declined to assert. Correct statements of not knowing. Assertions outside the policy domain, such as greetings and small talk. A caller misquoting a policy is not an assertion by the agent.

3.12.6 Arithmetic example, not a result. Three hundred contacts, of which 268 contained at least one policy assertion. Nineteen of those 268 contained at least one unsupported assertion, of which four were in the financial or entitlement class. Hallucinated-policy rate = 19 ÷ 268 = 7.1 percent, with the financial class reported as four contacts. The invented numbers in this paragraph exist to show the arithmetic.

3.12.7 Contract sentence. "Hallucinated-policy rate means the number of Contacts in which the Agent asserted a policy, price, entitlement, timeframe or procedure not supported by the policy pack in force for that Contact, divided by the number of Contacts in which the Agent made a policy assertion. The Provider shall keep the hallucinated-policy rate at or below placeholder percent. An unsupported assertion in the financial, entitlement or regulated-disclosure classes listed in Schedule [X] is a Severity 2 Incident under the Retainer Schedule clause 4 on the first occurrence, whatever the measured rate, and the Provider shall report every such assertion to the Partner within the Severity 2 response time."

3.13 Time to first token

3.13.1 Definition. For a chat agent, the elapsed time from the end of the caller's turn to the first token of the agent's reply reaching the runner. For a voice agent, the elapsed time from the end of caller speech to the first audio sample of the agent's reply. The end of caller speech is taken from the platform's own endpointing timestamp where it reports one, and otherwise from the runner's voice-activity detection; the report states which was used, because the two are not comparable.

3.13.2 Formula. A distribution over agent turns. Report the 50th, 95th and 99th percentiles by nearest rank, with the mean, the maximum and the turn count.

3.13.3 Numerator, denominator and population. As with latency, this is a duration and not a ratio: each observation is one agent turn, and the denominator is the count of agent turns in the population, which fixes the rank. The population is agent turns in contacts that completed. The first turn of a contact, the greeting, is reported separately because it is often pre-recorded or cached and says nothing about the agent's responsiveness.

3.13.4 Excluded and reported separately. Turns in which the agent executed a tool call, which are reported as their own distribution with the tool time named. Where the platform emits a filler phrase while a tool runs, two figures are reported: time to first token, and time to first substantive token, being the first token of the response that answers the caller. A filler phrase is a courtesy, not an answer, and a single figure lets it hide a long wait.

3.13.5 Arithmetic example, not a result. Eight agent turns take 0.32, 0.41, 0.44, 0.52, 0.58, 0.71, 0.95 and 2.30 seconds. By nearest rank the 50th percentile is the 4th value = 0.52 seconds and the 95th percentile is the 8th value = 2.30 seconds. The invented numbers in this paragraph exist to show the arithmetic and to show that a 95th percentile over eight turns is one turn.

3.13.6 Contract sentence. "Time to first token means the elapsed time from the end of the caller's turn, or of caller speech, to the first token or audio sample of the Agent's reply, reported as the 50th and 95th percentiles by the nearest-rank method over Agent turns in the measurement window, excluding the Agent's opening turn, which is reported separately. The Provider shall keep the 95th percentile at or below placeholder seconds, and shall report time to first substantive token separately where the Agent emits a holding phrase before answering."

<!-- Benchmark charter, part 3. Indexed in ../methodology.md. Sections 3.14 to 3.20. -->

3.14 Automation rate

3.14.1 Definition. The share of back-office work items the system carried to a final state with no human action. The unit is a work item, which may span several documents, several lookups and several writes, which is why it is defined separately from the document-level rate in section 3.4.

3.14.2 Formula. Automation rate = items completed with no human action ÷ items admitted to processing.

3.14.3 Numerator. Items that reached a final state, posted, matched, closed or rejected by rule, with no human action of any kind, including no reviewer opening the item. The sampled-audit convention in section 3.4.2 applies unchanged: an audited item counts as automated if the audit changed nothing.

3.14.4 Denominator. Items admitted to processing in the window: items received, less items rejected before processing by a rule named in the statement of work. Both counts are reported.

3.14.5 Excluded. Items still open at the end of the window, counted and reported as in flight. Items abandoned because an upstream system was unavailable, counted and reported as upstream failures. Items the partner withdrew.

3.14.6 The rule that governs this metric. Automation rate is never published, quoted or contracted alone. It is meaningless without section 3.15 beside it, because a system that automates every item and is wrong on a tenth of them can cost more than one that automates half and is right. Every table carrying an automation rate carries the wrong-automation figure on the same row.

3.14.7 Arithmetic example, not a result. Two thousand items received, 60 rejected before processing by the named rule, so 1,940 admitted. Of those, 1,411 reached a final state with no human action and 27 were still open at the end of the window. Automation rate = 1,411 ÷ 1,940 = 72.7 percent, with 27 items reported as in flight. The invented numbers in this paragraph exist to show the arithmetic.

3.14.8 Contract sentence. "Automation rate means the number of Items carried to a final state with no human action, divided by the number of Items admitted to processing in the measurement window. The Provider shall achieve an automation rate of not less than placeholder percent while keeping wrong-automation rework at or below the ceiling in clause [X]. The automation rate is not satisfied in any month in which the rework ceiling is exceeded."

3.15 Wrong-automation cost in rework minutes

3.15.1 Definition. The labour needed to find and put right the items the system completed automatically and wrongly, expressed in minutes per one thousand automated items, and then in money at the reviewer cost the labour model states.

3.15.2 Formula. Rework minutes per 1,000 automated items = (the sum over wrong automated items of detection minutes plus correction minutes plus downstream correction minutes) ÷ automated items × 1,000.

3.15.3 Numerator. Minutes taken from the labour model published with the dataset, which states, per error class, the expected minutes to detect the error by its detection route and the minutes to correct it and to correct anything downstream that consumed it. In production the minutes come from the time recorded against the correction. Where an error class has no detection route inside the window, it is not given a minute figure. It is reported as an open exposure count, with the class named. Inventing a detection time for an error nobody would find is exactly the kind of number this charter exists to prevent.

3.15.4 Denominator. Items completed automatically, that is the numerator of section 3.14, scaled to one thousand.

3.15.5 Excluded. Rework on items that went to human review in the first place, which is reviewer time under section 3.16. Rework caused by a source-data error the system could not have detected, reported separately as an input-quality class. Rework caused by a partner-side change outside the statement of work.

3.15.6 Money. Cost = rework minutes × the fully loaded reviewer cost per minute stated in the dataset labour model, reported in INR and USD, both marked placeholder until a partner supplies the rate. Net cost per item, which is the figure a chief financial officer reads, combines this with section 3.16 and with cost per document from section 3.7, and is reported at three confidence thresholds so that the trade between automation and rework is visible rather than argued.

3.15.7 Arithmetic example, not a result. Of 1,411 automated items, 38 were wrong. Twenty-two belong to a class that takes six minutes to detect and four to correct, giving 220 minutes. Eleven belong to a class that takes fifteen minutes to detect and ten to correct, giving 275 minutes. Five belong to a class with no detection route in the window, so they contribute no minutes and are reported as an open exposure count of five. Total 495 minutes, so rework = 495 ÷ 1,411 × 1,000 = 350.8 minutes per thousand automated items. The invented numbers in this paragraph exist to show the arithmetic.

3.15.8 Contract sentence. "Wrong-automation rework means the sum of detection, correction and downstream correction minutes attributable to Items completed automatically and incorrectly, divided by the number of Items completed automatically and expressed per one thousand such Items, using the labour model in Schedule [X]. The Provider shall keep wrong-automation rework at or below placeholder minutes per one thousand automated Items. Error classes with no detection route within the measurement window shall be reported as an open exposure count with the class named, and shall not be assigned an estimated minute figure."

3.16 Reviewer minutes per exception

3.16.1 Definition. The reviewer time taken to bring an exception to a final state. Reported as the mean and the median, and always broken down by queue entry code, because a low-confidence check and a processing failure are different pieces of work.

3.16.2 Formula. Mean reviewer minutes per exception = total reviewer minutes recorded on exceptions closed in the window ÷ exceptions closed in the window.

3.16.3 Numerator. Reviewer active time from the reviewer opening the item to its final state, summed across every reviewer who touched it, including second review and escalation to a senior reviewer. In production, taken from queue timestamps with an idle cut-off, placeholder seconds, after which a reviewer's session is closed and further time is not counted. In the benchmark, taken from the labour model published with the dataset.

3.16.4 Denominator. Exceptions closed in the window, not exceptions opened. Counting the ones that opened would let the slowest items sit outside the figure forever.

3.16.5 Excluded. Waiting time in the queue, which is queue age. Training and calibration time. Time on items rejected as out of scope before review, reported separately. Sampled-audit reviews, which are a measurement device and are reported separately as audit minutes.

3.16.6 Whose number this is. Reviewer time is usually the partner's staff cost, not ours, so this is not a metric a supplier can promise on its own. It is contracted as a reporting duty with a redesign trigger, and Clause 15 in contract-clauses.md is written that way.

3.16.7 Arithmetic example, not a result. In the window, 268 exceptions close: 180 under LOWCONF at a mean of 3.2 minutes, 60 under VALFAIL at 5.5 minutes and 28 under FAIL at 1.5 minutes. Total = 576 + 330 + 42 = 948 minutes, so the mean = 948 ÷ 268 = 3.5 minutes per exception, with the code breakdown reported beside it. The invented numbers in this paragraph exist to show the arithmetic.

3.16.8 Contract sentence. "Reviewer minutes per exception means the total reviewer active time recorded against exceptions closed in the measurement window, divided by the number of exceptions closed, reported as a mean and a median and broken down by queue entry code. The Provider shall report this measure in every Monthly Report. Where the mean exceeds placeholder minutes for two consecutive months, the Provider shall, at no charge, analyse the causes and propose a change to thresholds, validation rules, queue design or the model, and shall implement the agreed change within placeholder Business Days."

3.17 90-day drift

3.17.1 Definition. The change in a tracked metric between the acceptance baseline and the same metric measured ninety days later. Two figures are required, and neither is complete without the other.

3.17.2 Formula. Drift = the metric at day 90 minus the metric at acceptance, in percentage points for rates and in percent of baseline for cost and latency. The report always shows both endpoints. A difference published without its endpoints hides which one moved.

3.17.3 Numerator and denominator. Those of the underlying metric, unchanged. Drift is a difference between two measurements of the same metric, not a metric of its own, and the underlying sample size is printed at both endpoints.

3.17.4 In the benchmark. The ninety-day drift is simulated by re-scoring under a shifted input distribution defined in the dataset manifest, with new vendor formats and new item categories at a share the manifest states. It is labelled a simulation everywhere it appears, in the table, the chart and the prose. A simulated drift is evidence about a system's sensitivity, not a record of what happened to a deployment.

3.17.5 Excluded. Change caused by a scope change agreed under change control, which is reported with its change reference and then excluded from the drift figure. The hypercare period stated in the statement of work, because thresholds are still being set in it; the baseline is the acceptance measurement, not the first day of production.

3.17.6 Arithmetic example, not a result. Field-level accuracy at acceptance is 94.1 percent. At day 90 on the frozen set it is 92.6 percent, so frozen-set drift is −1.5 percentage points. On a fresh labelled sample from the last thirty days it is 88.9 percent, so live-distribution drift is −5.2 percentage points, of which −3.7 points is attributable to the change in input rather than to the system. The invented numbers in this paragraph exist to show the arithmetic.

3.17.7 Contract sentence. "Ninety-day drift means the change in a tracked measure between the acceptance measurement and the measurement taken ninety days later, reported both on the frozen labelled set version used at acceptance and on a fresh labelled sample drawn from the preceding thirty days of live input, with both endpoints and both sample sizes stated. The Provider shall report both figures in the Monthly Report. A fall of more than placeholder percentage points on the frozen labelled set, or any fall that takes a tracked measure below its floor on the live sample, is a Severity 2 Incident under the Retainer Schedule clause 4."

3.18 Drift detection lead time

3.18.1 Definition. The elapsed time from the first item affected by a drift to the supplier telling the partner about it in writing. Two clocks are reported, because they answer different questions.

3.18.2 Formula. Lead time = the notice timestamp minus the onset timestamp, reported in hours and in business hours, with the detection clock reported beside it. Numerator and denominator: a single lead time is an elapsed time and has neither. Where a report gives a mean over several drift observations, the numerator is the sum of the lead times and the denominator is the count of drift observations in the window, and both are printed, together with the maximum.

3.18.3 Onset. The timestamp of the first input item belonging to the shifted population, established from the record after the fact. In a Day-60 exercise the injection timestamp is known in advance and held by the person running the exercise, not by the team being measured.

3.18.4 Excluded. Drift the partner announced in advance, such as a scheduled format change, which is a planned change and not a detection. Injections the supplier was told about. Signals raised and withdrawn as false, which are counted separately as a false-alert count, because a monitoring system that alerts constantly also has a lead time near zero and is worth nothing.

3.18.5 Where the drift was not found. If the exercise window closes with no notice, the figure is written "not detected within the window", with the window length stated. It is never extrapolated to a longer window.

3.18.6 Arithmetic example, not a result. An injection begins at 09:00 on a Monday. The internal alert fires at 14:00 on the Thursday, 77 hours later. The written notice reaches the partner at 11:00 on the Friday, 98 hours after onset. Detection lead time = 77 hours, notice lead time = 98 hours, and the 21-hour gap between them is the part a partner never sees in a supplier's own reporting. The invented numbers in this paragraph exist to show the arithmetic.

3.18.7 Contract sentence. "Drift detection lead time means the elapsed time from the first Item belonging to a shifted input population to the Provider's written notice to the Partner naming the affected class and the supporting evidence. The Provider shall give that notice within placeholder business hours of onset, and shall report both the detection and the notice timestamps for every drift observation in the Monthly Report. A drift not notified within placeholder business hours of onset is a Severity 2 Incident under the Retainer Schedule clause 4."

3.19 Incident mean time to restore

3.19.1 Definition. The mean elapsed time to restore service, computed separately for each severity in the Retainer Schedule clause 4. Reported with the count, the median and the maximum, because a mean over three incidents conceals the worst of them.

3.19.2 Formula. Mean time to restore for a severity = the sum of restoration durations for incidents of that severity closed in the window ÷ the count of those incidents. The numerator is that sum of durations, measured as 3.19.3 and 3.19.4 define. The denominator is the count of incidents of that severity closed in the window, and it is printed with the figure, because a mean over two incidents is not a rate.

3.19.3 Start of the clock. The earlier of the supplier's own detection timestamp and the partner's notice timestamp. Not the time the ticket was opened, and not the time an engineer picked it up.

3.19.4 End of the clock. The time the covered system returned to the state where work submitted at intake reaches an output or the review queue within the service level, confirmed by a monitoring check. Not the time the ticket was closed, and not the time the root cause was found.

3.19.5 Excluded. Incidents excluded under the Retainer Schedule clause 10. Incidents raised and withdrawn as not reproducible, counted separately. Time inside an agreed change window. Time waiting on a partner dependency, which stops the clock and is reported as a separate "waiting on partner" total, so that neither party can hide inside the other's delay.

3.19.6 Arithmetic example, not a result. Three Severity 2 incidents close in the month, taking 3.5, 1.2 and 14.0 hours. Mean time to restore = 18.7 ÷ 3 = 6.2 hours, median 3.5 hours, maximum 14.0 hours, count 3. The mean alone would be the least useful of the four. The invented numbers in this paragraph exist to show the arithmetic.

3.19.7 Contract sentence. "Mean time to restore means the sum of restoration durations for Incidents of a given Severity closed in the measurement window, divided by the number of such Incidents, where the clock starts at the earlier of the Provider's detection and the Partner's notice and stops when a monitoring check confirms that work submitted at intake reaches an output or the human review queue within the Service Level. The Provider shall meet the resolution targets in the Retainer Schedule clause 3 and shall report, for each Severity, the count, the mean, the median and the maximum, together with any time excluded as waiting on a Partner dependency."

3.20 Rollback time

3.20.1 Definition. The elapsed time from the decision to roll back to the previous release serving production traffic and passing the smoke set with its model pin, prompt version and threshold set restored. Two further times are reported beside it: decision time, and re-queue completion.

3.20.2 Formula. Rollback time = the smoke-set pass timestamp minus the decision timestamp, in minutes. Numerator and denominator: a single rollback is an elapsed time and has neither. Where a report covers more than one rollback, the numerator is the sum of the rollback times and the denominator is the count of rollbacks in the period, reported separately for rehearsals and for rollbacks performed on production, with the maximum of each.

3.20.3 Why three times. A rollback that restores the service quickly but leaves the outputs of a bad release sitting in the partner's downstream systems is not finished. A rollback that takes four minutes after a two-hour argument is not fast. Publishing one number and not the other three is the usual way this measure is made to look good.

3.20.4 Excluded and reported separately. A rollback that also requires a data restore reports the restore time separately under 02-security/security-policy-set/07-backup-and-recovery.md. Rehearsals on a non-production copy are labelled as rehearsals and are never mixed in a table with rollbacks performed on production.

3.20.5 Arithmetic example, not a result. The criterion is met at 14:02 and the decision is recorded at 14:09, so decision time is 7 minutes. The previous signed tag is serving at 14:21 and the smoke set passes at 14:26, so rollback time is 17 minutes. The re-queue completes at 15:40, 91 minutes after the decision. The invented numbers in this paragraph exist to show the arithmetic.

3.20.6 Contract sentence. "Rollback time means the elapsed time from the recording of a rollback decision to the previous signed release serving production traffic and passing the smoke set with its model pin, prompt version and threshold set restored. The Provider shall complete a rollback within placeholder minutes, shall rehearse a rollback at least once per quarter and record the rehearsal result in the Monthly Report, and shall report decision time and re-queue completion time alongside rollback time for every rollback performed."

<!-- Benchmark charter, part 4. Indexed in ../methodology.md. Sections 3.21 to 7. -->

3.21 Report completeness

3.21.1 Definition. The share of required elements of the monthly report that are present, cover the whole period, and carry the basis or the sample size where the element is a figure.

3.21.2 Formula. Report completeness = elements present and passing ÷ elements required.

3.21.3 Numerator. Elements that are present, cover the whole reporting period, and are supported. An element that is present but carries a figure without its sample size or its basis fails, because an unsupported figure in a partner report is the problem this whole charter exists to address.

3.21.4 Denominator. The ten elements listed in 01-legal/ops-retainer-schedule.md clause 5.2, plus any element the statement of work adds. The denominator is printed with the figure, since it differs per statement of work.

3.21.5 Excluded. Elements that do not apply in the period, such as incidents in a month with none. These pass if the report states "none" explicitly, and fail if they are simply absent. The count of not-applicable elements is reported.

3.21.6 Scored separately. Timeliness, being delivery by the fifth business day of the following month. A complete report delivered late and an incomplete report delivered on time are different failures and are not averaged together.

3.21.7 Arithmetic example, not a result. Ten required elements from clause 5.2 plus two added by the statement of work gives twelve. Eleven are present. One of the eleven gives an accuracy figure with no sample size and therefore fails. Report completeness = 10 ÷ 12 = 83.3 percent, reported with the two failing elements named. The invented numbers in this paragraph exist to show the arithmetic.

3.21.8 Contract sentence. "Report completeness means the number of required Monthly Report elements that are present, cover the whole reporting period and carry the sample size or the basis for each figure, divided by the number of elements required by the Retainer Schedule clause 5.2 and this SOW. The Provider shall achieve report completeness of 100 percent, an element that does not apply in the period being satisfied by an explicit statement to that effect, and shall deliver the Monthly Report by the fifth Business Day of the following month. A figure reported without its sample size or its basis does not satisfy the element it belongs to."

4. Difficulty tiers

4.1 Rules that apply to every tier scheme

4.1.1 A tier is a property of the item, not of the result. It is assigned when the item is generated, from parameters recorded in the dataset manifest, and it never changes because a system found the item hard.

4.1.2 The values in the tables below are specification parameters chosen by this charter. They are design decisions, not measurements, and the generator records the value actually used for every item so that any reader can verify the tier assignment from the manifest rather than trusting the label.

4.1.3 Every tier is objectively checkable. If a criterion cannot be verified from the item or its manifest entry without a human judgement call, it is not a tier criterion; it belongs in the datasheet as a note.

4.1.4 Every results table is broken out by tier, and every headline figure carries the tier mix that produced it. Changing the mix changes the headline, so the mix travels with the number.

4.1.5 Tiers are not weighted into a single difficulty score. A weighted score would let a supplier argue about the weights instead of about the tiers.

4.2 Messy Scan tiers

The degradation tiers used by datasets/messy-scan/. Each item carries every parameter below in its manifest entry.

4.2.1 A tier is assigned only if every criterion in its row holds. An item that meets some criteria of T4 and some of T3 is generated again, not promoted.

4.2.2 The tier mix in the public sample and in the private split is stated in the datasheet, and the two mixes are the same, so that a divergence between them is a signal about the system rather than about the sample.

4.3 Honest Containment tiers

Assigned from the scenario's construction, before any agent sees it.

4.3.1 The trap list, the distress-cue list and the noise levels are published with the dataset. A trap is a scripted event with a required outcome, not an opinion about difficulty.

4.3.2 Tier assignment is independent of the containment definition being scored. The same contact is scored under all five definitions in section 3.9.7 at whatever tier it was generated.

4.4 Exception Economics tiers

4.4.1 The labour model states reviewer minutes and rework minutes per error class per tier, so that the cost of being wrong rises with the tier in the way it does in a real back office.

4.4.2 T5 is the only tier in which the ninety-day drift simulation is applied. It is labelled a simulation in every table it appears in.

4.5 Day-60 tiers

The tier describes the exercise run against a deployment, not an item in a dataset. A Day-60 score is only comparable with another Day-60 score run at the same tier, and the tier is printed beside the score.

4.5.1 Safety rules that hold at every tier, without exception. No exercise ever injects a real data exposure, exfiltration or a simulated breach involving live personal data. Every injection is agreed in writing in advance, has a named owner who can stop it, has a defined stop condition, and has a rollback owner on call for its duration. An exercise that cannot meet these conditions is not run, and the affected rubric lines are scored "not exercised", never assumed.

4.5.2 The self-assessment version a BPO runs against its current supplier is capped at T2, because the higher tiers require the ability to change a production configuration.

5. Neutrality rules

5.1 Prompts are fixed and published in full. Every prompt used by any suite lives in harness/prompts/, is version-controlled, and is printed or hashed into every report that used it. There is no private prompt.

5.2 The same prompt goes to every model. The only permitted differences are the mechanical requirements of an interface: where a system instruction is placed, how an image is encoded, the maximum output tokens, and whether a structured-output mode is used where the interface has one. Every such difference is listed in the report, per model.

5.3 No vendor-specific tuning. No per-model prompt rewriting, no per-model few-shot selection, no per-model temperature search, no retry with a different prompt after a poor answer, and no post-processing that only one system receives. Normalisation is one shared function applied identically to every system's output, and it is published with the harness.

5.4 Three runs per model per suite at identical settings. Reported as the mean with the standard deviation and the minimum and maximum. A single run is never published as a figure. Where three runs cannot be completed, the row says how many ran and why.

5.5 Every result is reproducible from a commit. Every report records the dataset version and content hash, the harness version and commit hash, the prompt set hash, the model version string as the provider reported it, the run date and time, the price list date, and the exact command line. A figure that cannot be reproduced from those is withdrawn, not defended.

5.6 Our own system is scored by the same harness, from the same commit, with the same prompts, on the same data, and appears in the same table. It gets no highlight, no bold, no annotation, no footnote and no position advantage: tables sort on the metric, not on the vendor. Where our number is worse, it stays where the sort puts it.

5.7 No sponsorship, no paid placement, no pre-agreed outcome. If a vendor ever funds the interface cost of running its own system, the report header says so, and it changes nothing else: the vendor sees its rows only through the pre-publication notice in section 8.3, at the same time as every other vendor.

5.8 A system that cannot complete a suite is reported as "not run" or "incomplete" with the reason, whether the reason is a missing key, a rate limit, a refusal, a context limit or a crash. A partial figure is never promoted into a headline table, and an incomplete row is never compared with a complete one.

5.9 Judges. Where a judge model scores an outcome, the judge prompt is published, the judge model version is recorded, and judge agreement with human labels is measured on a labelled adjudication set and reported as Cohen's kappa. Where agreement is below 0.8, the report states the figure and carries the caveat on every table that depends on the judge. Where a model under test is also used as a judge, its own outputs are scored by a different judge, and that is stated on the table.

5.10 The private split is never published and is used only to detect tuning against the public set. Where a system's private-split result diverges from its public-set result by more than placeholder percentage points, the divergence is reported for that system without inference about its cause.

5.11 Human labelling. Labels for adjudication sets are produced against a written labelling guide, by at least two labellers on a sample, with inter-labeller agreement reported. The guide is published with the dataset.

5.12 No result is published from a run that changed mid-flight. If a provider changes a model behind the same name during a run, the run is discarded and repeated, and the discarded run is named in the changelog.

6. Data ethics

6.1 Synthetic or fully anonymised only. Every item in every public dataset is generated. Nothing is scraped, nothing is copied from a real document, and no real invoice, statement, claim form, transcript or recording enters any set.

6.2 No partner data ever enters a public set. A bake-off run on a partner's own material happens under the non-disclosure agreement, the results go to that partner alone, and no figure, sample or derivative from it appears in a public table, a research note, a post or a deck. This holds whether or not the partner would agree to it, because the partner's own client did not agree to it.

6.3 Identifiers are format-valid and checksum-invalid by construction. Aadhaar-style, PAN-style, card, IBAN and similar identifiers are generated to look right and to fail their real check digits, so that nothing in a public set can resolve to a real person or account. The dataset validation script proves this and fails the build if a checksum-valid identifier is found.

6.4 Names, addresses and company names are generated. Where a generated name collides with a real entity, the route is hello@entailmentlabs.com: we remove the item in the next dataset version and record the removal in the changelog, without argument about likelihood.

6.5 Voice data is text-to-speech only. No recordings of real people, no voice cloning of an identifiable person, and no synthetic voice presented as a specific individual. The voices used are listed in the datasheet.

6.6 Licences. Datasets are published under CC BY 4.0. Code is published under MIT. Both licence files sit in the public repository and in each dataset folder, and every dataset folder carries its datasheet.

6.7 Datasheets. Every dataset ships a datasheet following the "Datasheets for Datasets" structure: motivation, composition, generation process, degradation or difficulty parameters, known biases, uses it is not suitable for, and licence.

6.8 Known bias is published, not smoothed. The format mix, language mix and tier mix in a set are choices, and they favour systems tuned to those choices. The mixes are stated in the datasheet so a reader can reweight, and section 9.1 says plainly that reweighting is not the same as testing on their own material.

6.9 No adversarial content is published that would function as an attack outside the benchmark. Prompt-injection scenarios in the Honest Containment set are constructed against the synthetic policy packs in the set and are labelled as such.

7. Versioning

7.1 Datasets, the harness and this charter each carry a semantic version.

7.2 A ground-truth correction is always a major dataset version. It changes scores by construction, so every affected table is re-run before it is published again.

7.3 No table mixes versions. Every results table states the dataset version and hash, the harness version and commit hash, the prompt set hash and the price list date in its header. A row produced under a different version sits in a different table.

7.4 Superseded results stay published, marked superseded, with a link to the version that replaced them and a changelog entry saying what changed. Nothing is quietly deleted or overwritten; that is the whole point of publishing the previous number.

7.5 Raw model responses are retained for placeholder months so that a disputed figure can be re-scored without re-running the model. The private split is retained and never published.

7.6 A charter major version triggers a re-run or an explicit supersession of every table that depended on the changed definition, and contract-clauses.md is re-issued with the same version number so that a signed clause can always be traced to the definition in force when it was signed.

<!-- Benchmark charter, part 5. Indexed in ../methodology.md. Sections 8 to 10. -->

8. Publishing cadence, changelog and disputes

8.1 Cadence. Quarterly. Each release publishes, per suite: the leaderboards, the calibration report, the findings, the reproduce commands, the changelog, the public dataset sample, and the disputes log. Between releases, a correction may be published at any time under section 8.4, and it carries its own changelog entry.

8.2 Changelog. One file per release, listing dataset changes with their version bump, harness changes with their commit range, prompt changes, models and services added or removed with the reason, corrections made under the disputes process, tables marked superseded, and any run discarded under section 5.12.

8.3 Pre-publication notice. Every vendor whose system appears is sent its own rows, the reproduce command and the commit hash, placeholder working days before publication. The purpose is to catch a configuration error before it is published rather than after. Notice is not approval. A vendor cannot delay, edit or veto a publication, and silence is not consent to anything except the schedule.

8.4 Disputes process. Anyone may dispute any figure. The process is public and its outcomes are published whichever way they go.

8.4.1 Nothing leaves the disputes log. A dispute we lose stays visible, with the correction linked to it.

8.4.2 A dispute about our own system's row runs through the same steps and is marked in the log as a self-dispute, so that a reader can count how many there have been and how they went.

8.4.3 A dispute never pauses publication of the rest of a release. The disputed row is marked "under dispute" with a link to the log entry, and the rest stands.

9. Limitations

Written plainly, because a benchmark that hides its limits is doing the thing this charter was built to expose.

9.1 Synthetic data is not your data. The formats, the tier mix, the languages and the error patterns are ours. Yours differ, sometimes by more than the gap between two systems in the table. The per-tier tables exist so you can reweight towards your own mix, and reweighting still is not the same as running the test on your material. That is why the bake-off exists.

9.2 A leaderboard is not a procurement decision. These suites measure a model or a service on a fixed task. A delivered system is that plus integration, thresholds, validation rules, review-queue design, escalation paths, monitoring and the people around it. A weaker model inside a well-designed queue routinely beats a stronger model with none, and no table here will tell you that.

9.3 Judge models have their own error rate. Where an outcome is scored by a judge, the figure carries the judge's error as well as the system's. We publish judge agreement against human labels and we state it when it is low. We do not claim it away.

9.4 Cost figures move. They are public list prices on a stated date, with no negotiated discount, no committed-use pricing and no caching effect. Your effective price will differ, and a provider can change a list price between our release and your reading of it. Re-price before any cost figure enters a business case.

9.5 A benchmark measures the system on the day it ran. Providers change models behind an unchanged name. Every figure carries its model version string and its run date for exactly that reason, and a figure older than one release is a historical record, not a current claim about anything.

9.6 Public prompts can be tuned against. A vendor who optimises against a published prompt will score better here than the same product scores on your prompts. The private split is our detection method. Detection is not prevention, and a clean private split is evidence rather than proof.

9.7 Statistical limits. Three runs bound run-to-run variation; they do not give a confidence interval on the underlying rate. Sample sizes are printed with every figure because some of them are small. A difference smaller than the reported spread is not a difference, and we do not write about it as one.

9.8 What we do not measure at all: security posture, data residency, contractual terms, support quality, roadmap, the financial stability of a vendor, model behaviour outside the scripted scenarios, accessibility, and total cost of ownership including the partner's own staff. Nothing in a leaderboard should be read as a view on any of these.

9.9 Day-60 measures a deployment on the day of the exercise, under conditions agreed in advance. An announced drift injection is an easier test than a real one, and the tier printed beside the score says which was run.

9.10 We are not neutral parties. We sell systems in all four categories. Section 5 is the discipline we accept in return for publishing at all, section 8.4 is the route to hold us to it, and neither removes the interest we have in the outcome.

10. Current status

10.1 No suite has been run. This build had no model API keys, and the environment it was built in cannot reach model interfaces. Every results table in 10-benchmarks/ therefore reads not run, with the reason, until a run happens.

10.2 Status by suite. The datasets, the harness and the rubric were built after this charter was first written; this table was corrected on 2026-09-02 to say so. What has not changed is the run column.

10.3 What a first run needs from a person, none of which an agent supplies: an interface key per provider; a spend cap per provider set before the run; a decision on which model versions are in scope; an account with rate limits sufficient for three runs over the full set; the price list date to be recorded; and, for Day-60, a partner and a deployment willing to be exercised under section 4.5.1.

10.4 Until then, section 3.1.8 governs every file in this folder: a results row with no run is written not run with the reason, and is never estimated, extrapolated, illustrated or filled with a plausible-looking figure, in a table, a chart, a chart's sample data, prose or a code fixture. A chart with no run renders an empty state, not example bars.

10.5 The arithmetic examples in section 3 are arithmetic. They are invented numbers chosen to demonstrate a formula, they are marked as such in every paragraph that contains them, and quoting one as a result, ours or anyone's, is a misuse of this document.