Honest Containment

A contact is contained when all three of the following hold. First, the stated intent was resolved: the outcome recorded matches the ground-truth resolution for the scenario, or in production the transaction the caller asked for was completed. Second, no human was requested and none joined: the caller did not ask for a person at any point, and no person joined the contact, took a callback or completed work on it afterwards. Third, there was no repeat contact from the same person about the same intent within seven calendar days of the first contact ending.

Containment = contacts meeting all three conditions ÷ contacts admitted to the agent.

Contacts meeting all three conditions in 3.9.1. Resolution is judged against the scenario's ground truth by rule checks on the recorded outcome, with a judge model where the outcome is expressed in language rather than in a field; the judge prompt is published and its agreement with human labels is reported (section 5.9). Repeat contact is taken from the hidden caller script in the benchmark, and from the contact record matched on caller identifier and intent class in production.

Every contact routed to the agent, including contacts whose intent is outside the agent's configured scope. This is the rule most often varied. Narrowing the denominator to in-scope intents is the single easiest way to raise a containment figure, so the in-scope-only figure is also reported, in its own column, beside this one.

Contacts that never reached the agent: abandoned before connect, or a telephony failure before the first agent turn; these are counted and reported separately. Test and monitoring contacts. Duplicate legs of the same contact: a transfer produces one contact, not two.

A contact that ended because the caller hung up. A contact where the agent asserted a resolution the ground truth does not support. A contact where the caller asked for a person and the agent declined or deflected, however the contact then ended. Ending a contact is not resolving it.

Every agent is scored under all five and the spread is published.

"Containment means the number of Contacts in which the caller's stated intent was resolved in accordance with the outcome definitions in Schedule [X], no person was requested by the caller and none joined or performed work on the Contact, and no further Contact was received from the same caller regarding the same intent within seven calendar days, divided by the number of Contacts routed to the Agent, including Contacts whose intent falls outside the Agent's configured scope. The Provider shall achieve containment of not less than placeholder percent. A Contact that ended without resolution is not contained, however it ended."

False containment is the share of contacts counted as contained under a named reference definition that are not contained under section 3.9.1. It is always reported against a named reference, because "false" only means anything relative to the claim being tested.

False containment = contacts contained under the reference definition but not under section 3.9.1 ÷ contacts contained under the reference definition.

Contacts that pass the reference definition and fail at least one of the three conditions in 3.9.1, broken down by which condition failed: not resolved, person requested and not provided, repeat contact within seven days. The breakdown is published, not just the total, because the three failures have different costs to the BPO.

Contacts counted as contained under the reference definition. A contact that the reference definition does not count as contained cannot be falsely contained, and is outside this metric.

Contacts excluded from section 3.9 under 3.9.5. Contacts where the ground truth marks the intended outcome as ambiguous by design, which are reported as a separate ambiguous count.

"False containment means the number of Contacts counted as contained under the containment definition used in the Provider's or a third party's own reporting, which are not contained under clause [X], divided by the number of Contacts counted as contained under that definition. Where the Provider reports containment under any definition other than clause [X], the Provider shall report the false containment rate against that definition in the same report, with the breakdown by failing condition."

Three numbers, always reported together, because one alone can be gamed by escalating everything or nothing.

A correct escalation is one where the ground truth requires escalation, the agent escalated within the turn budget the scenario states from the trigger appearing, and the escalation was not preceded by an out-of-policy commitment. Ground truth requires escalation where the scenario carries any of: an explicit request for a person, a distress cue on the published cue list, an out-of-policy request the agent must not fulfil, an identity that cannot be verified, or a policy trigger named in the scenario's policy pack.

For recall, contacts whose ground truth requires escalation. For precision and quality, escalations the agent made. The combined figure, where a report gives one, is the share of all admitted contacts where the escalate or do-not-escalate decision matched the ground truth, and any table carrying it says which of the three it is.

Scenarios labelled "either outcome acceptable", which are counted and reported separately and never scored as an error in either direction. Contacts that never reached the agent. Escalations caused by a platform failure rather than by a decision, which are counted as processing failures.

"Escalation recall means the number of Contacts requiring escalation under Schedule [X] in which the Agent escalated within the turn budget stated for that trigger, divided by the number of Contacts requiring escalation. Escalation precision means correct escalations divided by all escalations made. The Provider shall achieve escalation recall of not less than placeholder percent and escalation precision of not less than placeholder percent, and shall ensure that not less than placeholder percent of escalations reach the destination and carry the context fields listed in Schedule [X]. A failure to escalate on an explicit request for a person, or on a distress cue on the published cue list, is a Severity 2 Incident under the Retainer Schedule clause 4 regardless of the measured rate."

The share of contacts in which the agent asserted at least one policy, price, entitlement, timeframe or procedure that the policy pack supplied to it for that scenario does not support.

Hallucinated-policy rate = contacts containing at least one unsupported assertion ÷ contacts containing at least one policy assertion. The rate over all admitted contacts is reported beside it, so that an agent that avoids the subject entirely is visible rather than flattered.

Contacts with one or more unsupported assertions. Detection is a rule check first, on every quotable value in the policy pack, prices, fees, periods, thresholds, eligibility conditions, followed by a judge model for assertions expressed in language. Every judged contact enters the adjudication pool, and a human-labelled subset measures judge agreement (section 5.9). Assertions are also classified by consequence: financial or entitlement, regulated disclosure, procedural, and incidental. The financial and regulated counts are published separately and are never averaged into a single comfortable number.

Contacts in which the agent made at least one policy assertion, as identified by the same rule and judge pass.

Statements the agent hedged and offered to check, which are counted separately as declined to assert. Correct statements of not knowing. Assertions outside the policy domain, such as greetings and small talk. A caller misquoting a policy is not an assertion by the agent.

"Hallucinated-policy rate means the number of Contacts in which the Agent asserted a policy, price, entitlement, timeframe or procedure not supported by the policy pack in force for that Contact, divided by the number of Contacts in which the Agent made a policy assertion. The Provider shall keep the hallucinated-policy rate at or below placeholder percent. An unsupported assertion in the financial, entitlement or regulated-disclosure classes listed in Schedule [X] is a Severity 2 Incident under the Retainer Schedule clause 4 on the first occurrence, whatever the measured rate, and the Provider shall report every such assertion to the Partner within the Severity 2 response time."

For a chat agent, the elapsed time from the end of the caller's turn to the first token of the agent's reply reaching the runner. For a voice agent, the elapsed time from the end of caller speech to the first audio sample of the agent's reply. The end of caller speech is taken from the platform's own endpointing timestamp where it reports one, and otherwise from the runner's voice-activity detection; the report states which was used, because the two are not comparable.

A distribution over agent turns. Report the 50th, 95th and 99th percentiles by nearest rank, with the mean, the maximum and the turn count.

As with latency, this is a duration and not a ratio: each observation is one agent turn, and the denominator is the count of agent turns in the population, which fixes the rank. The population is agent turns in contacts that completed. The first turn of a contact, the greeting, is reported separately because it is often pre-recorded or cached and says nothing about the agent's responsiveness.

Turns in which the agent executed a tool call, which are reported as their own distribution with the tool time named. Where the platform emits a filler phrase while a tool runs, two figures are reported: time to first token, and time to first substantive token, being the first token of the response that answers the caller. A filler phrase is a courtesy, not an answer, and a single figure lets it hide a long wait.

"Time to first token means the elapsed time from the end of the caller's turn, or of caller speech, to the first token or audio sample of the Agent's reply, reported as the 50th and 95th percentiles by the nearest-rank method over Agent turns in the measurement window, excluding the Agent's opening turn, which is reported separately. The Provider shall keep the 95th percentile at or below placeholder seconds, and shall report time to first substantive token separately where the Agent emits a holding phrase before answering."

<!-- Benchmark charter, part 3. Indexed in ../methodology.md. Sections 3.14 to 3.20. -->