Messy Scan

Field-level accuracy = correct field instances ÷ field instances assessed.

Field instances whose returned value matches the labelled value under the match rule for that field type. Match rules, which are the rules already used in 06-delivery/build-standards.md section 6.1: exact for identifiers and codes; normalised for dates (compared as ISO 8601 calendar dates), amounts (compared as a numeric value plus a currency code, with separators, symbols and whitespace removed), and names (case, punctuation, diacritics, honorifics and multiple spaces normalised); free-text fields scored only where a tolerance rule is stated, and reported separately from the exact and normalised fields. A field the ground truth marks absent is correct only if the system returns an explicit null or empty value for it.

Every field instance in the ground truth for the documents scored, plus every field instance the system returned that has no counterpart in the ground truth. A returned field that should not exist counts as one assessed instance and is always incorrect. This is the point of the rule: a system cannot raise its accuracy by inventing fields, and it cannot raise it by returning fewer.

Fields outside the published schema for that document type. Field instances whose ground truth is marked unreadable by the labeller, which are counted and reported as a separate line rather than scored. Documents that failed to process at all, which are excluded from this metric and counted under the processing-failure line, never dropped from the manifest.

Overall, per field, per document type, per tier and per language, with three-run mean and spread.

"Field-level accuracy means the number of field instances whose returned value matches the labelled value under the match rules in Schedule [X], divided by the number of field instances assessed, where field instances assessed comprise every field instance in the labelled set for the documents scored plus every field returned by the System for which the labelled set holds no counterpart. The Provider shall achieve field-level accuracy of not less than placeholder percent, measured on the labelled set named in clause [X] at each measurement point."

Straight-through-processing rate = documents released with no human touch ÷ documents admitted to processing. Straight-through means zero human touch, not "little human touch", and not "approved quickly".

Documents whose output was released to the downstream system without any human action of any kind: no review-queue entry, no field correction, no approval action, no reviewer opening the item. One exception, stated so that measurement does not corrupt the measure: a document drawn into the sampled-audit queue (AUDIT in 06-delivery/build-standards.md section 7.1) still counts as straight-through if the audit reviewer changed nothing, because the audit is a measurement device applied after release. If the audit review changed the output, the document is not straight-through and is counted as an exception under section 3.5.

Documents admitted to processing in the window: documents received, less documents rejected before processing by a pre-processing rule named in the statement of work. Both counts are reported, so the effect of the admission rule on the rate is visible.

Documents rejected before processing by a named rule, which are counted and reported separately. Documents still in flight at the end of the window, which are counted and reported as in flight; the window is chosen so that the in-flight share is small enough to state.

Overall and per tier, per document type and per language, always beside field-level accuracy on the same row. Straight-through rate on its own is not a result: a system that releases everything untouched scores 100 percent and may be wrong on most of it.

"Straight-through-processing rate means the number of Documents released to the Partner's downstream system with no human action of any kind, divided by the number of Documents admitted to processing in the measurement window, where a Document drawn into the sampled audit is treated as straight-through only if the audit did not change its output. The Provider shall achieve a straight-through-processing rate of not less than placeholder percent while maintaining field-level accuracy at or above the floor in clause [X], and neither measure is satisfied without the other."

Exception rate = documents that entered human review ÷ documents admitted to processing.

Documents with at least one review-queue entry under the codes LOWCONF, VALFAIL, FLAG, FAIL or DRIFT, plus documents drawn into the sampled audit whose output the reviewer changed. A document counts once however many fields triggered it and however many times it re-entered the queue; the count of re-entries is reported separately.

Documents admitted to processing, the same denominator as section 3.4. The harness asserts the identity: straight-through rate plus exception rate equals one. A supplier's figures that do not satisfy that identity are using two different denominators, and the report says so.

Sampled-audit draws that confirmed the output unchanged, which are reported separately as the audit rate. Documents rejected before processing by a named rule. Queue entries created by the partner for training or calibration, which are counted separately.

Overall, by entry code, per tier and per confidence threshold, because the exception rate is a function of the threshold and moves the moment the threshold does.

"Exception rate means the number of Documents that entered the human review queue under any entry code other than a sampled audit that resulted in no change, divided by the number of Documents admitted to processing in the measurement window. The Provider shall keep the exception rate at or below placeholder percent, measured over a calendar month at the confidence thresholds recorded in the configuration in force during that month, and shall state any threshold change and its date in the Monthly Report."

A review queue is built on a confidence threshold. If the confidence is not calibrated, the threshold routes the wrong items, and both the accuracy floor and the exception ceiling become unreachable at the same time. Calibration is therefore a delivery metric, not a research one.

Expected calibration error = the sum over bins of (instances in the bin ÷ total instances) × the absolute difference between the accuracy of the bin and the mean reported confidence of the bin.

The unit is one field instance for the document suites, one decision for the voice and back-office suites. For each bin: accuracy of the bin = correct instances in the bin ÷ instances in the bin; mean confidence of the bin = the mean of the reported confidences in the bin. The weight of each bin is instances in the bin ÷ total instances carrying a reported confidence. Ten equal-width bins on the interval zero to one are the default; any other binning is stated with the figure, because the bin count changes the number.

Instances for which the system reports no confidence, which are counted and reported as unscored for calibration with their share of the total. A system that reports no confidence at all is reported as "no confidence output". It is never reported as an expected calibration error of zero. Instances excluded from accuracy under section 3.3.4 are excluded here too.

Two figures always travel with the expected calibration error: the reliability diagram, and the high-confidence check, which is the accuracy of instances whose reported confidence is at or above 0.95. The high-confidence check is what a threshold actually depends on.

"Expected calibration error means the weighted mean absolute difference between the accuracy and the mean reported confidence of each of ten equal-width confidence bins, measured over field instances carrying a reported confidence. The Provider shall keep the expected calibration error at or below placeholder, and shall ensure that field instances returned with a confidence of 0.95 or above are correct in not less than placeholder percent of cases. Where either measure is missed, the Provider shall recalibrate the confidence thresholds at no charge and restate the resulting exception rate before the change is applied."

<!-- Benchmark charter, part 2. Indexed in ../methodology.md. Sections 3.7 to 3.13. -->

Cost per document = total run cost ÷ documents admitted to processing.

Every charge incurred to process the documents in the run: input, output and any separately billed reasoning tokens; per-page or per-request charges for document services; the charges for retries and for calls that failed and were repeated; and, for a self-hosted model, the compute rate stated in harness/prices.yaml multiplied by the measured occupancy. Charges are taken from published list prices on a stated date. The price list date is printed with the figure.

Documents admitted to processing, as in section 3.4. Cost is reported per document, and also per page and per thousand documents, because document size differs between suites and between partners.

Human review labour, which is measured separately in section 3.16 and combined with this figure only in the net cost per item in the Exception Economics report. One-off build and integration cost. The cost of running the harness itself. Negotiated discounts, committed-use pricing and any private rate, all of which are excluded because they are not reproducible by a reader; a partner's own effective price is expected to differ, and section 9.4 says so plainly.

"Cost per document means the total provider and compute charges incurred in processing Documents in the measurement window, at the list prices in force on the date stated in the Monthly Report, divided by the number of Documents admitted to processing. The Provider shall keep the cost per document at or below placeholder in the currency stated in clause [X], and shall notify the Partner within placeholder Business Days of any provider price change that would take the figure above that ceiling, together with the options to restore it."

Latency is a distribution over documents, not a ratio. Report the 50th, 95th and 99th percentiles by the nearest-rank method, with the mean, the maximum and the count.

From the moment the document is admitted to processing, which the harness records immediately before the first provider call and which in production is the intake timestamp, to the moment the complete validated output is written, or the moment the document enters the review queue, whichever comes first. Both endpoints are recorded in the audit trail.

Latency is a duration, not a ratio: each observation is one document's elapsed time, so there is no numerator to count. The denominator is the count of documents in the population, and it is what fixes the rank a percentile lands on. The population is documents that completed, in the window or in the scored split. The population size is printed with every percentile, because a percentile over a handful of documents is one document.

Time waiting in the review queue, which is queue age and belongs to operations, not to the system. Documents that failed to process, whose times are reported separately as failure times. Time spent in rate-limit backoff, which is reported twice, once with backoff included and once without, because backoff is a property of the account, not of the model.

"Latency means the elapsed time from admission of a Document to processing until the complete validated output is written or the Document enters the human review queue, whichever occurs first, reported as the 50th, 95th and 99th percentiles by the nearest-rank method over Documents completed in the measurement window. The Provider shall keep the 95th percentile at or below placeholder seconds. Time spent in the human review queue is not included, and is reported separately as queue age."