Entity extraction quality evaluation

Current contracts: guide omk-entity-annotation/v11, corpus omk-entity-quality/v13, checker omk-entity-critical-checks/v5. Production prompt-json knowledge-extraction-v29 and strict knowledge-extraction-v30, response Schema v4, run v5 and quality projection v4 have separate identities; none inherits historical v14/v18 model-quality or Gold conclusions. The separately identified bounded retry/budget policy also requires its own fixed measurement. Quality v4 retains v3 grouping by complete policy and initial invocation caps, whose v2→v3 change was BREAKING-COMPARABILITY; one final selected birth admission contributes once per run. Independently versioned human sealing keeps current records in the denominator as execution unknown, with unsupported-schema and legacy dispositions separately visible; it does not supply admission or semantic acceptance.

The separately registered historical v14 real-window baseline owns its original evidence and limits. It is not a current strict paired experiment or independent quality acceptance. Human-confirmed cross-window organization is implemented through the entity-link journal and library without merging source identities; its proposals follow the separate association evaluation protocol. Neither organization decisions nor historical extraction evidence establish automatic cross-Agent identity or reuse benefits.

1. Versions and scope

This guide follows the identity rules and extraction contract, covering object granularity, complete names, configuration knowledge, critical sources and build identities. Exact positions do not establish semantics or artifact identity. Recorded-output replay and authored responses verify representation, admission and review candidates, not new model effectiveness.

Current test/fixtures/entity-extraction-review-v13.json retains v12's 60 public synthetic windows and 199 critical mentions unchanged and adds four development windows for the work-record/concept boundary; the total is 64 windows and 212 critical mentions, with 62 development and two reserved validation windows. The frozen corpus owns complete-name alternatives, identities, knowledge roles and itemwise rationales. It is neither a hidden blind set nor representative real logs, and contains no real message text. Viewed output windows serve only as development regressions, without blind-gain claims. New validation windows must be grouped separately by project/conversation and determined before output, without informing prompt tuning. Real logs need separate minimization, authorization, grouping and review.

Raw byte digests of the prompt, corpus, both language guides, checker and runtime artifacts bind the measurement identity. Updating current documentation also changes guide digests in new review packages; regenerate and freeze those packages. Old outputs, annotations, strict failures and scores remain unchanged and are not rescored under current rules. Different measurement identities cannot be compared directly; their frozen originals remain traceable through the corresponding Git revision and issues/PRs.

Guide v11 and corpus v13 make the Issue/PR work-record boundary explicit under identity rules section 2.4. This is BREAKING-COMPARABILITY with guide v10/corpus v12, alongside prompt-json v19/strict v20; previous annotations, frozen assets, original outputs and reported scores retain their identities. Checker v5 is unchanged: it presents unannotated extra entities for itemwise review rather than adding an automatic semantic verdict or a post-generation blacklist. New cases are development regressions, not fresh blind-quality evidence.

This referential-context correction retains guide v11's annotation/scoring rule identity, the original bytes of the 64-case corpus v13, and checker v5 source. Version notes and diagnostic content in the current bilingual guide pages do change, so their raw bytes and guideDigest change. New prompt-json v23/strict v24 requests require a new measurement identity and a newly frozen review package; old permissions, runIds and receipts cannot be reused. Historical frozen packages retain their original bilingual guide bytes, outputs and scores. An unchanged guide rule version does not mean an unchanged current page digest.

The subsequent v25/v26 correction preserves independently evaluated content objects when their containing files are also mentioned. It retains guide v11 rules, corpus v13 and checker v5; the new content/container synthetic diagnostic has its own identity and denominator. A file-only authored response can pass structural admission while existing required-mention checks still expose the missing content object; the file description does not substitute for that identity. Reverse cases cover file-only topics, background containment and explicit aliases. Real seen-source replication exposed the omission and remains development evidence; neither these regressions nor prompt demonstrations establish stable improvement or independent acceptance. The current guide page bytes and new guideDigest change, requiring a newly frozen measurement package; historical scores and outputs do not change.

The v27/v28 rules and output self-check within the same invocation clarify whole/part identity with BREAKING-COMPARABILITY. When a reference requires a whole independently reviewed in the source, returning its guidance or mentioning the whole in a part's description does not cover that identity. A source discussing only the part or file does not force an additional owner. Record uncertain part eligibility separately from a known whole omission, without widening references after output. The eight demonstrations, guide v11 rules, checker v5 and main-corpus denominators remain unchanged; current guide-page bytes and guideDigest change and require a newly frozen package. Seen diagnostics and authored responses do not establish that the new rule eliminates omissions, improves stability or completes human acceptance.

The v29/v30 source-claim clarification is BREAKING-COMPARABILITY for prompt rules and the output self-check within the same invocation. Evaluate descriptions, qualifiers and mention rationales independently: an exact locator or basis: explicit cannot validate an added role, containment or ownership assertion. Same-review/PR association does not supply those relations. A confirmed unsupported assertion remains an error even when its external truth is unknown; genuine phrase ambiguity remains uncertain and cannot count as passing evidence. Interpret complete spans and whole-object designations in context rather than treating a word as proof of an invented canonical identifier.

The eight demonstrations, registered v19–v28 raw-byte constraints, guide v11 rules, checker v5, 64-case/212-critical-mention denominators, public Schemas, wire, storage and single policy stay unchanged. Current guide-page bytes and guideDigest change and require a new frozen package; historical packages, runs, scores and review evidence retain their original identities. Source-claim diagnostics are development evidence, not proof that v29/v30 output passes, generalizes or completes independent human acceptance. No invocation stage is added.

Checker v5 enforces the two-record gate: goldReady requires completed author review and attested independent human review; either missing record keeps it false, and either request for changes blocks both readiness flags. Preserve historical packages and their reported readiness under their original measurement identities. Historical production strict v16 changed prompt bytes, version and complete system digest with BREAKING-COMPARABILITY, without changing wire/domain Schemas, run storage or pipeline policies. Updating this guide changes new preparation digests, so changed pending plans require a newly prepared frozen bundle and matching receipts before capture; old runIds and experiment permissions cannot authorize changed requests. Old frozen assets, results and receipts stay unchanged, without migration or rescoring. Currently registered raw UTF-8 prompt hashes are frozen through test/measurement-governance/prompt-registry.ts and prompt-registry-freeze.test.ts. This registry is not a complete archive of every historical prompt; strict v16 needs its original complete asset and corresponding revision to recompute its recorded digest.

The four complete revision-2 input/output demonstrations have undergone AI author review and are present in the production prompt. They are not an evaluation split, unseen validation or Gold; replaying their authored answers cannot establish model effectiveness. The separately labeled partial-admission diagnostic reuses the fourth example, showing rejection of a nonmatching-quote sibling while retaining the valid candidate. Its intentionally bad sibling describes host behavior, not desired model output. These offline engineering checks do not complete a real-model comparison or independent quality threshold.

2. Annotation units and allowed expressions

Freeze source messages, roles, omitted records and window limitations before annotating each item:

Annotation	Rule
Critical mention	Names and references needed for retrieval, identity or understanding knowledge, rather than every noun. Keep message index, verbatim quote and zero-based occurrence; count overlapping matches and use UTF-16 half-open spans.
Allowed boundary	alternatives lists equivalent boundaries agreed before outputs, such as a complete code identifier with or without backticks, or a short name and qualified description denoting the same object. An arbitrary broad quotation containing the name is not a hit; broader boundaries are review candidates.
Identity group	Mentions with the same entityKey must merge; different groups must remain separate. Model IDs and labels are not gold. Align source spans before comparing mention pairs.
Granularity and links	Annotate allowed referentKinds, identity status, explicit candidates, component and collection membership/completeness. Alternatives need prior agreement; uncertain sources retain uncertainty.
Knowledge roles	Annotate source anchor, subject, object and interpretation first. Relations may use synonymous wording. A human checks semantics, direction, negation, conditions, time and corrections against the full window; missing keywords are not errors. Declare coverage: required or if_selected before outputs.

A collective reference is one collection mention, not duplicate member mentions at the same span. Instances/versions differ from their component; environmental conditions alone do not force new identities. Distinguish tools from logs, plans from executions, replacement from rename. Matching names or paths are insufficient to merge. Truncated sources do not automatically require empty knowledge: an unresolved subject may retain a source assertion and pending investigation, without filling missing identity or generalizing conclusions. If two identity schemes are reasonable, adjudicate first or explicitly allow alternatives before outputs; this contract does not encode conflicting identity partitions, so unresolved disputes cannot be frozen.

Mention boundaries follow object meaning under identity rules section 2.3, rather than minimizing string length. “当前 main 的被测代码” and “enabled 字段” still denote the same objects and can be listed before new measurement outputs; “main 上的改造” discusses a change property, so the code-state mention should select main inside it. Do not mechanically strip “字段”, “的” or “上的”, or accept any sentence containing a short name. The program still matches predeclared alternatives exactly, without automatic trimming or regex-based semantic judgment. Recompute occurrence when narrowing to a repeated short name: the second main has occurrence 1, regardless of the unique longer phrase.

Judge type evidence separately from identity resolution: names, calls, failures or “node” labels alone do not prove components/deployed instances. A known identity with unknown type stays proposed object with type uncertainty; explicit module roles or deployment instances must not be downgraded. Qualified descriptions and short names may be prefrozen alternatives, never widened after new outputs. Distinguish shared references from their properties: their/它们 refers to a collection; a property phrase cannot identify just one member. Read relation and polarity together without double negation; “not failed” does not mean “passed”, and “not prohibited” does not mean “required”. Per-case semantic criteria preserve these judgments without keyword classifiers.

Review ordinary objects, abstract concepts and named objects case by case, including whether descriptions faithfully explain contextual meanings. Semantic category is not referentKind: a concept and a particular object may both be object without sharing identity. A product named “Plan” is not necessarily a plan. Record omissions, false extractions and unsupported categories/meanings beyond critical mentions; incomplete annotation does not support overall precision/F1. Inspect entity coverage even with no knowledge proposals; empty knowledge alone does not establish reliable associations.

Retained itemwise semantic review checks that central concepts are not replaced by statistics/adaptation objects. Short concept spans may overlap complete compound-object spans, without duplicating the same span; words inside product names do not automatically become concepts. An assistant-selected answer is not user clarification: retain the original ambiguous mention as unresolved with same-level candidates (or empty candidates plus a reason when levels differ), and only reconsider earlier mentions upon explicit clarification. Each statement has one proposition with uniform modality/polarity. Separate positive/negative requirements and their citations; context cannot restrict a single negative polarity to the latter clause, and a collective predicate cannot be mechanically split into member facts. Itemwise review applies these standards; structural admission uses no keyword inference or result completion.

Referential-context review applies existing identity and qualifier rules. Identify objects in teaching examples, hypotheticals or counterexamples within their own context rather than merging them into the actual current object by name or transferring their version, environment or result into its description/qualifiers. Each mention and rationale must support the object to which it is bound. Merge references when the source explicitly confirms one object across a quotation; quotation marks or an example label must not automatically split it. Without confirmed correspondence, retain an unresolved identity and source-supported candidates; an assistant's continuation cannot supply clarification. Exact mention and identity-group checks do not establish qualifier support, which still needs itemwise review against the full window.

Granularity review checks both omissions and excessive splitting: determine whether the source describes properties/behavior of a thing or independently defines, compares, refers to or associates an object before choosing identity. Ordinary properties can be statements about their owner; independent metrics/operation concepts must not be collapsed merely because they involve statistics/processes. Missing proper names or knowledge candidates is not an exclusion rule. Values, units, conditions and unconfirmed outcomes remain in statements. Critical mention matches do not prove this boundary correct. Record extra entities, actual knowledge wording and absent optional knowledge case by case; self-review is not independent evidence. Never alter identity schemes after output.

Review modal negation against the source meaning. These are examples of equivalent expressions, not a keyword classifier:

Source assertion	Allowed expression	Unsupported inference
No need to modify SDK	descriptive/negative, “needs modification”; negates necessity in this scenario.	Modification is prohibited or did not happen.
Should not modify SDK	normative/negative, “modify”.	Modification is unnecessary or the rule was followed.
SDK modification was not prohibited	descriptive/negative, “modification is prohibited”; negates prohibition status.	Explicit authorization, a requirement to modify or actual modification.
Explicitly allowed/prohibited SDK modification	permission/positive or negative, “modify”; retain authority and scope.	Permission in other scenarios or execution results.

Reported requirements retain normative/permission modality with source_assertion evidence; context preserves speaker, conditions, time and scope. Descriptive statements can also retain source assertions, without becoming direct observations. Negated necessity/prohibition status requires that predicate in relation rather than reducing every relation to the action. Read “cannot modify” in context to distinguish incapability from prohibition. A context mentioning “no need” cannot repair a statement already expressing prohibition. Retain unknowns or omit statements when the meaning cannot be expressed unambiguously. Cite modalities and conditions separately; missing optional knowledge is not_observed, never a modality pass.

Branch/tested-state review follows identity rules section 3.1: distinguish branch objects from separately referable version states; missing SHAs do not imply unresolved identity. Freeze allowed levels first. Branch subjects still qualify this tested state/source time. Distinct commits stay separate; dates alone do not split branches. Unknown reproduction identifiers and unreleased status remain explicit.

Configuration review follows identity rules section 2.2: selected knowledge retains exact values, types, owners, scope, conditions, source reasoning, and unknowns affecting judgment/reuse. An existing owner may be the subject with field and value stated; field endpoint differences are only review cues. Description, qualifiers, or accessible quotations do not establish complete statements. Each role declares coverage: required for mandatory facts or if_selected for retention when selected. Missing critical values in related knowledge are information loss; absent optional knowledge is not_observed; absent mandatory facts are coverage gaps. These differ from entity omission. A none policy cannot contain required roles.

Work-record review follows identity rules section 2.4: retain independently discussed Issue/PR concepts, but do not create standalone entities for individual numbered records or invent concepts by stripping identifiers. Useful record numbers, links, statuses and speaker attribution remain in sources or case context. Source-supported extra objects still need to satisfy this scope; a record entity is an output-scope error, not automatically an invented source fact. Do not generalize this boundary to files, deployment instances, versions or all specific objects. Freeze these decisions before new outputs; explicit post-output catalog/knowledge corrections preserve original revisions and do not raise the original run's score.

2.1 Itemwise complete-name audit

1. Read the source, identity group and every boundary for each existing critical mention. Audit the same occurrence rather than substituting another same-named mention elsewhere.
2. Consider type words, articles/demonstratives, owners, location and time qualifications; add only contiguous noun descriptions centered on the same referent. “已发布 r2”, “Its test instance sieve/dev01” and “昨天的更新活动 U1” can be qualified names. Stop before pass/failure/assignment predicates.
3. Check the phrase's actual head: “项目 Pavo 的 a.md” denotes the file and “客户端 L 的 errorStatus” the field, permitting overlap with shorter owner mentions. Neither entire phrase denotes its owner. Plural member lists are not individual names; Session counts are not aliases for Session.
4. Recompute occurrence and UTF-16 bounds independently for each added quote; preserve old alternatives, itemwise reasons and rejected examples. Matching stays exact, without accepting arbitrary containing spans or replacing review with suffix regexes.

The audit covers every existing critical mention, without claiming all possible synonymous/nested descriptions or annotating every noun. A newly discovered post-output gap retains that run's strict miss and stop, with repairs in an explicitly versioned revision. Recorded outputs only check coverage and excluded boundaries offline, never new model-quality evidence.

2.2 Critical-source targeted diagnostic v1

test/fixtures/entity-source-coverage-v1.json is a separate synthetic development case, omk-entity-source-coverage/v1: three messages, two objects, two identity mentions and five critical knowledge sources (purpose, current limit, applicability condition/exception, late-event constraint, and subsequent correction/unverified limit). It is not added to the main corpus and does not change the critical-mention denominator.

Before a model diagnostic, freeze this window, exact critical source positions, required knowledge and allowed expressions, current prompt/Schema, runtime and dependency digests. Authorize the specific model, budget and transfer separately. Authored responses are not model outputs. Empty knowledge remains structurally valid but leaves the five predeclared obligations uncovered. Ordinary repeated names need not all be saved.

Review each item: whether its statement retains the judgment and conditions, constraints, exceptions and corrections; whether its directly located citation supports that statement; whether mention identities/rationales match actual locations; and whether plans, source assertions and unverified limits remain intact. A mention helps users locate text but does not replace a statement's critical citation. Access to the complete message, a broad quote containing keywords, identical endpoints or a citation attached to another statement do not automatically establish semantic coverage. Reusing critical checker v5 anchor/statement lists only provides review candidates; semanticReview remains pending.

The predeclared targeted threshold requires recorded planned calls without call/parse/structural failures, correct identities for both objects and itemwise author agreement on all five knowledge/source obligations. Stop on missing coverage, definite errors or annotation disputes. Missing independent evidence remains explicit. This threshold does not replace section 3's original critical regression threshold or establish overall quality/stability. Only authored-response regressions have completed: zero new model calls and unverified automatic extraction effectiveness.

2.3 Version-artifact targeted diagnostic v1

test/fixtures/entity-version-artifact-v1.json is a separate synthetic development case, omk-entity-version-artifact/v1: two messages, two objects (a frozen archived corpus and a continuing prompt) and one required no-mixing obligation. It is not added to the main corpus and does not change the critical-mention denominator.

The predeclared check: a frozen, archived artifact with a version identifier and digest takes version referent identity, while the continuing prompt stays object with its current version only as a qualifier. Typing the archived corpus as object must surface a referent_kind mismatch in the offline checker. Historical knowledge-extraction-v18 introduced one synthetic demonstration of this distinction (synthetic-version-artifact-v2, asset bundle entity-extraction-five-positive-strict-v4); the demonstration is prompt material, not an evaluation split or evidence of semantic improvement. Model-produced type judgments remain subject to itemwise review; semanticReview stays pending.

The v18 correction retains the source-supported qualifier “当前版本 v14”; v17's demonstration had supplied the unsupported full identifier knowledge-extraction-v14. This changes prompt bytes and measurement identity (BREAKING-COMPARABILITY), with no new model calls or quality conclusion. The original input, first four demonstrations, no-fewshot control and wire identity stay unchanged. The v4 bundle binds the original v3 digests and a fresh AI author review; it registers no new live collector arm. The v3 assets and v17 runs retain their original bytes and facts. The v3/v4 verifiers check their corresponding historical v17/v18 builds and must run after rebuilding those revisions. Strict v20 reused the five demonstration texts unchanged with a new rule prefix. Strict v22 keeps those serialized bytes and adds a sixth temporal demonstration; its full system is not the v4 bundle's frozen v18 candidate and needs its own measurement identity. The v21/v22 temporal correction is BREAKING-COMPARABILITY without changing collection scope or public Schemas; it affects only new requests, leaves old outputs and scores unchanged, and does not guarantee semantic correctness.

2.4 Referential-context targeted diagnostic v1

test/fixtures/entity-context-boundaries-v1.json is a separate synthetic development asset, omk-entity-context-boundaries/v1: three cases, seven messages and eleven predeclared critical mentions. They check separation of isolated example objects, merging an explicitly identical object across a quotation, and retaining ambiguity when correspondence is unconfirmed. It contains no private conversation text and does not enter corpus v13's 64-case or 212-critical-mention denominator.

Authored-response regressions reuse exact localization, mention identity pairs and itemwise semantic review. Wrong merges, wrong splits and unsupported candidate selection must yield locatable check differences. Context transferred through descriptions/qualifiers still needs itemwise identity/unsupported_qualifier decisions, without a new NLP host filter. Correct locations, matching identity pairs or valid structure cannot automatically establish qualifier correctness. Test-authored judgments verify pending/blocked handling, not model output or independent human review.

prompt-json v23/strict v24 add referential-context guidance and a seventh complete wire demonstration. The first six demonstrations retain their serialized bytes; wire, public Schemas, storage, policies and invocation counts remain unchanged. This is BREAKING-COMPARABILITY; old runs and original scores are not recalculated. The motivation was a cross-context identity/version-qualifier defect in seen real development output, used only as a regression cue rather than unseen validation. Before model diagnostics, freeze these cases, allowed identities and qualifiers, current byte digests, model and budget, then retain itemwise review. These engineering regressions establish neither real v24 effectiveness nor overall quality improvement or formal Gold.

3. Layered results and acceptance threshold

Layer	Recorded checks	Limits
Calls and capture	Separate call failures, JSON parse failures, cancellation and unattempted items.	Internal checker exceptions stop the run instead of becoming model failures; no retries hide failures.
Structural admission	Envelope Schema rejection, rejected mentions/entities/proposals and itemized reasons.	Structural rejection of valid JSON is a captured output, never capture_failures. Rejected item counts include cascading consequences, not independent mistakes.
Critical mentions	Exact matches/missing/ambiguous mentions, broad-boundary candidates and critical mention recall.	Annotation is not exhaustive: precision/F1 are null; extra mentions need human review and are not automatically false positives.
Identity	Required same/different mention pairs, wrong merges/splits and granularity/candidate/component/member differences.	Unaligned mentions or missing references are not_evaluable, never passes. Correlated pair counts are not independent-sample accuracy.
Knowledge links	Statements linked to the source anchor, endpoints, relation, modality and polarity; reversed endpoints raise review prompts; no statement is not_observed.	Matching endpoints do not establish correct relations or truth. Other statements sharing an anchor, broad citations and synonyms require human review.
Knowledge coverage obligations	Each role preserves coverage, interpretation, and references to all accepted statements. Without structural rejection or knowledge candidates, required becomes required_not_observed, and if_selected becomes not_observed. Any candidates require needs_review; structural rejection yields not_evaluable.	Citation/endpoint matches do not establish values, types, owners, or reasoning; review alternate evidence/subjects against the full text, without keyword-based passing judgments.
Independent semantics	Per-case semanticChecks cover conditions, time, negation, assertions/observations, plans/completion, case organization and late corrections.	optional permits no knowledge; omission differs from error. Knowledge under none requires an issue. Empty results do not establish reliable knowledge links.

Frozen synthetic critical regression threshold: every planned call has a record; no call/parse failure, cancellation, unattempted item or structural rejection; 100% critical mention matches; all same/different mention pairs and identity links match without not_evaluable; independent human review approves generated knowledge roles, conditions and status with no unresolved disputes. Omitted optional knowledge supports entity-only conclusions and leaves knowledge-link coverage insufficient. This tool never automatically declares the entire threshold passed: semanticReview stays pending, with unverified scope reported. Predeclared required facts need itemwise coverage confirmation; structural rejection does not establish semantic absence from lack of accepted knowledge.

This threshold covers specific boundaries, not overall quality. Two repeats supply at most two samples; passing both does not establish population stability or cross-Agent benefits. Real-log sampling and thresholds are separately predefined, never adjusted after seeing outcomes.

4. Independent review, disputes and freezing

1. The implementer/author checks semantics, source closure, positions, grouping and contracts case by case, recording rationales in omk-entity-author-review/v1. authorReview: completed and exploratoryReady: true support continued engineering and exploratory evaluation without requiring the user to review every case. Self-review supplies no independent human evidence; goldReady stays false. Missing independent review is a reported coverage limit, not a blocker for this work; the independent quality completion threshold is unchanged.
2. A person independent of the authors checks each case's guide, allowed boundaries, identity groups, roles and semantic criteria. Complete review-template.json with approved or changes_requested per case, rationale, reviewer, timestamp and independence declaration. An independent model judge does not equal independent human review.
3. Adjudicate disputes before seeing outputs, recording the question, resolution and independent adjudicator. Unresolved disputes use changes_requested. Corpus/guide/checker changes require a version change and new bundle; editing a receipt cannot approve a different digest.
4. Both author and independent records cover every case, bind the measurement identity and predate output capture. Either supplied record requesting changes makes both readiness flags false; self-review cannot override independent disagreement. The checker rejects author self-attestation, omissions, wrong digests and post-hoc approval. It checks record closure and declarations, not actual reviewer identity or judgment correctness; attested_independent_human is a declaration, not an effectiveness conclusion.
5. The measurement identity combines corpus, bilingual guide, compiled checker and production runtime digests, dependency manifest/lockfile digest, Node version, raw UTF-8 prompt digest and versions. The runtime digest uses omk-entity-measurement-runtime/v2: normalized relative paths and raw bytes of all regular dist assets, excluding only the unrelated Next.js subtree studio/web. Relevant runtime or asset changes still invalidate identity; a web-only rebuild does not. This scope change is BREAKING-COMPARABILITY with earlier bench digests: prepare a new bundle and matching receipts or live plan, while retaining old evidence with its original frozen host; no migration or rescoring occurs. The shared helper is included in checker/collector digests, including entity-link review. The bundle preserves both original guide files and a combined reading copy. The manifest also records observed sourceRevision.commit and sourceRevision.dirty outside measurement identity; both are null when Git provenance cannot be established. A dirty checkout's commit does not reproduce its uncommitted source or prove which source produced an existing build. Runtime promptHash uses canonical JSON hashing rather than the raw UTF-8 digest here; do not interchange them.

5. Offline review tool and evidence

Update compiled artifacts, then prepare an external bundle:

yarn build:runtime
node dist-scripts/bench/entity-extraction-review.js \
  --corpus test/fixtures/entity-extraction-review-v13.json \
  --output /private/tmp/omk-entity-review-unique

The output must be absolute, outside the repository, with an existing parent and nonexistent destination; symlinks into the repository are rejected. It preserves raw corpus, bilingual guide, review.html with positions and annotations, review template, model inputs/digests and measurement manifest. Without an independent receipt, goldReady: false even after completed author review. The default template requests changes and has no signature; it cannot approve anything without completion.

Optional --self-review /absolute/self-review.json validates an author record (measurement identity, author, timestamp, per-case decisions and rationales) and saves self-review.json. It has no independence attestation and cannot pass as human evidence via --review. Optional --review /absolute/review.json validates an independent receipt; optional --captures /absolute/captures.json checks existing captures only. No executor entry exists: this command makes zero model calls. Preparing a bundle or signing annotations does not authorize data transmission or model execution.

Capture format omk-entity-captures/v1 contains the same measurementId, one split, one or two planned repeats, start time, executor/model and records with caseId, repeat index, frozen input digest and result. Results are output (verbatim text and optional original usage), call_failure or cancelled, with optional executor-reported durationMs and costUSD; omit unknown values instead of inserting zero. Unknown, duplicate, excess and input-mismatched records are rejected. Raw captures and admission results remain intact. Unattempted records are itemized; unchecked critical mentions stay outside the matching denominator and have their own count. Aggregate only reported usage, duration and cost. Total cost is unknown if any recorded attempt lacks reported cost, while the reported subtotal remains visible; partial cost coverage is not the whole-run cost.

Structural rejections do not fail capture status; call/parse failures, cancellation or incomplete plans cause a nonzero exit. Successful exit only means offline inspection completed. Raw outputs, itemized admission, original usage and every failure remain available; semantic review stays pending. Captures approved only by the author are exploratory_semantic_review_pending, never an independent acceptance pass. Subsequent model evaluation runs only within concrete authorization and obtains itemized independent output review.