How it works

OMK keeps one source of truth for evaluation: a host compiles local inputs into a host-neutral measurement contract, Evaluation Core seals and executes that contract, and every CLI or Studio view is projected from validated Core artifacts.

Single-run evaluation flow

After the host resolves inputs and assembles runtime capabilities, Core processes a single evaluation through these stages. Artifacts show the flow of evidence; the host owns file persistence and presentation.

flowchart TD
    I["Evaluation definition<br/>Cases, targets, evaluator declarations and configuration<br/>Comparisons, analysis and decision rules"]
    P["Runtime policy<br/>Concurrency, retries, budgets, caching and evidence requirements"]
    R["Host-provided capabilities<br/>Executor implementations and evaluator runtime implementations<br/>Analysis algorithms and storage ports"]

    subgraph CORE["eval-core"]
        PRE["1. Prepare<br/>Validate definitions and capabilities; bind declarations to implementations"]
        PLAN["SealedRunPlan<br/>Seal the plan, identities and digests"]
        EX["2. Execution<br/>Run targets and cases"]
        EB["ExecutionBundle<br/>Actual outputs, usage and execution evidence"]
        EV["3. Evaluation<br/>Invoke bound evaluator implementations"]
        VB["EvaluationBundle<br/>Metrics, scoring evidence and missing states"]
        AN["4. Analysis<br/>Follow the analysis graph for differences, uncertainty and coverage"]
        AB["AnalysisBundle<br/>Analysis results"]
        Q{"Decision policy configured?"}
        DE["5. Decision<br/>Apply analysis and evidence conditions to produce a conclusion and reasons"]
        DR["DecisionResult"]
        REP["6. EvaluationReport<br/>Link stage evidence<br/>Separate run, evidence and conclusion states"]

        PRE --> PLAN --> EX --> EB --> EV --> VB --> AN --> AB --> Q
        Q -->|Yes| DE --> DR --> REP
        Q -->|No| REP
        EB -.-> AN
        EB -.-> REP
        VB -.-> REP
        AB -.-> REP
    end

    I --> PRE
    P --> PRE
    R -. Capability binding .-> PRE
    REP --> OUT["Host persistence and presentation<br/>CLI, Studio or your application"]

• Declaration and implementation: The definition declares the evaluator, configuration and output metrics. The host supplies the scoring code; preparation binds and validates the two. This does not score an output twice.
• Execution and evaluation: Executors perform the target task. Evaluators inspect actual outputs using, for example, deterministic assertions or an LLM judge.
• Analysis and decision: Analysis produces results such as differences and intervals. An optional decision policy then applies evidence conditions to reach a conclusion. Without a decision policy, execution, evaluation, analysis and reporting remain available.

The diagram shows the normal stage order and omits failure and interruption branches. Failure, cancellation or budget exhaustion can stop work early; later stages follow the available evidence and policy. A run need not complete every stage or produce a verdict. Its result can include artifacts from completed stages, and a failed run may have no report. Run completion, evidence completeness and conclusion sufficiency are separate states.

Each stage retains its own artifacts. Reusing existing artifacts through staged interfaces requires validation of identities, digests and upstream links before rescoring, reanalysis or redecision; reuse does not make arbitrary artifacts comparable.

The important boundaries

• The host owns effects. File discovery, Git materialization, credentials, environment access, progress text, report directories, Studio, and browser opening stay outside Core.
• Core owns measurement meaning. Dataset projection, Target behavior, evaluator instruments, metrics, sampling units, comparison families, analysis parameters, missing-evidence policy, budgets, and Decision policy are sealed before the first Target call.
• Runtime identity is evidence. Provider, model, effort, tools, sandbox, protocol, skill isolation, and fingerprint assurance are explicit. An adapter that cannot satisfy a declared capability fails before measurement instead of silently dropping it.
• Gold is isolated. Executors see only execution inputs. Evaluators receive only their declared evaluation projection. Gold remains analysis-only and cannot leak into generation or scoring.
• Events are observational. Slow, absent, or failing progress consumers cannot change authoritative Bundles or the final Report.
• Persistence is immutable. Each run publishes its Run Plan, Execution, Evaluation, and Analysis Bundles, and Evaluation Report as one digest-linked artifact set. Corruption or broken lineage fails explicitly.
• Studio is a projection. UI cards and pages can be rebuilt from Core artifacts and never become a second measurement model.

Source dependency model

knowledge/ owns knowledge construction and maintenance as an umbrella domain: pure content contracts, entity and library rules, extraction, cross-window linking, application flows, adapters, and carrier lifecycle. CLI and Studio consume these responsibilities directly. The pure core is limited to root TypeScript modules, entities/, and entity-library/; source reads, model calls, and persistence belong to execution subdomains and adapters within the umbrella. Reference checks establish structural and source-location integrity, not support for a claim. Candidate extraction and review, immutable revisions, entity-library organization, and carrier maintenance are implemented. The extraction and library specifications own their scope and limits; a concept diagram establishes neither automatic linking nor verified benefits.

Directories under src express domain ownership rather than one mechanical repository-wide layering scheme. Three dependency kinds are reviewed separately:

• Runtime implementation edges must remain acyclic. A domain implementation may depend on facts it consumes or lower-level capabilities, but it may not create a reverse dependency through a facade, dynamic import, or utility module. The graph retains value imports into contracts; an audited cycle is registered by its complete domain and intra-cycle edge topology, so any new return path invalidates the registration. Non-literal dynamic imports across TypeScript and executable JavaScript sources are likewise fail-closed unless their importer, expression, and canonical source digest are explicitly registered.
• Contract edges share stable data shapes across domains. A bidirectional domain relationship is valid only when its return edge has been audited and registered; architecture tests reject new bidirectional relationships and stale registrations.
• Composition edges belong to delivery and host entry points such as cli, dsh-plugin, and eval-workflows/hosts. They may assemble domains and effects, while domain implementations may not import delivery composition.

shared is a cross-domain leaf and depends only on itself. eval-core is the deterministic measurement kernel on Node.js: host-neutral means it depends on no CLI, filesystem, network, environment variable, concrete executor or Studio, while the Node.js runtime itself (including node:crypto for canonical digests) is an accepted dependency and hashing is not injected. eval-runtime is the service-host runtime layer, and every module inside it is reachable from a published entry — a machine-checked invariant (test/architecture/production-reachability.test.ts), not a prose promise. Its canonical façade compiles ordinary evaluate() input into existing Core contracts, and its foundation assembles explicit ports and Core built-ins. It also owns two published surfaces that hosts call directly: the Judge／Evaluator implementations (judges/, evaluators/, custom-evaluator.ts) and the host-verification surface (conformance/: checkExecutor, checkRuntime, run*Conformance, plus debugEvaluator), which is how a host proves its own adapter conforms before shipping it. Product workflows stay in eval-workflows; filesystems, directories, persistence, provider runtimes, and UI remain outside Core and are assembled by hosts.

eval-core/primitives owns shared ordering, provenance trust ordering and JSON Pointer traversal. It is a dependency-free leaf below contracts: contracts, Core stages, Runtime and workflows consume it directly, without maintaining parallel implementations.

eval-core/verify owns conformance decisions: given declared contract shapes plus observed facts, it decides whether reality satisfies a plan. contracts keeps schema, identity and digest computation only, and must not import verify; the layering guard now registers only normalization and digest helpers still living in the contract layer, and that debt may only shrink.

Core stages share run-resource acquisition and disposal, abort forwarding, elapsed-time projection and the default event buffer through eval-core/runtime. Budget outcome projection, trust minima and topological ordering are shared invariants. Analysis schedules one ready node at a time; Series schedules a ready frontier. These explicit ordering policies retain existing event order. Execution and Evaluation keep their own record, cache identity, timeout-result and Bundle contracts.

eval-core ← eval-runtime ← product orchestration
                ↑                   ↑
                └──── hosts ────────┘
                        ↑
                    CLI / DSH

The arrow points from consumer to dependency. eval-runtime may depend on Core and type-only Executor contracts; it must not import eval-workflows, provider implementations, or delivery surfaces. eval-workflows reuses Runtime foundation leaf modules instead of importing the canonical user façade or maintaining a second lifecycle implementation.

Knowledge responsibilities retain separate dependency boundaries within one source owner:

knowledge/
├── contracts.ts  # knowledge content and evidence contracts
├── history.ts    # immutable revision rules
├── store.ts      # storage ports and canonical identity
├── tags.ts       # tag contracts
├── validation.ts # deterministic reference and evidence validation
├── entities/     # pure window identity, link review and invocation contracts
├── entity-library/      # pure cross-window organization contracts and revisions
├── extraction/   # selected windows, model protocols and output admission
├── entity-linking/      # cross-window candidate and review execution
├── application/  # shared use cases and local application assembly
├── adapters/     # model transport, source reads and file persistence
└── artifacts/    # knowledge carrier lifecycle

The pure core depends only on itself and zod, never on execution subdomains or host effects. Application flows consume extraction, linking and adapters; linking also consumes library application behavior and execution adapters. Existing adapter relationships with extraction and linking remain explicit. Runtime and type-only edges are audited separately by architecture guards; grouping these subdomains does not make every dependency point only toward the core. Entities remain here because window identity, knowledge bindings and library organization form one current responsibility; an independent top-level entity lifecycle is not required. This source reorganization changes no public exports, model invocations, prompt bytes, Schema identity or persisted workspace paths, and adds no aggregate entrypoint or compatibility shim.

Extraction study responsibilities

Keep extraction studies within the existing responsibility layers, with study-* module names. A study freezes and compares use of production capture, generation and replay; it does not own another extraction engine. A new aggregate knowledge/study/ directory would mix deterministic review with host I/O unless it recreated the same layer boundaries. The current implementation already shares production generation and run storage, so relocation alone would not remove a dependency or a second implementation.

Responsibility	Current owner	Boundary
Corpus, frozen plan and capacity	extraction/study-corpus.ts, study-plan.ts, study-capacity.ts	Validate explicit inputs and identities; do not read sources or dispatch models.
Semantic review, analysis and panel decisions	extraction/study-review.ts, study-analysis.ts, study-panel.ts	Derive conclusions from frozen evidence and declared receipts; AI review cannot become independent human acceptance.
Execution and inspection use cases	application/study-runner.ts, study-local.ts, study-inspection*.ts	Assemble existing production ports, persist attempts and coordinate inspection workers; do not duplicate generation or rewrite evidence.
Files, host identity and worker input	adapters/study-files.ts, study-host.ts, study-inspection-*.ts	Own filesystem reads, build fingerprints and bounded input preparation; keep these effects outside review calculations.
CLI and Studio delivery	cli/runtime/knowledge-study*.ts, studio/application/knowledge/study-inspection.ts	Expose the shared use cases and project safe results; delivery does not define semantic acceptance.

This is a source ownership decision, not a new public study API. Keep the named modules discoverable through the study specification; do not add a convenience barrel that restores cross-layer aggregation. Revisit a separate subdomain when an actual independent consumer or lifecycle requires a narrower contract. Any later move must preserve prompt bytes, frozen identities and historical evidence, and must audit runtime and type-only edges separately.

Knowledge artifact lifecycle capabilities share one ownership boundary:

knowledge/artifacts/
├── contracts.ts  # artifact identity and experiment roles
├── skills/       # skill frontmatter, hard rules, and workflow definitions
├── doctor/       # static and model-assisted artifact health checks
├── authoring/    # sample generation and controlled skill evolution
└── sources/      # canonical source fingerprints and distributable-tree identity

Governance consumes authenticated evaluation and observation evidence; it does not recompute Core scores or decisions. These capabilities remain separate subdomains so that sharing one lifecycle owner does not turn knowledge/artifacts into a generic utility layer. The enclosing knowledge path keeps carrier meaning distinct from measurement artifacts; the retired top-level src/artifacts path remains absent.

Evaluation inputs, evaluator instruments, and Gold calibration share the workflow boundary, while effectful runtime readiness belongs to executors:

eval-workflows/
├── analysis/           # reusable workflow-owned statistical primitives
├── artifact-store/     # Core artifact persistence, discovery, and overlays
├── assertions/         # authored assertion adaptation and score layers
├── gold/               # human-gold datasets, calibration, and CLI support
├── hosts/              # OMK host adapters and shared assembly, with a separate dependency boundary
├── input-compilation/  # host inputs → host-neutral measurement definition
├── inputs/             # evaluation config, samples, and schemas
├── instruments/        # evaluator configuration and frozen prompt assets
├── projections/        # authenticated downstream views of Core artifacts
├── sample-generation/  # evaluation case generation, constraints and source material
├── resume-disposition/   # persisted-run integrity and resume disposition
├── measurement/        # product scoring, analysis nodes, and evaluator implementations
└── orchestration/      # product orchestration, persistence, and injected Runtime consumption

executors/
├── contracts/          # executor ports, runtime identity, result, and trace facts
├── core/               # shared invocation mechanics: subprocess, http, limits, and usage
├── mock-runtime/       # mock runtime materialization for SDK hooks and CLI config dirs
├── preflight/          # host tool, file, environment, and custom-executor readiness
├── script/             # user-defined script command executor
└── <provider>/         # provider-specific runtime implementations

Product orchestration lives in eval-workflows/orchestration: it consumes an injected Runtime and persists evaluation results. createProductionEvaluationWorkflow describes this responsibility; concrete provider selection and resource assembly belong to the delivery entry or shared host modules. This internal rename does not add a package export or a compatibility forwarding module.

The canonical Runtime entry evaluate.ts re-exports its existing API from internal evaluation/ modules. Input and evaluator capture feed Definition compilation; preparation seals and runs the contract; reuse performs Core admission before rescore, reanalysis or redecision. A single result-state module owns authenticated results and prepared capabilities. Capture and compilation cannot depend on run state, and internal modules cannot import the public façade. These modules add no package subpaths and retain the existing identity and measurement contracts.

Artifact discovery, local and Git source resolution, and content-addressed copies belong to knowledge/artifacts/sources. Doctor, installation and evaluation consume that shared implementation. Batch discovery that requires evaluation samples stays in eval-workflows/inputs/batch-discovery.ts; the source layer must not import Workflow.

eval-workflows/instruments and eval-workflows/gold do not own measurement meaning: they adapt judge execution and Gold calibration into the instruments and analysis contracts owned by Core. Likewise, executors/preflight reports environment readiness facts, while eval-workflows/hosts/composition/preflight.ts decides workflow admission from binding declarations. These subdomains remain separate dependency-graph vertices even though they are physically grouped.

Product orchestration consumes an explicitly injected EvaluationRuntimeProvider. Product compilation supplies Definition, Policy and run metadata through EvaluationExecutionInput; orchestration never constructs a Core engine, provider adapter or resource lease. Concrete assembly belongs to hosts. Runtime owns single-run execution and Series preparation/execution. Workflow may orchestrate concurrent independent Series members while Core alone owns their scheduling, retry, timeout, budget and measurement contracts.

eval-workflows/hosts/
├── composition/       # shared registration, binding, preflight and assembly
├── local-task/          # Trusted local task application, collection and acceptance adapters
├── input-resolution/  # host request and content resolution
├── adapters/          # provider execution protocol bridges
├── evaluators/        # product judge factory wiring
└── resource-leases/   # resource materialization, access and cleanup

Sibling directories represent responsibilities. Node-specific implementations stay within their responsibility, such as resource-leases/node.ts, rather than mixing environment and responsibility categories. Shared types live in types.ts; adapters, evaluator wiring and resources cannot import composition, input-resolution or the aggregate host entrypoint.

eval-workflows owns the complete OMK product evaluation implementation, including hosts. Directory ownership does not permit arbitrary dependencies: other subdomains such as orchestration, input-compilation and measurement cannot import hosts. The dependency graph recognizes it as a separate composition boundary, consumed directly by CLI and DSH.

Host assembly consumes product declarations and injects Runtime capabilities into Workflow. Lower layers cannot import eval-workflows/hosts, including type-only imports. Product measurement implementations remain in eval-workflows/measurement; generic execution and lifecycle bridges remain in Runtime. The obsolete Workflow Runtime directory and forwarding wrappers are removed without a 0.x compatibility path. Correct Core/Runtime contracts take precedence over existing Workflow/CLI behavior. Legitimate missing capabilities belong in the responsible lower layer rather than a product execution bypass.

Source ownership and composition consumers

Source directories express ownership; package.json#exports selects public entrypoints. A directory need not become a public API. This inventory records domain boundaries without forcing all domains into one pipeline.

Owner	Public entrypoints and actual consumers	Ownership and verification boundary
Core / Runtime	eval-core, package root and eval-runtime; service callers and product Workflow	Core execution and measurement contracts, Runtime adoption and generic scoring; verify published packages, contracts, scheduling and cancellation
Workflow	eval-samples, projections; CLI, DSH, Studio and artifact evolution	Product declarations, versioned scoring/analysis, orchestration and evaluation stores; verify compilation, projections and release decisions
Executors	Internal calls from host adapters, judges, doctor, sample, evolve and observation analysis	Shared invocation protocols and mechanics, not an obsolete evaluation path; verify arguments, environment, traces, usage and errors
Knowledge	Internal consumers in CLI, Studio, Workflow and observation	Pure content/entity/library contracts; extraction, linking and persistence; carrier authoring, health checks and source resolution; verify immutable histories, source identity, recovery, isolated copies, installation and doctor
Observability / Diagnosis	MCP, DSH, CLI and Studio; observation storage parses the diagnosis contract	Separate evidence, signals and diagnoses; verify provenance, coverage and stable contracts
Evidence	Product domains and delivery entrypoints	Cross-source storage layout and association; does not replace evaluation artifact validation or decide scores/releases
CLI / DSH / MCP / Studio	omk, dsh-plugin, mcp / omk-mcp, studio	Own entry protocols, context, identity and presentation policy; verify real commands, plugins, services and views
Shared	Internal leaf utilities	No domain decisions; verify atomic files, locks and basic data operations

CLI and DSH evaluation entrypoints call eval-workflows/hosts/application.ts. The application resolves and compiles requests, assembles the Runtime, and saves sidecars. orchestration/evaluation-service.ts owns preparation, single runs, resume and independent Series through injected capabilities. Other Workflow subdomains cannot import hosts.

CLI retains argument conversion, environment classification, executable/credential resolution, progress and Studio launch. Node provider registration lives in hosts/composition/node-runtime.ts. DSH supplies its agent context, executor factory and judge invocation; it does not import CLI modules. Batch uses the same application for each child; the CLI supplies its per-item request conversion. The shared host boundary has these internal consumers:

Module	Actual consumers	Contract, failure and resource boundary
application.ts	CLI and DSH evaluation entrypoints	Product use case with explicit host capabilities and presentation callbacks
composition/node-preflight.ts	Application composition	Explicit compiled input, environment and project root; lazy checks propagate failures through existing admission rules
input-resolution/node-cli-evaluation-resolver.ts	Application	Shared product request resolution; the retained request protocol name does not make it CLI entry policy
input-resolution/node-sample-content-resolver.ts, input-resolution/safe-http-content-resolver.ts	Shared resolver above	File/network content resolution and session cleanup; verify source restrictions, cancellation and content identity
composition/runtime-registry.ts, composition/judge-provider-identity.ts	CLI and DSH	Explicit registration and judge identity; shared registry does not select providers or read credentials on behalf of an entrypoint
composition/assembly.ts, composition/runtime.ts, composition/builtins.ts, composition/preflight.ts	CLI and DSH via Runtime provider	Binding, registration, admission and run-resource wiring; hosts implement cleanup while Runtime controls lifecycle
adapters, evaluators, resource-leases	Composition above; DSH also uses resource interfaces	Provider bridges, evaluator wiring and resources cannot depend back on entry composition or bypass boundaries through aggregate exports

Direct service evaluate() uses Runtime/Core. CLI and DSH call the Workflow application, which owns compilation, host assembly and injected Runtime. Generation, repair and auxiliary analysis may directly use ExecutorFn. Observation facts reach Studio through diagnosis and domain projections. Extract shared mechanics according to those contracts. The user facade's event consumption and product run-lease wrapper do not constitute duplicate scheduling merely because both invoke Core.

Shared hosts remain because CLI and DSH actually reuse binding, registration and resource assembly; entry-specific policy belongs to its entrypoint. Internal tests follow host adapters, Runtime, product measurement and projections. Historical Schema/instrument versions are distinct from npm 0.x compatibility and remain governed by public references and evidence-reading needs. The pure knowledge-content core is implemented in knowledge/contracts.ts, validation.ts, history.ts and store.ts; filesystem adapters and extraction orchestration remain outside that pure core, within the knowledge umbrella.

Sample input admission

Input schema validation belongs to inputs/schemas, executor selection to hosts/input-resolution, shared API input policy to hosts/adapters/shared, and provider mapping to each provider adapter. Validation at authoring, CLI admission, and the independently callable runtime boundary protects different entry points; it is not a reason to bypass runtime validation or move host behavior into Core.

Provider mechanics and measurement adaptation

Ordinary ExecutorFn calls serve generation, judges and auxiliary analysis. Measurement adapters make stricter capability, provenance and failure-evidence commitments. The current implementation separates reusable mechanics from policy; the ordinary interface must not constrain Runtime capabilities.

Path	Shared mechanics	Contract differences retained
Codex CLI	Argument construction in executors/openai/codex/cli-arguments.ts; subprocess handling, event normalization and Trace parsing	Ordinary calls select read-only sandboxing; measured calls explicitly select sandbox, effort, strict configuration and environment inheritance, and bind executable identity
Codex SDK	Provider protocol and Trace	Ordinary calls own a per-call isolated home; measured calls bind SDK identity and Trial resources, whose lifetime cannot be delegated to an ordinary-call wrapper
OpenAI HTTP	Token-usage primitives; the measured HTTP adapter uses an identified transport and protocol infrastructure	Ordinary calls read environment configuration and use call timeouts; measured calls inject transport/credentials, use Core cancellation and output bounds, and retain fail-closed parsing
Custom executor	Subprocess infrastructure	Ordinary command output differs from the versioned measurement request/response exchange; concatenating wrappers does not make these one protocol

Codex argument extraction preserves argument order, prompt bytes and each caller's empty-value handling. Environment selection, timeout, error envelopes and cleanup remain outside the argument utility. Ordinary calls own call-level timeouts; measurement follows Core attempt cancellation without adding a second measurement timeout. Workspaces remain separate per Trial, retries retain the same Trial state, and failure usage and Trace retain their declared semantics.

Any further extraction must verify arguments, environment, output, Trace, usage, cancellation, errors and cleanup together. Reuse existing shared mechanics and retain adaptation where contracts differ; fewer functions are not grounds for merging measurement identities or changing protocols. Relevant regressions live in test/executors, test/eval-workflows/hosts and test/eval-runtime; product scoring and statistics tests live in test/eval-workflows/measurement.

Evidence persistence and cross-source association have one non-decision boundary:

evidence/
├── storage/  # canonical layout, names, bundles, discovery, and integrity checks
└── graph/    # doctor, eval, and observe evidence relationships

Evidence stores, validates, locates, and links facts. It does not score outputs, derive diagnoses, or decide knowledge lifecycle actions. Public package entrypoints live at their owning domains' natural index.ts or semantic file; package.json#exports is the only publication allowlist. Internal production modules import concrete domain files rather than those package barrels.

Diagnosis and Observability have an explicit boundary. Observability produces trace, inbox, and experience facts and may read only stable types or parsers under diagnosis/contracts. The downstream diagnosis/observe-producer.ts consumes those Observability facts to derive Diagnosis. Neither side may access any other private implementation across this boundary.

Observability subdomains

The src/observability root provides three stable entrypoints: experience.ts, server-side application.ts, and browser-safe presentation.ts. Studio consumes domain queries, review operations and display semantics through these entrypoints; local Agent inventory retains agents/index.ts, and type contracts are consumed through contracts/ and view-models/. Private implementations belong to vertical subdomains:

observability/
├── agents/  # local Agent discovery, inventory and log collection
├── analysis/  # coverage, gap signals and observation analysis
├── contracts/  # stable facts and data contracts
├── conversation/  # conversation catalog, windows and debugger projections
├── experience/  # experience facts, report derivation and text signals
├── inbox/  # observation inbox, review and feedback projections
├── prompts/  # frozen review prompts and document loading
├── skill-health/  # skill chain, health checks and advisories
├── soft-standards/  # derived-standard extraction, storage and execution
├── trace/  # source parsing, source-neutral IR and attribution
└── view-models/  # type-only projection contracts

test/architecture/import-boundaries.test.ts allows only these stable entrypoints for Studio, covering .js and extensionless Next imports, re-exports and literal module loads. Private subdomains cannot import the root application/presentation entrypoints. The client runtime closure guard separately prevents Node capabilities from reaching presentation consumers; view-models/ supplies types only. test/architecture/architecture-docs.test.ts reconciles every directory tree listed in this document (knowledge, knowledge/artifacts, eval-workflows, executors, eval-workflows/hosts, evidence, observability) with the source tree, in both languages.

Trace's message-classification.ts decides message origin and protocol semantics. Experience's text-signals.ts decides hard-rule, progress, and delivery signals. Adapters therefore do not depend backwards on downstream experience projections. Old root paths are not retained as re-exports or compatibility shims.

Scoring and release decisions

Assertions and rubric judges remain separate evaluator instruments. Analysis derives assertion layers, judge replicates and ensembles, dimensions, composite values, Bootstrap comparison families, and agreement tables without collapsing their identities.

Missing, invalid, failed, unavailable, and not-started observations are not zero scores. Coverage remains explicit through the graph. omk.release-decision/v7 requires complete evidence, exact Analysis binding, resolved Bootstrap Monte Carlo uncertainty, and a practical-effect confidence lower bound before returning one of PROGRESS, CAUTIOUS, REGRESSION, NOISE, UNDERPOWERED, or SOLO; configured but unmeasurable cross-judge agreement in any applicable rubric dimension gates positive comparisons as CAUTIOUS, and the sample-size guard compares observed comparison units with a preregistered fixed or a priori requirement. A display score or point estimate never replaces the registered Decision.

Cost, usage, duration, operational status, evidence status, conclusion status, and lineage are orthogonal facts rather than extra scoring dimensions. Independent --repeat runs form an Evaluation Series; cross-run stability is not inferred from one run.

See Composite scoring and Statistical rigor.

Observation pipeline: source-neutral Trace IR

omk observe does not disguise Codex, Claude Code, or OpenClaw logs as one another. trace/source.ts discovers and reads files, detects formats, and groups sessions; record parsing for each of the five source formats lives in its own trace/adapters/ directory (claude, codex, markdown, openclaw, qoder). All paths produce the same Trace IR before attribution, segmentation, and measurement:

flowchart LR
    S0["trace/source.ts<br/>discovery · detection · grouping"] --> C["adapters/claude"] --> IR["Trace IR"]
    S0 --> X["adapters/codex"] --> IR
    S0 --> M["adapters/markdown"] --> IR
    S0 --> O["adapters/openclaw"] --> IR
    S0 --> Q["adapters/qoder"] --> IR
    IR --> A["lifecycle correlation and skill attribution"]
    A --> S["segment"]
    S --> R["health · inbox · experience"]

The IR distinguishes message, tool_call, tool_result, usage, lifecycle, agent_activity, observed_effect, and unknown events. observed_effect records an effect the runtime observed (for example, which files a command changed): it is not a model-initiated call, so it never enters tool counts. User messages carry a human, runtime, skill-context, or synthetic origin, so injected instructions, environment context, and tool results cannot inflate human-turn metrics. Tool calls retain provider namespaces. Outcomes use success, failure, cancelled, and unknown; ambiguous outcomes remain unknown.

Identifiers have separate jobs: rootRunId groups a task tree, runId identifies a concrete task, traceId identifies an evidence stream, and each segment gets an independent sample ID. Every load also records an ingestion summary, so malformed, unrecognized, filtered, or partial source data cannot masquerade as complete observation coverage.