eval-runtime API layers

package.json#exports is the supported boundary. The API allowlist locks every value and type below. All entries are ESM-only.

oh-my-knowledge

The recommended ordinary-user entry. It exposes exactly the same canonical Runtime façade as oh-my-knowledge/eval-runtime: evaluate, prepareEvaluation, evaluateSeries, prepareEvaluationSeries, rescore, reanalyze, redecide, assessComparability, saveEvaluationResult, loadEvaluationResult, checkRuntime, checkExecutor, checkContentStore, their stable errors, and their public model types. Core engines, builders, registrations, and adapters are intentionally absent.

oh-my-knowledge/eval-runtime

The canonical API for application developers:

Export	Purpose
evaluate	Run one explicit solo, paired, or independent-group evaluation design, including multi-arm and multi-metric comparisons.
prepareEvaluation	Seal and inspect the exact Definition, Policy, Plan, Runtime resolutions, digest, and work estimate before any Target or Evaluator call.
evaluateSeries	Run a preregistered number of separate member evaluations under one fixed measurement design and summarize one numeric Analysis result across Runs.
prepareEvaluationSeries	Seal every Series member, its membership, Series plan, and total work estimate before the first Target call.
rescore	Reuse one authenticated Execution stage, then run Evaluation, Analysis, Decision, and Report under a newly sealed declaration.
reanalyze	Reuse authenticated Execution and Evaluation stages, then run Analysis, Decision, and Report under a newly sealed declaration.
redecide	Reuse authenticated Execution, Evaluation, and Analysis stages, then run a newly declared Decision and Report.
assessComparability	Assess two authenticated canonical Run results at evaluation, analysis, or decision scope without rerunning either Target.
saveEvaluationResult	Persist one authenticated canonical result through an injected host ContentStore; the versioned envelope is always classified as Gold.
loadEvaluationResult	Resolve and re-admit a stored result against an exact PreparedEvaluation, using an independent host verifier before restoring provenance authority.
EVALUATION_RESULT_MEDIA_TYPE	Versioned media type for the stored result envelope.
EvaluationResultStoreError	Stable, redacted storage, resolution, verification, plan, or content failure.
checkRuntime	Exercise exactly one injected Runtime component through its versioned behavioral probe. Supported kinds are Executor, Custom Evaluator, Judge, execution or evaluation cache, ContentStore／ContentResolver, and WorkspaceProvider.
checkContentStore	Exercise a host ContentStore／ContentResolver pair for descriptor integrity and stability, idempotent writes, and round-trip value, classification, and media type; host exceptions are reduced to stable reason codes.
checkExecutor	Exercise an Executor through success, failure, cancellation, cleanup, and measurement checks.
RUNTIME_CHECK_RESULT_SCHEMA_VERSION	Version tag shared by all serializable checkRuntime result envelopes.
EvaluationConfigurationError	Stable caller-configuration failure with a public code and no rejected payload.
EvaluationEventConsumptionError	Stable, redacted observer／event-stream failure that retains the terminal EvaluationResult when available.

Content storage conformance uses ContentStoreCheckInput, ContentStoreCheckResult, and ContentStoreConformanceCheck.

Unified checks use the discriminated RuntimeCheckInput, RuntimeCheckKind, and RuntimeCheckResult types. Component inputs and results are ExecutorRuntimeCheckInput, ExecutorRuntimeCheckResult, EvaluatorRuntimeCheckInput, EvaluatorRuntimeCheckResult, JudgeRuntimeCheckInput, JudgeRuntimeCheckResult, CacheRuntimeCheckInput, CacheRuntimeCheckResult, ContentStoreRuntimeCheckInput, ContentStoreRuntimeCheckResult, WorkspaceProviderRuntimeCheckInput, and WorkspaceProviderRuntimeCheckResult. Probe details use EvaluatorConformanceProbeSources, EvaluatorConformanceProbeInput, EvaluatorConformanceCheck, EvaluatorConformanceResult, JudgeConformanceProbeCase, JudgeConformanceProbeInput, JudgeConformanceCheck, JudgeConformanceResult, CacheConformanceProbeInput, CacheConformanceCheck, CacheConformanceResult, WorkspaceProviderConformanceProbeInput, WorkspaceProviderConformanceCheck, and WorkspaceProviderConformanceResult.

Each checkRuntime() call checks one component, not a whole host composition. Its schemaVersion, qualified runtimeKind, versioned checkStandardId, and evidenceLevel: 'behavioral-probe' state exactly what was observed. A passing result does not attest Runtime identity, provider quality, security isolation, durability, or end-to-end construct validity. After checking individual components, run the intended composition with evaluate().

Cache probes perform writes and intentionally leave harmless entries, so they require a caller-owned unused probeNamespace and a disposable namespace. A timeout bounds the caller's wait but cannot cancel an arbitrary cache promise. ContentStore probes also write an idempotent public value. WorkspaceProvider probes open trial leases and observe request forwarding, retry reuse, distinct active roots, Target access, and close() calls across success, failure, and cancellation; they do not prove descriptor bytes, physical deletion, or sandbox containment. Judge probes perform up to four real calls and may cost money, so allowExternalCalls: true is mandatory. Every publicProbeText is transmitted to that provider inside a generated rubric prompt and must contain only harmless public data. The result reports the observed invocation count, declared cost-reporting mode, and measured provider costs without retaining prompts, outputs, provider exceptions, cache entries, content payloads, workspace roots, locators, or credentials.

Evaluation-cache, Custom Evaluator, and Judge probes include concurrent canonical Core coordinates; their ports must remain correct when calls overlap. The current Core execution-cache read path is checked as it is actually invoked and makes no additional concurrency claim. Workspace timeoutMs bounds the check's wait for close(), but cannot stop an already-running provider promise, so providers must still implement bounded local cleanup. Cancellation checks require the Custom Evaluator or Judge call itself to reject after abort; a later successful settlement is not certified merely because Core reports the Run as cancelled.

Mock interception types are MockInterceptionDescriptor, MockInterceptionInput, MockInterceptionPlan, MockInterceptionProvider, MockInterceptionOpenRequest, MockInterceptionLease, MockInterceptionAccess, MockInterceptionRequest, and MockInterceptionDecision.

Public model types are Artifact, ArtifactKind, ArtifactSource, Variant, VariantExecution, RuntimeContext, AllowedToolsInput, AllowedToolsPlan, McpConfigDescriptor, McpConfigInput, McpConfigPlan, McpConfigProvider, McpConfigOpenRequest, McpConfigLease, McpConfigAccess, WorkspaceDescriptor, WorkspaceInput, WorkspacePlan, WorkspaceProvider, WorkspaceOpenRequest, WorkspaceLease, WorkspaceAccess, ContentDescriptor, ContentValue, ContentStoreRequest, ContentStore, ContentResolver, ExecutionCache, ExecutionCacheEntry, EvaluationCache, EvaluationCacheEntry, ExecutorIdentityVerifier, ExecutorIdentityVerificationRequest, ExecutorIdentityVerification, EvaluationInfrastructure, Dataset, Sample, EvaluationExecutor, Executor, InvokeExecutor, SessionExecutor, ExecutorSessionContext, ExecutorSessionAttempt, ExecutorSession, ExecutorCapabilities, ExecutorInvocation, ExecutorResult, Evaluator, ExactMatchEvaluator, RetrievalEvaluator, RetrievalMetricIds, AbstentionEvaluator, AbstentionMetricIds, ToolTrajectoryEvaluator, ToolTrajectoryMatchMode, RubricJudgeEvaluator, RubricJudgeMember, RubricJudgeAggregation, CustomEvaluator, CustomEvaluatorInvocation, CustomEvaluatorResult, CustomEvaluatorBinding, CustomEvaluatorContent, Metric, Judge, Rubric, Experiment, SamplingDesign, AnalysisRequest, CohortFilter, Comparison, ComparisonFamilyMember, CompositeMetricComponent, CompositeAggregation, Decision, FamilyDecisionCriterion, Policy, StagePolicy, RetryPolicy, RetryBackoff, FailurePolicy, CachePolicy, EvidencePolicy, BudgetPolicy, BudgetScope, RunBudgetScope, AttemptBudgetScope, ProviderCostLimit, EvaluateInput, EvaluationRunOptions, EvaluationResult, PreparedEvaluation, PreparedEvaluationPlan, RuntimeCapabilityResolution, EvaluationWorkEstimate, EventObserver, Clock, AssessComparabilityInput, EvaluationComparabilitySubject, and EvaluationComparabilityAssessment. Series uses EvaluationSeriesInput, EvaluationSeriesStability, EvaluationSeriesRunOptions, PreparedEvaluationSeries, EvaluationSeriesWorkEstimate, EvaluationSeriesMemberResult, EvaluationSeriesResult, EvaluationSeriesStabilityResult, and RunStabilityValue. Executor behavioral checks use ExecutorCheckInput, ExecutorCheckResult, and RuntimeConformanceCheck.

Policy uses independent execution and evaluation StagePolicy values. Each stage seals its own concurrency, timeout, and optional RetryPolicy; retry error codes are host-defined stable identifiers, and RetryBackoff is an explicit none, fixed, or exponential union. FailurePolicy is also discriminated: only failure-threshold carries maxFailures. BudgetPolicy exposes run, stage, coordinate, and attempt scopes with auditable invocation, active-duration, wall-clock, and provider-cost limits. Provider-cost admission is deliberately fixed to bounded overshoot; onUnreportedProviderCost selects fail-closed or unverifiable handling. CachePolicy independently controls Execution disabled／reuse／replay-only and Evaluation disabled／reuse. EvidencePolicy independently selects full, reference, digest, or none through output, trace, and evaluatorEvidence, plus one classification ceiling. The façade only materializes these declarations into the Core Measurement Policy; all scheduling, timeout, retry, cancellation, budget accounting, cache validation, evidence capture, and failure-threshold behavior remains in Core.

EvaluateInput.infrastructure accepts explicit host-owned ContentStore and ContentResolver ports. A reference capture writes through the one store shared by execution output, trace, and Evaluator evidence. If a downstream Evaluator consumes reference-captured output or trace, the resolver retrieves it. Core derives the canonical digest before storage, verifies the returned descriptor, and rechecks resolved value digest, classification, and media type before evaluation. The façade captures method bindings during prepare and rejects a missing required port, an input-removing capture mode, or an insufficient classification ceiling before the first Target call. Store implementations and credentials never enter the sealed Definition. The returned descriptor does enter the run artifact, so an optional uri must be a stable, opaque, credential-free locator rather than a physical path or signed URL. digest and none remain valid for output or trace that no Evaluator consumes.

The same EvaluationInfrastructure may provide independent ExecutionCache and EvaluationCache ports. Execution reuse accepts only deterministic Executors and requires an independent ExecutorIdentityVerifier that binds the captured callable and all behavior-affecting dependencies to a stable attestation. Running checkExecutor() or trusting the Executor's self-reported fingerprint is not that verification. replay-only fails on a miss without calling the Target, while Evaluation reuse may be enabled independently. The façade validates required ports and verification during prepare; Core remains the sole owner of cache keys, entry validation, hit provenance, and write timing.

RuntimeContext contains only reproducible host-defined JSON values. A Variant selects a logical workspace with a content-addressed WorkspaceDescriptor, or a WorkspacePlan containing one default plus bySampleId overrides; null explicitly disables the default for that sample. The Executor owns the corresponding WorkspaceProvider. Its stable providerId, version, and optional measurement-relevant fingerprintFacets participate in Runtime identity, while credentials, CAS locators, caches, and base directories remain inside the provider closure. Both canonical and advanced JSON adapters bind that provider identity into the final Executor fingerprint. prepareEvaluation() seals descriptors without opening a lease. At execution time the provider must verify the requested immutable descriptor and return a fresh WorkspaceLease with an absolute, trial-private root. The same WorkspaceAccess is reused across retry attempts for that trial, then the Runtime calls close() after success, failure, timeout, or cancellation. OMK itself never adds the physical root or provider-private state to a Definition, result, or error; an Executor must likewise avoid returning locators in its own output or trace. Invoke and session Executors receive only { descriptor, root }; they cannot close another component's lease. Reusing a lease object across trials or an active physical root fails closed, and a root whose cleanup fails remains quarantined in that process. open() and close() must be bounded local resource work. A lease provides measurement isolation, not a security sandbox; containing untrusted code remains the host's responsibility.

A Variant may also set execution.allowedTools to one exact list or an AllowedToolsPlan with default and bySampleId. OMK sorts the list for identity but never unions lists across samples. [] denies every tool; a null sample override explicitly restores the Executor runtime default. The Executor must declare capabilities.toolPolicy: 'allow-list' and strictly apply the allowedTools received by execute() or openSession(). Undefined means runtime default. OMK fails before execution if the capability is absent, but the capability is self-reported: do not declare it when the backend cannot enforce the exact list. Neither tool names nor workspace controls enter Gold or evaluation-only context. Skill discovery, installation, and name resolution remain host／Workflow responsibilities; Runtime seals only the final execution contract and does not add an allowedSkills policy. checkExecutor() currently rejects workspace- or tool-policy-enabled declarations because its generic probes cannot prove their isolation or enforcement; run a real Evaluation until dedicated conformance probes are available.

For native MCP configuration, a Variant selects a secret application/json McpConfigDescriptor directly or through a McpConfigPlan; null disables the default for that sample. The Executor must pair capabilities.mcp: 'native-config' with an McpConfigProvider. Provider identity and the selected descriptor participate in Runtime and execution-coordinate identity, while credentials, locators, and config bytes stay inside the provider. The provider returns canonical JSON matching the declared digest and byte size. Runtime opens one fresh lease per Trial, reuses it only for retries of that Trial, exposes only { descriptor, config } to the Executor, and closes the lease on every terminal path. OMK never serializes config content into its result or error, but Executors remain responsible for not echoing secrets in output or trace. checkExecutor() rejects MCP-enabled declarations until a dedicated isolation probe exists; use a real Evaluation. Discovery, defaults, and product-specific Workflow assembly are outside this Runtime port.

For pre-tool-call mocking, a Variant selects a secret MockInterceptionDescriptor directly or through a MockInterceptionPlan; null disables the default for one sample. The descriptor must use MOCK_INTERCEPTION_PLAN_MEDIA_TYPE (application/vnd.omk.mock-interception-plan+json) and identify an aggregate plan whose bytes bind strictness, first-match rule order, and ordered payload descriptors. The Executor must pair capabilities.mockInterception: 'pre-tool-call' with a MockInterceptionProvider. Provider identity and the selected descriptor enter Runtime and coordinate identity, while plan bytes, rule bytes, payloads, and locators stay inside the provider. The provider verifies the descriptor before returning a fresh attempt-scoped lease. Invoke Executors receive MockInterceptionAccess on their invocation; Session Executors receive it on each ExecutorSessionAttempt, never on openSession(). Runtime validates the request and mocked／pass-through／denied decision, closes the lease only after the Target call settles, rejects reused leases, lifts output and trace classification to secret, and redacts provider failures. Retry attempts always receive fresh state. A strict miss must be denied before any real tool call. checkExecutor() rejects mock-enabled declarations until a real tool-call probe exists.

const executor: Executor<string, undefined, string> = {
  executorId: 'acme.agent/v1',
  version: '1.0.0',
  schemas: { input: z.string(), output: z.string() },
  workspaceProvider: {
    providerId: 'acme.cas-workspace/v1',
    version: '1.0.0',
    async open({ descriptor, runId, trialId }) {
      const root = await materializeFreshOverlay(descriptor, { runId, trialId });
      return { root, close: () => removeOverlay(root) };
    },
  },
  async execute({ input, workspace, signal }) {
    return { output: await runAgent(input, { cwd: workspace?.root, signal }) };
  },
};

const variant: Variant<string, undefined, string> = {
  variantId: 'workspace-agent-v1',
  artifact: { name: 'workspace-agent-v1', kind: 'agent', source: 'inline', content: '...' },
  execution: { executor, workspace: workspaceDescriptor },
};

Sample.executionContext is per-sample input visible to the Executor; Sample.evaluationContext is per-sample input visible only to the Evaluator. These sample projections do not describe the host environment.

EvaluationExecutor is the union accepted by a Variant. Executor and its explicit alias InvokeExecutor run one stateless omk.invoke/v1 callback; omitting protocol means invoke. SessionExecutor requires protocol: 'session' and opens one isolated, newly allocated ExecutorSession object for each Core trial; object reuse across trials or Runs is rejected. ExecutorSessionContext exposes the Target projection plus stable runId and trialId, but never Gold or evaluation-only context. Retries call the same session's execute() with a fresh ExecutorSessionAttempt, including attemptId, attemptNumber, and the Core AbortSignal. Coordinate-derived attemptId values may recur in a separate Run, so provider idempotency keys must also be namespaced by runId or an equivalent provider-session scope. Success, failure, timeout, and cancellation all end with exactly one close(). openSession() and close() must be bounded local lifecycle work; opening is resource acquisition rather than a measured provider attempt, so billable or model work belongs in execute(). A session is temporary and run-scoped; it is not a cross-Run conversation or memory store.

EvaluationResult preserves every field of the Core EvaluationRunResult and adds the effective runId, definition, policy, and analysisResults. The last field is a read-only analysisId index over the exact Core Analysis records, not a second analysis implementation. Execution and evaluation evidence remain under artifacts, the decision remains under artifacts.decision, and the public report remains under report.

saveEvaluationResult() accepts only an original authenticated result with a complete Execution／Evaluation／Analysis source chain, stores a versioned canonical JSON envelope through the caller's ContentStore, and returns a content-addressed ContentDescriptor. The envelope contains the sealed Definition and Dataset Gold through the result, so Runtime always writes it with classification: 'gold'; hosts must apply matching access controls. loadEvaluationResult() requires the caller to prepare the exact declaration again, keeps the requested descriptor immutable, verifies the outer value plus every reference-captured output, trace, and Evaluator evidence value through the injected ContentResolver, then asks an EvaluationResultVerifier to authenticate the exact envelope before Core re-admits every bundle and the report. The verifier is a host trust boundary: its EvaluationResultVerification must explicitly list independently authenticated provenance Bundle digests, cache receipt digests, and Decision policy-execution digests. Runtime passes only those facts to Core and never derives verified receipts from the stored Bundle's own claims; merely recomputing the public envelope checksum is insufficient. Store, resolver, and verifier implementations may use files or databases, but Runtime never discovers either and redacts rejected promises as well as malformed return values. A different plan, incomplete result, cloned source on save, missing referenced content, tampered content, or insufficient verification fails closed. The related types are SaveEvaluationResultInput, LoadEvaluationResultInput, EvaluationResultVerifier, EvaluationResultVerificationRequest, and EvaluationResultVerification.

EvaluateInput contains only the measurement declaration. EvaluationRunOptions contains run-scoped runId, cancellation, progress observation, report annotations／summaries, event-buffer capacity, and clock. Omitting runId generates one and returns it as EvaluationResult.runId. prepareEvaluation(input) captures all mutable declarations, materializes defaults, resolves Runtime capabilities, and seals the Core Plan without calling a Target or Evaluator. Its frozen PreparedEvaluation exposes the exact definition, policy, plan, complete-contract planDigest, resolvedRuntimes, and estimatedWork; run(options) executes that same sealed Plan without re-reading the input or recompiling. Planned coordinates exclude retries and early termination, while duration and provider cost remain explicitly uncertain until execution.

assessComparability() consumes two exact authenticated EvaluationResult objects returned by canonical evaluation or by loadEvaluationResult(). A plain clone or deserialized document has no source authority and is rejected instead of being treated as verified. comparisonScope selects the deepest invariant contract to compare, while each EvaluationComparabilitySubject explicitly maps the intentionally changed Variant on the left and right. The returned EvaluationComparabilityAssessment is authored by Core and keeps designStatus, evidenceQualificationStatus, and overall comparabilityStatus separate. A mapped subject change is identity information, not a design mismatch; incomplete Runtime assurance remains conditional rather than being silently promoted.

evaluateSeries() measures fixed-design repeatability. repeatCount preregisters the complete number of member Runs; the full Dataset, SamplingDesign, Evaluators, Analysis graph, policies, Runtime identities, and measurement seed are captured once and sealed into every member before execution. Each Run is one experimental unit. Target trials, retries, and Rubric Judge replicates are nested measurements and never increase Series runCount. Members execute sequentially with Execution and Evaluation cache disabled, cannot be replaced after failure or cancellation, and every missing slot remains visible in coverage. stability.sourceAnalysisId selects an existing scalar Analysis result, or an interval's point estimate through the explicit interval-estimate projection. The built-in returns mean, Bessel-corrected sample variance with denominator n - 1, standard deviation, minimum, maximum, and range; it deliberately emits no verdict or confidence interval. It completes only when every preregistered slot is eligible and comparable; otherwise the stability record is inconclusive rather than a complete-case estimate. These are descriptive repeatability statistics under one sealed design, not evidence of cross-environment reproducibility or independent-and-identically-distributed sampling. Seed-aware Executors therefore receive the same sealed trial seeds in every member; a design that intentionally varies Run-level seeds is a different experiment and is not represented by this first façade.

prepareEvaluationSeries() performs the same capture and returns a single-use PreparedEvaluationSeries with the exact definition, Core plan, every member Plan, and estimatedWork, without calling a Target. seriesInstanceId names one intentional execution and must be unique when a caller intends a distinct Series. The default requires complete member evidence and accepts conditional comparability because ordinary callback identities are self-declared; callers may explicitly allow partial evidence or require compatible comparability. No result-dependent stopping, member replacement, cache reuse, or implicit Decision is provided. A caller cancellation stops future members and is forwarded to Core; the terminal result is cancelled, retains the planned member outcome list, and does not manufacture a completed Series Analysis or Report.

rescore(), reanalyze(), and redecide() take a complete new EvaluateInput, an exact prior EvaluationResult, and optional EvaluationRunOptions. The new declaration is sealed before reuse. Core recursively verifies that the retained source capability matches every skipped stage: changed Target input cannot be hidden behind rescore, changed Gold or Evaluator cannot be hidden behind reanalyze, and changed Analysis cannot be hidden behind redecide. A clone, deserialized report, or Bundle JSON has no in-process source authority and is rejected. Reused upstream bundles retain their original identities and historical evidence; every executed suffix stage receives the new Run identity, emits the only new progress events, and consumes only the new Run's suffix budget. The façade does not reconstruct evidence, rerun skipped callbacks, or duplicate Core scoring, statistics, Decision, Report, budget, cache, or scheduler behavior.

SamplingDesign supports a one-Variant solo quality profile, complete-block paired comparisons, and fixed-quota independent comparisons. It is the only owner of paired／independent semantics. A solo design may declare clusterKey; Core then treats whole clusters as the experimental and resampling unit. A Comparison declares one control, one or more treatments, and the Metrics to analyze; it contains no duplicate sampling discriminator. evaluators may contain multiple exact-match, retrieval, abstention, tool-trajectory, Rubric Judge, or custom evaluators, provided evaluator and metric IDs are unique.

RetrievalEvaluator is the source-neutral binary-relevance top-k preset. It reads a unique ordered document-ID array from an explicit output or trace JSON Pointer and a non-empty set of unique relevant IDs from Sample.expected. Its four RetrievalMetricIds are bounded, higher-is-better sample Metrics: Recall@k uses all known relevant documents as its denominator, Precision@k always uses k (missing result slots are non-hits), Reciprocal Rank@k uses the first relevant rank, and nDCG@k uses binary gain with log2 discount. The ranking is truncated before measurement. Duplicate or malformed IDs and empty relevance sets produce invalid evidence; they are never deduplicated, clamped, or converted to NaN. A summary mean over the Reciprocal Rank Metric is MRR. Cutoff, pointers, Metric IDs, and algorithm identity are sealed into the Definition and Runtime fingerprint.

ToolTrajectoryEvaluator deterministically compares ToolCallInfo.tool names from one complete omk.source-neutral-trace/v2 against tool names projected only from Sample.expected. Its explicit ToolTrajectoryMatchMode is exact-order, same-tools, contains-in-order, or contains-any-order; the names describe the actual-to-expected relationship without ambiguous subset／superset orientation. Matching is case-sensitive and preserves repeated-call multiplicity. Every call status participates because this Metric measures the agent's call decisions, not whether tools succeeded. Empty actual trajectories are valid. Empty expected trajectories are valid only for exact modes, where they assert that no tool should be called; contains modes reject the vacuous condition. Invalid trace／Gold becomes invalid evidence, while an unresolved pointer remains Core not-evaluated evidence. The boolean observation never copies the sensitive trajectory.

A RubricJudgeEvaluator is an explicit judge panel. judges contains one or more RubricJudgeMember values; replicateCount repeats only that member's measurement and never reruns the Target. One panel may expand to at most 1,000 member × replicate coordinates. RubricJudgeAggregation must select equal-member mean or a weighted-mean whose positive weights cover every member and sum to one. The only supported missing rule is require-complete: one unavailable planned member or replicate removes that Target × Sample × Trial panel reading from analysis rather than averaging the survivors. Raw member and replicate records remain in the Evaluation Bundle.

A CustomEvaluator is the canonical one-Metric callback extension. It declares explicit input bindings, serializable parameters, a sample-scope Metric, schema parsers, and measurement-relevant identity facets. The callback receives only the values selected by its bindings; it cannot inspect the full sample or execution record. It returns one score, missing, invalid, or stable failed result. Core remains the only owner of concurrency, timeout, budget, cancellation, evidence capture, and failure redaction. The callback contract is stateless, parallel-safe, and cancellation-cooperative; stateful lifecycle integrations use /advanced. One evaluator cannot emit multiple Metrics or claim ensemble coordinates.

Numeric and boolean custom Metrics require an explicit higher-is-better or lower-is-better direction. Categorical, text, and ranking Metrics cannot declare a scale or direction. The canonical progress/v2 Decision currently accepts only higher-is-better, because silently applying its positive-effect rule to a lower-is-better scale would reverse the verdict.

implementation.version, schema fingerprintFacets, and implementation fingerprintFacets are mandatory identity declarations. OMK never fingerprints Function#toString(). Callers must change one of these facets whenever callback code, dependencies, schemas, or provider configuration changes measurement behavior. Binding and value schemas validate without coercion, defaults, or field removal. CustomEvaluatorContent carries an explicit classification for evidence or invalid values; undeclared source values are never passed to the callback.

independent requires an explicit allocation for every Variant plus global and per-stratum minimum sample counts. The seed, optional stratumKey, weights, and minima are sealed before any Executor call. Core assigns each sample to exactly one Variant, reuses that assignment across repeated trials, and fails before execution if any minimum cannot be met. It analyzes each comparison with the unpaired percentile-bootstrap estimator; it never relabels independent data as paired.

Analysis is always explicit and preregistered. Top-level analyses[] accepts named summary, quality-interval, comparison-interval, comparison-family, composite-quality-interval, and composite-comparison-interval requests. Summaries expose numeric mean, boolean rate, and numeric quantile; individual intervals use an explicit confidence level and resample count. A comparison family declares at least two globally named contrasts plus one family-wise confidence level. Its bonferroni-percentile-bootstrap method seals every member at marginal confidence 1 - (1 - family level) / family size, then produces one Core-verified simultaneous-family table; the family level is a nominal target whose coverage depends on the marginal Bootstrap intervals having their stated coverage. It never fabricates p-values or selects family members after observing results. Each request may apply one sealed Dataset cohort filter. Rubric panels aggregate replicates within members, members by the declared rule, and repeated Target trials within a sample before bootstrap. Paired and independent members retain their respective resampling units. Metric direction is never used to silently flip a signed result. An analysis decision selects one interval. A comparison-family decision selects the outer family and supplies one raw-effect FamilyDecisionCriterion per member; its explicit all rule releases only when every simultaneous interval satisfies its inclusive bounds, blocks a proven violation, and otherwise stays not-decided. An empty analysis list intentionally retains typed evaluation evidence without fabricating statistics or a composite verdict.

A composite request names its derived [0, 1] higher-is-better Metric with compositeMetricId, then declares at least two CompositeMetricComponent values and one explicit CompositeAggregation. Components must have unique source Metric IDs, positive weights that sum exactly to one, and the only v1 aggregation is { method: 'weighted-mean', missing: 'require-complete' }. Boolean Metrics and bounded numeric Metrics with a monotonic direction are supported; the request cannot override their sealed scale or direction. Runtime materializes the derived Metric and, for a composite comparison, adds it to the selected Core Comparison. Core remains the sole owner of normalization, panel aggregation, unit-first composition, missing-component exclusion, resampling, coverage, and source-row lineage.

const analysis: AnalysisRequest = {
  analysisId: 'overall-quality',
  analysisKind: 'composite-quality-interval',
  compositeMetricId: 'overall-quality',
  variantId: 'prompt-v2',
  components: [
    { metricId: 'correct', weight: 0.6 },
    { metricId: 'rubric-quality', weight: 0.4 },
  ],
  aggregation: { method: 'weighted-mean', missing: 'require-complete' },
  confidence: { method: 'percentile-bootstrap', level: 0.95, resamples: 1_000 },
};

The entry deliberately exposes no Definition builder, Runtime registry, Core Target, lifecycle adapter, or raw Rubric factory. Artifact is what is evaluated, Variant binds it to an Executor, config, and runtime context, and control／treatment roles exist only inside an explicit Comparison.

Built-in abstention and mixed retrieval evaluation

Start with the four-step guide for runnable commands, data rules, service integration points, and expected output.

AbstentionEvaluator is a built-in evaluators[] declaration, available through the package root and /eval-runtime. It compiles to the existing Core contracts and produces two boolean AbstentionMetricIds. There is no separate evaluation factory, dataset selector, or execution lifecycle.

const abstention: AbstentionEvaluator = {
  evaluatorKind: 'abstention',
  evaluatorId: 'abstention',
  ranking: { source: 'output', pointer: '/solutionIds' },
  shouldAbstainPointer: '/shouldAbstain',
  metricIds: {
    abstentionCorrect: 'correct-abstention',
    falseAbstention: 'false-abstention',
  },
};
// evaluate({ dataset, evaluators: [abstention], ...hostDesign })

ranking must select the final ordered recommendation ID list in output or trace, after any application threshold. A successful execution with a valid empty list is abstention. Null, missing output, blank/non-string/duplicate IDs, execution failure and timeout are not successful abstention. IDs are case-sensitive without trimming or deduplication. shouldAbstainPointer selects an explicit boolean in Sample.expected; null and non-booleans are invalid evidence, missing Gold paths are rejected before execution. The evaluator does not infer labels from empty relevance or interpret business review status. Gold bindings are not passed to the Target.

Metric	Applicable population and meaning	Direction
abstentionCorrect	Valid samples with shouldAbstain: true; true for an empty list, false for any nonempty list	Higher is better
falseAbstention	Valid samples with shouldAbstain: false; true for an empty list, false for a nonempty list	Lower is better

Use summary statistic: 'rate' for these metrics. The opposite population is missing with reason abstention-not-applicable, excluded under exclude/v1. It is never given a fabricated zero or one. Missing output or trace bindings are reported by Core as unavailable before the evaluator runs; missing Gold paths fail preparation. Show the full Analysis coverage, including included (actual denominator), missing, invalid, sourceUnavailable, evaluationFailed, alongside Execution coverage. Zero applicable observations retain a non-completed Analysis state. Conditional correct-abstention rate is not end-to-end delivery rate: one successful empty response and nine timeouts can yield 100% among valid responses but only 10% successful delivery among all ten expected-abstention requests. Repeated trials still use the declared Core experimental unit and reduction, not an implicit per-attempt denominator.

The versioned omk.eval-runtime.abstention/v1 instrument seals pointers, metric IDs and the successful-empty-list protocol into Definition and Runtime identity. It does not change retrieval v1 formulas, scoring, prompts or Core schemas. Do not compare this metric directly with historical scores that count empty Gold as perfect recall; establish an explicit baseline for the new instrument. Natural-language refusals and safety-filter responses are outside this list protocol.

The complete offline example is the single file examples/eval-runtime/retrieval-abstention.mjs, including its host-owned helper functions. After building the repository, run node examples/eval-runtime/retrieval-abstention.mjs. A separate service can copy this example file and install OMK; no credentials or business network are needed for the synthetic Executor.

The example prepares and validates labels before execution, defaults to rejecting pending samples, and permits explicit exclusion with an audit of source revision, counts and reasons. Business quality.reviewStatus is interpreted only by that helper. The actual selected Dataset is sealed by Core. It combines built-in retrieval and abstention with an independent Custom Evaluator for forbidden-ID hits. All samples carry a relevance binding. An empty relevant set on negative samples retains the existing retrieval v1 invalid observation; retrieval analyses explicitly select the answerable cohort and exclude those negatives before aggregation. Abstention remains applicable to its own populations. The independent forbidden-hit example checks the first three final results, includes only valid samples with a nonempty forbidden list, and makes no claim that this list enumerates all wrong results. A correct retrieval and a forbidden hit can coexist.

The example also scopes correct-abstention analysis to unanswerable, false-abstention analysis to answerable, and forbidden-hit analysis to has-forbidden. This keeps coverage.planned, unavailable evidence and the actual denominator within each applicable population, including when execution fails before an evaluator can read its labels. A failure in another population must not appear as this metric’s unavailable evidence.

Dataset preparation and the forbidden-ID helper are example code, not additional OMK public APIs. Callers may replace them without changing the built-in abstention instrument. This release adds a composable scoring capability rather than prescribing a seven-metric suite.

oh-my-knowledge/eval-runtime/advanced

Low-level host assembly and extension SPI. Applications should prefer evaluate().

Export	Purpose
runEvaluation	Run an already assembled Core Definition, Runtime, and Policy.
EvaluationEventConsumptionError	Event-consumption failure for runEvaluation.
createEvaluationRuntime	Assemble Executor／Evaluator registrations and Core built-ins.
EvaluationRuntimeAssemblyError	Stable registration or resolution failure.
createExactMatchDefinition	Build an exact-match paired Core Definition.
createPairedComparisonDefinition	Build a one-metric paired Core Definition.
createMeasurementPolicy	Materialize Core Policy defaults, including explicit EventWriter delivery mode.
createExactMatchEvaluator	Create the built-in exact-match Evaluator port.
createInvokeExecutorIdentity	Declare an omk.invoke/v1 Executor identity.
createSessionExecutorIdentity	Declare an isolated omk.session/v1 Executor identity.
createRuntimeIdentity	Declare another host Runtime identity.
createJsonExecutorAdapter	Adapt a typed JSON callback to a Core Executor.
createJsonSessionExecutorAdapter	Adapt a typed, per-trial JSON session lifecycle to a Core Executor.
createRubricJudgeKit	Derive matching Rubric Definition, Metric, context, and registration fragments.
createRubricJudgeEvaluationContext	Combine criterion context for multiple Rubric kits.
createRubricJudgeRegistration	Combine multiple Rubric kit bindings.
runExecutorConformance	Run the low-level Executor conformance probe.
assertExecutorConformance	Throw when a conformance result failed.
RuntimeConformanceError	Stable conformance assertion error.
createNodeEvaluationClock	Supply the default Node.js Core clock.
EXACT_MATCH_EVALUATOR_IMPLEMENTATION_ID	Built-in exact-match implementation ID.
createExactMatchEvaluatorIdentity	Inspect the exact-match Runtime identity.
INVOKE_JSON_INPUT_SCHEMA	Default JSON input schema identity.
INVOKE_JSON_OUTPUT_SCHEMA	Default JSON output schema identity.
INVOKE_JSON_TRACE_SCHEMA	Default JSON trace schema identity.
SESSION_JSON_INPUT_SCHEMA	Default session JSON input schema identity.
SESSION_JSON_OUTPUT_SCHEMA	Default session JSON output schema identity.
SESSION_JSON_TRACE_SCHEMA	Default session JSON trace schema identity.
createExecutorFnAdapter	Bridge the legacy ExecutorFn.
createSameProcessExecutorAdapter	Implement explicit in-process Executor lifecycle SPI.
createSameProcessEvaluatorAdapter	Implement explicit in-process Evaluator lifecycle SPI.
createRubricJudgeCriterion	Construct a raw Rubric criterion.
createRubricJudgeInstrument	Construct a raw frozen Rubric instrument.
createRubricJudgeRuntimeConfig	Construct raw Judge Runtime config.
createRubricJudgeEvaluatorDefinition	Construct a raw Rubric Evaluator Definition.
createRubricJudgeMetricDefinition	Construct the raw 1–5 Metric.
createRubricJudgeEvaluatorIdentity	Derive the raw Rubric Evaluator identity.
createRubricJudgeEvaluator	Construct one raw Rubric Evaluator port.
createRubricJudgeEvaluatorRegistration	Combine raw Rubric bindings.
rubricJudgeInstrumentId	Derive the built-in instrument ID.

Run and assembly types are RunEvaluationInput, EvaluationEventObserver, CreateEvaluationRuntimeInput, EvaluationRuntimeSupportPorts, and RuntimePortRegistration. Builder types are ExactMatchDefinitionBuilderInput, ExactMatchTarget, PairedComparisonDefinitionBuilderInput, EvaluationRuntimeTarget, MeasurementPolicyBuilderInput, MeasurementStagePolicyInput, MeasurementRetryPolicyInput, MeasurementRetryBackoffInput, MeasurementFailurePolicyInput, MeasurementEvidencePolicyInput, MeasurementEventDeliveryInput, and CreateExactMatchEvaluatorInput. Identity and JSON adapter types are InvokeExecutorIdentityDeclaration, SessionExecutorIdentityDeclaration, RuntimeIdentityDeclaration, CreateJsonExecutorAdapterInput, CreateJsonSessionExecutorAdapterInput, JsonExecutorInvocation, JsonExecutorInvocationResult, JsonSessionExecutorContext, JsonSessionExecutorAttempt, JsonExecutorSession, RuntimeValueParser, AllowedToolsInput, AllowedToolsPlan, WorkspaceDescriptor, WorkspaceInput, WorkspacePlan, WorkspaceProvider, WorkspaceOpenRequest, WorkspaceLease, and WorkspaceAccess. Judge types are OmkLlmJudgeEffort, OmkLlmJudgeInvocationPort, OmkLlmJudgeInvocationRequest, OmkLlmJudgeInvocationResult, CreateRubricJudgeKitInput, RubricJudgeKit, CreateRubricJudgeEvaluatorInput, RubricJudgeEvaluatorBinding, and RubricJudgeEvaluatorDefinitionBuilderInput. Conformance types are ExecutorConformanceProbeInput, ExecutorConformanceResult, and RuntimeConformanceCheck. Legacy and lifecycle SPI types are CreateExecutorFnAdapterInput, ExecutorFn, ExecutorInput, ExecResult, ExecutorFnInputMapper, ExecutorFnResultMapper, CreateSameProcessExecutorAdapterInput, CreateSameProcessEvaluatorAdapterInput, SameProcessExecutorImplementation, SameProcessEvaluatorImplementation, SameProcessResourceLeaseAccess, SameProcessRunScope, and SameProcessOperationScope.

Advanced budget-builder types are MeasurementBudgetPolicyInput, MeasurementBudgetScopeInput, MeasurementRunBudgetScopeInput, MeasurementAttemptBudgetScopeInput, and MeasurementProviderCostLimitInput.

Advanced adapter types additionally expose McpConfigAccess, McpConfigDescriptor, McpConfigInput, McpConfigLease, McpConfigOpenRequest, McpConfigPlan, McpConfigProvider, MockInterceptionAccess, MockInterceptionDecision, MockInterceptionDescriptor, MockInterceptionLease, MockInterceptionOpenRequest, MockInterceptionProvider, and MockInterceptionRequest.

MeasurementCachePolicyInput is the cache-policy input accepted by the advanced policy builder.

oh-my-knowledge/eval-runtime/contracts

Versioned wire contracts for adapter and trace authors:

• Rubric identities and schemas: RUBRIC_JUDGE_EVALUATOR_IMPLEMENTATION_ID, RUBRIC_JUDGE_BINDINGS, RUBRIC_JUDGE_INSTRUMENT_SCHEMA_VERSION, RUBRIC_JUDGE_CONTEXT_SCHEMA_VERSION, RUBRIC_JUDGE_EVIDENCE_SCHEMA_VERSION, RUBRIC_JUDGE_INSTRUMENT_SCHEMA, RUBRIC_JUDGE_CONTEXT_SCHEMA, and RUBRIC_JUDGE_EVIDENCE_SCHEMA.
• Rubric types: RubricJudgeInstrument, RubricJudgeRuntimeConfig, RubricJudgeConfig, RubricJudgeCriterion, and RubricJudgeTracePolicy.
• Trace values: SOURCE_NEUTRAL_TRACE_SCHEMA_VERSION, SOURCE_NEUTRAL_TRACE_SCHEMA_DESCRIPTOR, SOURCE_NEUTRAL_TRACE_WITHOUT_MOCKS_SCHEMA_DESCRIPTOR, SourceNeutralTraceSchema, SourceNeutralTraceWithoutMocksSchema, SourceNeutralMockStatsSchema, parseSourceNeutralTrace, and attachSourceNeutralMockStats.
• Trace types: SourceNeutralTrace and SourceNeutralMockStats.

Migration

The 1.0.0-beta canonical entry replaces the previous assembly-first surface. The general façade also replaces the earlier fixed { executor, control, treatment, evaluator } input with { variants, evaluators, comparisons, analyses }; Executor and config now live under each Variant's execution, Sampling Design alone selects paired or independent semantics, and every summary or interval is an explicit named analyses[] request. Remove comparisonKind and replace the redundant analysis: { analyses: [...] } wrapper with analyses: [...]; neither old shape is read or detected. Move runId, signal, onEvent, clock, annotations, summaries, and eventBufferCapacity from the declaration into the optional second EvaluationRunOptions argument; omitted runId is generated. A Decision optionally selects one interval or one explicitly bounded comparison family by analysisId. Rubric evaluation requires judges + aggregation; the singular judge + model + effort shape is not accepted. Policy fields are grouped under execution, evaluation, failure, budget, and evidence; the earlier flat concurrency, timeout, invocation, failure, and classification fields are not accepted. There is no 0.x compatibility reader, old overload, or legacy-shape detector. Move low-level imports from oh-my-knowledge/eval-runtime to oh-my-knowledge/eval-runtime/advanced; wire schemas remain at /contracts. createEvaluationEngine has one meaning and one home: import the full staged engine from oh-my-knowledge/eval-core; use advanced runEvaluation when a preassembled Runtime, Definition, and Policy only need a standard complete run. New hosts should import evaluate, prepareEvaluation, or checkExecutor from the package root. The /eval-runtime entry remains the explicit equivalent for consumers that prefer domain-qualified imports.

Budget limits now live under explicit scopes: replace budget.maxInvocations with budget.run.maxInvocations. The old form is neither read nor detected.

Use oh-my-knowledge/eval-core for custom analysis graphs, staged replay, transported custom comparability policies, or artifact admission that does not fit the canonical complete-result contract above. Deep implementation imports are unsupported.