Composite scoring

Evaluation Core treats scoring as an authenticated evidence graph, not a mutable report row. The historical five-layer contract remains stable—assertion, llm, judge, dimension, and composite—but those names identify responsibilities rather than five consecutive averages.

criterion observations          raw rubric readings
          │                              │
          ▼                              ▼
 assertion-layer table      replicate → ensemble tables
     fact / behavior                    │
          │                       dimension table
          └──────────────┬───────────────┘
                         ▼
                  composite table
                         │
             Bootstrap comparison family
                         │
                 Release Decision

Assertion layers

Each Boolean criterion is explicitly bound to fact, behavior, or excluded-mixed-layer with a finite positive weight. Classification is sealed in Analysis parameters; the implementation does not infer it from assertion names or evaluator IDs.

For one Target／Sample／Trial coordinate, an observed assertion layer is:

layerScore = 1 + passedObservedWeight / observedWeight × 4

The result is rounded to two decimals on a 1–5 scale. Structural non-applicability is excluded from assertion scoring coverage. Analysis Bundle v2 still retains the rectangular input coordinate, classifies it separately as notApplicable, and authenticates its row identity and reason through notApplicableRows, so it does not degrade evidence completeness. Missing, invalid, failed, unavailable, and not-started observations remain explicit coverage states and never become false. If no weight was observed, the layer is missing rather than zero.

Implementation: omk.assertion-layer-table/v1 in src/eval-workflows/measurement/analysis/assertion-layer.ts.

Judge and dimension derivation

Raw rubric readings retain their evaluator, metric, instrument, ensemble-member, replicate-group, replicate-index, Sample, Trial, and sampling-unit identities.

• The replicate table averages observed readings for one planned member and preserves non-observed rows.
• The ensemble table gives each observed member mean equal weight and reports agreement only when the required evidence exists.
• The dimension table seals per-sample applicability and weights. It emits a weighted mean only when every planned dimension is observed; missing evidence fails closed.

This prevents retry attempts, judge repeats, ensemble members, Trials, and independent Runs from being collapsed into the same statistical unit.

Composite derivation

Sealed parameters bind up to three present layers: fact, behavior, and judge. The judge source is either an ensemble consensus or a dimension aggregate.

composite = mean(observed present layers)

The aggregate is rounded to two decimals. An absent layer is structural non-applicability; a present but missing layer remains explicit missing evidence. If no planned layer is observed, the composite is missing rather than numeric zero. Every source group and binding is retained in lineage, and transported tables are recomputed during validation.

Implementation: omk.composite-table/v2 in src/eval-workflows/measurement/analysis/composite-table.ts.

What the composite can answer

The composite is a comparison signal inside one sealed design. It is useful for a preregistered treatment-minus-control Bootstrap comparison when Dataset, Target conditions, evaluator identities, policy, and layer bindings are held fixed.

It is not an absolute psychometric level. Equal weighting is pragmatic, assertion pass rates and rubric scores have different measurement properties, and designs with different present layers measure different constructs. Do not rank unrelated artifacts, datasets, or runs by their raw composite values.

Decision boundary

A composite score or positive point estimate cannot authorize release by itself. omk.release-decision/v7 consumes the exact Composite and Bootstrap-family v2 results plus every applicable rubric-dimension Judge Ensemble. It first checks complete evidence and source lineage, then applies dissent and unmeasured-uncertainty gates to every configured dimension. Statistical significance uses unrounded tail evidence and fails closed when finite-resample Monte Carlo uncertainty crosses the significance threshold. A positive comparison passes the practical-effect gate only when its persisted four-decimal percentile lower bound is greater than or equal to triviallySmallDifference; a large point estimate with an uncertain lower bound remains CAUTIOUS.

Its six conclusions are PROGRESS, CAUTIOUS, REGRESSION, NOISE, UNDERPOWERED, and SOLO. A normal release route requires a decided PROGRESS carrying release-gates-passed. When any applicable Judge Ensemble has dissent or cross-judge agreement cannot be estimated for control or treatment, a positive comparison becomes CAUTIOUS; deterministic evaluations without a Judge Ensemble are unaffected. A nonsignificant paired comparison uses complete pairs for the sample-size gate; an independent comparison uses its smaller observed arm. The sealed requirement is either an explicit minimum or a recomputable a priori paired-comparison plan; no observed run variance enters that plan. Missing intervals and Monte Carlo-indeterminate significance remain not-decided; multi-treatment results use the worst registered conclusion; cross-run stability belongs to an Evaluation Series rather than a single-run score. Historical release policies v1 through v6 and Bootstrap family v1 remain available as semantic implementations. The assignment-aware schema cutover versions their Runtime identities and intentionally does not accept pre-cutover Plans for exact replay.

See Statistical rigor for uncertainty, agreement, debiasing, and coverage gates.

Comparability invariant

The five-layer meanings, frozen scoring prompts, Bootstrap formulas, missing-evidence semantics, and length-debias toggle anchor cross-version comparison. A change to these semantics requires an explicit BREAKING-COMPARABILITY review. A presentation-only Studio projection must never redefine the score or Decision.