Observe. Measure.
Know.
OMK makes every knowledge change in your AI application evidence-backed.
Observe real-world performance, measure version differences, and determine whether the change is effective and the version is ready to ship.Use doctor → eval → studio before shipping, then close the loop with observe → sample → evolve.
Observe real-world performance, measure version differences, determine what works and what can ship
doctor, eval, studio, promote, observe, sample, and evolve are not isolated commands. Together they turn knowledge changes into evidence, and production gaps into reviewable drafts that can become the next fixed evaluation set.
doctor pre-ship
Is the artifact coherent enough to evaluate? doctor catches structural, dependency, safety, and measurability problems before an A/B run wastes tokens.
$ omk doctor my-skill --dimensions audit.yaml
- Triggers / docs / instructions / deps / tools / safety / examples
- --repeat: parallel sampling + k/n consensus
- Custom endpoint dimensions: call an API for deep review
eval on release
Is v2 really better than v1? eval fixes the model and samples, changes only the artifact, then reports a verdict with confidence intervals and failed cases.
$ omk eval --control v1 --treatment v2
- Bootstrap CI / length-debias / saturation curves on by default
- Krippendorff α: judge↔human agreement on a gold set
- Blind A/B · judge ensemble · multi-run variance
observe in production
What did real use expose? observe turns session logs into knowledge-gap signals; from-traces drafts samples first, and reviewed drafts can become fixed eval samples.
$ omk observe ~/.claude/sessions
- Failure rate / latency / cost broken down per artifact
- Knowledge-gap inbox for review and triage
omk sample --from-tracesdrafts samples for review
Rigor is the foundation, not an add-on
Five often-overlooked distortions decide whether a comparison is trustworthy. omk builds every defense into the foundation, so you don't have to enable them one by one.
Peer tools typically cover only one or two of these. omk's choice: build credibility into the foundation rather than leave it optional.
A side-by-side under one set of criteria
Criteria come from common LLMOps selection axes (metric library / judge / CI / observability / collaboration) plus measurement validity & reliability — not rules tailored to omk. On several axes omk doesn't win, and we mark that honestly.
| Capability | omk | promptfoo | DeepEval | LangSmith |
|---|---|---|---|---|
| Measurement credibility · validity / reliability | ||||
| Statistical significance (CI / tests) | ✓ Bootstrap | — | — | — |
| Judge ↔ human reliability (agreement) | ✓ Krippendorff α | — | — | — |
| Evaluation bias control (length-debias) | ✓ default | — | — | — |
| Evaluation capability | ||||
| Assertion / metric library breadth | ✓ 30+ | ✓ | ✓ | ◑ |
| RAG-specific metrics | ◑ 3 | ◑ | ✓ rich | ◑ |
| LLM-as-judge | ✓ | ✓ | ✓ | ✓ |
| Engineering & collaboration | ||||
| CI/CD integration (exit-code routing) | ✓ | ✓ | ✓ | ◑ |
| Onboarding speed / config simplicity | ◑ | ✓ very fast | ◑ | ◑ |
| Experiment tracking / tracing | — | — | ◑ | ✓ strong |
| Hosted SaaS dashboard / team collab | — | — | ✓ | ✓ |
| Ecosystem & integration | ||||
| Community size (GitHub stars, 2026-04) | nascent | 9k+ | 12k+ | commercial |
| Native Claude Code skill | ✓ | — | — | — |
Full comparison (8 tools × 30+ dimensions, incl. RAGAS / OpenAI Evals / lm-eval-harness / inspect-ai) is in the comparison doc, current as of 2026-04 — spot something stale? send a PR. Takeaway: no silver bullet — omk's tradeoff is "statistical credibility by default"; want a SaaS dashboard, pick LangSmith; want academic benchmarks, pick lm-eval-harness.
Use it right inside your coding agent
One line — omk install omk-agent-skill — installs the official Agent Skill into the Claude Code / Codex it detects locally (--to all writes to all). After that, /omk works out of the box in Claude Code; in Codex and others, just run the omk CLI.
No commands to memorize — state your goal in plain words, and the agent locates the skill from context and picks the right command.
Before your next release, let the data speak first.
From a bare number to a conclusion that holds up
No files to touch — omk init scaffolds two skill versions and three cases, and omk eval produces an HTML report plus a one-line verdict in under 5 minutes.
