omkDid the knowledge change really make it better?

Measurement-grade evaluation & iteration for what you feed an LLM — prompt, RAG, skill, agent. Hold the model fixed, vary only the knowledge, and get statistically comparable diagnostics instead of vibes.

Quickstart

Why measurement

GitHub

Three stages — doctor / eval / observe

doctor gates suite health before you spend a cent; eval scores knowledge variants through a five-layer pipeline; observe tracks runtime regressions. One framework, three jobs.

Comparable, not vibes

Bootstrap CIs on the difference, judge-ensemble agreement, length-debiasing. A reported delta means a real difference within measurement noise — not a number that moved.

evolve — keep only proven gains

Propose an edit, gate it on a statistically significant held-out improvement, and report an unbiased score on a locked test set. Refuses gains indistinguishable from judge noise.

Agent Skill onboarding

Install the official omk Agent Skill with npm plus `omk install omk-agent-skill`, then drive doctor, eval, sample, evolve, observe, and studio from natural language inside your coding agent.