Spec Conformance Is the Metric Benchmarks Skip

SWE-bench Verified is the number every vendor leads with, and by August 2026 the frontier model on that leaderboard was clearing it at 95.0 percent, with SWE-bench Pro at 80.3 percent [1]. That number says an agent can produce a patch that makes a test suite pass on a curated set of GitHub issues. It says nothing about whether the code an agent just wrote in your repo matches what your team meant when it filed the ticket, and that gap is where most AI-generated code actually fails.

The benchmark treadmill

Benchmarks for coding agents have multiplied past SWE-bench precisely because pass or fail against a fixed test suite misses real failure modes. A recent survey of ten open-source benchmarks lists Terminal-Bench for shell fluency and Senior SWE-Bench for maintainability and design judgment, among several others built to catch what the original SWE-bench cannot see [2]. Each new benchmark is an admission that the previous one measured something narrower than correct code.

None of them measure conformance to a specific project's stated intent, because none of them have access to it. SWE-bench grades against the fix that was eventually merged for a real issue. Senior SWE-Bench grades against senior-engineer judgment calls. Both are proxies for the acceptance criteria a team would have written down if it had a spec in the first place [2].

What a passing score hides

Teams running agents at scale are seeing the proxy gap show up as review debt. One verification guide's synthesis of 2026 industry data found that under high AI adoption, median pull request review time rose five times and incidents per pull request tripled, while PR size grew 51 percent [3]. The same guide cites a 2,500-review, 3.2-million-line study showing human defect detection collapses past 400 lines, close to the size of an average AI-authored PR now under review [3].

Model quality has not closed the gap. Syntax pass rates have climbed toward 95 percent across recent model generations, but security pass rates have stayed flat at 45 to 55 percent, and one cited study puts average package hallucination at 19.6 percent across sixteen models [3]. Newer models write code that looks more finished. They are not, by these numbers, writing code that is more correct.

DeviQA's 2026 survey of QA teams describes the same blind spot from the other direction. AI-assisted code is now the default assumption on the QA floor, and low security-defect counts get read as proof the code is safe when they are really proof that QA is doing exactly the job it was configured to do [4]. Requirement fidelity, whether the implementation does what the ticket asked and not just what its own tests check, is the failure mode neither the benchmark nor the QA pipeline is built to catch, because the agent that wrote the code often wrote the tests too [4].

Specs turn looks right into is right

This is the argument for treating a specification as an executable contract instead of documentation that trails the code. One spec-driven development guide published in late August frames the shift directly: traditional specs are read by humans, spec-driven specs execute as validation gates, and the build fails when the code diverges from what is written [5]. That is a categorically different check than the tests pass. A test suite verifies whatever the agent decided to test. A spec conformance check verifies whether the criteria a human actually wrote down were met.

The regulatory timeline is making this less optional. The EU AI Act Omnibus entered into force on July 27, 2026 and pushed back high-risk compliance deadlines, with standalone high-risk systems coming into scope on December 2, 2027 and AI embedded in regulated products by August 2, 2028, and fines up to fifteen million euros or three percent of global turnover for breaches [5]. Once a spec can be pointed to as compliance evidence, a green CI pipeline stops being an adequate answer to how a team knows its system does what it is supposed to do.

What a spec check does that a benchmark can't

• A verdict per acceptance criterion, met, unmet, or uncertain, instead of one aggregate green checkmark
• Code with no governing spec flagged as drift, not silently merged as passing
• Deterministic evidence first, test results and file diffs, with LLM judgment used only where that evidence is missing
• Run on every pull request in CI, not benchmarked once per model release

acceptance_criteria:
  - id: AC-3
    text: "Refund endpoint rejects orders older than 90 days"
    verdict: unmet
    evidence:
      test: "tests/refunds_test.py::test_stale_order_rejected — FAILED"
      file: "src/refunds.py:42 — no date check present"
  - id: AC-4
    text: "Refund endpoint logs a reason code on rejection"
    verdict: uncertain
    evidence:
      note: "no deterministic test covers this; judged against src/refunds.py:58"
drift:
  - file: "src/refunds_export.py"
    reason: "no approved spec in specs/ governs this file"

onspec doesn't write the spec and it isn't a test runner. It reads whatever acceptance criteria a team already committed to specs/, anchors each one to test results and file evidence, falls back to an LLM judgment only where deterministic evidence is missing, and flags the code no approved spec covers. It closes the gap between the benchmark passed and the spec was met, nothing more.