STAGE: Blog
Frameworks shipping with knowledge: benchmarking agents on one they haven't seen
Next.js 16.3 bundles version-matched docs so agents stop guessing. We ran the same idea on a framework AI has effectively never seen (full disclosure: ours, SpyneJS) — and the result we trust most is the one that went against us.
In June, Vercel shipped something quieter than a feature but even more consequential: Next.js 16.3 now writes a version-matched AGENTS.md block into your project and points coding agents at documentation bundled with your installed version. The block opens with a warning written for machines: this is not the Next.js you know — APIs, conventions, and file structure may differ from your training data.
The largest React framework is telling AI agents, in its own scaffolding, that their memory of the framework cannot be trusted — and shipping the corrective knowledge alongside the code. The Docs are readable as Markdown by appending .md to any URL, and Errors come with copyable fix prompts. The framework has started serving knowledge, not just code.
It names a real problem, and the problem is not just staleness — it is confidence: agents write yesterday's framework with exactly the assurance of today's, and nothing in the prose tells you which you got. Vercel's answer is to anchor generation in served, version-matched knowledge — the model reads instead of recalls.
We have been shipping knowledge for the inverse case. Our framework occupies effectively no space in any model's memory — nothing stale to correct — but the agent doesn't arrive empty: it carries JavaScript, the DOM, a decade of engineering pattern, and that competence is exactly what produces the confident wrong moves. So the knowledge base's job is not correction but direction — channeling a vast general prior into a framework it has practically never seen. Which makes it a clean boundary case: whatever the agent gets right about the framework must have come from the knowledge it was served.
Every framework now faces the same question: how much of what an agent needs to know do you ship, and how much do you leave to guessing?
#Two answers to one problem
Here is the map for everything that follows. One axis: how much training prior a model carries about your framework. The other: whether knowledge is served at generation time. Next.js lives in the top row — enormous prior, now supplemented by served docs. Our experiment lives in the bottom row: SpyneJS, a behavior-first frontend framework in continuous production since 2019 that occupies effectively no space in any model's training memory. Full disclosure before anything else: I built SpyneJS — read every number that follows with that in mind.
The asymmetry is the shape of the evidence. On the Next.js side we have structural evidence: what 16.3 ships, and what we observed porting Vercel's own tutorial application. On the SpyneJS side we have measurements: scored builds, versioned knowledge rounds, and a harness. We did not run Next.js generation through a scoring harness — our suites are SpyneJS-specific — so nothing here is a head-to-head benchmark. What the two rows share is the mechanism under test: served knowledge versus guessing.
And “no prior” is not a slogan; it is a floor we measure — and it means the framework-specific prior. General competence arrives intact; our baseline arm — agents given the tasks with no knowledge base at all — measures what that competence alone produces on a framework the model has effectively never seen. That floor has a number, and you will meet it below: it is the 344.
#External specs are compensation
Step back from frameworks for a moment. The past two years produced a wave of external instruction files — AGENTS.md, CLAUDE.md, spec-driven workflows, rules directories. It is tempting to read these as a new practice. They are not new. They are the senior engineer's mental model, finally written down: where behavior lives, what may touch what, which patterns are load-bearing and which are legacy. Before AI, that spec existed — it was carried in heads, transmitted in code review, enforced in standup. The new workforce cannot attend standup, so the spec had to become a file.
Which raises the question these files quietly concede: external specs are compensating for what the code's structure does not say. The more a framework's architecture encodes its own rules — where things go, what owns what, what is allowed to reach what — the shorter the compensating spec can be, and the more of the remaining knowledge can be served as versioned fact rather than tribal instruction. That is the design bet SpyneJS makes, and it is why it made a good test subject: a framework whose structure is explicit enough that a knowledge base can describe it completely, serving an audience that knows nothing.
The spec was always there. It used to be called knowing the codebase. AI didn't create the need for it — AI created the need to write it down.
#One application, two experiments
We needed an application that was canonical, non-trivial, and indisputably home turf for the incumbent — so we used Vercel's own: the Next.js Learn dashboard, the tutorial app Vercel uses to teach the App Router. We ported it to SpyneJS faithfully — the reference and the port are both public — then had AI agents extend both far beyond the tutorial: search, pagination, record creation and editing, deletion, customers, responsive tables.
Faithful meant verified, not eyeballed. Every ported data predicate was checked 13-for-13 against the reference implementation's own API endpoint, and templates were compared byte-for-byte where the designs coincide. Where this article says the two apps do the same thing, that claim has a test behind it, not a screenshot.
From that one application, two separate experiments — sharing the knowledge base and nothing else. Experiment one, the agentic comparison: frontier agents build and extend both apps, every session audited — qualitative, and the source of every knowledge-base revision. Experiment two, the scripted benchmark: single-shot tasks derived from the knowledge base's own records, run across models and knowledge arms and scored — quantitative, and the source of every number in this article. (Experiment one required frontier models on both frameworks equally; experiment two ran across model tiers.) Keeping them separate is the point: the build comparison is argued from public code, the numbers are measured, and neither borrows the other's authority.
#Experiment one: build, audit, revise
The SpyneJS side ran as an instrumented loop. Claude Code built all twelve builds, on claude-fable-5, under a served knowledge base: the SpyneJS Knowledge Base, a versioned set of operation cards and construction records describing how the framework is actually used. A second agent family — Codex — ran independent audit sessions over the same work. Every session, builder or auditor, ended with a structured self-audit: what went wrong, what evidence, what rule would have prevented it. Audit findings became knowledge base revisions — versioned rounds, shipped like software — not workarounds patched into the app. Twelve builds; six scored rounds; the knowledge base moved from round r1 to r10 over the program.
The most common failure in the audits was not faulty reasoning — it was correct reasoning from stale or missing precedent, and the audit line that kept recurring was some version of: my reasoning wasn't wrong, my precedent was. That is precisely the failure Vercel's warning block anticipates — we just got to watch it happen under instrumentation, and fix the precedent instead of scolding the agent.
Two details from the loop earned their keep. First, the two families' audits independently converged on the same weaknesses — one family building, the other auditing, the same cards and the same hazards flagged from different directions. Second, precision cut both ways: the cards that helped most were the ones precise enough to say no — to refuse a plausible-but-wrong application of a pattern, not merely to describe the right one.
#Experiment two: the numbers — and what we won't claim
Across the measured arms, serving the knowledge base took rule violations from 344 — the baseline arm: no knowledge base served, the model's existing competence working unaided — to 19 on the same task suites — label it before reading it: co-evolved benchmark, not a holdout result. Audit findings drove revisions and revised rounds re-ran the suites, so 344→19 measures how completely agents adhere to served knowledge on documented operations. It is an iteration curve, not proof of generalization; the architectural argument later in this article is argued, not measured. Within that stated scope: recognition — whether an agent identifies which documented operation a task calls for, as assessed by our judge model — reached 67 of 70 with the full knowledge base served, measured on locked prompt-variant assignments (close paraphrase, natural request, underspecified), a caveat that ships with the number because the variant mix moves it. On feature-level acceptance checks, full-knowledge arms passed 100%.
- The two experiments: experiment one — the agentic loop (twelve builds, six scored KB rounds r1→r10; Claude Code on claude-fable-5 built; Codex audited independently); experiment two — the scripted API benchmark. The numbers above come from experiment two.
- Denominators: baseline 344 violations over 435 attempts; full-stack 19 over an identical 435 (413 completed generations; 22 provider/parse failures, which carry zero violations by construction and are counted in the denominator); L1 per arm — sonnet 30/30, gpt-low 30/30, gpt 11/11 (one gpt attempt unscored: parse failure, counted as an attempt).
- Models and semantics: claude-sonnet-5 (alias pin, live-verified 2026-07-22 — no dated snapshot exists, so we say so) and gpt-5.5-2026-04-23 at default and low effort; one independent generation per attempt, k per cell, no auto-retries, failures counted.
- The receipts: violation taxonomy, harness scoring rules, judge-model prompt, and the per-cell results table are in the benchmark protocol — published with this article, not after it, pinned at the commit that scored these numbers. A self-test ships with it: clone anonymously, run it, and watch the planted violations get caught.
- What was not done: no untouched holdout suite existed in this program — that is exactly what the standing re-test adds next — and no Next.js arm was scored.
#The result we trust most is the one against us
We handed agents six probe tasks the knowledge base deliberately does not cover — with an explicit affordance to say this isn't documented. Across 29 judged attempts, they honestly declined in exactly 4; the rest improvised, confidently — one improvisation was a competent nine-file drum machine wearing the framework's idioms without the framework's backing. The probes, per-model outcomes, and the honest declines are published — task prompts verbatim, outcomes per model, as a failure finding, not a success. Served knowledge dramatically improves what agents build. It does not yet make them honest about the edges of what they know — and we are redesigning for citation-first output because of it.
Beyond that, the refusals are simple: nothing here is a general better-than-Next.js result, a productivity multiplier, or evidence about your codebase. One application, our harness, our suites.
Serving knowledge fixed what agents build. It did not fix what agents admit they don't know. Both results are in the data, so both are in the article.
#Where the two frameworks diverged
The port and experiment one surfaced a structural difference — and it deserves both readings. Extending the Next.js app beyond the tutorial meant, in several places, crossing the model's boundaries: 'use client' islands wired through useSearchParams/usePathname/router.replace for live search, and cache coherence restored by hand with revalidatePath after mutations. A Next.js practitioner will say — correctly — that this is the model: the server/client boundary is the design, and explicit invalidation is a documented, first-class mechanism. Our read, stated as a read: in this app they functioned as escape valves — points where the framework asks you to leave one model to accomplish the feature, then reconcile the two by hand. Both descriptions fit the same commits; the repos are public so you can apply your own.
The same features on the SpyneJS side were expressed inside the vocabulary the application already had: search and pagination ride the existing route channel with one URL-writing view, so bookmark, back button, and keystroke arrive by one path; deletion disposes rows by absence from the server's answer, keeping the server the authority. New behavior, once expressed, was available to the whole application through the same channels everything else uses — no second model to reconcile with the first.
The audits supply the mirror image: the SpyneJS side was not friction-free. Agents mis-placed state, mishandled replayed events, got ordering wrong in ways the audit trail records. The difference was where the fixes went — every correction landed in the versioned knowledge base, raising the floor for the next build; none landed as an app-level workaround. Escape valves patch the application. Our failures patched the knowledge.
And symmetry demands the costs that are ours by design, not by agent error: SpyneJS has a fraction of Next.js's ecosystem, no server-rendering or streaming story, an RxJS-shaped learning curve, and a production history concentrated in far fewer organizations. If your constraints live there, the divergence above should not move you.
#A standing test, not a verdict
One number in this article will be wrong soon, and that is the point: the knowledge base is versioned, the task suites derive from it, and this test re-runs as both evolve. The harness, prompts, and scoring methodology are public in the benchmark protocol; future rounds — including, if the suites can be made framework-fair, a scored Next.js arm — will publish against the same ladder of claims, negative results included. Vercel ships knowledge with Next.js now. So do we. The interesting race is not between frameworks — it is between served knowledge and guessing. And the number we will be chasing hardest next is not 19; it is 4 of 29.
If the experiment itself interests you, the application shell that the agents built against is a command away — npx spyne-cli create-app — and the knowledge base the agents read is on npm as @spynejs/kb. Read the docs, or start from the examples. Everything claimed above links to something you can run.
#Added: the companion measurements
Added Aug 19, 2026An early reader correctly noted that comparison figures I had summarized in correspondence did not appear in this article. They belong to a different instrument than the scripted benchmark above, and they are published here, scoped as what they are.
These are not scored-generation results — the scripted benchmark still has no Next.js arm, exactly as stated above. They are two simpler kinds of measurement from the build program: runtime latency probes, identical instrumentation injected into both live deployments (same session, same machine, live payment simulation active on both throughout, settlement detected by mutation-observer on the rows container, 20–24 samples per interaction (cold-load: 20 and 12; see the receipts), reported p50/p95), and program-level agent-token totals from the twelve builds.
- Sort click → table settled: Next.js 132/181 ms, SpyneJS 24/200 ms (p50/p95) — ~5× at p50 to SpyneJS; p95s comparable under live-feed churn.
- Deep page jump → settled: Next.js 130/294 ms, SpyneJS 58/201 ms — ~2× at p50 to SpyneJS.
- Status toggle → first row change: Next.js 11/14 ms, SpyneJS 68/93 ms — ~6× to Next.js. Both sit under the ~100 ms perceptual-instant threshold; React's optimistic microtask flush paints sooner than our channel→replay→render pass, and the mechanism is the honest explanation.
- Cold load → first data: Next.js 115/145 ms (server-rendered rows present), SpyneJS 772/1191 ms (SPA bootstrap-complete; rows render one frame later) — ~6.7× to Next.js. These are architecture-appropriate proxies, not one metric: the SSR document and the SPA bootstrap are different events, and we report them as such.
- Agent tokens: the SpyneJS side cost ~1.9× aggregate tokens across the program — the price of building on a framework the models have never seen, on the incumbent's home terrain, and we state it plainly rather than netting it against anything.
The summary sentence we stand behind: in-page data interactions settle ~2–5× faster on the SpyneJS side; first paint of data on cold load and the perceived toggle flip are faster on the Next.js side — each architecture wins exactly where its model predicts. The probe code, method notes including discarded measurement approaches, and raw sample arrays are published in the benchmark protocol's comparison directory.