# Scaling the Harness: Auditing My Personal Agent Against a Berkeley Paper *A new paper argues the next bottleneck in agentic AI is the system around the model, not the model. So I handed it to my agent and asked it to audit itself.* > arXiv:2605.26112 says agent performance now comes from six system components around the foundation model: memory, context construction, skill routing, orchestration, and governance. I ran the paper through my personal agent setup as a self-audit. Verdict: half the paper was already implemented through months of small adoptions, two named failure modes (stale-but-confident memory, confident-but-unchecked skills) were real gaps, and both got fixed the same afternoon by the agent itself. This is the full process: paper to audit to shipped enhancements to this post. Published: 2026-07-10 · Reading time: 6 min · Tags: ai, ai-agents, claude-code, personal-agent, agentic-ai, harness, context-engineering, memory, skills, system-design, arxiv, automation Canonical URL: https://huytieu.com/blog/scaling-the-harness/ Author: Huy Tieu (huytieu.com) --- > **TL;DR**: I gave my personal agent a fresh arXiv paper ([Scaling the Harness in Agentic AI](https://arxiv.org/abs/2605.26112)) and one instruction: map it against your own setup, adopt what is genuinely better, then write about it. What happened: > - The paper's claim: agent capability now comes from the **harness** (memory, context, skill routing, orchestration, governance), not just the model. Six components, three named failure modes. > - The audit found my vault already covered ~half of it, accumulated through months of stealing patterns from other people's repos. > - Two real gaps matched the paper's two scariest failure modes: **stale-but-confident** memories and **confident-but-unchecked** skill outputs. > - The agent shipped fixes for both (a new `/memory-hygiene` skill and a post-condition rule), wrote the audit report, and then wrote this post. > - Takeaway: you don't need to build a research harness to benefit from this paper. Treat it as a checklist and audit whatever agent setup you already run. ## One prompt, whole loop This post exists because of a single message I typed into [Claude Code](https://claude.com/claude-code) this morning: > Try: "from this paper, map to the current problem/situation of the current setup and historical working, map out the concept that is useful and enhancement needed. then enhance, also update or write a new blog about that process. https://arxiv.org/pdf/2605.26112" That's it. Read a paper, compare it against a living system, decide what to adopt, apply the changes, document everything. A year ago this was a weekend. Today it is one prompt, because the system around the model already knows how to do each step. Which, funnily enough, is exactly what the paper is about. ## The paper in one diagram [From Model Scaling to System Scaling: Scaling the Harness in Agentic AI](https://arxiv.org/abs/2605.26112) (Shangding Gu, UC Berkeley, May 2026) argues that once models cross a capability threshold, long-horizon agent performance is decided by the **harness**: the structured system layer around the foundation model. The paper decomposes an agent into six components: ``` user / task | +--------- orchestration (O) ----------+ | | | memory --> context --> [ MODEL ] --> skill | | (M) builder (C) (R) router (S) | | | +------- governance / verify (G) ------+ | environment ``` Model scaling improves R. Everything else is system scaling. The author backs it with a comparison of Claude Code, OpenClaw, and his own Python reference harness [CheetahClaws](https://cheetahclaws.github.io), and names one failure mode per bottleneck: ``` bottleneck failure mode system move ------------------ --------------------- ---------------------------- context governance exposure without access minimum sufficient context trustworthy memory stale-but-confident re-verify at retrieval skill routing confident-but-unchecked post-condition checks ``` Those three failure-mode names are the best thing in the paper. They are precise enough to grep your own system for. ## The audit: my vault vs six components My setup is a [terminal-first personal agent](/blog/how-i-moved-my-entire-life-to-the-terminal-building-a-personal-ai-agent-system/): an Obsidian vault as the database, Claude Code as the runtime, ~70 skills, a persistent file-based memory, and a nightly self-evolve routine. The agent mapped each paper component to what actually exists here: ``` paper component my implementation verdict ------------------- ---------------------------------- -------- R reasoning Claude (model routing: Opus thinks, n/a Sonnet executes) M memory ~/.claude memory files + index GAP C context builder worker-writes-file rule, terse ok index, lazy-loaded skill resources S skill router 70+ skills, description matching GAP O orchestration lead session + typed worker agents ok G governance propose-only gates, review gates, partial verifier passes ``` The interesting part is *why* the "ok" rows were ok. None of them were designed from a framework. They accumulated: the worker-output-to-file rule came from a painful 13-minute agent transcript, fresh-context isolation came from dissecting someone's repo where a 5x speedup shipped hallucinated findings, the verifier pass came from the same place. The paper calls these context governance and communication fidelity. I called them "stop doing the thing that burned me." Same destination, and honestly the scar-tissue route teaches harder. ## Gap 1: stale-but-confident memory The paper's sharpest observation: memory failure is asymmetric. A stale memory rarely fails to be *retrieved*. It fails by being retrieved **confidently** after the world changed. ``` write: "webhook lives at /publish-social" [2026-05] | (env changes: webhook dies) [2026-07] | recall: "webhook lives at /publish-social" <- still ranks high, agent acts on it still sounds sure ``` I have felt this exact bug. My memory store holds ~150 facts: file paths, Slack channel IDs, Jira board numbers, routine IDs, API endpoints. Some are wrong right now and I don't know which. The n8n webhook my blog pipeline "remembers" was already flagged INACTIVE once; the memory that recorded the flag is itself aging. The paper's move: make trust a runtime decision, not a property of the stored item. Verify against the live environment, and carry per-entry confidence and recency as first-class fields (that part is lifted from the CheetahClaws memory design). What the agent shipped: a `/memory-hygiene` skill. Monthly sweep, and it's deliberately cheap: - Split each memory's claims into **environment-dependent** (paths, URLs, IDs: verifiable) vs **preference** (tone rules, people facts: not verifiable, only checkable for contradictions). - Verify the first bucket with one-call checks (`ls`, `curl -sI`, `gh repo view`). Hard cap of a minute per memory. Hygiene, not investigation. - Stamp `last_verified` and `confidence` into each file's metadata. - Drifted facts get corrected in place; obsolete memories get **proposed** for archive, never silently deleted. - Each sweep ends in a scorecard (verified / unverifiable / drifted / archived) with deltas against the previous sweep, because one sweep tells you the state and the trend tells you whether your memory is getting better or quietly rotting. The paper calls this longitudinal evaluation; the whole section 5.2 boils down to "measure the trend, not the episode." ## Gap 2: confident-but-unchecked skills The symmetric failure on the action side: a skill or subagent returns plausible output that nothing downstream validates. As skills multiply, the paper notes, your failure mode shifts from a *missing* capability to a *present-but-unverified* one. With 70+ skills in this vault, that line hit home. Some of my skills already verified themselves, again from scar tissue. My [dogfooding pipeline](/blog/testing-katalon-true-platform-with-an-ai-agent/) cross-checks the AI test runner's self-reported verdict against Playwright ground truth, because I learned early that [a passing demo can lie](/blog/the-demo-that-lied/). The blog pipeline curl-verifies the live URL after deploy. But that discipline lived inside individual skills. Nothing forced a *new* skill to have it. Now it is a standing rule in the vault's constitution (CLAUDE.md): ``` mutate -------------------> report "done" BEFORE (hope) mutate --> observe artifact --> match? --> report AFTER (rule) (curl the URL, | re-fetch the no --> report the issue, screenshot failure, plainly the post) ``` The rule's key clause: the check must observe the **artifact**, not the tool's return value. A 200 from the deploy API is not the page being live. A "message sent" response is not the post rendering with its link card. Fetch the thing back. ## What I refused to adopt My adoption skill has one core principle, learned the hard way: **judge, don't copy**. Every idea gets compared head-to-head against what already exists, and adopting requires being strictly better on a named dimension. Two of the paper's ideas lost: - **Skill routing as a learned policy** (online subtask estimates, confidence-aware escalation). At single-user scale, description matching plus a static model-routing table works fine. A learned router is maintenance cost with no traffic to learn from. - **Building a reference harness and benchmark suite.** That is a research-lab deliverable. The useful residue for me is the metric vocabulary (trajectory quality, verification cost, pass^k for flakiness), which I filed for my day-job work evaluating AI test agents, where "the test passed once" is exactly the reliability trap the paper describes. Rejections get written down with reasons, so future-me doesn't re-litigate the same paper from scratch. ## The meta part Here is what I keep chewing on. The process that consumed this paper *is* the paper's thesis running live: ``` paper (pdf) --> [skill: /lizard] ......... S: routed to the right procedure --> reads own memory index .... M: what do we already do? --> compares head-to-head ..... C: minimum sufficient context --> ships 2 changes ........... O: multi-step, no hand-holding --> audit report + this post .. G: every change is reviewable ``` The model did the reading and the judging. But the reason one prompt was enough is that a skill already encoded the adopt-or-reject procedure, a memory already knew the house rules, and a governance habit already demanded the audit trail. Swap in a smarter model next year and all of that still has to exist. The vault got measurably better today: two failure modes the paper names are now guarded against, and the fix for one of them (the post-condition rule) hardens every future skill by default. That compounding is the point. Model scaling I can only wait for. **System scaling is the part I get to do myself.** If you run any kind of persistent agent setup, skim [the paper](https://arxiv.org/abs/2605.26112) with three grep queries in mind: where am I stale-but-confident, where am I confident-but-unchecked, and what is my minimum sufficient context? You almost certainly have all three bugs. I did.