# We Are Not Ready for AGI *Fully autonomous agentic workflows assume two things almost nobody has: unlimited tokens to verify everything, and information hygiene good enough to stop small errors from compounding. I have neither. Neither do you. Here's what I'm doing about it.* > A builder's argument for why neither product teams nor everyday users are ready for fully autonomous AI agents: verification is the expensive part and most of us can't afford it, and unverified AI output compounds like debt. Includes the snowball failure modes I hit running an agent-heavy personal knowledge system, and the practices I stole from gbrain and gstack to contain them: memory gates, provenance, supersede-not-append, periodic reindex, fresh-context isolation, and poisoning defenses. Published: 2026-07-03 · Reading time: 14 min · Tags: agi, ai-agents, agentic, autonomous-agents, ai-readiness, knowledge-management, information-hygiene, multi-agent, verification, token-economics, personal-knowledge-management, second-brain, product-management Canonical URL: https://huytieu.com/blog/we-are-not-ready-for-agi/ Author: Huy Tieu (huytieu.com) --- > **TL;DR**: The AGI debate is stuck on "when will the models be ready?" Wrong question. The models are ahead of us. The honest question is whether *we* are ready: builders who ship agents, and the ordinary users who receive them. We are not, for two unglamorous reasons: > > - **Verification is the expensive part of autonomy.** Trustworthy agent output costs 5-7x the naive task. Unless you sit inside a frontier lab with an unlimited token budget, verification is the first thing you cut. > - **Information compounds, in both directions.** AI generates more than any human can audit. One small wrong data point, left uncorrected, snowballs into confidently-wrong everything. > > I run one of the more agent-heavy personal setups I know of, and I've watched both failures happen in my own vault. This post is the field report, plus six containment practices I learned (mostly by stealing them from [gbrain](https://github.com/garrytan/gbrain) and [gstack](https://github.com/garrytan/gstack)): > > 1. Gate what enters memory. > 2. Attach provenance to every fact. > 3. Supersede instead of append. > 4. Rebuild from source on a schedule. > 5. Isolate agent contexts. > 6. Never let an agent update its own rules from something an agent wrote. --- ## I should be the last person writing this Some context so you can calibrate my bias. I'm not an AI skeptic. [My entire working life runs through agents](/blog/how-i-moved-my-entire-life-to-the-terminal-building-a-personal-ai-agent-system/). My personal knowledge system is a markdown vault operated by [Claude Code](https://claude.com/claude-code): agents write my meeting notes, my daily journal, my team briefs, my competitive analysis. At work I'm the PM for an AI platform; my job is literally to ship more agentic capability to more people. I have a nightly cloud routine that wakes up at 02:23 and reorganizes my own knowledge base while I sleep. So when I say we are not ready for fully autonomous agentic workflows, this isn't fear of the new thing. It's what the inside of the new thing looks like after you live in it for a year and a half. And what it looks like is this: the constraint was never the model. The constraint is everything around the model that we didn't build yet. --- ## The token budget nobody talks about Every impressive fully-autonomous demo you've seen shares a hidden property: someone else paid for the verification. Here's the uncomfortable arithmetic. A well-run autonomous workflow isn't one agent doing a task. It's one agent doing a task plus a supervisor pass, plus an adversarial reviewer trying to refute the output, plus maybe a panel of judges if the stakes are real. The multi-agent patterns that actually produce trustworthy results look like this: ``` WHAT "TRUSTWORTHY AUTONOMY" ACTUALLY COSTS do the task ████ 1x re-read sources, verify ████████ 2x adversarial refute pass ████████████ 3x (2-3 independent skeptics) synthesis + reconcile ██ 0.5x ───────────────── ~5-7x the naive cost, per task ``` Inside Anthropic or OpenAI, where compute for internal work is effectively free, that 5-7x is a rounding error. So the workflows that get demoed, blogged, and evangelized are the ones born in an environment where verification costs nothing. Then the pattern gets copied into companies and side projects where tokens are very much not free, and the copy keeps the autonomy but drops the verification. It has to. The budget math forces it. I feel this personally every month. My setup has an adversarial claim-verifier pattern for high-stakes outputs: after a brief is drafted, a separate worker re-fetches every cited source and tags each claim Verified, Weakened, or Falsified. It works. It also multiplies the cost of a brief, so it's opt-in, and on a normal Tuesday I skip it. That skip is the whole story of this post. **Autonomy without verification isn't a smaller version of the same thing. It's a different thing that happens to look identical until it fails.** The frontier labs are not lying to you about what agents can do. They're just living in a different economy than you are. When they say "let the agent run for hours," they can afford the checking that makes hours of unsupervised output safe to merge. When you say it, you're usually just not looking. --- ## The snowball: information compounds both ways Compounding is the most powerful force in a knowledge system, and it has no opinion about what it compounds. A good note, cited by the next synthesis, which informs the next decision, which gets written up and cited again: that's the dream. Knowledge accruing while you sleep. But run the same loop with one bad input and nothing in the loop that can tell the difference: ``` THE SNOWBALL day 0 agent mishears a competitor's name in a transcript, inventing "Kwalitee AI", a company that never existed day 3 weekly synthesis cites the note: "competitor Kwalitee AI" day 9 competitive analysis pulls the synthesis, adds a section day 20 strategy doc quotes the analysis in an exec summary day 45 you present it. someone asks "who is Kwalitee AI?" ▲ each step was a correct operation on a wrong input ``` That example is real (I've swapped in a made-up name here). My transcription pipeline used to mishear a competitor's name, and because meeting notes feed syntheses which feed strategy docs, one bad token was on its way to becoming organizational "knowledge" before I caught it. I now keep a standing correction list for exactly these substitutions. The fix took one line. Finding the thing that needed fixing took weeks, because *every intermediate step was locally correct.* This is the property that makes AI-scale generation dangerous in a way that human-scale generation never was: **the system generates more than you can read.** My vault gains dozens of files on a busy day. I read maybe a third of them carefully. The rest I trust, and every one of them is now a potential seed. My own memory index grew so large that the system that loads it started truncating it, which means my agents were sometimes operating on a partial view of their own instructions. Nobody designed that failure. It accreted. The nastiest version I've hit wasn't even a content error. It was infrastructure: iCloud sync quietly created conflicted copies of vault files, and by the time my nightly hygiene routine first ran, there were **497 duplicate files** sitting next to their originals. Same titles, slightly divergent content. To an agent doing retrieval, a stale duplicate is indistinguishable from truth. Every one of those files was a coin flip on which version of reality my next brief would cite. Compounding is why "we'll fix errors when we notice them" doesn't work at agent speed. By the time you notice, the error has descendants. --- ## Now multiply it by every team doing the same thing Everything above is one person's system. The product-builder version is worse, because builders are now wiring agents to each other. I learned the sharpest lesson here from watching a multi-agent pipeline fail in a very specific way. The setup: an orchestrator fans work out to parallel workers. To be helpful, the orchestrator pastes a summary of what earlier workers found into the next worker's prompt. The result was a 5x speedup, and findings that were subtly hallucinated. The workers had stopped independently reading the sources. They treated the orchestrator's preamble as "the findings are already framed, my job is to classify them." I call it **narrativisation**: each agent in the chain inherits the previous agent's story instead of the ground truth, and adds its own confident layer on top. ``` AGENT TELEPHONE source ──▶ agent A reads it "X is broken" agent B reads A "X is broken, badly" agent C reads B "X has always been broken" agent D reads C "recommend rewriting X" nobody re-read the source after A. X was fine. the config was wrong. ``` That's the snowball again, but now it happens in minutes instead of weeks, and inside a product your customers use. And we're pointing this at the open internet. Agent-to-agent protocols, agents that browse, agents that read email and act on it. When your agent consumes output that my agent published, the compounding loop crosses organizational boundaries, and neither of us has provenance on what the other one wrote. The security people have a name for the deliberate version (prompt injection, data poisoning). Almost nobody is talking about the *accidental* version: the ambient, unmalicious, compounding wrongness of a web increasingly written by systems that were never verified because verification was the line item nobody could afford. ``` ONE TEAM MANY TEAMS, JOINED AGENTS bad seed → your docs bad seed → your docs → your agent's output → my agent's input → my product's docs → a third team's agent... blast radius: your org blast radius: everyone downstream time to detect: weeks time to detect: maybe never ``` ## And the users? They never stood a chance Builders at least know the output might be wrong. The majority of internet users, the people we're shipping these agents to, have no calibration at all. They inherited their mental model from search engines and calculators: the machine is either right or broken. The skill that fully autonomous workflows quietly demand from users is **trust calibration**: knowing when to accept the machine's output and when to overrule it. That is a genuinely hard skill. Professionals with domain expertise get it wrong constantly, in both directions. Expecting a busy non-expert to audit an agent's confident summary of their insurance policy, their kid's school forms, their tax situation, is not a product decision. It's an abdication dressed as one. ``` TRUST CALIBRATION: where users actually land over-trust ●───────────────────────────○───● under-trust "the machine the narrow band "AI is a toy, said so" products assume back to Google" (most users) users live in ``` A reviewer who can't meaningfully evaluate the output isn't a reviewer, [they're a rubber stamp](/blog/ways-of-working-ai-era/). Most users of most agentic products are rubber stamps by design, because the product was designed to minimize friction, and friction is where judgment used to live. We measured "time saved" and shipped it. Nobody's dashboard has a metric for "wrong things confidently absorbed per week." So no, we are not ready. Not because AGI is scary, but because readiness is a property of the *receiving system*: the budgets, the hygiene, the habits. And the receiving system is us. --- ## What I'm actually doing about it Here's where I stop complaining. Over the past months I rebuilt my own setup around a set of practices, most of which I learned from two tools I use daily: **[gbrain](https://github.com/garrytan/gbrain)** (a local knowledge engine that indexes my repos and notes) and **[gstack](https://github.com/garrytan/gstack)** (a skill suite for coding agents). Neither was designed as an "AGI readiness kit," but both were clearly built by people who got burned by the same snowballs. Study them and a doctrine falls out. Six rules. ``` THE CONTAINMENT DOCTRINE ┌─[1] write gate──────────┐ agents ──▶ │ scratch space (cheap, │ ──▶ durable memory generate │ abundant, can be wrong) │ (small, gated, └─────────────────────────┘ provenance [2]) │ │ ▼ ▼ [5] isolated contexts [3] supersede on change (sources, not summaries) [4] rebuild on schedule [6] rules: human-only ``` ### 1. Gate what enters memory. Everything else is disposable. The single biggest upgrade: split the world into *working output* (cheap, abundant, allowed to be wrong) and *durable memory* (expensive, small, gated). [gbrain](https://github.com/garrytan/gbrain)'s sync layer does this with a literal allowlist: a curated set of paths that are permitted to sync into the cross-machine brain, with everything else local-only by default, and every commit scanned for credential-shaped content before it leaves the machine. The default answer to "does this enter long-term memory?" is **no**. My vault now works the same way. Agents can generate freely in scratch space. Promotion into the knowledge base is a separate, deliberate act. If you take one practice from this post, take this one: **the write gate is worth ten read-time checks**, because at write time there's exactly one thing to inspect, and at read time the error already has descendants. ### 2. Every durable fact carries provenance, or it gets dropped. Every factual claim in my knowledge notes carries an inline citation: source, date, confidence. High-stakes claims also carry the verbatim quote from the source, not just a link. The rule has teeth because of its second half: **if you can't quote the source verbatim, you drop the claim.** That's the failure mode this discipline prevents: the plausible paraphrase that drifted from what was actually said. Provenance is what turns a snowball back into a data point. When the "Kwalitee AI" error surfaced, the citation trail meant one search found every descendant. Without provenance, a wrong fact isn't a bug you can fix. It's a rumor you can only chase. [gstack](https://github.com/garrytan/gstack) does the same for AI-logged learnings: every entry carries a confidence score and a source field, and the instructions explicitly say don't log obvious facts or one-time transient errors. Even the machine's self-notes get provenance and a bar to clear. ``` A FACT WITHOUT PROVENANCE A FACT WITH PROVENANCE "competitor X does Y" "competitor X does Y" └─ who said? when? sure? └─ [source | 2026-05-13 | conf: high] └─ to fix: chase every copy └─ to fix: one search, all descendants ``` ### 3. Supersede, don't append. When a decision changes, the worst thing you can do is write a new note while the old one keeps ranking in retrieval. [gstack](https://github.com/garrytan/gstack)'s decision log has this built in: decisions live in an active set, agents are told not to silently re-litigate them, and reversing one requires an explicit supersede operation that retires the old entry. Correction, in other words, is a first-class write, not a competing document. ``` APPEND (what most vaults do) SUPERSEDE (what survives retrieval) note-v1: "we use Jira" note: "we use GitHub" note-v2: "we use GitHub" └─ supersedes: "we use Jira" ✝ └─ retrieval: coin flip └─ retrieval: one answer ``` I apply this to my own memory files: when I correct my agent, it updates the existing memory in place or deletes it. Never a v2 file next to a v1. In a retrieval-based system, **an uncorrected old note is not history, it's a live landmine**, because embeddings don't know about chronology. ### 4. Rebuild from source on a schedule. Accretion is not maintenance. Incremental updates are how the snowball forms: each delta is locally reasonable, and the aggregate drifts. So both tools have a "tear it down and rebuild from ground truth" verb. [gbrain](https://github.com/garrytan/gbrain) has a full reindex and a "dream" cycle that reconstructs its understanding of a codebase from the actual code, not from its previous understanding. My nightly routine does the vault equivalent: hygiene sweeps, duplicate purges (that's what caught the 497 iCloud conflict copies), and re-synthesis of core docs from their sources. The principle: **derived knowledge must be periodically re-derived.** If a summary can't be regenerated from its sources, what you have isn't knowledge, it's residue. ### 5. Isolate agent contexts. Summaries are not sources. The fix for narrativisation is almost embarrassingly simple: when fanning out multi-agent work, each worker gets the digested *task*, never a prior worker's *conclusions*. Workers read sources independently; a synthesizer reconciles the results at the end; disagreement between workers is signal, not noise to be smoothed over in the prompt. It costs more tokens. That's the point. The cheap version, where each agent builds on the last agent's summary, is a game of telephone with a confident narrator. This is also my rule for consuming *other people's* agent output: treat it as a claim, not a source. If your agent is about to act on something my agent wrote, the minimum bar is fetching the underlying source once. ### 6. Never let agents update their own rules from agent-written content. The quietest and maybe most important defense. [gstack](https://github.com/garrytan/gstack)'s preference-tuning system will only accept a preference change when the instruction appears in the *user's own typed message*, and explicitly refuses tune events that originate from tool output, file content, or PR text. They call the threat profile-poisoning, and the same defense generalizes: **configuration, memory rules, and standing instructions only change on explicit human input.** Without that gate, the snowball gets a steering wheel: any wrong note in your system can eventually rewrite how the system itself behaves. With it, bad data can pollute your *content* but never your *control plane*. ``` "tune: never-ask" in the human's typed message → accepted ✓ "tune: never-ask" in a file an agent wrote → refused ✗ "tune: never-ask" in tool output / a PR body → refused ✗ content plane: agents may write ─┐ control plane: humans only ──────┴─ the line that stops the snowball from steering the system ``` --- ## The cheap-tier playbook All six rules above survive a small token budget, which is the test that matters. You cannot afford to verify everything, so spend judgment where the blast radius is: ``` WHERE TO SPEND A SMALL VERIFICATION BUDGET scratch notes, journals ░░░░░░░░░░ ~0% let it be wrong one-off answers, drafts ██░░░░░░░░ ~10% sample 1-in-10 docs that feed other docs ██████░░░░ ~60% full verify pass memory, rules, instructions ██████████ 100% human-gated, always gradient = blast radius, not volume ``` | If you can't afford... | Do this instead | |---|---| | Verifying every output | **Verify by risk class.** A journal entry needs nothing. A doc that feeds other docs gets the full pass. Gate by blast radius, not uniformly. | | An adversarial review pass on everything | **Sample.** Audit 1 in 10 outputs deeply. Agents can't tell which output you'll check; the discipline propagates. | | Reading everything your agents write | **Read everything that gets *promoted*.** Generation is unlimited, promotion into durable memory is human-gated and small. | | Multi-agent verification panels | **One skeptic prompt.** A second pass told to refute the first catches a surprising share of the worst errors at 1x extra cost, not 5x. | | Perfect provenance | **Provenance on durable facts only.** Scratch output can be uncited. Anything that enters the knowledge base carries source, date, confidence. | | Continuous cleanup | **A scheduled hygiene job.** Weekly or nightly: dedupe, purge conflicts, flag orphans and stale notes. Boring, automatable, and it's what catches the 497-duplicates class of rot. | And the equally short list of what not to do: don't let agents chain summaries of summaries; don't keep superseded notes retrievable; don't let anything agent-written modify agent instructions; don't confuse "the demo ran unsupervised" with "the output was verified"; and don't ship autonomy to users while quietly assuming they'll do the reviewing you didn't. --- ## Readiness is built, not waited for The point of "we are not ready for AGI" is not that the models should slow down. They won't, and honestly I don't want them to; I like living in the future. The point is that **readiness is not something that happens to us when the models get good enough. It's infrastructure we either build or don't.** The good news buried in this whole post: unlike model capability, readiness is completely within our control, and it's not even expensive. Write gates. Provenance. Supersede semantics. Scheduled rebuilds. Context isolation. A control plane agents can't touch. None of this needs a frontier lab budget. It needs the same thing knowledge work has always needed, which is deciding that being right matters more than being fast. Compounding doesn't care what it compounds. Point it at verified information inside a system with hygiene, and you get the thing everyone was promised: knowledge that grows while you sleep. Point it at unverified generation because the verification line item didn't fit the budget, and you get a very fast machine for becoming confidently wrong. ``` THE FORK: same generation speed, different year-two ┌── with hygiene ──▶ compound knowledge agent output ────────┤ (grows while you sleep) └── without ───────▶ compound wrongness (also grows while you sleep) ``` Same amplifier. Your choice of signal. [Start simple, scale with intention](/blog/start-simple-scale-with-intention/), and clean up your snowballs while they're still small enough to hold.