# I Gave Claude Code Computer Use *Only OpenAI's Codex desktop app has computer use, and I live in the CLI. So I had my agent take the feature apart and rebuild it for the terminal.* > Claude Code and the Codex CLI can't see or touch the screen; only OpenAI's Codex desktop app has computer use. So I pointed my coding agent at the desktop app, had it reverse-engineer how the feature works (the @oai/sky helper, a unix socket, a JSON-RPC protocol), and rebuild it from scratch on public macOS APIs as computer-harness. This is what it found and what it built, dead-ends included. Published: 2026-07-23 · Reading time: 7 min · Tags: ai, agents, macos, computer-use, claude-code, open-source Canonical URL: https://huytieu.com/blog/computer-use-that-doesnt-take-over-your-mac/ Author: Huy Tieu (huytieu.com) --- I run two coding agents all day, Claude Code and the Codex CLI, and neither of them can see my screen. They can read my files and run my commands, but they cannot look at a window or click a button. The one place computer use actually works is OpenAI's Codex desktop app, and that is the one place I do not live. The frustrating part is that the Codex app is genuinely good at it. It drives one app while I keep working in another, no cursor jumping, no focus stealing. I wanted that, but in the terminal where I actually work, not in a chat app I keep open in a corner. So I did the obvious thing and handed the problem to my agent. I told it the Codex app clearly has this feature, go find out how it is built, and figure out whether we can have our own. Most of what follows was not my hands. I described the frustration and the constraints and kept pointing it at the next real problem; it did the reading, hit the dead-ends, and wrote the code. This is what it came back with. ## What it found when it opened the Codex app Everything it needed was already on my Mac, inside `ChatGPT.app`. The desktop app bundles its own Node runtime, and inside it a package called `@oai/sky`, version 0.4.20, described in its own README as "the model-facing computer use API." The verbs are what you would guess: `click`, `type_text`, `get_app_state`, `list_apps`. That part is only a client. On macOS the client does no work itself. It talks over a **unix domain socket** to a separate, code-signed helper called `SkyComputerUseService`, using **length-prefixed JSON-RPC**. The agent found the live socket sitting in a group container on my machine and the service already running as its own process. The model-facing library is a thin shell around a local RPC call to a helper that holds the real permissions. The design worth stealing is the interaction model. `get_app_state(app)` hands the model a screenshot of one window **plus a numbered accessibility tree** of that window, and the model acts by naming an element index or a coordinate. It is not squinting at a full-screen image and guessing pixels. It reads structure and points at it. ## The engine is just Apple's APIs The thing that turned "interesting" into "let's build it" was this: there is nothing proprietary in the engine. Every capability maps onto a documented Apple API. Synthesizing input is CoreGraphics `CGEvent`. Reading structure is the `AXUIElement` accessibility tree, with `AXUIElementPerformAction` to press a button or set a field value straight through the API. Seeing a window that is buried behind others is `CGWindowListCreateImage`, which reads a window's backing store whether or not it is in front. OpenAI's advantage is not secret technology. It is a signed helper that has been granted Accessibility and Screen Recording, wrapped in a good interaction design. The permissions I can grant myself. The design is copyable on the merits. This is the same move [browser-use](https://github.com/browser-use) made for Chrome, and the one Cloudflare and others keep rediscovering for tool use. The real advantage is usually a thin layer over capabilities the platform already gives you. So the plan stopped being "understand it" and became "build the terminal version." A small daemon holds the permissions, and any CLI agent, Claude Code included, drives it by piping in a few lines. ## The first build took over my screen The first version worked and was a menace. It wired up the obvious path: move the global cursor, synthesize a click, type into whatever is frontmost. That drove a real app on screen, which felt like progress, until I tried to use my computer at the same time. I was reading something on my second display while it ran a demo on the main one, and it kept yanking my focus back across the screen. Every click and keystroke it sent landed system-wide, so every one stole the window I was working in. Which was the exact thing we were trying to avoid. It had rebuilt the hostage demo. I told it that was the whole point, not a detail. The fix was not a patch, it was a rule: global input can never be the default, and it must never fire while I am working. The tool grew two tiers, and the split between them became the point of the project. ## Calculator wouldn't cooperate The non-disturbing tier is supposed to act on a background window without touching the cursor. The mechanism is accessibility: instead of moving the mouse to a button, ask the accessibility API to press that button directly. No pointer, no focus change. The first test was Calculator, because it is right there. It went behind another window, focus stayed elsewhere, and the tool pressed its buttons through the accessibility API. Nothing happened. Focus stayed put, which was correct, but the display never changed. It tried the other background path, posting mouse events straight to Calculator's process id. Still nothing. The events reached the queue and the app ignored them. For a while it looked like the whole non-disturbing idea would die here. Then the agent worked out that Calculator is a Mac Catalyst app, an iPad app in a trench coat, with famously thin accessibility support. We were testing the architecture against its worst possible citizen. ## TextEdit did So it tried a normal AppKit app. Open a TextEdit document, push it behind System Settings, confirm System Settings has focus, and ask the accessibility API to set the document's text value directly. The sentence appeared in TextEdit. Focus stayed on System Settings the whole time. It captured the background TextEdit window to a PNG to prove it, without ever raising the window, and there was the line of text sitting in a window that was never in front and never stole my cursor. That was the moment it became the thing I wanted. The boring evidence, for the record: - Coordinate clicks and the accessibility tree both drive Calculator fine when it *is* in front. `6 × 7 = 42` went in through numbered element indices, and typing `12 + 30` through the keyboard produced the same 42. - Background, non-disturbing, through the accessibility API: text set into a TextEdit window while a different app kept focus, then read back from a background screenshot. - Background window capture works even when the window is fully covered. It fails only when the window is on another Space, because the OS has no pixels for it there. That one it could not beat, so it documented it. The rule that fell out: standard native apps get driven in the background, invisibly. Catalyst apps, games, and canvases with no real accessibility tree fall back to the foreground tier, which is a deliberate, announced choice. ## How Claude Code actually gets to use it The shape ended up matching my other tools: a small daemon that holds the permissions, and a heredoc CLI with the verbs pre-imported. Any agent that can run a shell command can now see and drive native apps. ```bash computer-harness <<'PY' st = get_app_state("Notes") # background screenshot + numbered a11y tree click("Notes", element_index=4) # AXPress, runs in the background set_value("Notes", 7, "hello") # set a text field directly, no keystrokes PY ``` That last part is the whole reason I started. Claude Code could already read files and run commands. Now, when it needs to check something in a real app, it runs `computer-harness`, gets back a screenshot and a numbered tree, and acts. The non-disturbing tier (`get_app_state`, `click`, `set_value`, `capture_window`) never moves my cursor. The foreground tier (`click_at_xy`, `type_text`, `press_key`) is there for apps with no accessibility handle, and it announces itself as the interruption it is. This is the third piece of the same workbench. browser-use's harness drives Chrome. computer-harness drives every other native app. And [TermDeck](https://termdeck.huytieu.com), the terminal workspace I live in, is where the agents run. My CLI agents are no longer blind. ## The part that should make you nervous Before the repo went public, the agent ran a security pass on its own work, because a tool that types into your apps and reads your screen deserves one. The honest findings, which the README now leads with: - **It runs arbitrary code as you** and can drive your input and read your screen. It is a "run code as me" tool, like a shell. Only pipe it code you trust. - **The daemon is a standing proxy to your Accessibility and Screen Recording grants.** Its control socket is owner-only, so no other user can reach it. Any process running as you can, whether or not that process was granted those permissions itself. On a single-user Mac that is a fair trade as long as you know it is true, so it is written down. - **Screen captures can contain secrets.** They are written owner-only, into a directory locked to your account, and nothing is uploaded. There is no network code in the tool at all; the only socket it opens is the local one. The review also caught the unglamorous stuff: my work email was about to ride into the public git history on every commit, and the capture files were being created world-readable before the umask got tightened. Both fixed before the first push. None of it was a clever exploit. Most security problems are not. ## Where this leaves me I did not set out to reverse-engineer anything. I was annoyed that the feature I liked lived in the wrong app, so I pointed my agent at the right one and kept it honest about what actually worked. It came back with a tool my CLI agents can call, that drives the apps I use without fighting me for my own cursor. The tool is MIT, macOS only, and not affiliated with OpenAI. I am going to keep using it until it gives me a reason to change it. Code and install: **[github.com/huytieu/computer-harness](https://github.com/huytieu/computer-harness)**.