# How I Test Katalon True Platform With an AI Agent (as the PM Building It)
> I'm the lead product manager building Katalon True Platform, an AI-native testing platform, and every release my AI agent tests it alongside our QE team. One terminal command runs the full software testing loop: collect the release scope, design test cases, execute with the AI test runner and Playwright, file findings back. This is how a PM learns the product's real capabilities and limitations, with real screenshots, video, and a narrated deep dive for QA engineers, QE, and testers.
Published: 2026-07-09 · Reading time: 11 min · Tags: qa, ai, claude-code, agentic-qa, katalon, katalon-true-platform, dogfooding, playwright, test-automation, quality-engineering, ai-testing-agents, software-testing, qa-automation, product-management
Canonical URL: https://huytieu.com/blog/testing-katalon-true-platform-with-an-ai-agent/
Author: Huy Tieu (huytieu.com)
---
> **TL;DR** I'm the product manager building an AI testing platform, and my AI agent tests that platform every release, alongside our QE team.
>
> - One terminal command takes a release milestone and turns it into organized test cases on [Katalon True Platform](https://katalon.com), executed through two lanes: the platform's AI runner and Playwright.
> - The point isn't replacing QE. It's a PM finally *feeling* the product's capabilities and limitations first-hand, on a schedule, with every friction filed back as a ticket. A self-improvement loop for the product.
> - The orchestration lives in a skill (a written-down workflow), not in scripts. The agent makes the calls; the platform does the work.
> - Real screenshots and real execution footage below, plus a narrated 4-minute deep-dive video, and a CTA at the end if you want to run the same loop yourself.
I'm the lead product manager for AI at [Katalon](https://katalon.com), building True Platform, an AI-native testing platform. And for a while something bothered me: I was shipping AI testing features I had never *depended on*. I read the tickets. I watched the demos. I sat in the syncs. But when a QE asked me "would you trust the AI runner with a release?", my honest answer was second-hand.
PMs write roadmaps about capabilities. Users experience limitations. The gap between those two is where products quietly rot.
So I made a rule: **every release, my AI agent tests True Platform using True Platform, right alongside the QE team.** Not a one-off experiment. A standing loop whose whole purpose is to make me collide with the product's real capabilities and real limitations, and turn every collision into a ticket.
The product under test is our platform. The test management tool is the same platform. And the whole QA workflow is orchestrated by an AI agent, [Claude Code](https://claude.com/claude-code), from my terminal: test design, test execution, defect filing, reporting. If you're a QA engineer, QE, SDET, automation tester, or a fellow PM wondering what testing with AI agents actually looks like in production (not a demo), this is a tour of a real run.
 *The loop that eats itself, on purpose: one product playing two roles, one AI agent turning the crank.*
---
## Watch it instead (4 minutes)
If you want the whole story in one sitting, here's the narrated deep dive: the concept, the terminal run, the platform structure, both execution lanes with the real runner footage, and the feedback loop.
Everything in the video appears below as screenshots and text, so read on if that's more your speed.
Prefer to read with music? The lo-fi set I generated for the video is here, full length:
01 · focus (dusty drums, warm Rhodes)
02 · rainy (rain, jazzy guitar)
03 · ship it (marimba at dawn)
---
## Why a PM runs tests (and why "dogfooding" is not a demo)
Our QE team designs the release test strategy, owns the quality bar, and catches what matters. My agent's lane is different and deliberately additive: it runs the *product-shaped* loop, using True Platform exactly the way we tell customers to, and reports where the product helped and where it fought back.
Dogfooding gets talked about like a vibe: "use your own product more." For this to teach me anything as a PM, it needed to be stricter, something that runs on a schedule and produces artifacts:
- **A fixed trigger.** Every release milestone, no exceptions, no mood-based testing.
- **Full traceability.** Every backlog item in the release either gets test cases or gets an explicit "not testable through the UI, here's why." Nothing silently dropped.
- **Two independent lanes.** The same behavior checked by the platform's AI runner (natural-language test cases) and by Playwright code, results landing in the same place.
- **A feedback loop with teeth.** Anything that blocks or annoys the agent while it operates the platform gets filed to the product backlog, labeled as a dogfood finding, same as a customer-reported bug.
That last one is the entire point. When the tester and the tested are the same product, and the person reviewing the run is the person writing the roadmap, friction stops being an anecdote and becomes a ticket with a name on it.
---
## One command in the terminal
The whole run starts with a single slash command in Claude Code. Here's the shape of a run, end to end (this is a replay of the actual orchestration flow):
What you're watching: the agent resolves the release milestone from GitHub, classifies what's actually testable, designs granular test cases with real test-design techniques (equivalence partitioning, boundary values, pairwise combinations), pushes everything to True Platform over [MCP](https://modelcontextprotocol.io), fires both execution lanes, then consolidates one report and files findings back.
The important part is what's *not* there: no glue scripts, no CI YAML, no hand-maintained test runner. The workflow is written down as a skill (a markdown playbook the agent reads), and the agent drives the platform's own APIs the same way a person would drive the UI.
```
skills, stacked like lego
/dogfood-release <- the orchestrator (phases, rules, gates)
|
+-- platform skill <- how to drive True Platform (lanes, tools)
|
+-- reporter skill <- how to ingest Playwright results
|
+-- house rules <- evidence discipline, atomic cases,
"file every friction" policy
```
I wrote before about [giving Claude Code my QA job](/blog/i-gave-claude-code-my-qa-job/); this is that idea graduated from "one testing task" to "an entire release, recurring."
---
## What lands on the platform
The agent doesn't dump tests in a pile. It builds the same structure a disciplined QA team would: one folder per release, one folder per backlog item, suites chunked so each run shares mental context.
Each folder holds granular cases: one action and one checkable expected result per step, written in plain natural language, self-contained enough that an AI runner (or a new teammate) can execute them cold. This release, 43 backlog items were triaged into 134 executable cases, and the sizing wasn't vibes: a small coverage model decomposes each item into atomic requirements and floors the case count with standard test-design techniques.
The execution history tells the release's story at a glance:
---
## Lane A: the platform's AI runner, watched frame by frame
The first lane is Run with AI: True Platform's agentic runner takes the natural-language cases and physically operates the product in a real browser. Pass or fail, every step comes back with a verdict and a screenshot.
The failures in that run are the good kind: assertion failures on real product behavior, each one traceable to a specific expectation. Granularity is what makes this readable. When a case checks exactly one thing, a red row tells you *which* expectation the product missed, not just "something in this area is off."
Here's the part I find genuinely fun: the platform records the AI runner working, so you can watch your test being executed like a screen share. This is an actual recording from the run, the runner signing in, opening the AI chat, sending a message, and judging the response timing:
And the session view puts it all side by side: the natural-language steps on the left, per-step verdicts in the middle, the video and metadata on the right. Note the traceability detail: the case description links straight back to the originating backlog item.
Drilling into any step gives you the evidence trail, each step timestamped with its own screenshot attachment:
---
## Lane C: Playwright code, same destination
Some checks want code precision: timing thresholds, streaming behavior, things where "about 100ms" needs to be an actual number. Those become Playwright specs in a small standalone repo. The specs run locally or in CI like any Playwright project, and the results ship into the same platform project via [@katalon/playwright-reporter](https://www.npmjs.com/package/@katalon/playwright-reporter), so both lanes' evidence lives in one place.
 *Manual-style AI execution and code-based test automation, converging on one results page.*
Two independent lanes matter more than they sound. When both lanes agree a behavior is off, confidence is high. When they disagree, that gap is itself information, usually about the test, sometimes about the product.
---
## The feedback loop is the product
Here's where the self-referential setup pays off. While the agent operates the platform, it keeps a running ledger of everything that got in its way: a validation quirk, a tool that behaved unexpectedly, a limit it had to work around. At the end of the run, each item gets deduplicated against the existing backlog and filed to GitHub with a dedicated dogfood label. No approval theater; findings flow straight to the team that can fix them.
 *Friction becomes a ticket every time, and closed tickets get re-tested without anyone remembering to.*
This week's run filed two findings and confirmed several known ones with fresh evidence. The details belong to the team's backlog, not this post, but the pattern is the lesson: **the most valuable findings didn't come from test cases at all. They came from the agent trying to use the platform and hitting walls.** A test case checks what you predicted. Dogfooding catches what you didn't.
And because closed items automatically get verification cases in the next release's run, the loop closes itself: finding, fix, proof of fix, all in the same system. That is the self-improvement loop, and it runs whether or not anyone remembers to schedule it.
---
## What this taught me about my own product
This is the section I actually run the loop for. As a PM, I now hold two lists I could never get from status reports:
- **Capabilities I can defend with evidence.** I've watched the AI runner hold a clean login-to-verdict arc across granular cases, seen step-level screenshots make failures un-debatable, and watched Playwright results land in the same history as AI-executed runs. When I talk about these in a roadmap review or a customer call, I'm quoting my own runs, not a slide.
- **Limitations I respect instead of hand-wave.** Where natural-language steps need to be atomic or verdicts get mushy. Where sessions, suite sizes, and integrations have edges. Every one of those edges is now a ticket with evidence attached, and several have already turned into shipped fixes that the next run verified automatically.
And the craft learnings, for anyone building the same muscle:
- **Granularity is the whole game for AI runners.** Our first pass bundled setup, navigation, action, and judgment into fat steps. Verdicts were mush. Splitting to one action + one checkable result per step turned the same coverage into clean, per-expectation signal.
- **The agent needs the workflow, not the keystrokes.** Everything durable lives in the skill: the phases, the naming rules, the "never silently drop an item" policy. The agent brings judgment; the skill brings discipline.
- **Structure is what makes agent output reviewable.** One folder per backlog item, one suite per feature cluster, one report per run. I review a single file and a platform link, not a pile of agent chatter.
- **Friction logging beats test cases for finding blind spots.** Predictable failures get caught by assertions. The surprising ones got caught because the agent was a *user*, with a user's right to be annoyed.
- **Dogfooding scales when an agent does the chewing.** The honest reason teams (and PMs) don't do this every release is cost. This costs me one command and a review.
```
before after
PM reads status PM reads evidence
| |
"AI runner works" 7 passed, 4 failed,
(slide-ware) video attached
| |
roadmap by vibes roadmap by collisions
```
The most exciting part, honestly: watching the platform's own AI runner execute a test my agent designed, recorded on video, judged step by step, and knowing both sides of that exchange are things we ship. The loop is young and the platform evolves under it weekly, which is exactly why it exists.
---
## Want to run the same loop?
Everything here is reproducible, and most of it is public:
1. **Get the platform.** [Katalon True Platform](https://katalon.com) is the AI-native testing platform in this post: AI test runner, test management, Playwright ingest, MCP server.
2. **Grab the skills.** The agent playbooks I use are published on this site: [katalon-trueplatform-testing](/skills/katalon-trueplatform-testing.SKILL.md) (the full lane map and MCP tool rules, also available as a [Claude skill bundle](/skills/katalon-true-test-claude-skill.zip)) and [auto-test](/skills/auto-test.SKILL.md) (requirement-to-report in one command). Drop them into Claude Code and adapt the project IDs.
3. **Start with one story, not a release.** Pick one requirement, let the agent design and run coverage for it, review the verdicts. Scale to a release milestone once you trust the loop.
4. **File the friction.** Whatever fights you, write it down where the builders live. That habit is the entire compounding effect.
If you try this, whether you're a QE, an SDET, or a PM who wants to actually feel your own product, I'd genuinely love to compare notes: [LinkedIn](https://www.linkedin.com/in/htieu) or [email](mailto:tqhuy1991@gmail.com). And if the loop breaks in an interesting way, even better. That's the good data.
---
## FAQ: testing with AI agents
**What is agentic dogfooding?**
Agentic dogfooding is using an AI agent to test your own product with your own product, on a fixed trigger (every release), with findings filed back to the backlog automatically. It differs from ordinary dogfooding ("use your product more") by producing artifacts: test cases, verdicts, evidence, and tickets.
**Can an AI agent really run manual test cases?**
Yes, if the cases are written for it. Katalon True Platform's AI test runner executes natural-language test cases in a real browser and returns a verdict, a screenshot, and a recording per step. The catch is granularity: one action and one checkable expected result per step. Fat steps produce mushy verdicts; atomic steps produce clean pass/fail signal.
**Does this replace QA engineers or automation testers?**
No. It runs alongside them. QE owns the test strategy and the quality bar; the agent adds a product-shaped lane that a PM can afford to run every release. It replaces the keystrokes, not the judgment. What changes is cost: coverage that used to take a team-day of clicking now costs one command and a review.
**Why would a product manager run tests at all?**
Because capabilities and limitations read differently from inside a run than from a status report. A PM who has watched their own product fail a granular assertion (on video) writes different roadmaps than one who hasn't. The loop turns "I think the AI runner handles this" into "here's the execution where it did, or didn't."
**How do AI agents file bugs?**
In this workflow the agent deduplicates each finding against the existing backlog first: known issues get the new evidence as a comment; genuinely new ones get filed as issues with a dedicated label, repro steps, expected vs actual, and a link to the test run as evidence.
**Where does Playwright fit if the AI runner can execute tests?**
Playwright covers checks that need code-level precision (timing thresholds, streaming behavior) and cross-browser evidence. Both lanes report into the same platform, so AI-executed and code-executed results share one history. Two independent lanes agreeing is a stronger signal than either alone.
If you want the terminal-first background for how my whole setup works, start with [how I moved my life to the terminal](/blog/how-i-moved-my-entire-life-to-the-terminal-building-a-personal-ai-agent-system/) and [the QA job handover](/blog/i-gave-claude-code-my-qa-job/). This post is just those two ideas shaking hands.