--- name: auto-test description: One command turns a True Platform requirement ID into a complete, traceable test run on Katalon True Platform — pull the requirement straight from True (no Jira), design coverage, reconcile tests (link existing / update stale / create missing), build a suite, Run-with-AI against the live app, and ship a beautiful HTML report that surfaces exactly where the app has fallen behind the updated requirement. Use when the user says "Auto test ", runs /auto-test, or asks to "test this story end to end" / "cover on True Platform". --- # Auto Test — `Auto test ` > **Give me a requirement ID. I pull the requirement from True Platform, figure > out the coverage, wire the tests into True Platform, run them against the live > app, and hand back a report that shows what passed, what broke, and where the > build is behind the spec.** **The requirement is the single source of truth, and it comes from the Katalon True MCP only.** This skill does not call Jira/ALM tools at all — if a story was authored in Jira, it must already be synced into a True Platform requirement (that's how it gets here). Everything is read from the True requirement object via `read_requirement` / `find_requirements`. This skill is the **single-command demo of True Platform × Claude Code**: a PM types `Auto test BSD-13` and the whole testing lifecycle runs itself. It is a thin **orchestrator** — all platform mechanics come from the [`katalon-trueplatform-testing`](../katalon-trueplatform-testing/SKILL.md) skill (read it; this skill assumes its tool map and lane rules). This skill adds the *requirement-first wiring*, the *link/update/create reconciliation*, and the *report*. **Golden rule (inherited):** platform objects → **Katalon MCP**; touching the live app UI → **Browser/Playwright**. Never scrape the platform with a browser. --- ## 0. The one-line contract ``` Auto test → requirement pulled · coverage designed · tests linked/updated/created → suite built · run executed on the live AUT · report shipped → gap called out: which updated ACs the app does NOT yet satisfy ``` Run the whole arc autonomously. Pause only for: missing credentials/AUT, a genuinely ambiguous project/repo, or a destructive/defect-filing step (see §7). --- ## 1. Inputs & flags | Input | Meaning | |---|---| | `` | The requirement reference — a True Platform requirement key/ID, or the external issue key (e.g. `BSD-13`) of a story already synced into a True requirement. Resolved via the True MCP. The only required argument. | | `--aut ` | Override the app-under-test URL (else resolve via `read_auts`). | | `--dry-run` | Plan coverage + reconcile tests, **do not execute**. Stop after `reconcile.md`. | | `--no-defects` | Skip the "file defects?" step. | | `--repo ` | Force a specific True Platform repository / Test Project. | The requirement is read from **Katalon True Platform** via the True MCP — no Jira/ALM call is ever made. Today's date anchors any "updated requirements" detection. --- ## 2. The pipeline (seven steps) ### Step 1 — Resolve True Platform context ``` list_projects → pick the project (honor explicit hint / current context) list_repositories(project_id) → resolve the one repository / Test Project ``` - One obvious case-insensitive match → use it. Honor `--repo`. Only ask when genuinely ambiguous (parent skill's resolution rules). ### Step 2 — Pull the requirement from True Platform (the source of truth) **This is the only place the requirement comes from — the Katalon True MCP. No Jira/ALM tool is called.** ``` find_requirements(project_id, query matching ) → locate the requirement read_requirement(requirement_id) → full body + ACs ``` - `` may be a True Platform requirement key/ID **or** the external issue key of a synced story (e.g. `BSD-13`). Match it against the requirement `key`, `name`/`summary`, or `externalIssue` fields returned by `find_requirements`. Capture the **True Platform requirement ID** — it's needed for linking in Step 4. - `read_requirement` gives the authoritative description, acceptance criteria, and fix version / target release as stored in True. Parse the **acceptance criteria** into a flat list. Detect a heading like **"Updated Requirements"**, **"New scope"**, **🆕**, or a dated block, and classify those ACs as **NEW** (the build may not cover them yet). Everything above it is **BASELINE**. - If the requirement body has no explicit ACs, derive them from the requirement description + PRD context (search `04-projects//` for a `prd.md`). - Write `requirement.md`: requirement header (True ID + key), BASELINE ACs, NEW ACs, fix version, True Platform requirement link. > The BASELINE/NEW split is the spine of the whole run — baseline cases should > pass; NEW cases are the ones that expose the requirement↔app gap. > **If `` resolves to no True Platform requirement,** stop and tell the > user: the story must be synced into a True requirement first (this skill does > not read Jira). Don't guess ACs from a ticket key alone. ### Step 3 — Design coverage (think before typing) - Apply ISTQB techniques from [`references/manual-test-design.md`](../katalon-trueplatform-testing/references/manual-test-design.md): equivalence partitioning, boundary values, decision tables (tip/tax/service combinations are a natural decision table), state transitions, error guessing. - Produce a **case-to-AC map**: every case names the AC(s) it verifies and is tagged `BASELINE` or `NEW`. Risk-based, not max count — a tight P0/P1 set. - Write `coverage.md` (techniques used · selected cases · case→AC map · deferred + rationale). ### Step 4 — Reconcile tests (the smart part) For the resolved requirement, decide per case — **link / update / create**: ``` find_test_cases_by_requirement(requirement_id) → existing linked cases ``` | Situation | Action | Tool | |---|---|---| | An existing case already covers a BASELINE AC and matches the current steps | **Keep & link** | `link_requirements_to_test_case` | | An existing case covers the AC but its steps are **stale** vs the updated requirement | **Update** the steps in place | `update_test_case` | | An AC (esp. a NEW one) has **no** covering case | **Create** | `create_test_case` → `link_requirements_to_test_case` | - Prefer updating an existing case over creating a near-duplicate. Match by title/AC reference, not fuzzy guessing — if unsure, create and note it. - Every created/updated case gets linked to the requirement so traceability rolls up in True Platform. - Write `reconcile.md`: a table of `case · action(linked/updated/created) · AC · BASELINE|NEW · case-id · link`. **If `--dry-run`: stop here.** Report the reconciliation plan and exit. ### Step 5 — Build / locate the suite - `find_test_suites` for an existing story suite; else `manage_test_suite` to create **` — Regression`** and add every reconciled case. - A True Platform "Test Plan" entity doesn't exist via MCP — a named suite + the fix-version association IS the plan (see capability boundaries in the parent skill §5). ### Step 6 — Execute (Lane A — Manual + Run with AI) ``` read_auts (ALWAYS immediately before the run; never reuse a stale AUT) create_manual_test_run(... aut_environment_id | default_aut_environment_url=<--aut or live url>) read_test_suite(suite) # pass non-empty case lists create_manual_ai_session(manual_execution_id, test_suites, aut_environment_id…) read_manual_ai_session(...) → POLL until every case leaves TODO/IN_TESTING # Per-case verdicts are FINAL here even if the run container still says INCOMPLETE. # Best-effort end the run (non-blocking), then read results and ALWAYS report. read_execution ; read_execution_test_results ; read_test_result → per-case verdicts ``` - Run-with-AI executes on **Linux Chrome latest** — say so; cross-browser isn't supported in this lane. For cross-browser/CI, promote to Lane C (Playwright). - If a run blocks on AUT/account/env, report the **blocked** state + the exact fixture needed — don't silently spin. - **End the run best-effort, then ALWAYS ship the report — never block.** Once every case has left `TODO`/`IN_TESTING`, the per-case verdicts are final even if the run *container* still shows `INCOMPLETE`. Try to end the run programmatically if a tool/True-Platform API allows. If you can't, add ONE line to the report — `▶ End this run on True Platform to finalize the platform-side report: ` — and keep going. **Do not halt waiting for the user to click End Run**, and do not gate report generation on a terminal run status: that produces no report at all. Always proceed to Step 7 and build `report.html` from the per-case results you already have. ### Step 7 — Read & report - Read results: `read_execution` → `read_execution_test_results` → `read_test_result` (capture status, error lines, screenshots/evidence). - Write `results.md` (run link · pass/fail/blocked counts · per-case findings with verbatim evidence). - **Generate `report.html`** — see §3. - **Defects** (unless `--no-defects`): for each real failure, propose a defect via the True MCP (`create_defect` needs a failed `test_result_id` + ALM project from `list_alm_integrations`). This is a True Platform tool — defect-to-ALM propagation is handled by True's own integration, not by this skill calling Jira. **Ask before filing.** > The traceability link back to the originating story lives in True Platform > (the requirement's `externalIssue`). This skill does **not** post comments to > Jira — surface the run link + gap summary in the report and let True's ALM > integration carry it back. --- ## 3. The report (the deliverable) Use the [`html-artifact`](../html-artifact/SKILL.md) skill to produce a self-contained, shareable HTML page. It must be genuinely beautiful — this is the demo's payoff. Required sections, in order: 1. **Header** — story ID + title, fix version, run timestamp, AUT URL, overall verdict pill (e.g. *6 passed · 5 failed · 0 blocked*). 2. **The gap, up top** — a callout box: *"The app satisfies all 6 baseline criteria but none of the 5 updated requirements added 2026-05-29."* This is the headline. 3. **AC ↔ Result matrix** — every AC as a row: `AC · BASELINE|NEW · test case · ✅/❌/⛔ · evidence link`. Color baseline-pass green, new-fail amber/red. 4. **Per-case detail** — steps, expected vs actual, screenshots, verbatim error lines. Evidence discipline: no failure claim without its proof. 5. **Recommended actions** — defects to file, the build work each failed NEW AC implies, and the True Platform suite/run links. 6. **Traceability footer** — True Platform requirement link, suite link, run link (plus the requirement's external issue link if True exposes one). One click from report → platform. See [`references/report-template.md`](references/report-template.md) for the layout, the design language, and the data contract. --- ## 4. Layout of a run ``` 04-projects//auto-test/-/ ├── requirement.md # parsed story: BASELINE vs NEW ACs ├── coverage.md # ISTQB note + case→AC map ├── reconcile.md # linked / updated / created table ├── results.md # run summary + per-case evidence └── report.html # the shippable report ``` Working files live in the project folder, **not `/tmp`** (vault hygiene). --- ## 5. Honesty rules (carry these, they're load-bearing) - **Evidence or it didn't happen.** Every pass/fail carries its artifact (screenshot, verbatim error, trace). If you can't produce evidence, downgrade the claim. - **A failing NEW-AC test is a SUCCESS for this demo** — it proves the workflow catches the requirement↔app gap. Report it as a *finding*, not a tooling error. Don't "fix" the app to make tests pass; the gap is the point unless the user explicitly asks to close it. - **Don't fabricate platform IDs or links.** If a tool didn't return an ID, say so. - Resolve project/repo from context; only ask when genuinely ambiguous. --- ## 6. When things are missing (graceful degradation) | Missing | Do | |---|---| | `` resolves to no True Platform requirement | **Stop and ask.** The story must be synced into a True requirement first — this skill never reads Jira. Don't fabricate ACs from a bare ticket key. | | No configured AUT | Pass the live URL via `default_aut_environment_url` (or `--aut`). | | Run-with-AI blocks (login/account) | Report blocked + the exact fixture/account/env needed. | | No ALM integration for defects | Skip filing; list the defects you *would* file in the report. | --- ## 7. Autonomy & gates Autonomous: pull (from True) → design → reconcile (link/update/create) → suite → run → report. **Gate (ask first):** filing defects, any destructive change, or editing the live app. State assumptions as you go and keep moving otherwise.