Reality Check Grows Up

A QA oracle for TUI, CLI, and web

microsoft/amplifier-bundle-reality-check · Active

The Frame

Reality Check proves software works where it'll actually be deployed — not just in the dev session.

It's a real Microsoft bundle, microsoft/amplifier-bundle-reality-check, created 2026-04-14. It acts as a QA oracle: validate the built software in the environment where it runs, then hand back a verdict.

And that validation isn't limited to one kind of app —

The Proof

It validates three surfaces via three specialized agents, routed by verification method.

The schema itself models software categories — web_app, cli_tool, api_service, library — and a ValidatorType of browser | cli | other. Multi-surface coverage is built in, not bolted on.

One of those three surfaces got solid ground first —

1

browser-tester — web (type: browser)

2

terminal-tester — CLI / TUI (type: cli)

3

generic-tester — other (type: other)

The Setup

Web landed first; the newer frontier was a real coding-agent TUI inside a Digital Twin Universe.

The browser-tester agent was the earliest merged PR (#1, 2026-04-21). The harder case: driving the OpenAI Codex CLI — an AI coding-agent TUI — installed as an isolated DTU community profile in an Ubuntu 24.04 container.

But inside that container, a working terminal app looked broken —

PR #1

browser-tester, web validation, merged 2026-04-21

Codex CLI

AI coding-agent TUI, an isolated DTU profile

Ubuntu 24.04

Container the Codex TUI is installed into

The Tension

The PTY emulator can't render spinners, so a working TUI looks frozen — and terminal runs churned instead of passing.

The emulator does not render spinners, progress bars, or loading indicators; the screen can look static while the app is actively processing. On top of that, interactive bash, dropped keystrokes, and 10-minute timeouts made terminal runs flaky.

So each cause got fixed at its source —

Looks frozen: PTY emulator can't render spinners or progress bars.
Spurious fails: interactive bash expanded ! in acceptance tests.
Dropped input: TUIs drop or misinterpret bundled keystrokes.
Premature stop: long runs cancelled at the 600s (10 min) default.

The Mechanism — terminal-tester

Bridge into the TUI correctly, and spurious failures disappear at the source.

Running set +H on connect stops ! in generated tests from expanding as bash history (PR #8) — in one run 2 of 20 criteria failed spuriously, flipping a pass to partial. And exec <id> without -- gives raw PTY passthrough instead of JSON-wrapped, broken rendering.

The other class of failure was the agent-under-test failing to run at all —

1

set +H on connect — stops ! tests failing spuriously (PR #8)

2

exec without -- — raw PTY passthrough, not JSON-wrapped output

3

separate sends — send text and its confirming key as separate calls

The Mechanism — the DTU

The Digital Twin Universe itself was hardened so the Codex TUI could run at all.

Piping OPENAI_API_KEY through codex login --with-api-key persists the credential to /root/.codex/auth.json, stopping 401s despite a valid env var (PR #23). And a config.toml with sandbox_mode='danger-full-access' is the real sandbox-free switch — not the CODEX_UNSAFE_ALLOW_NO_SANDBOX env var (PR #24).

With both classes of failure fixed, the payoff lands —

1

Auth (PR #23): codex login --with-api-key writes auth.json; adds codex login status

2

Sandbox (PR #24): config.toml with sandbox_mode='danger-full-access', approval_policy='never'

The Payoff

Terminal validation stopped churning — Reality Check is now one QA oracle across TUI, CLI, and web.

Validators are routed by verification method, and raising all eight recipe-step timeouts to 86400s (24h) stopped long TUI runs being cancelled prematurely (PR #11). Both convergence loops — intent derivation and report/gap analysis — are CLI-validated and capped at 3 attempts.

And the shape of that fix generalizes —

3

surfaces: TUI, CLI, and web

86400s

step timeout across all 8 steps, 3 recipes (PR #11)

3

max iterations on each convergence loop

The Takeaway

Validate where software deploys, fix flakiness at its source — and you get a QA oracle you can trust across every surface.

Reality Check began by proving software works where it will actually run. Each terminal-validation fix removed a specific spurious-failure or premature-cancellation cause — a qualitative improvement grounded in the fix PRs, maturing the oracle across TUI, CLI, and web.

Sources

Research Methodology

Data as of: 2026-07-21 (facts re-verified via gh against GitHub).

Feature status: Active — microsoft/amplifier-bundle-reality-check, created 2026-04-14, depends on microsoft/amplifier-bundle-digital-twin-universe (created 2026-03-25).

Research performed:

gh repo view microsoft/amplifier-bundle-reality-check --json name,description,createdAt,pushedAt,updatedAt
gh repo view microsoft/amplifier-bundle-digital-twin-universe --json createdAt,pushedAt,description
gh pr view 1 --repo microsoft/amplifier-bundle-reality-check --json title,mergedAt (browser-tester, merged 2026-04-21)
gh pr view 8 --repo microsoft/amplifier-bundle-reality-check (set +H, merged 2026-06-02); gh pr view 11 (timeouts 86400s, merged 2026-06-09)
gh pr view 23 / gh pr view 24 --repo microsoft/amplifier-bundle-digital-twin-universe (codex auth 2026-05-22, sandbox 2026-06-02)
ls agents/; grep -n 'terminal-tester for\|browser-tester for\|generic-tester for' README.md
grep -rn 'web_app\|cli_tool\|ValidatorType\|software_type' src/; grep -n 'set +H\|WITHOUT\|config.toml\|sandbox_mode' agents/ profiles/
grep -n 'max_while_iterations\|Capped at 3 attempts' recipes/reality-check-pipeline.yaml

Gaps: No quantified before/after iteration count exists in the repos, so the churn reduction is presented as a qualitative improvement only. Narrative figures such as "87 iterations", "14 minutes", and "51 terminal calls" were searched for (rg) and NOT found — treated as unverified and excluded. Local clones are shallow; all history/PR facts came from gh, not local git log.

Contributors: Local clone HEAD authors observed — RC PR #11 by David Koleczek, DTU HEAD (#26) by Brian Krabach; per-PR human authorship not independently confirmed for every PR.