Reality Check

Proving "done" means done

The Problem

When an AI says "done," it graded its own homework.

The agent verifies its work in the same conversation that wrote it. "Tests pass" gets declared — and the last real check falls back on you.

The team behind Reality Check names this gap in their own words.

Proof — In Their Own Words

The authors say AI software is verified in the very context it was built.

The microsoft/amplifier-bundle-reality-check README opens by naming exactly this failure mode — false "done" claims.

So what does an in-conversation check actually fail to prove?

README: AI-generated software is "typically verified in the same environment and context it was built in."
This "leads to agents claiming 'done' when the work doesn't actually meet the user's intent, hasn't been tested in a real environment."
Reality Check "closes that gap … so that 'done' means done."

The Tension

"Tests pass" can't honestly say a browser really rendered "Hello, {name}!"

A one-shot self-check inside the build chat cannot truthfully assert that a real browser loaded the form, typed a name, submitted, and saw the greeting render.

To prove it, you need a real target that forces real interaction.

Load Open GET / — the form with a name input and a "Greet me" button.
Type & submit Enter a name, POST to /greet.
Confirm Verify <h1 id="greeting">Hello, {name}!</h1> actually renders.

The Target — Greeting App

An 89-line Flask fixture, built solely to force real browser interaction.

bkrabach/reality-check-greeting-app is a tiny Flask app: two routes designed to require a real browser. Its description says so outright — an end-to-end testing fixture.

With a real target set, how does Reality Check verify against it?

89

lines inserted (5 files)

1

commit (2087bef, 2026-04-30)

2

routes: GET / and POST /greet

Flask

by bkrabach (Brian Krabach)

Mechanism — Define "Done"

First it asks "what does done mean?" before any validator runs.

The intent-analyzer — the "what does done mean?" agent — derives verifiable acceptance tests from user intent as the pipeline's first stage.

Then those tests are checked from outside the build chat.

Intent-analyzer runs first Derives acceptance tests from user intent, before any validator.
CLI-validated loop validate-acceptance-tests gates it, capped at 3 attempts.
Then validators run Tests route by method: browser / cli / other.

Mechanism — A Fresh Perspective

Each validator runs as a one-shot sub-session against a real deployed container.

Validators run separate from the build conversation, against software deployed in an isolated Digital Twin Universe / Incus environment — a fresh perspective, not the build chat.

And the point proves itself, live, in the bundle's own repo.

One-shot sub-session: intent-analyzer, browser-tester, generic-tester, terminal-tester, report each run separate from the build.
Real environment: the dtu-launch step deploys the software in a Digital Twin Universe (Incus) container.
Evidence, not a claim: the report schema requires a non-empty evidence string per pass/fail result.

Payoff — Caught In Its Own Repo

The bundle's own suite fails the exact way Reality Check exists to catch.

test_version "passed history" asserting a stale 0.1.0 — but the CLI actually reports 0.2.0. An in-conversation self-check that agreed with itself, yet doesn't match reality.

That's why "done" has to be evidence, not a claim.

# uv run pytest -q
1 failed, 88 passed in 7.78s   (89 total)

# tests/test_cli.py:42
assert "0.1.0" in result.stdout
AssertionError: "0.1.0" not in
  "amplifier-reality-check, version 0.2.0"

# uv run amplifier-reality-check --version
amplifier-reality-check, version 0.2.0

The Takeaway

"Done" isn't a claim you accept. It's evidence a fresh check hands you.

Replace self-graded assertions with independently-produced evidence — verified from a fresh perspective, in a real deployed environment.

Sources

Research Methodology

Reality Check bundle — Development Status :: 3 - Alpha

Data as of: latest commit 2026-07-09 on microsoft/amplifier-bundle-reality-check (26 commits). Greeting-app single commit 2087bef, 2026-04-30.

Repos: primary-source fixture bkrabach/reality-check-greeting-app; core feature microsoft/amplifier-bundle-reality-check. Independently re-verified in a fresh session by cloning both repos and re-running every cited command.

Commands run:

gh repo view bkrabach/reality-check-greeting-app — description & README fixture claim
git rev-list --count HEAD, git show --stat HEAD, wc -l app.py Dockerfile requirements.txt README.md — 1 commit, 5 files, 89 insertions
git show --pretty --date=iso HEAD — commit 2087bef, bkrabach, 2026-04-30
cat app.py — GET / form + POST /greet renders <h1 id="greeting">Hello, {name}!</h1>
cat README.md / pyproject.toml — self-verification framing; deps digital-twin-universe + gitea
grep -n 'one-shot sub-session' agents/*.md — intent-analyzer, browser/generic/terminal-tester, report
grep -n 'evidence…NonEmptyStr' report.py — RawResultEntry requires status (pass|fail) + non-empty evidence
grep max_while_iterations recipes/reality-check-pipeline.yaml — intent & report loops capped at 3
uv run amplifier-reality-check --help / --version — 3 subcommands; version 0.2.0
wc -l src/*.py tests/*.py, uv run pytest -q — 1714 src / 1769 test lines; 89 tests, 88 pass, 1 fail
git log --pretty='%an' | sort | uniq -c — contributor counts (26 commits)

Primary contributors: David Koleczek (19 commits, two Git identities), Brian Krabach (4), Manoj Prabhakar Paidiparthy / manojp99 (3). Greeting-app: bkrabach (sole author, 1 commit).

Gaps: Whether the failing test_version is a deliberate teaching artifact or an accidental stale test is not stated in the repo — presented as observed irony. The bundle.md version 0.2.1 vs pyproject/CLI 0.2.0 drift is factual but unexplained. Production-usage metrics (projects reality-checked, defects caught in the wild) were not found and are not asserted.