Case Study

Building a Complete
Regression Test Suite
in Hours

The amplifier-app-benchmarks Story

Open Source | MIT Licensed | Community Driven

Active

The Challenge

Testing Amplifier CLI
is Complex

Validate capabilities across multiple AI providers
Ensure bash, web, PDF, and agent tools work correctly
Catch regressions before users hit them
Traditional approach: weeks of development

The Solution

amplifier-app-benchmarks

A complete regression suite built with Amplifier itself, powered by Microsoft's battle-tested eval-recipes framework.

Docker Isolation

Each test runs in a clean container, ensuring reproducible results

Parallel Execution

Run multiple tests simultaneously with configurable concurrency

Semantic Testing

AI-powered evaluation that understands context and catches subtle failures

YAML-Driven

Declarative configuration makes tests readable and extensible

By The Numbers

Production-Ready in Record Time

10+

Critical Tests

3

AI Providers

Hours

Not Weeks

YAML

Driven Simplicity

Test Coverage

Comprehensive Capability Validation

Test	What It Validates
Provider Response	AI providers (Anthropic, OpenAI, Gemini) respond correctly
Bash Execution	Shell commands execute and return results
Agent Delegation	Task tool spawns and coordinates sub-agents
Web Search	Search capabilities find relevant, recent results
Web Fetch	Content retrieval from URLs works correctly
Recipe Listing	Recipe system discovers and lists available recipes
PDF Extraction	Reading and extracting content from PDF documents
AGENTS.md Injection	AGENTS.md files are properly loaded into context
Bundle Context	Bundle composition and context injection work correctly

Architecture

Clean & Extensible Design

data/ agents/ # Provider configs tasks/ # Test definitions run-configs/ # Suite compositions src/ # CLI entry point

eval-recipes Foundation

Built on Microsoft's proven evaluation framework

Type-Checked

Full type annotations with verification

Modern Tooling

uv for packages, ruff for linting, pre-commit hooks

Getting Started

Run in Minutes

Prerequisites: Python 3.11+, Docker, and API keys for your provider(s)

# Clone the repository
git clone https://github.com/DavidKoleczek/amplifier-app-benchmarks
cd amplifier-app-benchmarks

# Execute the regression suite
uv run amplifier-benchmarks run data/run-configs/regression.yaml

# Or install as a tool
uv tool install "git+https://github.com/DavidKoleczek/amplifier-app-benchmarks"
amplifier-benchmarks run config.yaml --max-parallel 15

Lessons Learned

Key Takeaways

01

Use the Tool to Build the Tool

The best validation of a dev tool is using it to build its own infrastructure

02

Stand on Giants' Shoulders

Leveraging eval-recipes provided Docker isolation and parallel execution out of the box

03

YAML-Driven Configuration Wins

Declarative configs make tests readable, maintainable, and extensible

04

Semantic Testing is Powerful

AI-powered evaluation catches nuanced failures traditional assertions miss

05

Open Source Multiplies Impact

Community contributions improve the entire Amplifier ecosystem

Sources

Research Methodology

Data as of: February 20, 2026

Feature status: Active

Research performed:

GitHub: gh repo view DavidKoleczek/amplifier-app-benchmarks — confirmed active, last updated 2026-02-10
Repository owner: DavidKoleczek (personal repo, not microsoft/)
Search: no local clone found in development workspace

Gaps: No local clone to verify exact test count, line counts, or license. "Hours" development time is qualitative, not measured. MIT license claim from repo — not independently verified.

Primary contributors: David Koleczek (repo owner)

Get Involved

Try It Today

Run the regression suite or extend it with your own tests

View on GitHub

MIT Licensed | Community Contributions Welcome

Building a CompleteRegression Test Suitein Hours

Testing Amplifier CLIis Complex