Parallel task execution with real-time monitoring
ramparte/amplifier-toolkit · Sam Schillace · February 2026
Agent workflows hit a wall when tasks pile up and run one at a time.
Tasks queued sequentially. A 50-task run waits for every predecessor to finish before the next begins.
No dashboard, no progress. You start a run and hope for the best—or grep through raw log files.
Orphaned processes, race conditions on shared state, no automatic recovery when a worker crashes mid-task.
Four layers, each with a single responsibility.
Source files: database.py, orchestrator.py, worker.py, dashboard.py, cli.py, migrate.py, __init__.py
No Redis, no RabbitMQ. A single SQLite file holds all task state. Durable, portable, zero-config.
N workers run in separate processes. N workers reduce wall-clock time proportionally to available parallelism.
Failed tasks re-queued automatically. Configurable retry count keeps transient failures from stalling runs.
FastAPI serves a live view over WebSockets. Progress, worker health, and task results update in real time.
Uses psutil to detect workers that vanished without updating state. Reclaims their tasks automatically.
Signal handlers let in-flight tasks finish cleanly. No half-written results or corrupted queue state.
FastAPI + WebSockets push updates as tasks complete. No polling.
Production workloads crash. Swarm is designed to keep going.
Transient failures—API timeouts, rate limits—are retried without manual intervention. Configurable limits prevent infinite loops.
psutil checks whether worker PIDs are alive. Tasks claimed by dead processes are automatically returned to the queue.
All state lives in SQLite. On restart, the orchestrator reads the queue, finds incomplete tasks, and resumes where it left off.
Production Evidence
swarm.db, swarm.db.backup, and swarm-full.log found in the wordbs project directory—confirming active production use with real task data.
Sequential execution vs. Amplifier Swarm.
Real example (wordbs project): 50 tasks processed via Swarm. With 4 workers, wall-clock time reduces to roughly ¼ of sequential execution.
Amplifier Swarm ships inside amplifier-toolkit.
Tasks are YAML entries. Each has a name, command, and optional dependencies.
The CLI launches the orchestrator, workers, and optional dashboard in one command.
Dependencies: click, rich, pyyaml, fastapi, uvicorn, websockets, psutil · MIT License
find . -name "*.py" | xargs wc -lParallel task execution, real-time monitoring, and crash recovery—in 2,496 lines of Python.
MIT License · v0.1.0 · ramparte/amplifier-toolkit