The build lab
What happens if
we actually build it?
What do AI agents actually do when they work on real software? Each experiment here is a real question with a real build behind it, and each one becomes a video.
The project briefs are ready to use. Recorded episodes and measured results will be added when they’re published.
Did my tiny AI learn—or just memorize?
Build a small text model, watch its output change, and design a test that training examples cannot answer for it.
Read the experimentCan an AI remember the app it built yesterday?
Compare a fresh session with a short, maintained project memory. Measure whether the agent repeats the same mistakes.
Read the experimentI gave my AI two manuals that disagree.
Build a document assistant that can show its sources, notice contradictions and stop when the evidence runs out.
Read the experimentUp next
Can AI agents work like a real team?
The next experiments test the idea Bookbag is built on: agents need a system around them, not just a better prompt.
- 04In planning
Can an AI developer pass QA without a human fixing its work?
A coding agent ships a feature. A strict QA agent reviews it. Nobody steps in. How much survives, and why does the rest fail?
- 05In planning
Five agents, one feature: where does coordination break?
Five agents on one codebase at the same time. Who overwrites whom, what conflicts, and what it takes to merge.
- 06In planning
Does an agent follow your architecture when nothing enforces it?
The rules are in the prompt. Nothing checks them. Does the work pass every test and still break the architecture?
- 07In planning
Instructions aren't permissions. What does an agent do with access it shouldn't use?
The prompt says don't. The tools say it can. Which one wins, and how often?
- 08In planning
Can you audit what an AI agent did six months later?
Pick a merged change from an agent and try to reconstruct who asked, who checked, and why it was allowed through.
- 09In planning
Can an AI software team maintain the same app for 30 days?
Shipping is day one. Bug reports, dependency updates and real users are the other 29.
Free: AI Agent Rollout Checklist
Are your agents running on prompts and hope?
Get the AI Agent Rollout Checklist: inventory, permissions, gates, audit and ownership for teams putting AI agents to work. Subscribe to get it, plus the results of each new experiment.
Free newsletter on Substack. Unsubscribe anytime.