Nightly builds for a national retail chain, and why I checkpoint agent runs
Packaging POS releases where a failed rollout stopped checkout and store revenue, with a rollback path per store. The instinct that made durable execution obvious twenty years later.
Early in my career I ran the nightly build cycle for Point-of-Sale software deployed across Giant Tiger stores nationally. Consolidate the day's completed changes, validate the executables and their runtime dependencies, package the release with its configuration and database components, regression-test it, then coordinate after-hours distribution to geographically dispersed stores — and in the morning, work out which installs failed and why.
The stakes were unusually legible. A POS failure stops checkout. Not "degrades the user experience" — the till does not work, the queue does not move, and the store stops taking money until it is fixed. There is no debate about severity and no negotiating with a product manager about priority.
That job installed three instincts I did not know I was carrying until I started building agent systems.
1. The unit of failure is the unit you designed
Rollouts failed constantly — network, dependency, configuration, version mismatch. That was normal. What mattered was the blast radius of one failure.
If a release is a single indivisible operation across four hundred stores, one bad install means the whole rollout is suspect and you are reasoning about a national estate at 3am. If each store's install is its own unit with its own verification and its own rollback path, one bad install is one store, and it is a phone call rather than an incident.
That is the same sentence I would now write about an agent run.
My first research agent ran its whole loop in one HTTP request. A transient page-fetch timeout on iteration four discarded three completed model calls — because I had accidentally made the entire run atomic. Rebuilding it so every unit of work checkpoints independently meant a failed step retried alone, with the work before it already durable and already paid for.
I did not arrive at that from reading about durable execution. I arrived at it from remembering what an all-or-nothing rollout felt like.
2. A known-good state to return to is not optional
Every release had a rollback path, so an individual store could return to a build that worked. This was not sophistication. It was the minimum, because the alternative — a broken till and no way back — was unacceptable in a way that concentrated the mind.
The agent equivalent is not "undo", which mostly does not exist. It is a replayable trace: a durable record of every message, every tool call, every argument and every result, complete enough that you can walk any past run step by step and see where it went wrong.
That is why my agent host persists conversations to Postgres and has a replay UI. Not for the demo. Because "it did something odd yesterday and I cannot see what" is the operational equivalent of a store with no known-good build.
It also turns out to have a second life. Once every model call is recorded with its inputs, its outputs and its price, the recording is a test fixture. My regression suite replays from stored traces at $0.00 per run, which means it runs on every push instead of whenever somebody remembers to authorise the spend.
3. Verify the install, do not trust the exit code
The part of the nightly cycle that took real time was not distribution. It was going through the morning's results working out which stores had actually received a working build — because "the installer reported success" and "the till works" are different claims, and only one of them mattered.
Modern equivalent: an agent that returns a confident final answer has told you it finished. It has not told you it was right.
This is why I now write evals as outcome assertions rather than step assertions — did the run find all 36 targets, did it spend under the ceiling, did it stop when told to — and why the failure states in my systems are explicit terminal states rather than the absence of a success. A run that dies mid-execution gets marked failed with a readable error. A run that fails to enqueue gets marked failed too, rather than sitting in a queued state forever, indistinguishable from one that is about to start.
"Still working" and "died four minutes ago" must never look the same in the UI. That is a lesson from a morning spent discovering that a store had been down since midnight because nothing had reported anything at all.
What is genuinely different now
I do not want to overstate the continuity.
The failure surface is probabilistic. A POS installer either ran or it did not. An agent can complete successfully and be wrong, which is a category of failure the retail work never had. No amount of checkpointing helps with that; only evaluation does.
Cost is a per-attempt variable. Retrying an install cost bandwidth. Retrying a model call costs money proportional to the conversation so far, which means retry policy is now a budget decision. That is why my agent tree has three different caps — a rate limit, a quantity limit, and a money limit — where a release process needed none of them.
Non-determinism at the routing layer. A build either includes a component or it does not. An agent may take a different path on the same input, so "replay and compare" is not a straightforward correctness check.
Why I bother writing this down
There is a fairly strong current assumption that agent engineering is a green field, and that experience predating it is a liability to be explained away in an interview.
I think the opposite. The hard parts of agent systems are release engineering, distributed failure handling, and operational observability — and those have thirty years of accumulated answers attached. The model call is the easy part. It is one HTTP request.
What makes a system trustworthy is everything around it: knowing what ran, knowing what it cost, being able to stop it, being able to see afterwards what happened, and having somewhere to go back to when it goes wrong. I learned all five of those from a job where getting them wrong meant a store could not sell anything.