A multi-minute agent run should not die on one transient failure

The first version of my research agent ran the whole loop inside one HTTP request. Submit a task, the request stays open, the agent calls the model, fetches a page, calls the model again, and eventually the response comes back.

It worked. It also had two properties I did not like once I looked at them properly:

• A run could take longer than the request could. Multiple model calls plus a page fetch is not reliably a sub-thirty-second operation.
• Any failure lost everything. A transient fetch timeout on iteration four discarded the three iterations before it. Nothing was persisted; there was nothing to resume.

The second one is the real problem. It is not that failures happen — it is that the unit of failure was the entire run.

The rewrite, in one sentence

Every unit of background work goes inside its own checkpointed step, so a crash or a transient failure costs only the step that failed, not the run.

Browser → POST /api/runs → insert `runs` row → inngest.send(...)
   → durable function → checkpointed step.run(...) per unit of work
   → each step written to Postgres → Supabase Realtime → live UI

The request returns the moment the row is inserted. Everything after that happens in a background function that can outlive it.

What "a unit of work" turns out to mean

Getting the granularity right is the whole design. Too coarse and you have not gained anything; too fine and you are checkpointing bookkeeping.

export const runAgentFn = inngest.createFunction(
  { id: "run-agent", retries: 3, triggers: [{ event: "agent/run.requested" }] },
  async ({ event, step }) => {
    await step.run("mark-running", async () => { /* ... */ });
 
    const planText = await step.run("plan", async () => { /* one model call */ });
 
    // the loop; each iteration and each tool call is its own step
    const { finalAnswer } = await runAgentIterations(messages, { step, /* ... */ });
 
    await step.run("finalize", async () => { /* ... */ });
  },
);

The steps are: mark running, plan, each model call, each tool execution, finalise. That last pair is where the value is. A tool call that times out retries by itself, with the conversation up to that point already durable — the model calls before it are not repeated, and more importantly not re-billed.

Retries make idempotency a design requirement, not a nicety

retries: 3 is one word in a config object and it changes what your code is allowed to do.

A step that may run more than once must be safe to run more than once. In practice that meant being deliberate about which side effects live inside which step:

• Reads and pure computation — free to retry, no thought required.
• The model call — retrying costs money but is semantically safe. This is the one place where the retry budget is a real budget.
• Database inserts — the dangerous one. A step that inserts a run_steps row and then does something that can fail will insert that row again on retry, and the trace grows a duplicate.

The fix is not clever: keep the write at the end of a step whose earlier part can fail, or give the row a key that makes the insert naturally idempotent. What matters is that you have to decide this per step, and the decision is invisible until the first retry happens in production.

The failures that used to vanish

The part I would emphasise to anyone doing this rewrite is not the happy path. It is that durable execution changes what a failure looks like, and you have to handle the new shapes explicitly or runs get stuck.

There are two of them, and both were silent in v1:

A run that fails to enqueue. The row is inserted, inngest.send(...) throws, and now there is a queued row that no background function will ever pick up. It sits in the list forever, looking like it is about to start. The user has no way to tell it apart from a run that genuinely is about to start.

A run that fails mid-execution. The function throws on iteration four. Without explicit handling, the run row stays running — a status that is now permanently wrong.

Both get an explicit terminal state:

} catch (err) {
  const message = err instanceof Error ? err.message : String(err);
 
  await step.run("mark-failed", async () => {
    await insertStep(runId, stepIdx++, "error", "Run failed", message);
    const { error } = await supabaseAdmin
      .from("runs")
      .update({ status: "failed", error: message, updated_at: new Date().toISOString() })
      .eq("id", runId);
    if (error) throw new Error(`Failed to mark run as failed: ${error.message}`);
  });
 
  throw err;
}

Two details worth pointing at.

The failure handler is itself a step, so recording the failure is durable too. If the process dies between throwing and writing, the retry writes it.

And it rethrows. Marking the run failed for the user is not the same as telling the platform the job succeeded. Swallowing the error here would mean a run the user can see failed, and an execution history that says everything is fine.

Live updates without polling

Because every step is written to Postgres the moment it happens, the browser does not have to ask what is going on. It subscribes to postgres_changes on the runs and run_steps tables, filtered by run id, and each row appears as it is inserted.

Two properties fall out of this that I did not fully anticipate:

• Closing the tab costs nothing. The run continues; reopening reloads the full history from Postgres and resubscribes. There is no in-memory session to lose.
• The live trace and the audit trail are the same data. I did not build a streaming channel and a history table. The history table streams.

That second one is the argument for doing persistence-first rather than streaming-first. If you stream from memory and persist as an afterthought, you end up with two representations that drift. If you persist and subscribe, there is only one.

What I would tell v1

The version of this I would build first, knowing what I know now:

1. Decide the unit of failure before writing the loop. It determines your step boundaries, and retrofitting them means restructuring the loop.
2. Give every run an explicit terminal state, including the ones that fail before they start. Enqueue failure is a real state, not an edge case.
3. Write every step as it happens, then subscribe to the table. Do not build a separate live channel.
4. Keep the loop itself free of the execution engine. Mine takes an interface it can satisfy in a test with two lines — which is a separate post, and the reason the safety properties are actually tested.

The durable function is src/inngest/runAgent.ts; the walkthrough of the whole flow is in HOW_IT_WORKS.md.