Durable Agent Execution

Durable Execution: Run the Agentic Loop as a Replayable Log

Background & Principle

An ordinary agent loop keeps its state in process memory: the messages array, the loop counter, local variables. The process dies, it's all gone. Durable execution engines (Temporal / Restate / DBOS) swap the model: every time your workflow function completes a "durable step," the engine writes that step's input and output into a journal (event log). After a crash, the engine reruns the same function on a fresh worker — but steps already in the journal aren't actually re-executed; their recorded return values are replayed back until execution catches up to the crash point, then continues.

This model is tailor-made for agents: LLM calls are slow and expensive (1–30s, real money), tool calls may have side effects, tasks span minutes. Temporal adds only about 10–50ms of overhead per durable step — negligible against an LLM call — in exchange for "kill the worker however you like, the agent resumes from the last successful step." DBOS is even lighter, storing the journal straight into your existing Postgres: one pip package, no separate broker or control plane.

Example

DBOS makes "which parts are durable steps" clearest — @DBOS.workflow is the replayable orchestration, @DBOS.step is a journaled atomic step:

from dbos import DBOS

@DBOS.step()                       # each step's return value is written to the Postgres journal
def call_llm(msgs): return client.messages.create(...)

@DBOS.step()
def run_tool(name, args): return TOOLS[name](args)

@DBOS.workflow()                   # on crash, resume from the last completed step
def agent(task):
    msgs = [{"role":"user","content":task}]
    for _ in range(30):
        r = call_llm(msgs)               # succeeded turns aren't re-sent on replay
        if r.stop_reason == "end_turn": return r
        out = run_tool(...)              # crash here → prior steps all skipped on restart
        msgs += [...]

You barely change the agent logic — you just wrap the "expensive / side-effecting" operations in a step and the loop in a workflow. The execution semantics go from "crash loses everything" to "crash resumes."

Checkpoint ≠ Durable Execution: Snapshots Can't Save a Half-Run Node

Background & Principle

Many assume "I use a LangGraph checkpointer / I persist state to a DB, therefore I'm durable." Not so. A checkpoint is a state snapshot saved after each super-step (node boundary); it gives you memory, human-in-the-loop, and node-level fault tolerance. But crash granularity is brutal: if a node internally calls an LLM, then sends an email, then crashes on the third line — on restart the entire node replays from line one: the LLM is re-called, the email re-sent. Diagrid's Checkpoints Are Not Durable Execution nails it: snapshots record "which cells you visited," not "which irreversible actions you took inside a cell."

LangGraph itself exposes three durability modes so you can pick the trade-off: exit (persist only on exit — fastest, no mid-run recovery), async (persist asynchronously — small chance of losing the last step), sync (persist synchronously before each next step — slowest but safest). Even sync only pushes recovery granularity to the node boundary; atomicity within a node is still on you. That's the fundamental gap versus step-level journal engines like Temporal/Restate.

Example

# ❌ Dangerous: one node holds two side-effecting actions
def node_book_trip(state):
    flight = charge_card(state)      # side effect 1: charge succeeds
    email  = send_confirm(flight)    # crash here → charge_card runs AGAIN on replay!
    return {"booking": email}

# ✅ Fix A: split into single-side-effect nodes; checkpoint boundary = side-effect boundary
graph.add_node("charge", charge_node)   # at most one irreversible action per node
graph.add_node("email",  email_node)
# compile with the safest mode, bind thread_id as the resume handle
app = graph.compile(checkpointer=saver, durability="sync")
app.invoke(inp, config={"configurable":{"thread_id":uid}})

The discipline: in a checkpoint engine, "one node = one safely-replayable unit." Slice nodes down to "at most one irreversible side effect" so the snapshot boundary actually aligns with the danger boundary. Otherwise, move up to a step-level journal engine (§03).

Exactly-Once Side Effects: Journal Your LLM & Tool Calls, Then Add Idempotency

Background & Principle

Durable execution's "exactly-once" has an easily misread boundary: what the engine guarantees is that a step's return value is materialized in the journal exactly once — on replay it won't call your function body twice. Restate's ctx.run(), Temporal's activity, DBOS's @DBOS.step all mean this. But the engine can't stop the wrapped call itself from producing duplicate effects on the outside world — if your step charges twice on its first execution, the journal faithfully records that result.

So the real discipline is two layers: (1) wrap every external call as a step/activity so the engine handles "don't re-execute on replay"; (2) for non-idempotent external APIs, carry an idempotency key inside the step so the downstream service handles "dedupe duplicate requests." Restate even supports request-level idempotency keys with automatic dedup. LLM calls, though "externally side-effect-free," must still be wrapped as steps — otherwise replay re-burns tokens for nothing.

Example

# Idempotency key = a stable, reproducible ID within the workflow (never random/timestamp!)
@DBOS.step()
def charge(user, amount, idem_key):
    return stripe.PaymentIntent.create(
        amount=amount, customer=user,
        idempotency_key=idem_key)      # ← Stripe dedupes; replay/retry charges once

@DBOS.workflow()
def checkout(user, cart):
    # key derived from workflow identity → unchanged across replays → naturally idempotent
    key = f"pay-{DBOS.workflow_id}"
    charge(user, cart.total, key)      # crash+replay: step already recorded → not re-executed
    send_receipt(user)                 # if it crashes here, charge won't fire twice

The key thing is the key's origin: it must derive from the workflow's durable identity (workflow_id + step sequence), so every replay computes the same key. Using uuid4() or a timestamp as the key is the classic trap — each replay makes a new key, downstream can't tell it's the same transaction, dedup fails.

Replay Determinism vs LLM Non-Determinism: Push the Uncertainty into Activities

Background & Principle

Replay works only if, when the workflow function reruns, its control flow follows the exact same path byte-for-byte — otherwise it replays into a branch that doesn't exist in the journal and the engine is lost. That forbids the workflow body from directly containing: random numbers, now()/clocks, direct network/file IO, and… direct LLM calls. All are "sources of non-determinism." This is a different determinism from Day 56's sampling determinism: that issue was "same prompt, same output"; here it's "same function reruns the same path."

The fix is uniform and elegant: wrap every source of non-determinism in a step/activity. On first execution it truly runs and its result enters the journal; on replay it doesn't run — it emits the old journaled value. So from the workflow's view, even a dice roll becomes deterministic. The LLM's non-deterministic output is "frozen" into the log, and the workflow body returns to a pure, replayable state machine. This also unlocks durable execution's killer capability: long suspend — with a durable timer + signal, an agent can "sleep" for days awaiting human approval, holding zero live process, and wake to resume the moment the signal arrives.

Example

# Temporal style: workflow deterministic, non-determinism goes to activities
@workflow.defn
class ApprovalAgent:
    def __init__(self): self._approved = None
    @workflow.signal                       # external approval arrives via signal
    def approve(self, ok): self._approved = ok
    @workflow.run
    async def run(self, task):
        plan = await workflow.execute_activity(  # LLM call = activity
            call_llm, task, start_to_close_timeout=TIMEOUT)
        # sleep until "someone approves" or 3-day timeout — zero live process
        await workflow.wait_condition(
            lambda: self._approved is not None,
            timeout=timedelta(days=3))
        if self._approved:
            await workflow.execute_activity(apply_plan, plan, ...)

Note call_llm is never called directly in the workflow body — it's always an activity. wait_condition + signal lets the agent hang for days without burning a cent of compute — the cleanest way to land an "async human approval queue" (Day 34).