Operating

How durability works

Durability is the product. This page explains the mechanism precisely: the queue, leases, checkpoints, event sourcing and replay — and what each one costs you when it is missing.

The cost argument, concretely#

Take a ten-node workflow that enriches accounts and emails a summary. Node 9 calls a provider that times out. What happens next is the entire difference between a durable engine and a script.

The queue and the worker#

Starting a run does not execute it. The API validates the request, writes a run row in queued, and returns. A worker picks it up later. That indirection is what makes the system survivable: nothing is holding your run in memory, so nothing can drop it.

Leases#

A worker leases a run rather than owning it. The lease is a row with an expiry, renewed every few seconds while the worker is alive. If the worker is killed, deployed over, or loses its network, the lease simply expires and another worker claims the run from its last checkpoint.

This is why a deploy in the middle of a run is uneventful. There is no drain step and no graceful-shutdown handler to get right — the failure mode is the same as a crash, and the crash path is the one that is tested constantly.

Checkpoints and commit boundaries#

Every node ends at a commit boundary. The node’s validated output, the inputs that produced it, and the idempotency key of any side effect it caused are written in one transaction. Either all of it lands or none of it does.

A checkpoint is a commit boundary that the engine will never cross again. Once node 6 is checkpointed, node 6 will not run a second time in this run — not on retry, not on resume, not after a worker crash.

Idempotency keys#

Checkpoints protect you inside Vectorbea. Idempotency keys protect the systems outside it. A tool call that creates a charge carries a key derived from the run id and node id, so if the engine is ever uncertain whether a call landed, the provider deduplicates it rather than charging twice.

1export const chargeCustomer = defineTool({
2  name: "stripe.charge",
3  risk: "critical",
4  timeoutMs: 30_000,
5
6  // The key is deterministic: the same node in the same run always
7  // produces the same key, so a replay can never double-charge.
8  idempotencyKey: ({ runId, nodeId }) => `${runId}:${nodeId}`,
9
10  async run({ input, connection }) {
11    return connection.post("/v1/charges", input);
12  },
13});

Event sourcing and replay#

A run’s history is an append-only event stream: run.accepted, node.started, node.committed, tool.called, gate.opened, run.resumed. Current state is derived from the stream, never stored as the truth.

That is what makes resume exact rather than best-effort. Resuming is not “work out roughly where we were”; it is folding the event log to a state and continuing. The same fold drives the replay scrubber in run detail, so what you watch afterwards is what actually happened, not a reconstruction.

Retries, timeouts and the dead-letter path#