Loop Engineering, Part 3: State, Recovery and the Right to Continue

The most dangerous question after a crash is not “where were we?” It is “did the action already happen?”

Imagine a loop that opens a pull request, publishes a release, sends a message, or deploys a service. The external system accepts the action. Before the loop records the result, the worker dies.

The next worker arrives to an honest blank:

• Retry, and you may do it twice.
• Assume success, and you may skip work that never happened.
• Start over from the chat summary, and you are trusting prose written before the crash.

This is where loop engineering stops being a clever prompt pattern and becomes systems engineering.

A loop earns the right to continue only when it can reconstruct the past well enough to make the next action safe.

The conversation is not the state

Long context windows are useful. Summaries are useful. Neither should be the source of truth for a durable loop.

Conversation is a working view. It is shaped for the model, compacted for cost, and sometimes disconnected from the world between turns. Durable state has a different job: preserve the facts another process needs to resume.

A fresh worker should be able to answer:

1. What goal is this loop pursuing?
2. Which steps definitely completed?
3. Which side effects are confirmed?
4. What evidence did the checker return?
5. What budget remains?
6. Is the run active, waiting, done, or failed?

If any answer exists only in the previous model’s memory, the loop is not resumable. It is merely hopeful.

Store events, not a polished story

The most useful primitive I found while building the NEXUS loop runtime was an append-only event log.

Every meaningful step becomes an event. The exact names vary by runtime; across the NEXUS loop and the Frontier human-approval rail, the useful vocabulary includes:

• grounding
• model
• tool_requested
• tool_result
• check
• retry
• compaction
• approval_requested
• approval_decided
• learning
• error
• status

Each event gets a monotonically increasing sequence number. The sequence becomes a cursor: “I have safely observed everything through event 41.”

This design gives you two things a summary cannot:

Replay

A new process reconstructs the run from facts in order. It does not need the old process or its private memory.

Resume

A client can disconnect, reconnect with its last cursor, and receive only the events it missed. A hard refresh becomes an inconvenience, not a lost run.

The event log should be the source of truth. Live notifications are only a wake-up mechanism. If a notification is missed, the cursor still finds the event.

Recovery requires ownership, not just replay

A resumable run can still execute twice.

A scheduler retries after a timeout. The old worker is slow rather than dead. A new worker acquires the same session, replays the same events, and both processes believe they own the next action.

The event log preserves order. It does not decide who may append the next event.

For a local runtime, an exclusive lease can be enough:

with store.lease(session_id):
    events = store.events(session_id)
    state = replay(events)
    advance_once(state)

For a distributed runtime, lease expiry is not enough. The old worker may wake after ownership has moved. Every mutation needs a fencing token that changes with the owner:

def append_event(session_id, event, fence):
    if fence != store.current_fence(session_id):
        raise StaleOwner("run ownership moved")
    store.append(session_id, event)

Resumable does not mean concurrently replayable. Before a fresh worker advances the run, it needs both the history and the current right to write.

Idempotency answers “may I try again?”

An append-only log tells you what was recorded. It does not automatically tell you whether an unrecorded external action happened.

For side effects, the loop needs an idempotency key: a stable identity for one intended action.

If the worker retries deploy:production:067138a, the deployment system should either:

• return the original result; or
• report the current status of that same action.

It should not start a second production rollout because the network response was lost.

Where the external system does not support idempotency, add a status lookup before retry. The unsafe option is blind repetition.

The honest state between intent and observation is unknown.

If the log contains an authorised tool request but no result, the run should not resume model or tool work as though nothing happened:

status = "failed"
reason = "incomplete_tool_turn"
error  = "external outcome unknown; reconciliation required"

Failing closed here is not the same as declaring that the external action failed. It means the loop cannot prove whether the action happened, so it has not earned another side effect.

This is why retries belong inside the loop contract. “Retry three times” is not resilience unless the operation is safe to repeat.

The log cannot contain what the observer missed

Durability does not repair incomplete observation.

In August, one of my scheduled pull-request review loops ran repeatedly against what appeared to be a stable six-item queue. A seventh pull request had opened, but it never appeared in the run snapshots. The log faithfully preserved every decision the loop made from an incomplete view.

That incident sharpened the contract:

• record the query scope, timestamp, pagination state, and source revision;
• distinguish “observed empty” from “observation failed”;
• treat partial pages, stale caches, and connector errors as evidence gaps;
• reconcile periodically against an independent listing or watermark.

Replay can recover everything the system observed. It cannot recover a fact the sensor never recorded.

Retry transient failure, not every failure

A rate limit, timeout, or temporary service error may deserve exponential backoff. An invalid request, denied permission, or failed policy check usually does not.

The loop should record every retry:

{
  "type": "retry",
  "operation": "model_step",
  "attempt": 2,
  "delay_seconds": 4,
  "error": "429 rate limited"
}

That record matters later. It shows whether the system recovered from a transient condition or spent its budget repeating a deterministic failure.

The breaker can then make a better decision:

• transient and improving: continue;
• deterministic and unchanged: hand off;
• ambiguous side effect: resolve status;
• budget exhausted: stop.

Completion proof belongs to the current run

Replay can show that an earlier worker passed a check. It does not prove the world still satisfies that check now.

A deployment may have drifted. A file may have changed after acceptance. A fresh run may inherit a green workspace produced by the previous run without performing the failure-first work required by its own contract.

The completion receipt should therefore bind:

• the run ID;
• the exact acceptance check;
• the artifact or environment revision;
• the evidence returned by the checker;
• a final execution of that check against current state.

A prior run’s victory is historical evidence, not permission for a new run to declare success.

Context must be managed separately from history

The durable event log grows. The model’s context window cannot grow forever with it.

Do not solve that by deleting history. Keep the full record durable, then build a bounded working context for each model step.

A practical context manager:

1. keeps the newest events verbatim;
2. compacts older events into a summary;
3. preserves unresolved decisions and critical evidence;
4. records that compaction happened;
5. stays inside a fixed token budget.

The model sees a useful working set. The system keeps the full truth.

That separation is easy to miss. If the compacted prompt becomes the only history, every summary mistake becomes permanent. If the full event log is always sent to the model, cost and context eventually explode.

Durable history and model context are different products.

Learning needs an outcome, not just activity

Once a loop is durable, it is tempting to make it self-improving: count what the agent used, reinforce it, and feed it into future runs.

Be careful. Activity is not success.

In NEXUS, this problem appeared in retrieval. A document could be boosted simply because search returned it often. That creates a popularity loop:

returned often -> ranked higher -> returned more often

Nothing in that cycle proves the document helped.

The fix is to record a trajectory:

query -> retrieved evidence -> action -> outcome

Then reinforce only from a positive, independent outcome: the source was cited, the answer was grounded, the check passed, or the task succeeded.

This principle applies beyond retrieval:

• Do not learn from a draft merely because it was generated.
• Do not learn from a tool merely because it was called.
• Do not learn from a plan merely because the agent sounded confident.
• Learn from work that survived the checker and produced a real outcome.

A loop that learns from its own activity can become confidently worse. A loop that learns from verified outcomes has a chance to improve.

That is a design requirement, not a claim that the runtime has solved general self-improvement. The recovery tests later in this article verify replay, ambiguity, and ownership; they do not verify that a learned policy improves future outcomes.

What the operating loop taught me

The theory became clearer after several weeks of hourly pull-request review runs:

Observation	What it proved
24 consecutive runs with an empty queue and zero mutations	Safe repetition often means doing nothing, repeatedly and verifiably.
Five reviews posted once, then skipped on later runs	Idempotency is visible as the absence of duplicate side effects.
A protected workflow-file conflict was detected and routed to a person without pushing a branch	Human handoff is a valid terminal disposition, not failed autonomy.
Connection errors and an idle timeout were recorded as non-runs with no observation	Infrastructure death must not become success-shaped state.
An existing resolver comment was repaired in place, then a candidate branch was verified upstream and skipped while unchanged	Stable identities and status checks make repeated operation safe.
A newly opened pull request was missing from repeated snapshots	A durable record can still preserve an incomplete reality perfectly.

The most valuable runs were not the ones with the most activity. They were the runs that could explain exactly why no mutation was owed.

The minimum durable loop

You do not need a distributed event platform to begin. A local JSONL file or small database can carry the fundamentals.

For each run, persist:

State	Minimum field
Identity	stable session ID
Purpose	goal and acceptance condition
Progress	append-only sequenced events
Side effects	idempotency keys and confirmed results
Judgment	checker verdicts and evidence
Control	attempt, cost, time, and remaining budget
Disposition	running, waiting, done, or failed
Learning	outcome reported separately from the attempt

Then prove one recovery scenario before adding scale:

1. Start the loop.
2. Let it take one real but reversible action.
3. Kill the worker between steps.
4. Start a fresh worker.
5. Confirm it replays the record, avoids duplicate action, and resumes from the correct point.

If that test is hard, it is telling you where the hidden state lives.

The durable runtime behind this work now keeps four focused recovery regressions:

test_store_appends_monotonically_and_resumes_from_cursor
test_resume_does_not_repeat_taken_turns
test_resume_with_ambiguous_tool_outcome_fails_closed
test_concurrent_runner_fails_fast_without_duplicate_work

....                                                                     [100%]
4 passed

These tests do not prove general autonomy. They prove four narrow claims this article depends on: ordered replay, no repeated completed turn, fail-closed ambiguity, and single-owner advancement.

A loop is a promise across time

The maker-checker-breaker pattern gives a loop bounded authority. Durable state gives that authority continuity without turning memory into guesswork.

The event log says what was observed. The cursor says what has been seen. The lease says who may advance. Idempotency says what may be retried. Reconciliation resolves what remains unknown. The checker says what passed now. The outcome says what is worth learning.

Together they answer the only question that matters after interruption:

Does this loop have the evidence to continue?

If yes, resume. If no, stop honestly and hand the uncertainty to a person.

That is not a limitation of autonomy. It is what makes autonomy trustworthy enough to repeat.

---

Start with the series: The Repeat Is the Product explains the six-part loop contract, followed by Maker, Checker, Breaker for bounded autonomy.