Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.

Agent state inventory — where in-process suspended state lives today

Status: stage 1 of the RFC #4002 sequence (re-entrant conversation-as-state turn model), per the maintainer response: inventory (this document) → journaled prototype with a kill-mid-turn replay test → handoff evaluation. Docs only; nothing here changes behavior. Every file/symbol reference below was verified against the v5 branch at the time of writing (line numbers drift; symbol names are the stable citation).

The problem

RFC #4002 proposes making the conversation the durable state and each agent turn a re-entrant function call, so an agent survives a pod roll because nothing about it is suspended in-process. Before that can be prototyped, we need an honest answer to the prior question: where does in-process suspended state actually live in hive today, and what happens to each piece when the pod rolls?

We already know the shape of the answer from operational history — the last month re-taught “in-process state dies on a pod roll” one subsystem at a time (#3771, #3817, #3968, #3792, #3989, #3997, #3876), each as its own incident-and-fix cycle. This document consolidates those lessons, verifies each one against current code, and sweeps for the state nobody has been burned by yet. Section 4 classifies every row for the RFC: absorbed by the conversation-as-state model, kept as infrastructure state, or owned by a backend CLI that hive cannot externalize without headless turns.

1. The current agent-loop model

1.1 Hosted agents: a tmux pane owned by the manager

A hosted agent is not a process hive controls turn-by-turn. It is a backend CLI (claude, codex, copilot, agy, goose, bob, …) launched into a detached tmux session that the spoke’s agent manager owns (src/pkg/agent/manager.go, Manager / AgentProcess; sessions are named hive-<agent> — manager.go:846). The manager’s contract with the agent is deliberately thin:

• Launch: launchInTmux creates the session with a deep scrollback (defaultTmuxHistoryLimit = 50,000 lines, raised from tmux’s 2,000 default by #3790 because the default silently capped both browser scrollback (#3694) and the full-log endpoint (#3693)) and a 500-column pane (defaultTmuxPaneWidth, #3878 — the CLI truncates tool-call lines at render time, so a narrow pane destroys log content before it ever reaches scrollback).
• Kick: the governor decides an agent is due (src/pkg/governor/governor.go, State.LastKick) and the manager types a prompt into the pane (deliverKickLocked).
• Observe: a 3-second poller captures the pane (pollTmuxOutputForAgent → AgentProcess.lastPaneCapture / LastPaneChange) and watchdogs infer progress from what the pane looks like — stall nudges, consent-screen dismissal, login detection (NeedsLogin), no-action detection via tool-marker counting (lastInferKickMarks).

Everything the manager knows about a running turn is therefore an inference over terminal text. The turn itself — the conversation, the tool calls, the partial work — is suspended inside the backend CLI’s process and rendered into a tmux buffer. Neither is a structure hive can checkpoint, hand off, or re-enter.

1.2 The backend CLI owns the conversation

Each backend records its own transcript, in its own format, in its own home:

• Claude Code: ~/.claude/projects/<project-hash>/*.jsonl (src/pkg/tokens/claude_scanner.go, ScanClaudeSessions)
• Copilot CLI: ~/.copilot/session-state/ (src/pkg/tokens/copilot_scanner.go, ScanCopilotSessions)
• bob: ~/.bob/tmp/<uuid>/chats/<session-id>.json (src/pkg/tokens/bob_scanner.go)

Hive reads these files today — but only for token accounting (src/pkg/tokens/collector.go), never for resume. A subtle and important consequence of the deployment layout: agent homes live on the PVC (HOME=/data/home — manager.go:1972; /data is the hive-data PersistentVolumeClaim, src/pkg/hub/saas_provision.go deployment template), so the transcript bytes survive a pod roll. What dies is the session: the CLI process, its in-flight turn, the tmux server whose scrollback was the observable surface, and any backend-internal suspended state. Hive has no contractual way to turn the surviving bytes back into a running conversation — each CLI’s resume semantics are its own, vary by backend, and are exactly the “(b) backend-native session resume” path the maintainer response classifies as a per-backend optimization, not a foundation.

1.3 The relay/scrape layer

The contributor relay (bin/contributor-relay.sh — a Node program despite the extension) is the WebSocket client that connects a contributor agent to a hive’s contribute plane. In interactive mode it drives the same pane-shaped contract from the outside: it types tasks into a tmux session and classifies the pane text to decide whether the agent is working, idle_complete, or blocked_on_human (classifyTmuxPane, contributor-relay.sh:1193, with per-backend regex branches).

This layer is where scrape brittleness becomes a bug class of its own. The agy branch of classifyTmuxPane carries the freshest scar (#4026): agy narrates in prose, a leftover “Running…” line from a previous task pinned the classifier to working forever, the relay kept renewing the task lease, and the task was never reclaimed. The fix — scope the activity scan to the pane tail — is a regex-window adjustment, which is the point: when the conversation is a terminal buffer, correctness lives in regexes over prose.

1.4 The contrast case: contribute headless mode

The same relay also has a headless mode (MODE_HEADLESS, contributor-relay.sh:85, added for #2538; per-backend one-shot invocations — claude -p "<prompt>", codex exec "<prompt>" — verified per backend as recently as #4007). In headless mode there is no pane and no scraping: hive constructs the prompt, the backend runs one bounded non-interactive turn, and completion is reported as a structured verdict — since #3997 including no_work_needed, which is a structured turn return in exactly the RFC’s sense. Lifecycle state is recorded as JSON (HEADLESS_STATUS_FILE, default /tmp/contributor-headless-status.json).

This is why the maintainer response targets the contribute plane for the prototype: the headless path already demonstrates hive-owned turns with structured returns. What it does not yet have is a journaled, re-enterable operation list — its status file is a coarse lifecycle marker in /tmp, not a recoverable turn journal.

2. Inventory: volatile state and its blast radius

Scope note: “volatile” here means lost or reset when the owning process dies — whether the state is a Go map, a process’s heap, or a file on a non-PVC path (/tmp, /var/run). Rows marked fixed are the incident history: state that was volatile, burned us, and is now persisted or re-derived; they stay in the inventory because they map the pattern the RFC generalizes. Verdicts: volatile (lost, with consequences), benign (lost, self-healing by design), fixed (was volatile, now durable or re-derived).

2.1 Hub process (src/pkg/hub)

Durable-by-design hub state that shares the same file (s.mu → saveCh → saveLoop, server.go:3453): the registry itself (row: none — see §3). Already-persisted leaf stores: timeline (/data/saas/timeline/), journey (/data/saas/journey-state.json), alert acks (/data/saas/hub-alert-acks.json), revokedSessions (PVC, audit F10 — in-memory-only would un-revoke sessions on a roll), hub banners (/data/saas/hub-banners.json), reachHistory (/data/reach-history.json), key generations (/data/saas/hub-generations.json).

2.2 Spoke process (src/cmd/hive, src/pkg/agent, src/pkg/governor)

2.3 Contribute plane (src/pkg/dashboard/contribute_ws.go + relay)

Count: 38 rows. The pattern across the “fixed” rows is uniform and worth stating because it is the RFC’s whole argument: every one was in-process state whose loss was discovered as a production incident, and every fix was either (a) write it to the PVC, or (b) re-derive it from something that already was. The conversation-as-state model is the claim that rows 16, 17, 31, 32 and 37 — the ones that are not fixable by pattern (a) or (b) because the state is a suspended process, not a value — need a third pattern.

3. The durable stores that already exist

The maintainer response cautions that the fleet “already has four durable-state stores that grew this way” and the RFC’s envelope “should consolidate or at least map onto these, not become store number five.” The verified list is actually longer, which sharpens the caution:

Three properties recur, and any RFC envelope that joins this list must keep them: single writer (every file has exactly one owning process and lock), atomic replace (tmp+rename everywhere; a crash mid-write never leaves a torn file), and fail-open loads with explicit staleness (maxStateAge, banner pruning, streak expiry on load — old state is discarded, not trusted). The counter-lesson is also on record: hub-generations.json deliberately fails closed on an unreadable file because re-deriving from a default would re-install superseded key material (master-key-rotation.md). State that changes behavior when lost must decide, per store, which failure mode is the safe one — a decision the envelope design cannot skip.

And the caution stands: nine stores grown one incident at a time is the argument for mapping the envelope onto existing stores (the contribute ledger dir is the natural home for turn envelopes of contribute tasks) and against minting /data/turn-envelopes.json as store number ten.

4. Classification for RFC #4002

4.1 Absorbed: turn-scoped state the conversation-as-state model owns

Rows 16, 31, 32, 34, 36, 37 — plus the verdict/cooldown rows (27–30) at the boundary. The maintainer response’s minimal envelope — task ref + conversation + verdict + cooldown clocks — falls directly out of this table:

• task ref: what leases + currentTask + generation fencing express today across three in-memory structures whose safety depends on dying together (rows 31/32/34). In the turn model the task ref is in the envelope, the claim machinery stays authoritative for who may hold it, and a re-entering process proves possession of the envelope instead of a live socket.
• conversation: what the tmux pane and the backend transcript are today (rows 16/17). Absorbed only on the headless path, where hive constructs the prompt and owns the transcript (§1.4).
• verdict: already structured and already durable (row 30) — #3997 built the RFC’s turn-return shape without calling it that.
• cooldown clocks: rows 27–29 stay in the contribute ledger, but the envelope must reference them, because a re-entered turn’s admission decision depends on them.
• The ops journal — the piece nothing in this inventory provides — is what makes row 16’s side-effect problem tractable: a turn that died after opening a PR must find that fact in its own journaled history on re-entry, not discover it by opening a second PR. That is stage 2’s kill-mid-turn replay test, and no current store records it.

Row 33 (assignmentTimes) is a judgment call: it is not turn state, but it is the same “enforcement that silently resets on a roll” class. It should either ride the contribute ledger like rows 27–30 or be explicitly documented as best-effort; today it is neither.

4.2 Infrastructure state: stays exactly where it is

Rows 1–15, 18–20, 23, 26, 38 and every store in §3. Registry, hive meta, key material and generations, channel/upgrade delivery, governor cadence and budget, breaker, heartbeat liveness, config overlays, claim dedupe. None of this is conversation; all of it already has a durability story (some of them paid for in incidents). The RFC should not touch these — and conversely, the envelope should not quietly absorb any of them, because their writers, locks and failure modes (fail-open vs fail-closed) were chosen per store.

4.3 Backend-owned: cannot be externalized without headless turns

Rows 17, 21, 24 and the backend-format transcript files of §1.2. For hosted tmux agents, the conversation belongs to the CLI: hive can read the bytes (token scanners do) but cannot construct, checkpoint, or re-enter the session, and every observation of it is a regex over pane text (#4026 being the freshest reminder of what that costs). These rows are why the prototype target is a contribute headless turn and not a hosted agent: on the headless path the pane-ownership problem does not exist, so the spike measures the turn model, not the scraping. Backend-native resume stays what the maintainer response called it — a per-backend optimization to evaluate after the envelope survives the replay test.

5. Residuals and open questions

Honest gaps in the current state of things, and in this inventory:

1. The persist window. /data/hive-state.json is written once per eval cycle and on clean shutdown. A SIGKILL between ticks loses up to one cycle’s worth of pause/override/LastKick mutations. Nobody has been burned hard enough to make it write-through; the RFC’s journal, if it lands, should not inherit this cadence.
2. Row 32’s coupling is undocumented elsewhere. Resolved by #5681. The warning was accurate and it was load-bearing: leases were made to survive restarts, and the fence would have broken silently had taskGen not been re-derived from the restored leases in the same change. The coupling is now stated in loadLeases itself and pinned by a test (TestLeaseRestart_GenerationAdvancesPastRestoredLeases), so it is no longer carried only by this document.
3. Rate limits reset on every roll (row 33). Known now; not yet an incident; cheap to fix inside the existing ledger dir.
4. Webhook loss (row 3) has no re-derivation path. It predates this effort and is unrelated to the turn model, but it is the one hub row where real external input is dropped with no retry on either side.
5. This inventory is a snapshot. It was compiled by reading v5 at one commit, seeded by the incident history and a sweep of sync.-guarded fields in the three big processes. The sweep was systematic but not mechanical; a lint that flags new in-memory maps in HubServer / Manager / ContributeWSHub without a durability comment would keep it from rotting. Line numbers cited will drift; symbols are the stable reference.