Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.
The agent turn model and where in-process state lives (#4002)
Status: spike / investigation — steps 1 and 2 complete, no production decision taken.
Step 1 of RFC #4002 documented how hive drives an agent turn today and where
per-agent state lives that would not survive a process restart. Step 2 now adds
an isolated, contribute-shaped prototype in pkg/turn: a re-entrant Step
function over a serialized conversation envelope, with journaled external
effects and crash/reload tests. Nothing constructs it in the live tmux agent
loop or contributor relay. Steps 3 (handoff path) and 4 (feasibility + migration
cost) remain untouched.
Every claim below carries a file:line citation against origin/v4 at the time
of writing. Line numbers drift; the function names are the durable handle. Where
the RFC’s framing does not survive contact with the code, this page says so
plainly — that is the main thing a spike is for.
Summary of findings
- There is no “turn” in the code. No
Turntype, no per-turn context, no per-turn goroutine, no turn return value. A turn is an emergent property of pane text: it begins when hive types a prompt into a tmux pane and ends when the pane matches an idle-input-prompt marker again. - There is no agent-owned conversation to externalize. Hive never holds the conversation. The backend CLI does, in its own private on-disk format, and hive treats it as opaque. Hive durably records the prompts it sent and a rendered-text transcript of the terminal, and neither is a conversation.
- Hive already externalizes a lot of per-agent state — via
/data/hive-state.json— but that state is control-plane state (paused, pins, restart counts, backoff ladders), not conversation state. - A restart does not resume a turn; it abandons one.
Restartkills the CLI and the tmux session outright, and the replacement receives no prompt at all until the governor’s next tick happens to find it due — the kick that arrives carries no signal that a restart occurred. - The one thing that does survive is accidental, not designed, and it works only when the tmux server outlives the hive process — which a pod roll or an image upgrade rules out.
1. How a turn is actually driven
1.1 The CLI is long-lived; the turn is not a process
The backend CLI is launched once per tmux session, as an interactive
foreground process typed into a shell pane, and it stays there across many
turns. launchInTmux creates a detached shell session
(src/crux/agent/manager_tmux.go:172, newSessionCommands, which issues a plain
tmux new-session -d at src/crux/agent/manager_tmux.go:201) and then types the CLI
invocation into that shell:
m.tmuxSendLiteralForAgent(agent, fullCmd) // src/pkg/agent/manager_tmux.go:553
time.Sleep(textToEnterDelay)
m.tmuxSendEntersForAgent(agent) // src/pkg/agent/manager_tmux.go:558
The composed command line is per-backend and is built as a string —
claude --model … --dangerously-skip-permissions
(src/crux/agent/manager_launch.go:849), copilot … --allow-all
(src/crux/agent/manager_launch.go:859), and so on.
Nothing in the kick path ever spawns a process. deliverKickLocked
(src/crux/agent/manager_kick.go:267) only manipulates the existing pane.
1.2 A turn starts with simulated keystrokes
SendKick (src/crux/agent/manager_kick.go:102) is the entry point; it delegates to
deliverKickLocked (src/crux/agent/manager_kick.go:267), which:
- clears stale input with
C-cthenC-u(Codex uses onlyC-u; Goose skips clearing) (src/crux/agent/manager_kick.go:318-330), - types the prompt literally, chunked, via
tmuxSendLiteralForAgent(src/crux/agent/manager_kick.go:365, implementation atsrc/crux/agent/manager_tmux.go:553→tmux send-keys -l), - submits with
tmuxSendEntersForAgent(src/crux/agent/manager_kick.go:396).
Pane writes return errors to the kick caller. A failed clear, prompt chunk, or
Enter stops delivery, increments the existing undeliverable counter, and leaves
LastKick and successful-kick history unchanged. A final readiness refusal or
restart during chunk delivery also returns an error, so synchronous dispatch
and asynchronous kick status cannot mistake a skipped delivery for success.
The governor evaluation log lists idle reviewer lanes under agents_skipped;
no synthetic kick timestamp delays their next check for work.
The prompt reaches the model the same way a human’s keyboard would. There is no API call, no stdin pipe, and no structured request that hive could inspect, replay, or hand to another process.
1.3 A turn “ends” when the pane looks idle
Nothing calls Wait() on the CLI for a turn — the process does not exit between
turns, so there is no exit to wait for. Turn completion is inferred from pane
text, by two independent mechanisms:
The readiness gate before the next kick.
waitForInputPromptForAgent (src/crux/agent/manager_tmux.go:496) polls
captureTmuxPaneForAgent and returns once paneShowsInputPrompt
(src/crux/agent/panesignal/panesignal.go:193) matches a per-backend marker string. SendKick
calls it at src/crux/agent/manager_kick.go:203. This gate is the only thing that
stops hive typing a new prompt on top of an in-flight response.
The gate is bounded by inputPromptTimeout (120s), which is why operator kicks
from the dashboard do not use SendKick directly. SendKickAsync
(src/pkg/agent/kick_async.go) keeps the fast, deterministic preconditions
synchronous and runs this gate plus the typing on a background goroutine, so the
HTTP handler cannot outlive an ingress idle timeout and report a succeeding kick
as a 504 failure (hivecommons/hive#5325). Delivery is deduplicated per agent, so
a retried click cannot type the prompt twice. The governor’s tick still calls
SendKick synchronously — it has no proxy in front of it.
The pane poller. pollTmuxOutputForAgent
(src/crux/agent/manager_poll.go:13) runs a 3 * time.Second ticker
(src/crux/agent/manager_poll.go:15) for the agent’s whole lifetime, diffing
captured pane content to maintain agent.LastPaneChange
(src/crux/agent/manager_poll.go:102) — hive’s only evidence that a running,
authenticated CLI is actually doing something.
Stall recovery is likewise textual: nudgeIfKickStalled
(src/crux/agent/manager_kick.go:977) compares paneContentHash(pane) against
agent.lastInferKickPane (src/crux/agent/manager_kick.go:918), and
paneShowsActiveWork (src/crux/agent/manager_kick.go:857) matches a spinner or a
token-counter marker.
1.4 Scheduling: one shared ticker, not a per-turn continuation
Correction to the task framing: src/pkg/agent/scheduler.go does not exist.
Kick timing lives in pkg/governor; kick text is built in pkg/scheduler.
Governor.agentsDueForKick(src/pkg/governor/governor.go:887) comparesnow - LastKickagainst each agent’s configured cadence (AgentCadence,src/pkg/governor/governor.go:25).Governor.Evaluate(src/pkg/governor/governor.go:377) calls it atsrc/pkg/governor/governor.go:485and returns the due list.- The driving loop is a single ticker in
main:deps.newTicker(initialEvalInterval)atsrc/cmd/hive/main.go:5772, loop atsrc/cmd/hive/main.go:6112, evaluation viadeps.runEvalatsrc/cmd/hive/main.go:5956(runEvalCycle,src/cmd/hive/main_eval_cycle.go:471), message assembly viasched.BuildKickMessagesWithSkipsatsrc/cmd/hive/main_eval_cycle.go:964(src/pkg/scheduler/kickmessage.go:261), and delivery viaagentMgr.SendKickwired intodispatchAgentKicksatsrc/cmd/hive/main_eval_cycle.go:1127.
This matters for the RFC: the scheduler is already stateless with respect to
turns. It does not hold a continuation, does not await turn N before
planning turn N+1, and reconstructs the entire decision each tick from
LastKick timestamps. The control loop is arguably closer to the RFC’s model
than the RFC assumes. The suspended state is not in the scheduler — it is in the
CLI subprocess.
2. Where per-agent state lives
2.1 Durable — survives a process restart
| State | Where | Citation |
|---|---|---|
| Pause flag (one bool per agent) | /data/hive.yaml via AgentConfig.Paused | src/crux/config/agent_config.go:219; writer SetAgentPausedAndSave src/pkg/config/config.go:6524 |
Pause provenance (PausedAt, PausedReason, PausedTrigger, PausedBy), CLI/model pins, model/backend overrides, restart count, LastKick, truncated kick history | /data/hive-state.json via snapshot.AgentState | src/pkg/snapshot/state.go:82-117; path src/cmd/hive/spokewire.go:3 |
| Watchdog failure count, crash-loop latch, backoff deadline, healthy-since, conditions | same file, snapshot.PersistedState.Watchdog | src/pkg/snapshot/state.go:48; watchdog.PersistedAgent src/crux/watchdog/reconciler.go:216 |
| Fleet-breaker engagement + held set | same file, BreakerState | src/pkg/snapshot/state.go:56 |
| Governor budget/spend/eval history, cadence overrides, ACMM level | same file | src/pkg/snapshot/state.go:16-42 |
| Full text of every delivered prompt | /data/prompt-history.jsonl (lumberjack-rotated JSONL) | src/pkg/dashboard/prompt_history.go:24; writer Server.RecordPrompt src/pkg/dashboard/prompt_history.go:361, wired at src/cmd/hive/main.go:4035 |
| Rendered terminal scrollback, per kick | /data/logs/kicks/<agent>/<ts>-<reason>.log | src/crux/agent/kick_logs.go:22 (defaultKickLogDir); writer archiveKickLogLocked src/crux/agent/kick_logs.go:178 |
| Token-usage summary | /data/token-summary.json | src/pkg/tokens/collector.go:202, :608-611 |
| Structured audit trail | /data/audit.jsonl, reloaded into a ring at boot | src/pkg/dashboard/audit.go:23, loadFromDisk :189 |
| Agent-name → UID allocation | durable source /data/.hive/uid-map.json, runtime copy /var/run/hive/uid-map.json | PersistedUIDMapPath / UIDMapPath src/pkg/agent/uidmap.go; load src/pkg/agent/manager.go |
| Backend CLI’s own session/credential files | the CLI’s own HOME / CODEX_HOME, rooted at /data/home | per-agent CODEX_HOME src/crux/agent/manager_env.go:510, helper src/crux/agent/manager_homes.go:56; per-agent HOME src/crux/agent/interactive_home.go:29; shared .claude bridged by symlink interactive_home.go:67 |
Two caveats on that table:
- The UID map is rooted at
/data/.hive/uid-map.jsonand copied to/var/runfor runtime consumers. Existing entries are preserved exactly; new agent names getmax(existing uid)+1, so adding an alphabetically earlier agent cannot shift established per-agent state. On the first boot without a persisted map, the entrypoint adopts the unique owner UID of/data/home/agents/<agent>, falling back to/data/agents/<agent>and then/data/beads/<agent>, before allocating fresh UIDs for names with no owned state. - The last row is the important one for the RFC and is discussed in §5.
2.2 In-process — lost on restart
All of the following are fields of AgentProcess
(src/crux/agent/manager.go:208) or goroutines owned by the manager. None is
written anywhere.
| State | Field / mechanism | Citation |
|---|---|---|
| Live output ring buffer | OutputBuffer *RingBuffer — pure in-memory circular slice, no persistence path | src/crux/agent/manager.go:252; src/pkg/agent/ringbuffer.go |
| Last captured pane content | lastPaneCapture []string, guarded by paneMu | src/crux/agent/manager.go:253 |
| Activity clock | LastPaneChange (guarded by paneMu) | src/crux/agent/manager.go:366 |
| Most recent kick text | LastKickMessage — replaced on every kick; the durable copy is truncated (KickRecord.Snippet) or lives in prompt-history | src/crux/agent/manager.go:259; the prompt-history store is contrasted with the 120-character snippet at src/pkg/dashboard/prompt_history.go:186-188 |
| tmux session/socket identity | tmuxSession, tmuxSocket | src/crux/agent/manager.go:266-267 |
| Per-launch cancel func and launch generation | cancel context.CancelFunc, launchGen int | src/crux/agent/manager.go:268, :404 |
| Launch-serialization flag | launching bool (guarded by m.mu) | src/crux/agent/manager.go:278 |
| One-shot bootstrap override | BootstrapOverride | src/crux/agent/manager.go:283 |
| Token-restart backoff ladder | lastTokenRestart, tokenRestartAttempts, tokenRestartGaveUp | src/crux/agent/manager.go:303, :322, :325 |
| Login / quota observations | NeedsLogin, QuotaExhausted | src/crux/agent/manager.go:344, :351 |
| Consent-screen watcher timers | consentSeenAt, lastConsentDismiss | src/crux/agent/manager.go:367-368 |
| Stall-watchdog per-kick state | lastInferKickAt, lastInferKickPane, stallNudgeSent, lastInferKickMarks, actionNudgeSent | src/crux/agent/manager.go:369-372, :405, :424 |
| Transient-API-error nudge cooldown | lastTransientNudge, transientNudgesThisKick | src/crux/agent/manager.go:388-389 |
| Un-archived-scrollback flag | kickLogPending (guarded by m.mu) | src/crux/agent/manager.go:417 |
| Sandbox / bob-key latches, last launch banner | sandboxResumeAfterCancel, awaitingBobKey, lastLaunchFailureBanner | src/crux/agent/manager.go:435, :444, :471 |
| Poller goroutines themselves | go m.pollTmuxOutputForAgent(agent, agentCtx) and siblings, tied to a per-launch context | src/crux/agent/manager_launch.go:246, :249, :256 |
| Blocked-action thrash windows | Manager.thrash map[string]*thrashState under its own thrashMu — deliberately not m.mu, to avoid re-entrancy from the output-capture goroutines | src/crux/agent/manager.go:567-571; thrashState src/crux/agent/manager_thrash.go:65; trip logic recordBlockedAndCheck src/crux/agent/manager_thrash.go:104 |
Note the asymmetry: several counters that exist precisely to stop a runaway
loop (tokenRestartAttempts, transientNudgesThisKick, stallNudgeSent) are
in-process only. A restart resets each of them to zero. The watchdog’s own
backoff ladder is persisted (src/crux/watchdog/reconciler.go:374) specifically
so that it doesn’t have this property; the manager-side nudge counters were not
given the same treatment. This is an observation about current behaviour, not a
bug report — see Open questions.
2.3 tmux state, and the one accidental survival path
tmux is a separate process tree with its own server per agent socket
(src/crux/agent/manager_tmux.go:118 builds tmux -L <socket>). What tmux holds —
the running CLI process and its scrollback — is therefore outside hive’s memory
and can outlive a hive process restart.
launchInTmux exploits this. If the pane already shows a CLI marker and no
relaunch was forced, hive adopts the surviving CLI instead of launching one:
if !agent.forceRelaunch && m.tmuxPaneHasCLIForAgent(agent) {
m.logger.Info("CLI already running in tmux pane, skipping launch", …)
…
go m.pollTmuxOutputForAgent(agent, agentCtx)
return nil
}
(src/crux/agent/manager_launch.go:235-281)
Three things about this are worth stating precisely:
-
It is one of two reattach points, and both are early returns rather than a recovery routine. The other is
ensureTmuxSession(src/crux/agent/manager_tmux.go:285), whose first act isif m.tmuxSessionExistsForAgent(agent) { return nil }(src/crux/agent/manager_tmux.go:286-288) — a surviving session is reused, not recreated. Boot reaches both:mainunconditionally callsagentMgr.Start(ctx, name)(viadeps.startAgent) for every enabled agent (src/cmd/hive/main.go:4514) and the reuse-vs-relaunch decision is taken inside. There is noAdopt,Reattach, orRecoverAgentsfunction; searching for one finds onlyRestoreBreaker(src/crux/agent/manager_pause.go:382), which restores control metadata and is explicitly documented as not touching agent state: “a boot restore must never change agent state, only reattach the breaker” (src/crux/agent/manager_pause.go:379-380).Because reattachment is emergent from two independent early returns rather than an explicit path, nothing in the codebase names it, tests it end to end, or reports whether it happened. The only trace is a log line (
src/crux/agent/manager_launch.go:236). -
The adoption test is a screen-scrape.
tmuxPaneHasCLIForAgent(src/crux/agent/manager_tmux.go:441) is one line:paneHasCLIMarker( m.captureVisiblePaneForAgent(agent)). Hive decides whether an agent’s conversation is still alive by pattern-matching rendered terminal output. -
It does not survive the deployment’s actual restart mode. In hosted operation hive restarts by rolling the pod or replacing the image, which destroys every tmux server in the container. The
kick_logs.goheader states this outright: graceful shutdown must archive scrollback because “a pod roll or hive image upgrade … destroys every tmux server in the container” (src/crux/agent/kick_logs.go:19-21,:334-336).
So the reattach path helps only when the hive process restarts inside a surviving container. For the case the RFC names — spoke rolls — it does not apply.
3. What a restart actually does to a turn
Restart (src/crux/agent/manager_restart.go:434) does not resume anything. In order:
C-cinto the pane and cancel the launch context (src/crux/agent/manager_restart.go:574-579).- Archive the scrollback before it is destroyed —
tearDownTurnLocked(agent, "restart")(src/crux/agent/manager_restart.go:585), with the comment thatkill-sessiondestroys the only record of the previous run (src/crux/agent/manager_restart.go:581-584). - Reap the CLI process and any UID-owned helpers
(
src/crux/agent/manager_restart.go:602-611). tmux kill-session(src/crux/agent/manager_restart.go:614).
The replacement CLI starts with no prompt at all in the default case.
buildBootstrapPrompt (src/crux/agent/manager_env.go:19) unconditionally returns ""
(src/crux/agent/manager_env.go:27), with the reasoning recorded inline: the
governor’s first eval cycle kicks all due agents with fully substituted
templates, whereas sending a boot prompt here leaked unsubstituted ${ISSUE_LIST}
placeholders to the agent (src/crux/agent/manager_env.go:20-25).
So a restarted agent sits idle at its input prompt until the shared governor
ticker (§1.4) next finds it due. Only the explicit override paths supply text
directly: RestartWithBootstrap (src/crux/agent/manager_restart.go:286) sets
BootstrapOverride, and RestartThenSendKick
(src/crux/agent/manager_restart.go:383) is documented as restarting “with a clean
slate”. Whichever path runs, delivery is gated on the new pane reaching its
input marker (deliverStartupKick, src/crux/agent/manager_kick.go:514).
Restart is a reset, not a resume. Whatever the agent had established in its conversation — plan, working context, partial tool results — is gone, and the next thing it receives is an ordinary scheduled kick that carries no knowledge that a restart happened or that earlier work exists.
4. The watchdog
src/pkg/watchdog/ observes liveness and, in acting modes, restarts.
Its input is an Observation (src/crux/watchdog/classify.go:47) whose
substantive fields are Pane string — the visible pane, deliberately excluding
scrollback so “stale markers in scroll history must not vouch for a dead CLI”
(src/crux/watchdog/classify.go:52-53) — plus boolean verdicts derived from that
pane and a LastChange timestamp from the poller.
It classifies each pane into one PaneClass
(src/crux/watchdog/classify.go:13-31): ready, shell-prompt,
auth-required, stuck-overlay, no-output, no-session, unknown.
Dead() (src/crux/watchdog/classify.go:37) returns true for shell-prompt,
stuck-overlay, no-output, and no-session. auth-required is deliberately
excluded, because restarting into a dead credential produces a restart loop
(src/crux/watchdog/classify.go:34-36). The pattern tables it matches against are
literal CLI UI chrome — "Login expired", "Please run /login"
(src/crux/watchdog/classify.go:99-104), "[Next]", "Choose an accent"
(src/crux/watchdog/classify.go:110-119).
Its action is r.fleet.Restart(ctx, name) (src/crux/watchdog/reconciler.go:661,
via planRestartLocked at :576 and restartDetached at :657) — i.e.
exactly the destructive path in §3.
What the watchdog does about lost in-process state: nothing, and by design it
cannot. It restores its own ladder across restarts
(Snapshot/Restore, src/crux/watchdog/reconciler.go:374 and :408), so it
“neither forgets a crash-loop nor re-runs a backoff ladder from the top”
(src/crux/watchdog/reconciler.go:401-402). It has no mechanism to restore the
agent’s conversation, because none exists to call. From the watchdog’s point of
view a restarted agent is a fresh agent that happens to keep its name and its
failure count.
This is the sharpest way to state hive’s current position: the control plane is already durable; the agent is not.
5. Honest feasibility notes
These are constraints observed in the code, not arguments for or against the RFC.
5.1 Hive does not own the conversation, and cannot read it
Every supported backend is an opaque interactive subprocess. Hive composes a
command line as a string (src/crux/agent/manager_launch.go:849,
src/crux/agent/manager_launch.go:859, :888),
types it into a shell, and from then on interacts only through keystrokes in and
rendered characters out.
Hive sets HOME and a per-agent CODEX_HOME
(src/crux/agent/manager_env.go:510, helper at src/crux/agent/manager_homes.go:56) so
that each agent’s CLI writes its session files somewhere hive controls the
location of. That is location control, not format control: nothing in
src/pkg/agent/ parses, writes, or migrates a backend session file.
Notably, no launch path passes a resume/continue/session-id flag to any backend. The composed command lines carry model and permission flags only. There is therefore no existing mechanism by which hive could ask a relaunched CLI to pick up where the previous one stopped, even for a backend that supports it.
Consequence for the RFC: “the conversation is the durable state” presumes the orchestrator can serialize and reload the conversation. Hive today can do neither. Any conversation-as-state design must either (a) confine itself to backends that expose a documented, stable resume mechanism and accept that the envelope is backend-specific and not portable, or (b) move agents off opaque interactive CLIs onto an API-shaped backend hive drives itself. Both are substantially larger than “externalize hive’s state”, and the choice between them is a real fork the RFC does not currently name. This spike takes no position on which.
5.2 The two durable records hive does keep are not conversations
It is easy to mistake either for one; neither is.
/data/prompt-history.jsonlholds the input half only. It is the fully-expanded prompt text of every kick (src/pkg/dashboard/prompt_history.go:24,:361). The alternatives — in-memoryLastKickMessageand the 120-characterKickRecord.Snippet— could not answer “what was my agent asked to do?”; the store’s own comment still contrasts itself with the 120-character snippet (src/pkg/dashboard/prompt_history.go:186-188). No model response, tool call, or tool result is recorded./data/logs/kicks/…holds rendered terminal text, captured withtmux capture-pane -p -J(src/crux/agent/terminal.go:109-111). It is a human debugging artifact, retained to 10 files / 64 MiB per agent (src/crux/agent/kick_logs.go:28,:39). It contains ANSI-rendered, width-wrapped, spinner-animated output with no message boundaries and no role structure. It is not machine-replayable into any backend.
Together these mean hive can already tell you what it asked and what the screen looked like, and still cannot reconstruct what the model’s state was.
5.3 The state envelope precedent already exists
Whatever shape a handoff envelope takes, hive has a working pattern for it:
snapshot.PersistedState (src/pkg/snapshot/state.go:16), saved atomically via
a .tmp + rename (src/pkg/snapshot/state.go:179-190), aged out at seven days
(src/pkg/snapshot/state.go:14), restored at boot by
restoreAgentRuntimeState (src/cmd/hive/state_restore.go). The watchdog’s
PersistedAgent (src/crux/watchdog/reconciler.go:216) shows how a subsystem
contributes a durable subset of its in-memory record to that envelope without
persisting the whole thing, including the judgement call about which fields
are safe to restore under which mode
(src/crux/watchdog/reconciler.go:401-407).
If step 2 externalizes manager-side state, this is the mechanism to extend rather than a new one to invent.
5.4 Everything hive knows about an agent’s progress is screen-scraped
This is the load-bearing constraint behind all of the above and is worth
isolating. Turn completion (§1.3), liveness classification (§4), stall
detection (src/crux/agent/manager_kick.go:977), auth failure
(src/crux/watchdog/classify.go:99-104), quota exhaustion
(src/crux/agent/manager.go:351), and even whether a surviving CLI can be
adopted (src/crux/agent/manager_tmux.go:441) are all decided by matching substrings
against rendered terminal output.
A structured turn return value — the RFC’s testability and subagent-sync benefits — is not a refactor of an existing structured signal. There is no structured signal. It would be a new capability, and it requires a channel to the backend that hive does not currently have.
6. Step 2 prototype: a re-entrant contribute turn
The additive prototype in pkg/turn takes the smallest
headless slice that exercises the RFC’s state claim without pretending the tmux
path can provide a structured conversation. Its public transition is:
Step(SessionEnvelope, optional initial TurnPlan)
-> (SessionEnvelope, TurnOutput, error)
It carries the operation-journal lesson previously exercised on the earlier
v5 line in #4053 onto current
v4, and adds the real atomic file store plus fail-closed handling for effects
that cannot be reconciled.
The first invocation binds the ordered plan into the envelope. A later process
receives only the serialized envelope and calls Step with no plan. The
envelope carries the conversation, task and agent identity, bound operation
plan, operation journal, status, and terminal contribute verdict. The return is
structured (status, done, verdict, rationale, landed effects); no pane
substring decides whether the turn finished.
This is a prototype boundary, not a new production path. It does not call an
LLM, replace a backend CLI, or run from bin/contributor-relay.js. A headless
backend adapter would produce the initial plan; the spike starts immediately
after that model call, where re-entry meets external side effects and where a
duplicate is more damaging than repeated inference.
6.1 Durable operation protocol
Each side-effectful operation has a stable key derived from:
v1 + sha256(session | kind | repo | target | body)[:32]
The model’s tool-call ID is recorded for traceability but excluded from the key, because a repeated inference can mint a new call ID for the same semantic effect. Keys are bound into the plan before persistence, so scrub-on-persist cannot change operation identity.
An operation crosses two durable boundaries:
- Write
intendedto the envelope and commit it before the external write. - Perform the write.
- Write
succeededorfailedand commit again.
On re-entry, succeeded short-circuits. intended is the ambiguous crash
window: the write may have landed before the process died, so the runner asks a
remote reconciler before proceeding. If no reconciler exists, it fails closed
with ErrAmbiguous; it does not blindly replay. This is stricter than
at-least-once delivery and deliberately trades manual recovery for never
silently creating a duplicate PR or comment.
FileStore makes those boundaries real rather than test-only: it serializes
through the fleet’s logscrub boundary, writes a mode-0600 temporary file,
syncs it, and atomically renames it over the prior envelope. The in-memory
conversation is not redacted or mutated; only the artifact that can cross a
process or spoke boundary is scrubbed.
6.2 What the replay test establishes
TestStepReentersAtEveryPersistenceBoundary drives a contribute-shaped turn
(comment → push → PR create → label), kills it before every intent, settle, and
terminal-envelope commit, discards all in-memory state, reloads only serialized
JSON, and re-enters. Every case must produce:
- exactly one call per external effect;
- the same ordered effect summary and
shippedverdict as an uninterrupted control; - no ambiguous journal entries at completion.
Separate tests prove that an ambiguous operation without reconciliation never
replays, a known-not-landed operation does run, a failed operation can retry,
semantic keys survive a freshly minted tool-call ID, and the persisted artifact
contains no credential-shaped plaintext. pkg/turn remains above the
repository’s 90% package coverage floor under go test -race.
6.3 What step 2 does not establish
- Inference is outside the envelope transition. A crash before the plan is committed repeats inference and may produce a different plan. The prototype protects landed effects, not token cost or plan determinism.
- Reconciliation is effect-specific. PR-by-head has a good remote surrogate; pushes and high-traffic comments do not. Those effects stop as ambiguous instead of claiming exactly-once semantics they cannot provide.
- There is no concurrency claim. The journal makes sequential re-entry safe. It does not let two processes race one envelope. Step 3 must reuse the contributor relay’s existing offer→claim ownership rather than create a fifth durable-state store or parallel claim system.
- No backend conversation is captured. The package proves the envelope and side-effect protocol after a structured headless response. It does not resolve §5.1’s backend-native-resume versus API-shaped-backend fork.
- The beads comparison is not measured. A bead checkpoint can carry task notes, but not this operation journal or a structured backend response today. Step 4 still needs the proposed kill-at-50% comparison before recommending the larger migration.
Open questions
Things this spike did not establish, and what would settle each.
- Do the supported backends expose a stable resume mechanism, and is its session format documented? Not investigated here — it is a property of claude / copilot / bob / agy, not of hive, and no hive code exercises it. Settling it requires per-backend version-pinned testing against the actual binaries, since an undocumented on-disk format is not a contract.
- Should the manager-side nudge and restart counters join the persisted
envelope?
tokenRestartAttempts(src/crux/agent/manager.go:322),transientNudgesThisKick(src/crux/agent/manager.go:389), andstallNudgeSent(src/crux/agent/manager.go:372) are loop-breakers that reset to zero on restart, while the watchdog’s equivalent ladder is deliberately persisted (src/crux/watchdog/reconciler.go:401-402). Whether the difference is intentional (these counters are per-launch by design, and a fresh launch legitimately deserves a fresh budget) or an oversight is not determinable from the code alone and was not raised as a defect by this spike. Confirming it needs the author’s intent, not more reading. - What is the actual observed restart rate, and how much work is lost per
restart?
The RFC’s motivation is “frequent spoke rolls”, but this spike found no metric that quantifies work discarded by a restart.An instrument now exists (src/pkg/agent/turn_loss.go); the question is awaiting fleet data rather than awaiting a way to collect it. See Measuring the motivation below. - Is
buildBootstrapPrompt’s path-building dead code, or a staging post? Since answered: dead code. The function used to assemble a candidate policy-file list and then discard it by returning""; the path construction has since been deleted, and the function is now an explainedreturn ""stub (src/crux/agent/manager_env.go:19-27,return ""atsrc/crux/agent/manager_env.go:27), with the removal recorded in its comment. - Are there non-tmux agent execution paths with different properties?
src/pkg/agent/sandbox_executor.goruns a different shape of execution (src/crux/agent/sandbox_executor.go:405composes its own command line) and inference backends are handled separately (IsInferenceBackend,src/crux/agent/manager_routing.go:23). This spike scoped itself to the tmux CLI path, which is the fleet’s normal mode; the sandbox and inference paths were not mapped and may not share these constraints.
Measuring the motivation (open question 3)
The RFC argues from “frequent spoke rolls discard agent work”. That premise was
unsized: RestartCount (src/pkg/snapshot/state.go) counts restarts and says
nothing about what any of them cost. Step 4’s feasibility call depends on the
difference, and the measurement is cheap next to a prototype — so it is now
instrumented, in src/pkg/agent/turn_loss.go.
What is counted. A turn is interrupted when hive tears down an agent’s CLI
and tmux session while a delivered kick’s output has not yet been rotated
(kickLogPending) — the restart and shutdown paths, funnelled through
tearDownTurnLocked. Ordinary kick rotation is deliberately excluded: a kick is
only delivered into a pane sitting at its input marker, so rotation means the
previous turn finished, and counting it would bury the signal in completed work.
Why two clocks, and why neither alone is honest. kickLogPending means
“output not archived”, which is not the same as “the turn was still running”: an
agent that finished and then sat idle for an hour still carries pending output.
Charging that hour to the restart would overstate the loss — an overclaim in
favour of the RFC, produced by the RFC’s own instrument, which is exactly the
result nobody should trust. So each record carries:
| Field | Meaning |
|---|---|
SinceKick | Teardown minus kick delivery. An upper bound on what the interruption could have cost — never a measurement of what it did. |
SinceOutput | Teardown minus the last observed pane change. A large value beside a large SinceKick says the agent was idle and little was lost. Nil means the pane poller never saw a change, which reads as unknown and must never be read as idle. |
Producing | Whether the pane changed after the kick landed — the agent did something this turn. Binary and threshold-free, which is why the aggregate carries it: “interruptions that hit an agent doing work” is answerable across the fleet without anyone first agreeing how many idle seconds count as idle. Unknown resolves to false. |
Bytes | Archived scrollback at teardown, a proxy for the volume of turn output discarded. |
Analysis discounts with SinceOutput; the instrument does not decide for it.
Deciding would be taking step 4’s judgement inside a tool built to inform it.
Where it lands. Per-agent aggregates plus a bounded tail of individual
records ride the existing /data/hive-state.json through
snapshot.AgentTurnLoss — not a sidecar, per the maintainer caution against
adding durable store number five — and are re-seeded at boot
(SeedTurnLoss), because a counter that reset on restart would reset on exactly
the event it exists to count. Each interruption is also logged as
audit: turn interrupted mid-flight, so the fleet-wide question is answerable
from log aggregation without reading every spoke’s PVC.
Reading it. The headline is producing / interruptions — the share of
teardowns that certainly discarded work — with upper_bound_s as the ceiling on
cost and the recent tail showing the shape (a few long turns vs. many short
ones):
jq '.agents | to_entries[] | select(.value.turn_loss)
| {agent: .key, interruptions: .value.turn_loss.interruptions,
producing: .value.turn_loss.producing,
upper_bound_s: .value.turn_loss.upper_bound_s}' /data/hive-state.json
What it still does not answer. It sizes what a teardown discards, not what recovery would cost, so it does not by itself settle the beads-checkpointing comparison raised on the issue — that needs the killed-at-50% experiment, which wants this baseline first. And it measures the tmux CLI path only: the sandbox and inference paths (open question 5) have different teardown semantics and are not instrumented here.
Status: instrument only. It changes nothing about when or how an agent is
torn down. tearDownTurnLocked replaces three hand-copied archive-then-clear
blocks with one funnel so the accounting cannot drift between the restart paths,
and is otherwise behaviour-preserving.
What this page does not say
It does not say the RFC is infeasible, and it does not say it is worthwhile. Step 4 is where that judgement belongs, and it should be made with §5.1’s fork named explicitly and Open question 3 answered.
Step 3 — the handoff evaluation — is now written up separately in
Evaluating a handoff path for the re-entrant turn model.
It re-checks §6.3’s residuals against v4, and reports that §5.1’s fork is no
longer deferrable: the envelope it describes is handoff-able on the headless
path only.