Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.
crux serve: the runtime over a local API
crux serve exposes a crux.Runtime over HTTP+JSON so a process other than
the one that started an agent can run, observe, kick and stop it. It is the
Stage 3 process boundary of the Crux design record
(design/crux.md, RFC
#13536, stage 2/3 tracker
#14138): a harness other
than hive drives agents through it, and the crux CLI has a remote mode that
talks to it. Hive’s optional durable lane participant uses
separate project-authenticated lane RPCs on this server; that does not change
its agent-control hosting mode or grant lane tokens agent-control authority.
The wire contract is the stdlib-only package crux/api
(src/crux/api), which a client can import without pulling in the runtime.
The server is crux/serve; the Go client is api.NewClient.
Start a server
cd src && go build -o crux ./cmd/crux
crux --config /etc/hive/hive.yaml serve
# serving 3 agent definition(s) on unix:///run/user/1000/crux/crux.sock
serve builds the runtime from --config (the enabled agents: of a
hive.yaml) and listens until interrupted, stopping nothing on exit: agents
it started keep running in tmux. Without --config it serves an empty
runtime and every agent must arrive as a definition on POST /v1/agents.
| Flag / env | Meaning |
|---|---|
--socket PATH, CRUX_SOCKET | unix socket to serve on. Default $XDG_RUNTIME_DIR/crux/crux.sock, else <tmp>/crux-<uid>/crux.sock. The directory is created 0700, the socket 0600, and it is never reachable at that path with wider permissions (it is bound under a temporary name, chmod’ed, then renamed). A socket somebody already answers on is refused; a stale one is replaced. |
--listen HOST:PORT | also serve on TCP. Refused without a token. |
--token T, CRUX_TOKEN | bearer token every request must carry. When set it gates the socket too. |
--lanes-config PATH, CRUX_LANES_CONFIG | opt into durable event/work lanes on this same server, backed by NATS JetStream. Lane project credentials are separate from agent-control credentials. |
--hive-callback ADDR, CRUX_HIVE_SOCKET (+ CRUX_HIVE_TOKEN) | dial hive’s callback endpoint (crux/api/callback) for the capabilities hive owns — scoped GitHub tokens, PR creation, Linear credentials, text/canary scanners, conditions — and feed the runtime’s seams from it instead of leaving them nil. Health reports it under hive_callback; an unreachable endpoint raises the hive_callback_unavailable condition. (#14246) When the runtime is a manager it also polls hive’s host policy from the same endpoint; see Host policy from hive. |
serve does not use the per-work-dir control socket that a foreground
crux run serves (<workdir>/crux.sock); the two are different things, see
crux-cli.md.
Fail closed
- No TCP without a token:
--listenwithout--token/CRUX_TOKENis a usage error. When the server has no token the handler accepts only requests it can see arrived over its own unix socket; anything else (including aHandler()mounted on a foreignhttp.Server) gets401. - With a token, every request on every listener needs
Authorization: Bearer <token>; anything else is401withWWW-Authenticate: Bearer realm="crux". - Socket mode
0600, directory0700. The directory must be a real directory (not a symlink) owned by the serving uid with no group/other bits, orListenrefuses to place a socket in it; a pre-existing world-readable<tmp>/crux-<uid>is an error, not a trust. - Request bodies are capped at 1 MiB and unknown JSON fields are rejected.
The API
All routes are under /v1. Bodies and responses are JSON; agent-control errors are
{"error": "...", "code": "unauthorized|bad_request|not_found|conflict|runtime_error"}.
Opt-in lane routes require a separate project bearer credential and capability
ACLs; see the lane routes and request types
and lane error codes and failure behavior.
| Route | Body | Runtime method | Returns |
|---|---|---|---|
GET /v1/health | {"version":"v1","agents":N} | ||
GET /v1/agents | Observer.AllStatuses | {"processes":[Process…]}, sorted by name | |
POST /v1/agents | {"name":…, "definition":{…}} | Definitions.AddAgent (when definition is present), Lifecycle.Start | 201 {"agent":…, "process":Process}; 400 bad name, 409 the name already has a definition, 422 the runtime refused it |
GET /v1/agents/{name} | Observer.GetStatus | Process | |
GET /v1/agents/{name}/logs?source=pane|buffer|full&n=N | GetOutput / GetBufferOutput / FullLogHeader+CaptureFullLog | {"agent","source","lines":[…]} or {"header","content"} for full | |
GET /v1/agents/{name}/logs?follow=1&n=N | GetBufferOutput, polled | text/event-stream of agent.log records {"agent","lines":[…]}; the first replays the last n buffered lines (none for n=0; the 200-line follow window when n is absent) | |
POST /v1/agents/{name}/kick | {"message":…} | Kicker.SendKick | {"agent","process"} |
POST /v1/agents/{name}/stop | Lifecycle.Stop | {"agent","process"} | |
GET /v1/events | watcher + Kicker.SetKickObserver + the crux.Host observers | text/event-stream of Event records | |
GET /v1/lanes and lane/group subroutes | Publish/recovery/receipt JSON; request shapes | lanes.Broker | Opt-in only; lane-project credential and capability ACLs; JSON or 204; full routes and responses |
Runtime families (#14138 stage 3)
Every method of crux.Runtime has a route; api.RuntimeRoutes is the
method → route table and crux/api/client’s parity test fails when a method
is added to the interface without an entry. The sidecar client
(crux/api/client.Runtime, selected by CRUX_MODE=sidecar) is written
against this table. Routes marked ⚑ are new in stage 3; the rest predate
it or are the CLI routes above under another method.
| Family | Route | Runtime method(s) | Notes |
|---|---|---|---|
| Lifecycle | POST /v1/agents/{name}/start ⚑ | Start | |
POST /v1/agents/{name}/restart ⚑ | Restart, RestartWithBootstrap ({"bootstrap":…}) | ||
POST /v1/agents/{name}/pause ⚑ | Pause, PauseBy, Supervisor.PauseByCause | body selects the variant | |
POST /v1/agents/{name}/resume ⚑ | Resume | ||
DELETE /v1/agents/{name}/restart-count ⚑ | ResetRestartCount | ||
GET /v1/agents/{name} | IsPaused, Observer.ResolveAgent | ||
| Observer | GET /v1/snapshots, GET /v1/agents/{name}/snapshot ⚑ | AllStatuses, GetStatus, GetStatusFast | full crux.Process snapshot, not the wire projection |
| Kicker | POST /v1/agents/{name}/kick/validate ⚑ | ValidateKick | 409 kick_refused |
GET /v1/agents/{name}/kick ⚑ | KickDispatchState | ||
GET /v1/agents/{name}/kicklogs[/{id}] ⚑ | ListKickLogs, ReadKickLog | ||
| Definitions | POST /v1/definitions, POST /v1/definitions/reconcile ⚑ | AddAgent, ReconcileAgents | |
DELETE /v1/agents/{name}, PUT /v1/agents/{name}/definition ⚑ | RemoveAgent, UpdateConfig | ||
PUT /v1/agents/{name}/overrides, PUT/DELETE /v1/agents/{name}/pins/{kind} ⚑ | SetModelOverride, SetBackendOverride, PinCLI, PinModel, UnpinCLI, UnpinModel | ||
POST /v1/fleet/modes/{clear,sync}, POST /v1/fleet/inference/refresh ⚑ | ClearModeOverrides, SyncModeFiles, RefreshInferenceRoutes | ||
| Fleet | GET /v1/fleet ⚑ | GetACMMLevel, SharedDevUID, Supervisor.ConsentWedgedAgents, CountAgentsWithModel | one read |
GET/PUT /v1/fleet/breaker, POST /v1/fleet/breaker/{engage,release} ⚑ | BreakerState, Restorer.RestoreBreaker, EngageBreaker, ReleaseBreaker | ||
PUT /v1/fleet/acmm, POST /v1/fleet/relaunch-bob ⚑ | SetACMMLevel, RelaunchBobAgentsAwaitingKey | ||
| Credentials | POST /v1/agents/{name}/login-code ⚑ | SubmitLoginCode | |
GET/PUT /v1/credentials/copilot ⚑ | CopilotToken, CopilotTokenSource, SetCopilotToken, ActivateCopilotToken | ||
GET /v1/credentials/claude, POST /v1/credentials/claude/reload ⚑ | ClaudeCredentialCandidatePaths, ReloadClaudeToken | ||
| Resumer | GET /v1/agents/{name}/session ⚑ | ResumeHandle, SessionID, SessionMissing | |
POST /v1/agents/{name}/session/kick ⚑ | SendResumeKick | 409 resume_busy, 410 resume_session_gone, 422 resume_unsupported map back to the crux.ErrResume* sentinels | |
| Supervisor | GET /v1/agents/{name}/supervision ⚑ | GovernorMayStart, AgentCredentialRefreshable, AgentHasValidCredential, AgentCapabilities, EffectiveBackend, InvocationMetadata, RestartTelemetry, StartFailureState, ProviderErrorBackoffRemaining, CrashOOMSuspected, AgentAuthAvailable | one read |
POST /v1/fleet/{startup-queued,restart-crashed,tokens/refresh,kicklogs/archive} ⚑ | MarkStartupLaunchQueued, CheckAndRestartCrashedAgents, RefreshAgentTokens, ArchiveAllKickLogs | ||
POST /v1/agents/{name}/tokens/refresh, POST /v1/agents/{name}/nudge ⚑ | RefreshAgentTokenFor, SendGitHubAPIBudgetNudge | ||
GET /v1/backends/{backend}/auth, POST /v1/audit, GET /v1/trajectory ⚑ | BackendAuthAvailable, RecordAudit, TrajectoryAgents | ||
| Restorer | POST /v1/agents/{name}/seed ⚑ | SeedPauseState, SeedRestartTelemetry, SeedTurnLoss, SeedLastKick, SeedKickHistory | one body, each field optional |
| Background | GET /v1/health | StartAgentTokenRefresh, StartCredentialWatchdog, StartCopilotSessionRefresh, StartTranscriptActivityRefresh | the loops run server-side; the client’s calls are no-ops |
| Authorizer | POST /v1/authorize ⚑ | AuthorizePROpen, AuthorizePushBranch, AuthorizeIssueOpen, AuthorizeMerge, AuthorizeReviewRequest | synchronous; {"action","agent","file_uid","kind","repo"}; 403 forbidden carries the refusal, any transport error is a refusal (fail closed) |
| Watchdog | GET /v1/watchdog/agents, GET /v1/agents/{name}/watchdog, PUT /v1/agents/{name}/watchdog/conditions ⚑ | watchdog.Source (AgentNames, Observe, LastProduction, SetConditions) | 501 not_supported when the runtime has no watchdog source; crux/runtime.NewWatchdogFleet builds the fleet over these in sidecar mode |
Process is the wire projection of crux.Process: name, id, state,
backend, model, pid, paused (+ reason, trigger, by), started_at,
last_kick, restart_count, last_restart_reason, kick_refused,
kick_refusal_reason, kicks_undeliverable, busy_condition,
start_failure_class. It is a subset by design: the full snapshot carries
engine-internal state that is neither stable nor meaningful off-process.
definition on POST /v1/agents is the JSON form of
crux/config.AgentConfig, the per-agent overlay hive reads from
agents_dir. The server applies the same gate and defaults the YAML loader
does: name must match ^[a-z0-9][a-z0-9_-]*$ (≤64 chars); id and
role default to the name, replicas to 1, enabled to true unless the
JSON says otherwise. A name the runtime already knows is a 409 conflict
rather than a silent start of the old definition; stop and re-post. Without
definition, name may also be an agent ID. Parsing the portable
AgentDefinition format is the caller’s job, as it is the host’s job for
the CLI (the CLI does it before sending).
Agents start under the server’s lifetime, not the request’s: the runtime ties an agent to the context it was started with, so a client disconnecting does not kill what it started.
Events
GET /v1/events is a server-sent-event stream; each record is
event: <type> then data: <Event JSON>, with : keep-alive comments
every 15 s. Event is {"time","type","agent","state","detail","data"}.
| Type | When |
|---|---|
agent.started, agent.stopped, agent.kicked | the API performed the action |
agent.state | the server’s watcher (polls AllStatuses every second) saw an agent change state for any reason: crash, restart, pause; state is the new state, detail the last restart reason when there is one |
agent.kick | the runtime’s kick observer fired (Kicker.SetKickObserver); detail is the observer’s event name |
agent.kick_outcome, agent.pause_transition, agent.relaunch, agent.pause_persist, agent.prompt, agent.sandbox_audit, runtime.audit, runtime.condition | the crux.Host observers, relayed when the runtime is a Host; data carries the typed payload (api.KickOutcome, api.PauseTransition, …). A sidecar hive demuxes these into the observers its boot sequence installs (cmd/hive’s sidecarHost) |
A subscriber that stops reading is dropped after 64 buffered events so one slow client cannot stall the others.
Host policy from hive (#14138 stage 4)
With --hive-callback, serve also reads the values an in-process hive
hands the manager through the crux.Host policy setters. They travel on
the hive callback endpoint (crux/api/callback, runtime → hive), not on
this API:
| Callback route | Feeds |
|---|---|
GET /v1/policy | a callback.Policy snapshot, polled every 15 s (hivecap.WatchPolicy): build → SetBuildInfo, explain_mode_default → SetExplainModeDefaultResolver, kick_log_source → SetKickLogSourceResolver, gateway_backends → SetGatewayBackendChecker, dead_session_recovery_owned_elsewhere → SetDeadSessionRecoveryOwner, sandbox (the agent_sandbox block) → SetSandboxConfig; repo_auto_merge: true installs the next row |
POST /v1/policy/repo-auto-merge | {"repo"} → {"enabled"} for SetRepoAutoMergeEnabledResolver, asked per merge authorisation |
The resolvers answer from the last snapshot (the manager reads them under
its own lock, so they never call out) and keep it through a hive outage;
the auto-merge call fails closed, refusing the merge. A value hive does not
report, or a hive without the policy capability (404 not_configured),
leaves the runtime’s own default in place.
Every crux.Host setter has a disposition in api.HostSetters, and
TestParityHostVersusSetterTable fails when one is added without an
entry. The remaining setters, all on the same callback endpoint:
| Setter | Callback route (capability) | Fail-closed behaviour |
|---|---|---|
SetInferenceCallbacks | PUT/DELETE /v1/inference/routes/{agent} (inference_routes): the manager pushes each agent’s inference route into hive’s proxy | route not installed; the agent runs without a per-agent route |
SetSandboxMutationBoundary | POST /v1/effects, POST /v1/effects/{id}/end (effects): two-phase claim/end over hive’s journaling boundary | mutation refused |
SetAppAuth | POST /v1/tokens with {agent, tier} and no repos (installation-wide, what AppAuth.ScopedToken(tier) mints in-process); hivecap.AppTokenWriter writes the token in place (O_TRUNC, 0600) to the agent’s existing cache file and nowhere else, never logging it. HIVE_PROXY_INJECT_GH_AUTH is refused in sidecar mode (ErrProxyInjectionUnsupported) rather than writing the real token | cache file untouched; the agent keeps its last token or runs read-only |
SetAgentMint | POST /v1/tokens/mint (agent_mint); a hive with mint disabled answers 404 not_configured, which the writer maps to the in-process “mint off” contract ("", no error) | no mint token |
SetBobAPIKeyResolver, SetBobKeySourceResolver | GET /v1/bob/credential (bob); cached 1 min runtime-side, a failed refresh serves the stale value for at most 2 min | empty key and source |
SetSpanStarter | — in-process-only: a span is a Go value wrapping a call in one address space. The sidecar’s manager starts spans on the OpenTelemetry global tracer (a no-op without an exporter in the runtime’s process); trajectory events do not depend on it | n/a |
The proxy seams are carried the same way: hivecap.Set.{Accounting, Usage, Canary} come from GET /v1/github/rate-limit, POST /v1/usage
and POST /v1/scan/canary, so a proxy built on them refuses writes while
hive is away and resumes on the next successful call. No setter is served
from a runtime: config key: every value is already resolved hive-side
from the live config, so a reload reaches the runtime on its next poll.
Supervision and failure semantics (#14138 stage 4)
In sidecar mode hive decides who keeps crux serve alive:
| Env | Values | Meaning |
|---|---|---|
CRUX_SUPERVISE | child (default), external | child: hive spawns crux serve itself (crux/runtime.Supervisor), waits for /v1/health, and restarts it with exponential backoff (1 s → 1 min, reset after 5 min healthy). external: systemd / a pod sidecar runs it; hive only reconnects. |
CRUX_BINARY | path or name | the executable child runs; default crux on $PATH. A missing binary is logged loudly and hive behaves as external. |
The child inherits hive’s environment plus CRUX_TOKEN; its listen flags
are derived from CRUX_SERVER (unix:// → --socket, host:port →
--listen). /v1/health now carries started_at and uptime_seconds
next to hive_callback.
When the socket is gone, whichever way it is supervised:
| Surface | Behaviour |
|---|---|
| governor loop | keeps ticking; nothing in hive blocks on the socket |
| writes with an error return (kick, stop, resume, …) | refused with crux.ErrRuntimeUnavailable (wraps api.ErrUnreachable). Kicks are not queued or replayed: the governor re-derives what is due on its next tick, so a replay buffer would only duplicate work after a long outage |
| observer reads | answer zero / unknown (unchanged from stage 3) |
| fleet report | crux_runtime_unavailable (component agent-runtime, severity high) raised on the first failed probe (every 15 s, or any failed call) and cleared on the first success; the dashboard renders it like every other runtime condition |
hive /api/health | crux_runtime: {mode, reachable, last_error, since, supervise, child{running, ready, pid, starts, last_exit}}; status becomes degraded while unreachable |
client Status() | {reachable, last_error, since, failed_calls, last_success} on api.Client/client.Runtime; a caller that cancelled its own context is not counted, an expired deadline is |
In-process mode is untouched: there is no socket, the condition never
fires, and /api/health carries no crux_runtime member.
Drive it with curl
S=/run/user/$(id -u)/crux/crux.sock
curl -s --unix-socket $S http://crux/v1/health
curl -s --unix-socket $S http://crux/v1/agents | jq '.processes[] | {name, state}'
# run from a definition (an agents_dir-style AgentConfig document)
curl -s --unix-socket $S -X POST http://crux/v1/agents \
-H 'Content-Type: application/json' \
-d '{"name":"fixer","definition":{"backend":"claude","model":"claude-sonnet-4-6","role":"fixer"}}'
curl -s --unix-socket $S 'http://crux/v1/agents/fixer/logs?source=buffer&n=50' | jq -r '.lines[]'
curl -s -N --unix-socket $S 'http://crux/v1/agents/fixer/logs?follow=1' # streams until Ctrl-C
curl -s --unix-socket $S -X POST http://crux/v1/agents/fixer/kick \
-H 'Content-Type: application/json' -d '{"message":"look at the failing build on main"}'
curl -s -N --unix-socket $S http://crux/v1/events # in another terminal
curl -s --unix-socket $S -X POST http://crux/v1/agents/fixer/stop
Over TCP with a token:
CRUX_TOKEN=$(openssl rand -hex 16) crux --config hive.yaml serve --listen 127.0.0.1:7777
curl -s -H "Authorization: Bearer $CRUX_TOKEN" http://127.0.0.1:7777/v1/agents
The hostname in a unix-socket URL is ignored; crux is a convention.
Drive it with the CLI
--server ADDR (or CRUX_SERVER) switches run, ps, logs, kick and
stop to the API; --token (or CRUX_TOKEN) carries the bearer token.
ADDR is a socket path, unix:///path, host:port or http(s)://host:port.
export CRUX_SERVER=/run/user/$(id -u)/crux/crux.sock
crux run examples/agents/customized-agent.yaml # definition parsed here, sent as JSON; follows output; Ctrl-C stops
crux run --detach fixer # a name already defined on the server
crux ps
crux logs -n 50 --buffer fixer
crux logs -f fixer # follow the ring buffer over SSE
crux kick fixer "look at the failing build on main"
crux stop fixer
In remote mode nothing is built in this process: no runtime, no control
socket, no --config (the server’s definitions are what exist). serve
itself cannot be run remotely. A 401 is reported with a hint to set the
token.
Drive it from Go
import "github.com/hivecommons/hive/crux/api"
c, err := api.NewClient(os.Getenv("CRUX_SERVER"), os.Getenv("CRUX_TOKEN"))
procs, err := c.List(ctx)
resp, err := c.Run(ctx, api.RunRequest{Name: "fixer", Definition: defJSON})
err = c.FollowLogs(ctx, "fixer", func(ch api.LogChunk) error { …; return nil })
err = c.Events(ctx, func(ev api.Event) error { …; return nil })
_, err = c.Stop(ctx, "fixer")
crux/api imports only the standard library, so a client binary does not
link the runtime. Errors from the server are *api.StatusError;
api.IsNotFound and api.IsUnauthorized classify them.
Client.WaitReady polls /v1/health for a caller that just launched
crux serve.
What this is not
For agent control it is not tenancy, quotas or hive-as-a-client: those routes
trust whoever can reach the socket or holds the token, and hive still constructs
*agent.Manager in-process. Those are the rest of Stage 3 in
design/crux.md.
The opt-in durable lanes are a separate project-authenticated surface with explicit capability ACLs, configured per-lane retention/storage bounds and per-group lease/ledger limits. Hive uses them as an ordinary lane client. This does not provide general multi-tenant isolation, change Hive’s agent hosting model or grant lane credentials agent-control privileges.