Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.

crux serve: the runtime over a local API

crux serve exposes a crux.Runtime over HTTP+JSON so a process other than the one that started an agent can run, observe, kick and stop it. It is the Stage 3 process boundary of the Crux design record (design/crux.md, RFC #13536, stage 2/3 tracker #14138): a harness other than hive drives agents through it, and the crux CLI has a remote mode that talks to it. Hive’s optional durable lane participant uses separate project-authenticated lane RPCs on this server; that does not change its agent-control hosting mode or grant lane tokens agent-control authority.

The wire contract is the stdlib-only package crux/api (src/crux/api), which a client can import without pulling in the runtime. The server is crux/serve; the Go client is api.NewClient.

Start a server

cd src && go build -o crux ./cmd/crux
crux --config /etc/hive/hive.yaml serve
# serving 3 agent definition(s) on unix:///run/user/1000/crux/crux.sock

serve builds the runtime from --config (the enabled agents: of a hive.yaml) and listens until interrupted, stopping nothing on exit: agents it started keep running in tmux. Without --config it serves an empty runtime and every agent must arrive as a definition on POST /v1/agents.

Flag / envMeaning
--socket PATH, CRUX_SOCKETunix socket to serve on. Default $XDG_RUNTIME_DIR/crux/crux.sock, else <tmp>/crux-<uid>/crux.sock. The directory is created 0700, the socket 0600, and it is never reachable at that path with wider permissions (it is bound under a temporary name, chmod’ed, then renamed). A socket somebody already answers on is refused; a stale one is replaced.
--listen HOST:PORTalso serve on TCP. Refused without a token.
--token T, CRUX_TOKENbearer token every request must carry. When set it gates the socket too.
--lanes-config PATH, CRUX_LANES_CONFIGopt into durable event/work lanes on this same server, backed by NATS JetStream. Lane project credentials are separate from agent-control credentials.
--hive-callback ADDR, CRUX_HIVE_SOCKET (+ CRUX_HIVE_TOKEN)dial hive’s callback endpoint (crux/api/callback) for the capabilities hive owns — scoped GitHub tokens, PR creation, Linear credentials, text/canary scanners, conditions — and feed the runtime’s seams from it instead of leaving them nil. Health reports it under hive_callback; an unreachable endpoint raises the hive_callback_unavailable condition. (#14246) When the runtime is a manager it also polls hive’s host policy from the same endpoint; see Host policy from hive.

serve does not use the per-work-dir control socket that a foreground crux run serves (<workdir>/crux.sock); the two are different things, see crux-cli.md.

Fail closed

  • No TCP without a token: --listen without --token/CRUX_TOKEN is a usage error. When the server has no token the handler accepts only requests it can see arrived over its own unix socket; anything else (including a Handler() mounted on a foreign http.Server) gets 401.
  • With a token, every request on every listener needs Authorization: Bearer <token>; anything else is 401 with WWW-Authenticate: Bearer realm="crux".
  • Socket mode 0600, directory 0700. The directory must be a real directory (not a symlink) owned by the serving uid with no group/other bits, or Listen refuses to place a socket in it; a pre-existing world-readable <tmp>/crux-<uid> is an error, not a trust.
  • Request bodies are capped at 1 MiB and unknown JSON fields are rejected.

The API

All routes are under /v1. Bodies and responses are JSON; agent-control errors are {"error": "...", "code": "unauthorized|bad_request|not_found|conflict|runtime_error"}. Opt-in lane routes require a separate project bearer credential and capability ACLs; see the lane routes and request types and lane error codes and failure behavior.

RouteBodyRuntime methodReturns
GET /v1/health{"version":"v1","agents":N}
GET /v1/agentsObserver.AllStatuses{"processes":[Process…]}, sorted by name
POST /v1/agents{"name":…, "definition":{…}}Definitions.AddAgent (when definition is present), Lifecycle.Start201 {"agent":…, "process":Process}; 400 bad name, 409 the name already has a definition, 422 the runtime refused it
GET /v1/agents/{name}Observer.GetStatusProcess
GET /v1/agents/{name}/logs?source=pane|buffer|full&n=NGetOutput / GetBufferOutput / FullLogHeader+CaptureFullLog{"agent","source","lines":[…]} or {"header","content"} for full
GET /v1/agents/{name}/logs?follow=1&n=NGetBufferOutput, polledtext/event-stream of agent.log records {"agent","lines":[…]}; the first replays the last n buffered lines (none for n=0; the 200-line follow window when n is absent)
POST /v1/agents/{name}/kick{"message":…}Kicker.SendKick{"agent","process"}
POST /v1/agents/{name}/stopLifecycle.Stop{"agent","process"}
GET /v1/eventswatcher + Kicker.SetKickObserver + the crux.Host observerstext/event-stream of Event records
GET /v1/lanes and lane/group subroutesPublish/recovery/receipt JSON; request shapeslanes.BrokerOpt-in only; lane-project credential and capability ACLs; JSON or 204; full routes and responses

Runtime families (#14138 stage 3)

Every method of crux.Runtime has a route; api.RuntimeRoutes is the method → route table and crux/api/client’s parity test fails when a method is added to the interface without an entry. The sidecar client (crux/api/client.Runtime, selected by CRUX_MODE=sidecar) is written against this table. Routes marked ⚑ are new in stage 3; the rest predate it or are the CLI routes above under another method.

FamilyRouteRuntime method(s)Notes
LifecyclePOST /v1/agents/{name}/start ⚑Start
POST /v1/agents/{name}/restart ⚑Restart, RestartWithBootstrap ({"bootstrap":…})
POST /v1/agents/{name}/pause ⚑Pause, PauseBy, Supervisor.PauseByCausebody selects the variant
POST /v1/agents/{name}/resume ⚑Resume
DELETE /v1/agents/{name}/restart-count ⚑ResetRestartCount
GET /v1/agents/{name}IsPaused, Observer.ResolveAgent
ObserverGET /v1/snapshots, GET /v1/agents/{name}/snapshot ⚑AllStatuses, GetStatus, GetStatusFastfull crux.Process snapshot, not the wire projection
KickerPOST /v1/agents/{name}/kick/validate ⚑ValidateKick409 kick_refused
GET /v1/agents/{name}/kick ⚑KickDispatchState
GET /v1/agents/{name}/kicklogs[/{id}] ⚑ListKickLogs, ReadKickLog
DefinitionsPOST /v1/definitions, POST /v1/definitions/reconcile ⚑AddAgent, ReconcileAgents
DELETE /v1/agents/{name}, PUT /v1/agents/{name}/definition ⚑RemoveAgent, UpdateConfig
PUT /v1/agents/{name}/overrides, PUT/DELETE /v1/agents/{name}/pins/{kind} ⚑SetModelOverride, SetBackendOverride, PinCLI, PinModel, UnpinCLI, UnpinModel
POST /v1/fleet/modes/{clear,sync}, POST /v1/fleet/inference/refresh ⚑ClearModeOverrides, SyncModeFiles, RefreshInferenceRoutes
FleetGET /v1/fleet ⚑GetACMMLevel, SharedDevUID, Supervisor.ConsentWedgedAgents, CountAgentsWithModelone read
GET/PUT /v1/fleet/breaker, POST /v1/fleet/breaker/{engage,release} ⚑BreakerState, Restorer.RestoreBreaker, EngageBreaker, ReleaseBreaker
PUT /v1/fleet/acmm, POST /v1/fleet/relaunch-bob ⚑SetACMMLevel, RelaunchBobAgentsAwaitingKey
CredentialsPOST /v1/agents/{name}/login-code ⚑SubmitLoginCode
GET/PUT /v1/credentials/copilot ⚑CopilotToken, CopilotTokenSource, SetCopilotToken, ActivateCopilotToken
GET /v1/credentials/claude, POST /v1/credentials/claude/reload ⚑ClaudeCredentialCandidatePaths, ReloadClaudeToken
ResumerGET /v1/agents/{name}/session ⚑ResumeHandle, SessionID, SessionMissing
POST /v1/agents/{name}/session/kick ⚑SendResumeKick409 resume_busy, 410 resume_session_gone, 422 resume_unsupported map back to the crux.ErrResume* sentinels
SupervisorGET /v1/agents/{name}/supervision ⚑GovernorMayStart, AgentCredentialRefreshable, AgentHasValidCredential, AgentCapabilities, EffectiveBackend, InvocationMetadata, RestartTelemetry, StartFailureState, ProviderErrorBackoffRemaining, CrashOOMSuspected, AgentAuthAvailableone read
POST /v1/fleet/{startup-queued,restart-crashed,tokens/refresh,kicklogs/archive} ⚑MarkStartupLaunchQueued, CheckAndRestartCrashedAgents, RefreshAgentTokens, ArchiveAllKickLogs
POST /v1/agents/{name}/tokens/refresh, POST /v1/agents/{name}/nudge ⚑RefreshAgentTokenFor, SendGitHubAPIBudgetNudge
GET /v1/backends/{backend}/auth, POST /v1/audit, GET /v1/trajectory ⚑BackendAuthAvailable, RecordAudit, TrajectoryAgents
RestorerPOST /v1/agents/{name}/seed ⚑SeedPauseState, SeedRestartTelemetry, SeedTurnLoss, SeedLastKick, SeedKickHistoryone body, each field optional
BackgroundGET /v1/healthStartAgentTokenRefresh, StartCredentialWatchdog, StartCopilotSessionRefresh, StartTranscriptActivityRefreshthe loops run server-side; the client’s calls are no-ops
AuthorizerPOST /v1/authorize ⚑AuthorizePROpen, AuthorizePushBranch, AuthorizeIssueOpen, AuthorizeMerge, AuthorizeReviewRequestsynchronous; {"action","agent","file_uid","kind","repo"}; 403 forbidden carries the refusal, any transport error is a refusal (fail closed)
WatchdogGET /v1/watchdog/agents, GET /v1/agents/{name}/watchdog, PUT /v1/agents/{name}/watchdog/conditions ⚑watchdog.Source (AgentNames, Observe, LastProduction, SetConditions)501 not_supported when the runtime has no watchdog source; crux/runtime.NewWatchdogFleet builds the fleet over these in sidecar mode

Process is the wire projection of crux.Process: name, id, state, backend, model, pid, paused (+ reason, trigger, by), started_at, last_kick, restart_count, last_restart_reason, kick_refused, kick_refusal_reason, kicks_undeliverable, busy_condition, start_failure_class. It is a subset by design: the full snapshot carries engine-internal state that is neither stable nor meaningful off-process.

definition on POST /v1/agents is the JSON form of crux/config.AgentConfig, the per-agent overlay hive reads from agents_dir. The server applies the same gate and defaults the YAML loader does: name must match ^[a-z0-9][a-z0-9_-]*$ (≤64 chars); id and role default to the name, replicas to 1, enabled to true unless the JSON says otherwise. A name the runtime already knows is a 409 conflict rather than a silent start of the old definition; stop and re-post. Without definition, name may also be an agent ID. Parsing the portable AgentDefinition format is the caller’s job, as it is the host’s job for the CLI (the CLI does it before sending).

Agents start under the server’s lifetime, not the request’s: the runtime ties an agent to the context it was started with, so a client disconnecting does not kill what it started.

Events

GET /v1/events is a server-sent-event stream; each record is event: <type> then data: <Event JSON>, with : keep-alive comments every 15 s. Event is {"time","type","agent","state","detail","data"}.

TypeWhen
agent.started, agent.stopped, agent.kickedthe API performed the action
agent.statethe server’s watcher (polls AllStatuses every second) saw an agent change state for any reason: crash, restart, pause; state is the new state, detail the last restart reason when there is one
agent.kickthe runtime’s kick observer fired (Kicker.SetKickObserver); detail is the observer’s event name
agent.kick_outcome, agent.pause_transition, agent.relaunch, agent.pause_persist, agent.prompt, agent.sandbox_audit, runtime.audit, runtime.conditionthe crux.Host observers, relayed when the runtime is a Host; data carries the typed payload (api.KickOutcome, api.PauseTransition, …). A sidecar hive demuxes these into the observers its boot sequence installs (cmd/hive’s sidecarHost)

A subscriber that stops reading is dropped after 64 buffered events so one slow client cannot stall the others.

Host policy from hive (#14138 stage 4)

With --hive-callback, serve also reads the values an in-process hive hands the manager through the crux.Host policy setters. They travel on the hive callback endpoint (crux/api/callback, runtime → hive), not on this API:

Callback routeFeeds
GET /v1/policya callback.Policy snapshot, polled every 15 s (hivecap.WatchPolicy): build → SetBuildInfo, explain_mode_default → SetExplainModeDefaultResolver, kick_log_source → SetKickLogSourceResolver, gateway_backends → SetGatewayBackendChecker, dead_session_recovery_owned_elsewhere → SetDeadSessionRecoveryOwner, sandbox (the agent_sandbox block) → SetSandboxConfig; repo_auto_merge: true installs the next row
POST /v1/policy/repo-auto-merge{"repo"} → {"enabled"} for SetRepoAutoMergeEnabledResolver, asked per merge authorisation

The resolvers answer from the last snapshot (the manager reads them under its own lock, so they never call out) and keep it through a hive outage; the auto-merge call fails closed, refusing the merge. A value hive does not report, or a hive without the policy capability (404 not_configured), leaves the runtime’s own default in place.

Every crux.Host setter has a disposition in api.HostSetters, and TestParityHostVersusSetterTable fails when one is added without an entry. The remaining setters, all on the same callback endpoint:

SetterCallback route (capability)Fail-closed behaviour
SetInferenceCallbacksPUT/DELETE /v1/inference/routes/{agent} (inference_routes): the manager pushes each agent’s inference route into hive’s proxyroute not installed; the agent runs without a per-agent route
SetSandboxMutationBoundaryPOST /v1/effects, POST /v1/effects/{id}/end (effects): two-phase claim/end over hive’s journaling boundarymutation refused
SetAppAuthPOST /v1/tokens with {agent, tier} and no repos (installation-wide, what AppAuth.ScopedToken(tier) mints in-process); hivecap.AppTokenWriter writes the token in place (O_TRUNC, 0600) to the agent’s existing cache file and nowhere else, never logging it. HIVE_PROXY_INJECT_GH_AUTH is refused in sidecar mode (ErrProxyInjectionUnsupported) rather than writing the real tokencache file untouched; the agent keeps its last token or runs read-only
SetAgentMintPOST /v1/tokens/mint (agent_mint); a hive with mint disabled answers 404 not_configured, which the writer maps to the in-process “mint off” contract ("", no error)no mint token
SetBobAPIKeyResolver, SetBobKeySourceResolverGET /v1/bob/credential (bob); cached 1 min runtime-side, a failed refresh serves the stale value for at most 2 minempty key and source
SetSpanStarter— in-process-only: a span is a Go value wrapping a call in one address space. The sidecar’s manager starts spans on the OpenTelemetry global tracer (a no-op without an exporter in the runtime’s process); trajectory events do not depend on itn/a

The proxy seams are carried the same way: hivecap.Set.{Accounting, Usage, Canary} come from GET /v1/github/rate-limit, POST /v1/usage and POST /v1/scan/canary, so a proxy built on them refuses writes while hive is away and resumes on the next successful call. No setter is served from a runtime: config key: every value is already resolved hive-side from the live config, so a reload reaches the runtime on its next poll.

Supervision and failure semantics (#14138 stage 4)

In sidecar mode hive decides who keeps crux serve alive:

EnvValuesMeaning
CRUX_SUPERVISEchild (default), externalchild: hive spawns crux serve itself (crux/runtime.Supervisor), waits for /v1/health, and restarts it with exponential backoff (1 s → 1 min, reset after 5 min healthy). external: systemd / a pod sidecar runs it; hive only reconnects.
CRUX_BINARYpath or namethe executable child runs; default crux on $PATH. A missing binary is logged loudly and hive behaves as external.

The child inherits hive’s environment plus CRUX_TOKEN; its listen flags are derived from CRUX_SERVER (unix:// → --socket, host:port → --listen). /v1/health now carries started_at and uptime_seconds next to hive_callback.

When the socket is gone, whichever way it is supervised:

SurfaceBehaviour
governor loopkeeps ticking; nothing in hive blocks on the socket
writes with an error return (kick, stop, resume, …)refused with crux.ErrRuntimeUnavailable (wraps api.ErrUnreachable). Kicks are not queued or replayed: the governor re-derives what is due on its next tick, so a replay buffer would only duplicate work after a long outage
observer readsanswer zero / unknown (unchanged from stage 3)
fleet reportcrux_runtime_unavailable (component agent-runtime, severity high) raised on the first failed probe (every 15 s, or any failed call) and cleared on the first success; the dashboard renders it like every other runtime condition
hive /api/healthcrux_runtime: {mode, reachable, last_error, since, supervise, child{running, ready, pid, starts, last_exit}}; status becomes degraded while unreachable
client Status(){reachable, last_error, since, failed_calls, last_success} on api.Client/client.Runtime; a caller that cancelled its own context is not counted, an expired deadline is

In-process mode is untouched: there is no socket, the condition never fires, and /api/health carries no crux_runtime member.

Drive it with curl

S=/run/user/$(id -u)/crux/crux.sock
curl -s --unix-socket $S http://crux/v1/health
curl -s --unix-socket $S http://crux/v1/agents | jq '.processes[] | {name, state}'
 
# run from a definition (an agents_dir-style AgentConfig document)
curl -s --unix-socket $S -X POST http://crux/v1/agents \
  -H 'Content-Type: application/json' \
  -d '{"name":"fixer","definition":{"backend":"claude","model":"claude-sonnet-4-6","role":"fixer"}}'
 
curl -s --unix-socket $S 'http://crux/v1/agents/fixer/logs?source=buffer&n=50' | jq -r '.lines[]'
curl -s -N --unix-socket $S 'http://crux/v1/agents/fixer/logs?follow=1'     # streams until Ctrl-C
curl -s --unix-socket $S -X POST http://crux/v1/agents/fixer/kick \
  -H 'Content-Type: application/json' -d '{"message":"look at the failing build on main"}'
curl -s -N --unix-socket $S http://crux/v1/events                            # in another terminal
curl -s --unix-socket $S -X POST http://crux/v1/agents/fixer/stop

Over TCP with a token:

CRUX_TOKEN=$(openssl rand -hex 16) crux --config hive.yaml serve --listen 127.0.0.1:7777
curl -s -H "Authorization: Bearer $CRUX_TOKEN" http://127.0.0.1:7777/v1/agents

The hostname in a unix-socket URL is ignored; crux is a convention.

Drive it with the CLI

--server ADDR (or CRUX_SERVER) switches run, ps, logs, kick and stop to the API; --token (or CRUX_TOKEN) carries the bearer token. ADDR is a socket path, unix:///path, host:port or http(s)://host:port.

export CRUX_SERVER=/run/user/$(id -u)/crux/crux.sock
crux run examples/agents/customized-agent.yaml   # definition parsed here, sent as JSON; follows output; Ctrl-C stops
crux run --detach fixer                          # a name already defined on the server
crux ps
crux logs -n 50 --buffer fixer
crux logs -f fixer                               # follow the ring buffer over SSE
crux kick fixer "look at the failing build on main"
crux stop fixer

In remote mode nothing is built in this process: no runtime, no control socket, no --config (the server’s definitions are what exist). serve itself cannot be run remotely. A 401 is reported with a hint to set the token.

Drive it from Go

import "github.com/hivecommons/hive/crux/api"
 
c, err := api.NewClient(os.Getenv("CRUX_SERVER"), os.Getenv("CRUX_TOKEN"))
procs, err := c.List(ctx)
resp, err := c.Run(ctx, api.RunRequest{Name: "fixer", Definition: defJSON})
err = c.FollowLogs(ctx, "fixer", func(ch api.LogChunk) error { …; return nil })
err = c.Events(ctx, func(ev api.Event) error { …; return nil })
_, err = c.Stop(ctx, "fixer")

crux/api imports only the standard library, so a client binary does not link the runtime. Errors from the server are *api.StatusError; api.IsNotFound and api.IsUnauthorized classify them. Client.WaitReady polls /v1/health for a caller that just launched crux serve.

What this is not

For agent control it is not tenancy, quotas or hive-as-a-client: those routes trust whoever can reach the socket or holds the token, and hive still constructs *agent.Manager in-process. Those are the rest of Stage 3 in design/crux.md.

The opt-in durable lanes are a separate project-authenticated surface with explicit capability ACLs, configured per-lane retention/storage bounds and per-group lease/ledger limits. Hive uses them as an ordinary lane client. This does not provide general multi-tenant isolation, change Hive’s agent hosting model or grant lane credentials agent-control privileges.