Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.
Hive operator reference
Fleet breaker
The dashboard top bar includes a fleet breaker: a circular pause/play control plus a status pill. The GET /api/breaker state is readable by authenticated viewers, but engaging or releasing the breaker is owner-only (POST /api/breaker/engage, POST /api/breaker/release).
Engage pauses every agent that is currently running and is not on_demand. Agents already paused before engage are skipped and keep their original pause reason. Release resumes only the exact agents the breaker paused and still owns (PausedTrigger == fleet-breaker); if an operator manually re-pauses an agent while the breaker is engaged, release leaves it paused. The breaker state is persisted so a crash/restart can still release the same captured set.
Fixes #2953.
Self-reported issues
When a hive notices an unusual error in itself — a restart storm, repeated
kick failures, a recovered panic, degraded model discovery, inference
auth failures or advisory post failures — it files an issue on
hivecommons/hive titled [selfreport] <class>: <short message>, or adds
itself to an open issue that already describes the problem (fingerprint
match, hive/selfreport + class match, or keyword match on a human-filed
issue). Joining posts one “Also seen on <hive>” comment per 24h and
bumps the Affected hives (N) block in the body, where each hive has its own
line — - <hive> (instance <id>) — owner(s): @login — seen N, last <time> —
naming the GitHub login(s) of the hive owner (dashboard.authorized_users
owner-role entries plus the device-flow operator) so a maintainer can see who
is affected. Each report also names the repositories the hive watches
(project.repos) and their owners, and the body block keeps an
Affected repos (N) / Affected repo owners (N) union across reporting
hives (hivecommons/hive#12338). Every selfreport carries
origin/selfreport (the provenance axis every hive-created issue records,
see labels-and-control-signals.md)
next to the deprecated hive/selfreport.
governor:
fleet_report:
enabled: true # unset = on for hub-managed hives, off (dry-run) for self-hosted
repo: hivecommons/hive
max_per_day: 5
include_repos: true # false: never name project.repos / repo owners (set this if any watched repo is private)
include_owners: true # false: render "(anonymous)" instead of the hive owner's GitHub login(s)
Rate limits: max_per_day new issues per hive per 24h, at least 1h between
new filings of different classes, no re-filing of a fingerprint closed
upstream while the hive keeps observing the condition (7-day cooldown from
the last observation), deferred while the GitHub API budget is not normal. Issues
are never auto-closed; the hive posts one “cleared” comment when the
condition stops. Hub-managed hives that cannot write to the report repo relay
through the hub (POST /api/selfreport/ingest, heartbeat bearer). The
dashboard shows “Self-reports: N open (last: <class> → #NNN)” in the Fleet
Report panel and the toggle/repo/cap under Settings → Governor →
“Self-Reported Issues”. Full details: fleet-report.md.
Agent CLI session recycling
Each agent’s CLI (Copilot, Claude, …) is a long-lived Node/V8 process whose
heap grows with every turn and never shrinks while idle. The health tick
samples the resident memory of the whole process tree under each agent’s
tmux pane and recycles a session that is over budget — but only while the
agent is idle at its input prompt with no kick pending, so a turn is never
interrupted. A recycle is an ordinary restart (last_restart_reason: session recycle): claims, queues and the kick log are preserved exactly as on any
other restart, and the governor re-kicks on the next cadence. After a recycle
the same agent is left alone for ten minutes even if it is over budget again.
agents:
session_recycle: # fleet default; on unless enabled: false
max_rss_mb: 1536 # recycle an idle session whose process tree exceeds this (MiB)
max_age: 6h # …or whose CLI session is older than this
min_idle: 2m # pane + transcript must be silent this long first
scanner:
session_recycle: # optional per-agent override; absent fields inherit the fleet value
max_rss_mb: 2048
The sampled value is shown as Session RSS (with the session age and a
↻N recycle count) in the agent detail panel. The log line is
agent session recycled with rss_mb, age and reason.
Agent CLI memory
Each subscription CLI keeps a private cross-session memory in the agent’s
home (Claude Code ~/.claude/projects/*/memory/, Codex $CODEX_HOME/memories*/,
Gemini ~/.gemini/GEMINI.md; Copilot memory is server-side). The
agents.cli_memory block, overridable per agent, decides what happens to it;
the full table is in
agent-configuration.md.
agents:
cli_memory: # fleet default; each key may be overridden per agent
auto_memory: false # pin the vendor auto-memory switch off (default)
clear_on_kick: false # wipe the agent's memory stores before every non-resume kick
drain_to_knowledge: false # first convert each memory fact into a draft knowledge entry
drain_on_write: false # also drain a Claude memory file as soon as it is written
scanner:
cli_memory:
auto_memory: true # this agent may take notes…
drain_to_knowledge: true # …which become drafts in bead-synth-wiki at the next kick
clear_on_kick: true # …and the private store is then emptied
| Key | Default | Effect |
|---|---|---|
auto_memory | false | Pins Claude Code (CLAUDE_CODE_DISABLE_AUTO_MEMORY=1), Codex ([features] memories = false) and Copilot CLI ("memory": false) auto-memory off at launch; true pins it on. |
clear_on_kick | false | Removes the agent’s own memory stores before each non-resume kick; shared homes (HIVE_SHARED_AGENT_HOME=1, uid 0) are never touched. |
drain_to_knowledge | false | Before the wipe, writes every memory bullet or paragraph as a state: draft entry into the bead-synth-wiki vault (knowledge.bead_synthesizer.vault_path), tagged cli-memory, agent:<name>, cli:<name> and repo:<org/repo> when derivable, deduplicated by content hash; a duplicate only gains the draining agent’s agent:<name> tag and keeps its state. Drafts are excluded from kicks until approved. |
drain_on_write | false | Needs drain_to_knowledge. Also drains a Claude Code memory file as soon as the session writes it, through a PostToolUse hook on Write|Edit|MultiEdit, for long-lived sessions that rarely end. |
For Claude Code, drain_to_knowledge also seeds a SessionEnd hook, and
drain_on_write a PostToolUse hook (matcher Write|Edit|MultiEdit), into
the Claude settings Hive manages: the shared /data/home/.claude/settings.json
for interactive agents and both settings files of inference agents. Each runs
hive memory-drain --cli claude --agent "$HIVE_AGENT", which asks the hive to
drain that session’s project memory into drafts and empty the files whose
every fact is stored. The hook always exits 0 and waits at most 10 seconds;
on a write outside ~/.claude/projects/*/memory/ it exits at once. Your own
hooks entries are kept; turning the keys off removes only Hive’s entry.
Each request is audited as cli_memory_drain_requested.
The drain logs one line per kick that found a store:
cli-memory: drained N facts from <agent> into bead-synth-wiki (dupes K, spooled S)
with drained, dupes, skipped (over the 200-facts-per-drain cap),
spooled and sources. A memory file whose facts did not all reach the vault
(cap hit, vault unwritable) is copied to
/data/knowledge/auto-memory-spool/<agent>/ before clear_on_kick wipes the
store, retried first on the agent’s next drain, and deleted from the spool
once every fact in it is stored. Review drafts in the dashboard’s
knowledge view or the vault directory and approve them through the normal
lifecycle (PUT /api/knowledge/entry/{id}/state); nothing promotes them
automatically.
A drained draft reaches no agent while it is a draft, even for an agent whose
knowledge.agent_scopes entry (or the primer’s include_states) admits
drafts: the kick primer, the knowledge advisor and every ?agent= TOC, entry
and effective read skip it, and the curator’s auto-promotion never selects
it. Review it under 🧠 Auto-memory drafts in the dashboard’s knowledge
view, where an owner can Approve (primed from the next kick) or
Discard (deprecated, never re-drafted); see
Auto-memory drafts.
The shared /data/home/.claude/projects/*/memory store that predates
per-agent CLI homes is not drained at kick time. Drain it once with
hivectl knowledge drain-memory --legacy (owner-only, audited as
knowledge_drain_legacy_memory): its facts become drafts tagged
agent:shared, and the files stay as they are unless --truncate is given,
which empties only files whose every fact is now in the vault. The
bead-synth-wiki vault must be connected (bead stores configured or
drain_to_knowledge on); otherwise the call returns 409.
GitHub API quota
Agents reach api.github.com through the Hive GitHub proxy, which injects the
agent-scoped App installation token and records charged requests. The App
installation quota is shared with the daemon’s scan, merge, and maintenance
loops, so the proxy enforces two safeguards:
-
agents.github_api_hourly_cap(default300) limits each agent to that many charged GitHub API responses in a sliding one-hour window. A charged response is any proxied response other than304 Not Modified; the proxy’s own token minting is not agent-attributed and is not counted. Set the value to0to disable the fleet default, or setagents.<name>.github_api_hourly_capto override one agent (0disables that agent’s cap). -
github.agent_reserve_floor(default400) protects the shared core bucket. When the latest rate-limit reading showscore.remainingbelow the floor, agent read requests receive the same proxy429even if the agent has not reached its personal cap. This keeps daemon merge/scan work from being starved by agent polling. Only readings from the hive’s own bucket update that snapshot: responses whoseX-RateLimit-Limitis 60 (GitHub’s per-IP unauthenticated bucket, e.g. an agent’sghcall without a token) or whose limit differs from the daemon’s are ignored, so they cannot trip the floor or the API budget mode. -
github.etag_cache_max_mb(default128, envHIVE_GITHUB_ETAG_CACHE_MAX_MB) andgithub.etag_cache_max_entries(default8192) bound the shared conditional-request cache that turns repeatGETs into free304s. When it is full, every new store evicts a body some poller is about to revalidate, and that poll is charged again. The ETag cache row under Debug → GitHub API budget (andetag_cache.bytes/max_bytesin/api/gh-rate-limits) shows the fill level; raising the cap from Settings → Forge App applies at once. Bodies are stored gzip-compressed, so the cap is several times its size in raw JSON.
Hub bearer validation and exhausted user tokens
The hub validates a Bearer token it has not seen before with one
GET /user using that token, then caches the login for 5 minutes. A token
GitHub rejects (401, or 403/429 with X-RateLimit-Remaining: 0) is now also
remembered: for 60 s by default, or until the token’s X-RateLimit-Reset
(capped at 15 minutes) when it is rate-limited. Without that negative entry a
client retrying with an exhausted token made the hub re-probe GitHub on every
request, charging the token owner’s own core bucket while it was already at
zero. The probe is recorded in /api/gh-rate-limits under caller
agent:hub-bearer-validate, so a flood of 401s from one dead token is visible
on the hub admin page; being a pass-through credential it never updates the
hub’s own core-bucket snapshot.
Keep personal tokens out of cluster secrets
GitHub’s 5,000 req/h core limit is per user, shared by every PAT and
OAuth token (gho_…, ghp_…, github_pat_…) that user owns, from every
host. A gh auth token pasted into a cluster secret therefore shares its
owner’s bucket with their laptop gh, the hub’s bearer validation of that
user, and any other service holding a copy. The hub itself authenticates as
the GitHub App (HIVE_HUB_GITHUB_APP_ID / HIVE_HUB_GITHUB_INSTALLATION_ID /
HIVE_HUB_GITHUB_APP_KEY_FILE, see env-vars.md) and uses
HIVE_HUB_GITHUB_TOKEN only as a fallback, logging a startup WARN naming the
identity it ended up with. Audit the rest of the fleet the same way:
- Find copies. For every cluster context, compare the first 8 characters
of each GitHub-looking secret value and
/data/gh-user-tokenfile against the owner’s token (kubectl get secret -A -o json, decode, never print the full value). Known holders in a hive deployment are the Actions Runner Controller (gha-runner-scale-set)githubConfigSecret, its generated*-listener-configsecret, and any sidecar that scrapes GitHub (dibs). - Move ARC to App auth. The runner scale-set listener long-polls the
Actions broker and acknowledges every message; with a PAT those calls are
charged to the PAT owner (~3,000 req/h at 350 ephemeral runners/h). Grant
the GitHub App repository Administration: read & write (or organization
Self-hosted runners: read & write for an org-level runner group),
then replace the
githubConfigSecretkeysgithub_tokenwithgithub_app_id,github_app_installation_idandgithub_app_private_key(the same PEM mounted on the hub ashive-hub-github-app-key), and delete the listener pod so the controller regenerates its*-listener-configfrom the new secret. App traffic lands in the installation’s own 15,000 req/h bucket. - Remove the PAT from the hub. With the three App variables set, unset
HIVE_HUB_GITHUB_TOKENondeploy/hive-hub; the startup log then readshub GitHub API identity selected mode=appinstead of theWARN, andGET /api/gh-rate-limitsreports the App installation’score.limit. - Rotate. Once no cluster holds it, revoke the leaked user token
(
gh auth refreshmints a newgho_), so stale copies fail closed rather than silently sharing the bucket again.
The proxy returns JSON shaped for the GitHub CLI:
{"message":"hive: agent GitHub API hourly cap reached (N/cap); stop polling and continue with local work","documentation_url":"…docs/operator-reference.md#github-api-quota"}
It also sets Retry-After to when the one-hour window (or observed reset) can
make progress, logs once at 80% of each agent window, and sends one
stop-polling nudge per agent window. Hive-authored write relays are not
agent-attributed and are not capped by this path.
Open-PR scans use github.graphql_pr_batch by default to move mergeability,
review-decision, linked-issue and CI-rollup reads into GitHub’s separate
GraphQL bucket. /api/gh-rate-limits reports these calls as caller
hive:pr_batch on endpoint /graphql and exposes graphql_pr_batch counters
for repositories, PRs, pages, REST fallbacks, errors, and the last GraphQL
query cost. Per-PR REST reads still happen when the batch is disabled, a repo’s
GraphQL query fails, or GitHub returns mergeable: UNKNOWN for that PR.
Webhook-driven PR cache invalidation
See GitHub webhooks: cadence, patch and reconcile for the cadence keys, the fallback to polling when deliveries degrade, and the accuracy guarantees of the ForgeState contract. Hive owners (not cluster operators) should start at the owner guide: it covers the ⚙ Settings → Forge App → GitHub webhooks section, which shows the payload URL, generates a per-hive secret and can create the org/repo hook.
A spoke accepts GitHub App webhooks on the public POST /api/webhook/github,
either delivered directly by GitHub or relayed by the hub (the hub forwards the
original signed body to the hive that manages the repository). The receiver
fails closed: it accepts a delivery only when X-Hub-Signature-256 verifies
against GITHUB_WEBHOOK_SECRET (exactly as the hub does), or when the hub’s
relay signature verifies against this hive’s own relay key (hosted spokes,
below). With neither configured it rejects every delivery with 503.
Deliveries for repositories the hive does not manage are ignored.
What the operator configures
- Secret (self-hosted only). Either press Generate secret in
⚙ Settings → Forge App → GitHub webhooks (the hive writes
/data/github-webhook-secret.key, mode 0600, and shows the value once), or setGITHUB_WEBHOOK_SECRETin the spoke’s environment (a Secret mount on Kubernetes; never inhive.yaml). The environment variable wins when both exist. A hosted spoke needs nothing here: it never holds a GitHub secret and verifies relays with its hub relay key instead (item 3). - GitHub App webhook. Point the App’s webhook URL at the hub
(
POST /api/github/webhook) for hosted spokes, or at the spoke’s own/api/webhook/githubfor a stand-alone hive (a repository or organization webhook with the same secret works too). Subscribe the App to these events:pull_request,pull_request_review,pull_request_review_comment,issue_comment,issues,check_suite,check_run,status,push, optionallyworkflow_job(Actions: read; lets the CI counts on the activity monitor move from the job payload for Apps that do not surface check runs), and, withgovernor.work_source.type: github_projects,projects_v2_item(organization-scoped; see Work sources). - Hub relay. The hub verifies the HMAC and relays the body byte-for-byte,
with the original signature and
X-GitHub-Delivery, to the spoke whose reported dashboard URL manages the repository. For a hosted hive it also addsX-Hive-Relay-TimestampandX-Hive-Relay-Signature-256(sha256=HMAC-SHA256 overtimestamp.delivery-id.body) keyed with that hive’s own relay key, which the hub mints at provisioning and delivers over the heartbeat (persisted on the spoke as/data/webhook-relay.key). The spoke rejects a relay whose signature does not verify, whose timestamp is more than 5 minutes off, or whose delivery ID it already accepted. Nothing to configure. - Pacing (optional).
governor.eval_interval_webhook_s(default900) is the base tick while webhooks are healthy;governor.webhook_wake_debounce_s(default10) andgovernor.webhook_wake_min_gap_s(default30) pace the early cycles described below.review.event_drivenadditionally letspull_requestdeliveries dispatch reviews early.
Verify: /api/status.webhooks.events_1h climbs after a push or comment,
/api/status.evalMode reads webhook, and the spoke log shows
github webhook wake: running an early eval cycle with the delivery’s reason.
Through the hub, every enrolled hive that tracks the delivery’s repository and has a dashboard URL receives the relay, so multiple hives (for example v5 and v6 spokes) tracking one repo all stay on webhook mode.
What a delivery does
pull_request,pull_request_review,pull_request_review_comment,check_suite,check_run,workflow_job,status,issue_comment(on PRs only) andpushinvalidate the cached PR detail and the GraphQL batch check-run/review entries for the affected PR and mark it dirty. Check and status events are matched by PR number and head SHA (fork PRs arrive without a PR list);pushinvalidates every open PR whose base or head branch is the pushed branch.issuesopened/labeled/unlabeledmark the issue dirty and wake evaluation;reopenedwakes evaluation too. To select the scanner for newly seen actionable issues, configure a top-level CEL trigger matchingevent.kind == "issue.opened". Agent Channels is not a webhook installer and only supports the governorkickruntime.pull_requestopened/synchronize/reopenedalso hands the PR to the sentinel, which evaluates it now instead of waiting for its reconcile cadence (#13379).projects_v2_item(created/edited/deleted/reordered/archived/restored/converted) patches the GitHub Projects work-source cache: the delivery names the item and its project, the hive re-reads that one item (one GraphQL call) and upserts it, or drops it ondeleted/archived. The payload carries no repository, so the spoke receiver routes it ahead of the repository-keyed events and the response carriesprojects_patch. The hub relay forwards repository-keyed deliveries only, so a hosted spoke using a Projects board needs the organization webhook pointed at its own/api/webhook/githubfor this event.- Every invalidating delivery wakes the governor loop. The wake is
debounced for
governor.webhook_wake_debounce_sso a push and its check fan-out coalesce into one cycle, and webhook-triggered cycles are at leastgovernor.webhook_wake_min_gap_sapart so a delivery storm stretches the batch instead of spinning cycles. The wake cycle does not re-list repositories. It plans from the governor’s cached actionable snapshot (the one the last cadence tick enumerated, patched by the targeted re-reads described below), so a wake costs the targeted re-read plus the dirty-PR detail refreshes and nothing else: repositories with no dirty PR reuse their previous GraphQL PR batch (graphql_pr_batch.webhook_skips), clean PRs are served from the PR detail cache pastgithub.pr_detail_ttl_s, and only the dirty PRs are re-read. The log line isgithub webhook wake: running an early eval cycle(reason,deliveries,coalesced,source=cached), followed byeval cycle planning from cached snapshot. - Webhooks are healthy while the last delivery for a managed repository is
newer than
2 × governor.eval_interval_s. The first healthy window puts the loop in webhook mode: the governor’s base eval interval rises togovernor.eval_interval_webhook_s(default900, capped bygovernor.eval_interval_max_s) — safe because the wakes above carry the latency — and/api/status.evalModereadswebhook,/api/status.webhooks.modereadswebhook. The API budget stretch still applies; the larger interval wins. - Fallback to polling (webhooks.md).
Webhook mode degrades when the window lapses while GitHub was active
(a cadence pass listed an item updated after the newest delivery) or when
deliveries reach the receiver but fail verification; a quiet repository
with no deliveries and no activity keeps the stretched tick. The loop
re-checks as the window closes (not just at the next stretched tick),
resets its ticker to
governor.eval_interval_s, logs one INFOwebhooks degraded; polling cadence restored(withreason), andevalModereadspoll,webhooks.modereadspollingwithsinceandmode_reason. The way back has hysteresis: deliveries must keep arriving for2consecutive healthy windows (webhooks.healthy_windowscounts them) before one INFOwebhooks healthy; webhook cadence engagedrestores the webhook cadence. pull_requestopened,reopened,synchronize,ready_for_reviewandreview_requesteddeliveries also queue an early review dispatch whenreview.event_drivenis on; see Event-driven dispatch.- A failed check reacts within seconds, not at the next sweep. A
check_run(completed, conclusionfailure/timed_out),check_suite(failure/timed_out) orstatus(failure/error) delivery is resolved to the open PR(s) at that head from the PR detail cache — one cached lookup, no extraGET /pullsfan-out; a head no cached PR matches is counted under/api/status.ci_failure_signals.unmatchedand left to the sweep’s dirty mark. Each match becomes a CI failure signal{repo, number, head, check, details_url, at}, deduplicated per head + check and kept for two hours. A hive-authored PR is attributed to the agent whose relay opened it (agent_pr_createdin the audit log; scanner when unattributed) and that agent is kicked at once with its FIX-BEFORE-NEW section naming the check and the log URL — the same section every regular kick carries, into which the signal is also folded untilci-failing.jsoncatches up, so nothing is lost if the kick is gated. A contributor PR nudges the automerge lanes for a targeted pass over that PR now (the contributor lane /stage/072path), with or without the ForgeState cache. Guards, in this order:governor.ci_failure_webhook_kick(defaulttrue;falserecords signals only), the repo pause (project.paused_repos), the agent’s config/runtime pause (the fleet breaker pauses every agent), governor kick eligibility, and one webhook-triggered kick or nudge per PR head pergovernor.ci_failure_kick_cooldown_s(default600). The activity log recordsci_failure_kick— “CI failed on #N (check) — kicked agent X” / “nudged contributor lane”. Agents never watch or poll CI themselves: detection is out-of-band via this webhook path, or the sweep as fallback. - Gated workflow runs are approved for trusted heads. GitHub holds
workflow runs for dependabot and first-time/fork contributors behind
“Approve and run workflows” (
action_required); such runs create no check runs, so CI never reports.EnrichCIStatusdetects them on pending heads and the CI caption readsCI waiting · N workflow runs need approval on GitHub. Right after it, the eval cycle runs the workflow-approval sweep, which approves a head’s gatedpull_requestruns at most once per PR head (#16603) when the head is trusted: a person admitted this head (a verified owner’s or merger’s dashboard Approve, recorded on the stage record and dropped by the next push — #15136; the automerge label on its own never counts, since it is bound to no head and names no approver), or the PR author or the run’s triggering actor is this hive’s App or is listed ingovernor.ci.auto_approve_workflow_actors(default['dependabot[bot]', 'renovate[bot]']; an explicit[]trusts only the App and owner admission). A trusted author covers every gated run on the head regardless of who pushed it — including commits a fork collaborator pushed to the author’s branch — so keep the actor list to bot logins unless that is acceptable for a human login you add. Trusting the App as triggering actor covers heads the App pushed itself (its own PRs, rebases, review-fix commits), and only when the run’s head repository is the PR’s base repository: a fork head is never trusted on the triggering-actor path, because a public App installed on a contributor’s fork pushes with the same<slug>[bot]login. Runs of other events (for examplepull_request_target) are counted in the caption but never approved. Guards:governor.ci.auto_approve_workflows(defaulttrue— unset means on;falseleaves the gate to a person), the repo pause, hold / exempt / sentinel labels, a freshGET /pulls/N(open, non-draft, same head, author, labels) before any write, a head-repository match on every run, and the write-back gate (ci_rerun). Approval starts CI only; review, hold and merge gates are unchanged. The sweep does not run on every cycle: it is an optional enrichment step, so it is skipped whenever the API-budget decision setsSkipOptional—conserveandcriticalbudget modes, off-cadence cycles undergovernor.optional_sweep_every_n_cycles, and cached-snapshot (webhook wake) cycles — and a gated head then waits for the next cycle that runs it.EnrichCIStatusstill runs on those cycles, so the gate stays visible. A403/404fromPOST /actions/runs/{id}/approve(the App lacksactions: write) is recorded once, never retried for that head, logged asworkflow approval refused; needs workflow approval on GitHub, and the caption becomesCI waiting · needs workflow approval on GitHubon every later cycle, including ones that skip the sweep. Any other failure of that POST — a5xx, a network error, a cancelled cycle, or a primary/secondary rate-limit403— is logged asfailed to approve gated workflow run; retrying next sweepand gives the head’s pass back, so the next sweep re-reads the PR and tries again (denials are not counted twice). That refusal is bound to the head: granting the App Actions: Read and write afterwards does not make the hive retry a head it already recorded a refusal for — the operator approves that run on GitHub, or a new push to the PR (a new head) or a hive restart (the ledger is in memory) lets the sweep attempt it again. The sweep uses the App token, so it needs Actions: Read and write on the App (github-app-setup.md); it runs alongside the older/ok-to-testpath, in which the contributor lane posts a comment and the repository’s ownhive-ok-to-testworkflow approves with itsGITHUB_TOKEN— operators who want to stay on Actions: Read and that path alone setgovernor.ci.auto_approve_workflows: false. Each approved run logs oneapproved gated workflow runINFO line; counts since boot are at/api/statusworkflow_approvals(approved,denied,permission_errors,last). Approved runs execute the head’s workflow files on the repository’s runners; operators on self-hosted runners who want an explicit opt-in should setgovernor.ci.auto_approve_workflows: false. - A recorded review verdict also wakes the governor loop. When the
relay records a verdict bound to a review dispatch, the head that wrote it
is free and the PR may be ready for its next head or its decision kick, so
the loop runs a cached-snapshot cycle now instead of waiting for an
unrelated delivery or
governor.eval_interval_webhook_s. Advisory verdicts do not wake it. The first verdict wakes at once; later ones insidegovernor.webhook_wake_min_gap_sfold into one wake at the end of the gap.governor.webhook_wake_debounce_sdoes not apply to this wake (its debounce is fixed at 0, so a verdict is never held back), and the spoke logsreview verdict wake: running an early eval cycle from the cached snapshot(#14705). - Every status publish (cadence or wake) invalidates the
/api/stage-trackercache and sends astage-trackerevent on/api/events, so the dashboard’s activity monitor refetches at once instead of waiting outdashboard.stage_tracker.cache_ttl_splus its poll. Invalidation only marks the cached body stale: the publish, and every webhook below, returns at once even while a tracker rebuild is running. The next poll starts one rebuild outside any lock; polls that land during it get the previous body withsource: staleandmax-age=0rather than queueing, so a slow build (one sentinel evidence read per open PR, on NFS in hosted hives) can never back up webhook handlers, the governor’s publish, or an unbounded line of dashboard requests (hivecommons/hive#14681). - The activity monitor row moves without waiting for a cycle. A delivery
that names a PR or issue this hive tracks (
pull_request*,check_suite/check_run,status,pushto a PR branch,issue_comment,issues— any action) also drops the/api/stage-trackercache at once and queues a targeted re-read of just that item: oneGETof the PR or issue, debounced bydashboard.stage_tracker.webhook_refresh_debounce_ms(default2000) and coalesced per item. The re-read patches that row of the status snapshot in place — labels, title, draft flag, last activity; a closed or merged item leaves — then drops the tracker cache again and sends thestage-trackerSSE nudge. The same re-read is handed to the governor, which upserts the item into its cached actionable snapshot — a new issue or PR appears, a closed or merged one leaves, a label change moves it between the actionable / held / sidelined buckets — so the wake cycle that follows plans against the patched item. No repository is re-enumerated. The read goes through the paced REST transport and is skipped outright while the API budget iscritical; it refreshes the PR detail cache, so the following cycle does not pay for it again. The re-read is also remembered fordashboard.stage_tracker.webhook_refresh_replay_window_s(default1800) and re-applied to every status snapshot published meanwhile: an eval cycle that enumerated the item before the close still publishes from that enumeration minutes later, and without the replay its publish put the closed row back until the next full enumeration.dashboard.stage_tracker.webhook_refresh: falseturns the re-read off (deliveries then still invalidate caches and wake the loop, and wake cycles fall back to a full enumeration because nothing else patches the snapshot). Independently of the re-read, the one payload fact the tracker trusts without a confirming read is a closed or mergedstateon the issue or pull request the delivery embeds: on any action (labeled,unlabeled,created, not onlyclosed) that row leaves at once, so a refused or failed re-read cannot leave a closed item listed open. As a backstop for deliveries that never arrived, on each status publish it also lists each repo’s items closed since its last look everydashboard.stage_tracker.closed_sweep_interval_s(default300; one list call per repo; negative disables) and evicts any still shown. - CI counts on the activity monitor move from the check payload. A
check_run,workflow_joborstatusdelivery is applied directly into the per-head check set the tracker’s CI chevron counts (CI 7/11): the named check is upserted by run id (or check name and producing App), a newer run replaces an older one, and the row is pushed over thestage-trackerSSE nudge without any GitHub request — a fan-out of twentycheck_rundeliveries costs zero requests. The payload-onlycheck_run/statusdelivery no longer queues the PR re-read above. When no check-run listing exists yet for that head (first delivery after a push, or a sidelined PR the eval cycle never enriched) one coalesced listing of the head’s check runs is scheduled instead (debounced by the samewebhook_refresh_debounce_ms), so the required-check set and thepass/totaldenominator are right from the first row. The polling path’sEnrichCIStatusstill refreshes every actionable PR each cycle and is the backstop for spokes without webhooks./api/status.webhooks.last_ci_event_at,ci_events_1h,ci_events_applied_1handci_refetches_1hshow whether check deliveries actually arrive and how many were served from the payload.
Polling cadence vs webhook patch vs reconcile
The full model — the three writers, the accuracy guarantees of #13373, every cadence key with its default, and the webhook-outage fallback — is webhooks.md. In brief: inbound webhooks cost nothing against the GitHub rate limit; outbound API calls do. The governor therefore keeps three distinct paths and never lets a delivery trigger a full re-list:
| Path | Trigger | GitHub cost | What it does |
|---|---|---|---|
| Reconcile | cadence tick — governor.eval_interval_webhook_s (default 900) while webhooks are healthy, governor.eval_interval_s otherwise | full: lists issues and PRs for every tracked repo | Authoritative. Rebuilds the actionable snapshot from scratch and overwrites every webhook patch. If the pass fails outright (every repo refused: rate limit, 429, 5xx) the previous snapshot — patches included — stays published, marked stale for every repo (stale_repos, repo card countsStale), and the next tick retries; a repo refused on its own is served from its last-good enumeration the same way. |
| Webhook patch | one delivery naming an issue/PR | one GET per item, debounced and coalesced | Upserts, moves or removes that single item in both the activity-monitor row and the governor’s cached snapshot. Counted in /api/status.webhookPatches. |
| Cached cycle | webhook wake, review wake, continuous-agent wake | dirty-PR detail refreshes only | Runs the governor’s decision pass (kicks, automerge, stage moves) from the patched snapshot without listing. source=cached in the wake log; counted in /api/status.cachedCycles. Before the first reconcile has produced a snapshot a wake enumerates instead. |
Cadence ticks reconcile; webhooks keep the picture current in between. The priority order is accuracy of issue/PR state, then latency, then API thrift, which fixes four rules for the patch path:
- The payload is a hint, never state. A delivery only names the item;
the GET decides what the row says. A
closeddelivery whose re-read finds the item open leaves it open. - Upserts are monotonic. A re-read whose
updated_atis older than the copy already held (two deliveries’ GETs landing out of order) is dropped. Each applied read is recorded in the snapshot’sitem_patches(fetch time,updated_at, head SHA,X-GitHub-Deliveryid) for audit. - Closed/merged removes at once; reopened re-adds — both after the confirming GET.
- Irreversible actions are freshness-bounded. Before an issue close or a
PR merge planned from the snapshot (stage sync, close-on-merge, automerge
sweep), the governor refuses outright while that repo’s share is stale and
otherwise re-reads the item once if its copy is older than 60 s
(
DecisionFreshnessBound), proceeding only if it is still open. Refused actions logirreversible action deferredand retry next pass.
Startup runs a full enumeration before the first decision pass; a cold cache
whose GitHub share failed is published stale, never as “0 items”.
/api/status.staleSince (and per-repo countsStaleSince) carries the
stale-as-of time and the Overview shows a stale · as of HH:MM badge.
Raising eval_interval_webhook_s lowers list-call volume further at the cost
of a longer window before a missed delivery is corrected by reconciliation.
Reading from the ForgeState cache (forgestate.enabled: true)
With the per-repo state cache on (forgestate: block in hive.yaml,
default off; #13373) the
governor work source stops listing GitHub on the cadence tick too
(#13375). The cache’s own
reconcile loop (forgestate.reconcile_interval_s, default 900, shortened
to one minute while a repo is stale or its drift is high) is the only
full list; every governor pass — cadence tick, webhook wake, review wake —
takes each repo’s open issues and PRs from the cached snapshot and classifies
them exactly as before. What changes:
- Startup gate. No decision pass runs before every repo has completed
its first successful reconcile. The loop logs
forgestate: waiting for the first reconcile before the first decision passonce a minute while it waits andfirst reconcile completewhen it is released. A repo whose cache is not ready in a later pass is served from its last-good enumeration and marked stale, like a transient GitHub refusal. - Stale is inherited. A repo whose last reconcile failed keeps its
last-good items and appears in
stale_reposwithstale_sinceset to the last good reconcile, so the same close/merge guard applies. - Freshness bound before a kick. Every GitHub ref in an assembled kick
goes through the same 60 s bound as close and merge: a copy younger than
the bound is kicked as-is, an older one is confirmed with one targeted
GET(dropped from the kick if it closed or merged meanwhile), and a ref in a stale repo is withheld (kick ref withheld: snapshot not fresh enough to act on). A kick whose refs all fail is not sent. - Single-item reads feed the cache. The webhook patch
GETand the freshness confirmationGETare observed into the cache (as a webhook patch carrying theX-GitHub-Deliveryid, or as a read-through), and the hive’s own label add/remove, issue close and PR merge are written through immediately, so the next pass sees them without waiting for a webhook. item_patchescovers every cached item. With the cache, the pass’sgenerated_atsays nothing about an item’s age, so the fetch time of every item served from the cache is recorded there; the closed attributed PR history scan is reused for one reconcile interval rather than re-run on every wake.- Automerge lanes read the same cache. The self-authored, trusted-author
and contributor-lane sweeps take their open-PR set from the cached
snapshot and run event-driven with a 5-minute floor pass; see
hive-merge.md→ “Event-driven sweeps on the ForgeState cache” (#13377).
Repos added to the config after boot list live until the next restart rebuilds the store with them. Enabling the cache changes no classification: only the two list calls per repo move behind it.
Rate-limit pacing priority
After a GitHub rate-limit refusal (a secondary-limit 403 or a primary-limit reset) the shared transport enters a slow-start caution window and paces requests about 2s apart. Pacing is prioritised so periodic housekeeping cannot starve the governor:
- the reconcile pass (
EnumerateActionable) runs high priority and is bounded only by its own deadline; - ordinary requests may queue at most 30s ahead;
- housekeeping (self-authored and queued automerge sweeps, the review-request
watcher, advisory digest lookups, the dashboard metrics and fleet-stats
collectors) runs low priority: a request that would queue more than 5s
ahead is refused immediately (
github pacing: request shed) instead of taking a slot, and a tick whose first slot is already further out than the tick could wait is skipped outright (low-priority work shed, logged at most once a minute per caller) and retried on its next tick; - the self-authored automerge sweep backs its tick off to at least 60s while the window is engaged and resumes its adaptive interval when it ends;
- GitHub App installation-token mints (and the other App-JWT endpoints) do not go through the pacer at all: they are a separate rate-limit bucket, and a mint that queued behind paced traffic failed every request that needed the token.
Event-driven readers (ForgeState phase 2d)
- The review-request watcher runs on events. The relay that submits
hive-reviewrequests (review_request_watcher.go) no longer polls its drop box every 10 s. A request file drop wakes it through a directory notification, andissue_comment,pull_request_review,pull_request_review_commentandissueslabeled/unlabeleddeliveries wake it for a pass (/api/status.webhooksdeliveries reportwoke_review_watcher). While webhooks are healthy the unprompted scan is a five-minute reconcile (ReviewRequestWebhookFallbackInterval); when webhooks go stale, or the drop box cannot be watched, the watcher returns to the legacy 10 s poll. A--reviserequest for a repo outsidereview.revise_reposis a policy denial (.denied, one WARN), not a retried failure. - The advisory issue is resolved from the ForgeState cache. When a
forgestatestore is bound to the GitHub client (forgestate.enabled, hivecommons/hive#13373),EnsureAdvisoryIssueanswers from the cache: the memoized advisory issue is read through the cache within its freshness bound (zero GitHub calls on a healthy cycle), a stale or missing cached copy costs one targetedGETof that issue, and a closed advisory issue is reopened in place and written through. The search → label list → title-scan path runs only when the cache cannot answer (repo not yet reconciled, or no labelled advisory issue to confirm).
/api/status and /api/gh-rate-limits expose
webhooks: {healthy, last_event_at, events_1h, invalidations_1h, targeted_refreshes_1h, tracker_invalidations_1h, mode, since, mode_reason, healthy_windows, verification} (the middle two count the single-item re-reads and the
tracker cache drops webhooks caused; mode is webhook | polling and
since when it began — see webhooks.md; verification is
github-signed, relay (hub-signed) or none);
/api/status adds evalMode (webhook | poll), evalIntervalS,
lastWakeReason, lastWakeAt and webhookWakes;
graphql_pr_batch.webhook_skips counts batch queries skipped because nothing
changed.
governor:
eval_interval_s: 300
eval_interval_webhook_s: 900
webhook_wake_debounce_s: 10
webhook_wake_min_gap_s: 30
dashboard:
stage_tracker:
webhook_refresh: true # re-read the PR/issue a delivery names
webhook_refresh_debounce_ms: 2000
webhook_refresh_replay_window_s: 1800 # re-apply re-reads to snapshots published meanwhile
closed_sweep_interval_s: 300 # list items closed since the last look and evict rows still shown; -1 disables
waiting_push_stale_h: 4 # hours without a push on a CI / Review row before the pill's age turns muted-amber
ci_activity_stale_m: 10 # minutes a CI snapshot may still claim a check running before the head's check runs are re-listed; -1 disables
toast_info_s: 15 # how long the tracker's result / waiting-on-you toasts stay (3-120 s); the merge banner stays twice as long
toast_error_s: 30 # how long a failed row action's toast stays (3-120 s); editable from the Activity Monitor's ⚙ → Toasts & timing
merged_hold_s: 8 # ceiling on the "Merging · merge landed" bridge; celebration fires on the merged fact (min 3)
HIVE_GITHUB_TOKEN permissions
github.token: ${HIVE_GITHUB_TOKEN} creates the main GitHub client when a GitHub
App is not configured. The code reads issues, PRs, commits, contents, checks,
commit statuses, contributors, and search results; it can also create/update
issues and comments, add/remove/create labels, open PRs, approve PRs for the
merge queue, and squash-merge queued PRs.
Minimum practical PAT permissions for full PAT-authenticated operation:
- Classic PAT:
repofor private repositories (public_repois enough only when every managed repo is public and no private org data is needed). - Fine-grained PAT: select the managed repositories and grant:
- Contents: read/write (branch pushes by agents and content reads)
- Pull requests: read/write (list, open, approve, merge)
- Issues: read/write (list, comments, labels, advisory issue)
- Checks: read-only and Commit statuses: read-only (CI/merge gating)
- Administration: read-only (branch-protection verification before merge requests)
- Metadata: read-only (required by GitHub)
If you use a GitHub App, configure equivalent repository permissions on the App
installation instead of broadening the PAT. A repo-moved GitHub App verdict
means the App/key/installation are healthy, but the hive still names the old
repository owner after an org transfer; update the configured project org rather
than trying to add the moved repository back to the old installation.
GitHub App permissions the hive needs, and what each one unblocks
The App’s repository permissions decide which steps the hive can take itself.
A step the App cannot take is not a question for a person to decide; it is
a capability gap the hive records as a structured blocker
(<!-- hive:blocker kind=capability:… -->, see
labels-and-control-signals.md),
parks the item on, and surfaces at ⚙ → Security as GitHub App permissions:
missing <permission> · N items blocked (#15099).
| Permission | Level | Without it the hive cannot… | Blocker kind |
|---|---|---|---|
| Contents | read/write | push branches, read files | capability:repo-write |
| Pull requests | read/write | open, approve, merge PRs | — (nothing starts) |
| Issues | read/write | comment, label, file issues | — (nothing starts) |
| Workflows | write | push any change under .github/workflows/* (GitHub refuses the push with refusing to allow a GitHub App to create or update workflow) | capability:workflows-permission |
| Actions | write | approve workflow runs from forks (the 085 Awaiting human: approve workflow runs pill) | — (hive:unauthorized) |
| Checks / Commit statuses | read | gate merges on CI | — |
| Administration | read | verify branch protection before a merge request | capability:branch-protection when a rule the App cannot satisfy stands |
| Metadata | read | anything (required by GitHub) | — |
Granting a permission on an existing installation needs the installation
owner to accept the new permission request in the App settings page the
dashboard links to (githubAppInstallURL). The hive re-checks on its own on
the next stage pass; Re-check permissions on a blocked row
(POST /api/repos/{owner}/{repo}/items/{number}/blocker/recheck) probes the
installation now and releases every item blocked on that permission when it is
there. GET /api/status carries the same picture as
capabilities: {missing: ["workflows"], blocked_items: N, by_permission: {workflows: N}}.
Onboarding errors point to the Getting Started guide
Every setup problem the dashboard’s 🧭 Guide (top navbar; also Getting
Started in the sidebar) has a step for is classified server-side and rendered
with a tip — New hive? Open the 🧭 Guide → step ‘Install the Forge App
(GitHub App)’ for the install URL and permissions checklist — whose link opens
the guide at that step. The classification rides the status payload as
onboardingIssues: [{kind, guideStep, message}], on system alerts as
onboarding: true + guideStep, and on the embedded deepHealth checks
(github_auth, repo_target) as onboarding/guideStep. Kinds:
github-app-not-installed, github-app-wrong-installation,
github-app-permissions, github-app-credentials, github-base-url
(→ step github-app), repo-target, no-repos (→ repos), no-agents
(→ add-agent), proxy-token (→ welcome). Error text is unchanged; the tip
is appended. While any onboarding issue is active the Guide chip stays visible
even if the checklist was dismissed earlier. The guide’s GitHub App step shows
the concrete install URL for the configured forge host (githubAppInstallURL,
GHES-aware; on an enterprise host with no github_app_slug it says what to
configure instead of linking to github.com). The guide, its entry points and
the ⚙️ Governor Config gear require a non-read role from GET /api/role —
a read-only viewer sees none of them and must be granted access via Manage
Access.
Runtime diagnostics (debug, GOMEMLIMIT, mem: log line)
Added for #14681 (hive process RSS growing hundreds of MiB/min with no pprof to name the culprit).
| Name | Source | Default | Purpose |
|---|---|---|---|
debug.pprof_addr | hive.yaml | 127.0.0.1:6060 | net/http/pprof listener. Loopback only: a non-loopback host (0.0.0.0, pod IP) is rejected at startup with a warning and the default is used instead. off disables the listener. Profiles are under /debug/pprof/; /debug/runtime prints the same counters as the mem: log line. |
GOMEMLIMIT | env | 85% of the cgroup memory limit | Go soft memory limit. When the env var is unset the hive reads /sys/fs/cgroup/memory.max (v2) or memory/memory.limit_in_bytes (v1) and sets MemoryLimitPercent (85%) of it via debug.SetMemoryLimit; an unlimited cgroup leaves the runtime default. Set the env var to override. Logged at startup as runtime: memory limit with `source=env |
mem: runtime snapshot | log (INFO, every 60 s) | always on | heap_inuse, heap_live, heap_objects, total_sys, next_gc, gc_cycles, goroutines, threads (OS threads from /proc/self/status), gomemlimit. Read from runtime/metrics, so no stop-the-world. |
The same snapshot is embedded in GET /api/health as runtime so the
dashboard health probe and kubectl exec … curl see goroutine count and heap
without the pprof port. Taking profiles on a pod:
kubectl -n <ns> port-forward deploy/hive 6060:6060 &
go tool pprof -top http://127.0.0.1:6060/debug/pprof/heap # inuse_space
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head # parked goroutines
cmd/hive flags and environment
Startup
| Name | Source | Purpose |
|---|---|---|
--config | flag | Path to hive.yaml; default /etc/hive/hive.yaml unless HIVE_CONFIG is set. |
HIVE_CONFIG | env | Sets the default of --config. An explicit --config outranks it — and the image ships one (CMD ["--config", "/etc/hive/hive.yaml"]), so entrypoint.sh appends --config "$HIVE_CONFIG" to the launch argv when the variable is set. Without that append the variable is inert in the container (#4973). |
HIVE_MODE=hub | env | Starts the hub server instead of a spoke dashboard. |
HIVE_HUB_PORT | env | Hub listen port in hub mode; default 3001. |
HIVE_SINGLETON_LOCK | env | Internal/escape hatch for the process singleton lock path; value off disables the guard. |
GitHub and dashboard auth
| Name | Purpose |
|---|---|
HIVE_GITHUB_TOKEN | Main PAT fallback when github.token is empty; also used for token identity/fleet stats fallback. Wrong-scope tokens fail with generic 403s at request time — see github-app-setup.md for the required scopes per ACMM tier. |
GH_APP_KEY_FILE | GitHub App private-key path fallback when github.key_file is empty. |
HIVE_GITHUB_ETAG_CACHE_MAX_MB / HIVE_GITHUB_ETAG_CACHE_MAX_ENTRIES | Size the shared GitHub ETag response cache (defaults 128 MiB compressed / 8192 entries); win over github.etag_cache_max_mb / github.etag_cache_max_entries. See GitHub API quota. |
HIVE_DASHBOARD_TOKEN | Shared dashboard/API token fallback for dashboard.auth_token. Any non-empty string is accepted with no strength check — generate one with openssl rand -hex 32; see env-vars.md. |
Hosted-spoke and fleet metadata
| Name | Purpose |
|---|---|
HIVE_HUB_URL | Hub URL override for heartbeat/spoke registration. |
HIVE_CLUSTER_ID | Hosted cluster ID override. |
HIVE_ID | Hive ID override. |
HIVE_LEVEL | ACMM level bootstrap/override used by hosted flows. |
HIVE_COVERAGE_BADGE_URL | Source of the coverage percentage on the ci-maintainer card: an http(s) badge URL (shields JSON or SVG) or repo://<ref>/<path> read from the primary repo via the App client. See env-vars.md. |
Dashboard metrics
See Monitoring a hive with Prometheus and Grafana for setup, metric families and scraping.
| Name | Purpose |
|---|---|
HIVE_METRICS_ENABLED | Registers Prometheus /metrics when set to 1, true, yes, or on; off by default (route not registered — scrapes 404) because it exposes estimated cost data. |
HIVE_METRICS_TOKEN | Required whenever metrics are enabled (#3804): scrapers must send Authorization: Bearer <token> (configure Prometheus bearer_token). Enabled-but-tokenless serves 403 with an error naming both variables, and the hive logs a startup warning; the cost/agent series are never served unauthenticated. See ServiceMonitor setup and alert runbooks for configuring scraper access and on-call alerts. |
Note on
tokens_24h: despite its name, the per-spoke heartbeat fieldtokens_24h(stored on the hub astotalTokens24h) is a cumulative total, not a rolling 24-hour window. Read it as lifetime token consumption for the spoke. See token-tracking.md for the full heartbeat and/api/saas/usagerollup details.
Knowledge: Context7 content caps
| Name | Default | Purpose |
|---|---|---|
HIVE_CONTEXT7_MAX_SNIPPETS | 40 | Snippets kept from one Context7 llms.txt fetch (document import or context7 connector). |
HIVE_CONTEXT7_MAX_SNIPPET_BYTES | 4096 | A single snippet larger than this is dropped whole. |
HIVE_CONTEXT7_MAX_TOTAL_BYTES | 16384 | Total content kept per fetch, marker included; values under 1024 are raised to 1024. |
Snippets are dropped whole (never clipped) by relevance to the topic, the
stored content ends with a … truncated: N snippets / M bytes omitted (size cap) line, and the fetch logs the library. Unparseable or non-positive values
fall back to the default. The kick primer’s own byte cap is the yaml key
knowledge.primer.max_bytes (default 4096); see
knowledge-curator.md.
Inference endpoint fallbacks
| Name | Default | Purpose |
|---|---|---|
HIVE_VLLM_ENDPOINT | unset | Comma-separated vLLM endpoint list; set only to a cluster-reachable endpoint. Unset disables the built-in vLLM gateway route. |
HIVE_LLMD_ENDPOINT | http://hive-llm-d-epp.hive-inference.svc.cluster.local:8000 | Comma-separated llm-d endpoint list. |