Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.
GitHub webhooks: cadence, patch and reconcile
This page is the operator’s model of how a hive keeps its picture of GitHub current, and what each of the moving parts costs. It is the Phase 4 document of the ForgeState epic (#13373, #13381). The receiver, the App/hub set-up and what each delivery invalidates are in Webhook-driven PR cache invalidation; the dashboard side is in dashboard.md.
The priority order throughout is accuracy → latency → API thrift. A delivery is a free hint; every outbound GitHub call is budgeted. The rules below exist so that the hive never trades the first for the third.
Setting up webhooks (owner guide)
Webhooks are optional. Without them a hive polls GitHub every
governor.eval_interval_s (default 300 s) and finds out about a review, a red
check or a new comment at the next tick. With them the hive is told at once:
the governor wakes early, CI-failure and review kicks land within seconds
instead of minutes, the base tick stretches to governor.eval_interval_webhook_s
(default 900 s), and the outbound API budget drops because each delivery costs
one confirming GET instead of a full re-list. Open ⚙ Settings → Forge App
→ GitHub webhooks on your hive, or click the webhooks tile on the
Governor card, to see which of the two cases below applies to you.
Hive never shares its own webhook secret. Every hive verifies deliveries with a key that belongs to it alone.
Hosted hive (hub-managed)
Nothing to do. GitHub sends App deliveries to the hub, the hub verifies them
and relays each one to every hive that manages that repository, signed with
the relay key it minted for your hive when it was provisioned. The Settings
section reports Receiver: hub relay and the events/hour count; the
Governor tile reads N/1h. If it stays at no events yet for an hour of
activity, confirm the hivecommons GitHub App is installed on the organization
or repositories your hive tracks (Settings → Forge App → Installation) — the
hub can only relay what GitHub sends it.
Self-hosted hive
Your hive needs an org- or repo-level webhook in GitHub that points at it and uses a secret only you hold.
- Generate the secret. Settings → Forge App → GitHub webhooks →
Generate secret. The hive writes it to
/data/github-webhook-secret.key(0600) and shows it once with a copy button; it is never shown again (regenerate to rotate). If you prefer to manage it yourself, setGITHUB_WEBHOOK_SECRETin the hive’s environment instead — the button then reports that the environment variable is in charge. - Create the webhook. If the section offers Create webhook for me,
click it: the hive’s GitHub App installation holds
organization_hooksorrepository_hookswrite permission and the hive creates the hook with the right URL, secret and events. Clicking it again is safe: when a hook for this hive’s payload URL already exists it is reported, not duplicated. Otherwise do it by hand in GitHub: organization (or repository) → Settings → Webhooks → Add webhook.- Payload URL: the value shown in the section, i.e.
https://<your-hive>/api/webhook/github - Content type:
application/json - Secret: the value from step 1
- SSL verification: enabled
- Events: Let me select individual events →
check_run,check_suite,issue_comment,issues,pull_request,pull_request_review,pull_request_review_comment,push,status,workflow_run(optionallyworkflow_job, andprojects_v2_itemif your work source is GitHub Projects).
- Payload URL: the value shown in the section, i.e.
- Verify. GitHub sends a
pingimmediately; the section’s Receiver card should read github-signed with a fresh last event, and the Governor tile moves from set up → toN/1hafter the first real event.
The payload URL comes from dashboard.public_url; if the section shows no URL
or the wrong one, set that key (or the hub-reported dashboard URL) to the
address GitHub can reach.
Troubleshooting
| Symptom (GitHub → Settings → Webhooks → Recent deliveries) | Cause | Fix |
|---|---|---|
401 invalid signature | Secret in GitHub differs from the hive’s | Paste the current secret into the GitHub hook (Edit → Secret), or regenerate in Settings and update GitHub. GITHUB_WEBHOOK_SECRET overrides the generated file — check which one the section says is active. |
503 webhooks disabled: no secret configured | Hive has neither a generated/env secret nor a hub relay key | Step 1 above. |
404 | Wrong path | The path is /api/webhook/github on the spoke (the hub’s App endpoint is /api/github/webhook — do not point an org hook there). |
200 but no events yet | Deliveries are for repos the hive does not manage | Check project.repos; deliveries for other repos are acknowledged and dropped. |
| Hosted hive, no events | App not installed on the repo / relay rejected | Install the App on the org or repo; the hub relays to the dashboard URL your hive reports, which must be reachable from the hub. |
| Deliveries time out | Hive not reachable from GitHub (private network, redirects) | GitHub (and the hub relay) follow no redirects and need a public HTTPS endpoint. |
Hub relay vs direct: a hive registered with the hub whose repos carry the hivecommons App receives relays as well as any direct hook you add; both are accepted, duplicates are dropped by delivery id. Only one is needed. A self-hosted hive with a private dashboard URL cannot receive relays and must use a direct hook.
Polling cadence vs webhook patch vs reconcile
Three writers feed one snapshot per repository. A fourth path writes PR claims to the duplicate-PR claim ledger and filters the actionable snapshot; readers use that ledger to keep covered issues out of the offerable set.
| Writer | Trigger | GitHub cost | Authority |
|---|---|---|---|
| Reconcile | the cadence tick: governor.eval_interval_webhook_s (default 900) in webhook mode, governor.eval_interval_s (default 300) in polling mode; the forgestate cache additionally runs its own reconcile_interval_s (default 900) per repo | full list of open issues and PRs per repo, ETag-conditional where supported | Authoritative. Rebuilds the snapshot and overwrites every patch. A failed pass keeps the last-good snapshot and marks it stale; it never publishes an empty one. |
| Webhook patch | one verified delivery naming an issue or PR | one GET of that item (1–3 calls with labels/head), debounced and coalesced per item | Hint + confirming fetch. The payload only names the item; the re-read decides what the row says. Upserts are monotonic by updated_at / head SHA, so out-of-order deliveries cannot roll state backwards. Closed/merged removes at once; reopened re-adds. The X-GitHub-Delivery id is recorded for audit. |
| Write-through | the hive’s own mutation (stage label, comment, merge, close, claim) | none beyond the mutation itself | Immediate. The cache reflects the mutation before the next read, without a confirming call. |
| PR claim ledger | a verified pull_request delivery with a claim-changing action | no GitHub calls for claim parsing, persistence or snapshot filtering | Payload-derived claims. Records open claims, releases dropped/unmerged claims and records merged claims; separate from the confirming item fetch. See PR claims and issue stages. |
Where they live:
- Reconcile — the governor’s
EnumerateActionablepass (src/pkg/github/client.go:1430records the newestupdated_atit listed, see fallback) and the ForgeStateReconcilewriter (src/pkg/forgestate/reconcile.go:18), whose per-repointervalshortens while drift is high (src/pkg/forgestate/reconcile.go:18). - Webhook patch — the legacy snapshot’s
ApplyWebhookItem(src/pkg/github/webhook_item_patch.go:291) fed byFetchWebhookItemSnapshot(src/pkg/github/webhook_item_refresh.go:75), and ForgeState’sApplyWebhook(src/pkg/forgestate/webhook.go:38) with itssupersedesmonotonic check (src/pkg/forgestate/store.go:400). - Write-through — ForgeState’s
WriteThrough(src/pkg/forgestate/writethrough.go:38). - PR claim ledger —
ClaimLedger.ApplyPRDeliveryinsrc/pkg/github/webhook_pr_claims.go, wired byprClaimsHookForinsrc/cmd/hive/webhook_pr_claims.go.
Between reconciles the picture is kept current by patches; a cadence tick in webhook mode is therefore a slow safety net, not the source of latency. Wake cycles (webhook, review, continuous-agent) plan from the patched snapshot and never re-list; see the table in operator-reference.md.
PR claims and issue stages
For a managed repository, pull_request actions opened, reopened,
edited, synchronize, ready_for_review, converted_to_draft and closed
parse the PR title, body and head branch with the same claim grammar as the
reconcile scan (Fixes #N, branch names and references). Label and
review-request actions skip this path. HandleWebhookEvent invokes the
registered PRClaimsHook before the receiver queues the confirming item
fetch; the claim write does not wait for that fetch or for a cadence tick.
ApplyPRDelivery first removes this PR’s unsettled claims, then records its
current claims under the ledger’s collision/rank rules. An edit dropping an
issue or a close without merge releases the claim; a merge records
MergedPR claims anchored at the merge time. The hook saves the ledger and
re-filters the cached actionable snapshot with FilterClaimedIssues, so
both the contributor offer gate and the governor see the claim immediately.
A save failure logs webhook PR claims: failed to persist claim ledger;
webhook PR claims applied to claim ledger reports the recorded, released
and suppressed counts. This is distinct from the PR row’s cache patch.
With the issue-stage lane enabled, an open hive-authored PR’s strong
same-repo closing claims also emit CoveredBy facts on opened, reopened,
edited and ready_for_review, moving those issues to 040-covered-by-pr.
This runs in the background with a two-minute timeout and can read issues
and write labels; the zero-call cost above applies only to the ledger and
snapshot work. External PRs and non-closing references remain the stage
sweep’s responsibility. See coverIssuesForWebhookPR in
src/pkg/github/webhook_pr_claims.go.
Merge trigger and ledger-to-stage backfill
- Merged delivery: after a merged delivery records at least one claim,
prClaimsHookForinvokescloseOnMergeForDelivery. Withissues.close_on_mergeenabled, this starts a backgroundCloseOnMergeLedgerSweeprestricted byOnlyPRRepo/OnlyPRNumberto that PR, bounded to two minutes. It re-reads the PR and issue and uses the scheduled sweep’s identity/trusted-author policy; the payload alone does not authorize a close. It does not wait for the scheduled sweep interval. - Ledger-to-stage backfill:
reconcileLedgerCoverRecordsruns every evaluation cycle, including cached cycles and those withSkipOptional.BackfillCoveredByLedgerchecks live open covering claims against existing stage-store records below040-covered-by-pr. A fresh PR read must confirm open/unmerged before aCoveredByfact repairs the record. It respects the issue-stage lane and the red-and-stale PR release valve, ignores merged or non-covering claims, and leaves unseen records to the sweep. Agreement between ledger and store costs no API calls; repairs can read PRs and write labels. This backfill repairs stage records, whereas the scheduledCloseOnMergeBackfillscans open issues for merged fixes.
Accuracy guarantees
These are the acceptance criteria of #13373 and the tests that hold them.
| Guarantee | How | Where |
|---|---|---|
| Closed on GitHub → gone within ≤ 10 s | a closed/merged delivery triggers the confirming GET; the re-read removes the item from the activity monitor and the governor’s snapshot; the debounce is dashboard.stage_tracker.webhook_refresh_debounce_ms (default 2000); the re-read is remembered for dashboard.stage_tracker.webhook_refresh_replay_window_s (default 1800) and re-applied to every status snapshot published meanwhile, so an eval cycle that enumerated the item open before the close cannot bring the row back | src/pkg/forgestate/webhook.go:38, src/pkg/github/webhook_item_patch.go:291 |
| Closed in the payload → gone on that delivery, read or no read | every issues, issue_comment and pull_request* payload embeds the item with its state; a closed or merged state evicts the activity-monitor row and the governor’s copy at once on any action (a labeled/unlabeled/created for an already-closed item says closed too), before and independently of the confirming GET, which a critical budget or the rate-limit caution window may refuse. Open-side fields (title, labels, bucket) still come only from the GET. Backstop: on each status publish, at most every dashboard.stage_tracker.closed_sweep_interval_s (default 300) one state=closed&since= list per repo evicts rows GitHub reports closed (hivecommons/hive#14159, #14064) | src/pkg/dashboard/webhook_tracker_refresh.go, src/pkg/dashboard/stage_tracker_closed_sweep.go, src/pkg/github/closed_since.go |
| Out-of-order deliveries are safe | upserts compare updated_at (and head SHA for PRs); an older read never replaces a newer copy; a removal leaves a tombstone so a late “open” read cannot resurrect it | src/pkg/forgestate/store.go:400 |
403 mid-reconcile keeps last-good + stale | a refused reconcile marks the repo stale and keeps the previous snapshot, patches included; counts are never published as zero | src/pkg/forgestate/reconcile.go:146, src/pkg/github/enumerate_lastgood.go:39 |
| Own mutation visible before the next read | write-through | src/pkg/forgestate/writethrough.go:38 |
| Freshness bound before irreversible actions | merge, close, claim and kick re-read an item older than DecisionFreshnessBound (60s) and refuse outright while the repo’s share is stale (irreversible action deferred) | src/pkg/github/decision_freshness.go:15, src/pkg/forgestate/store.go:334 |
| Startup gate | the first governor decision pass waits for a full reconcile of every repo (WaitReady); a cold cache whose fetch failed is published stale, never empty | src/pkg/forgestate/store.go:320 |
| Drift detection | each reconcile diffs cache vs API into forgestate_drift; above drift_threshold the repo’s reconcile interval shortens (the dashboard warning of #13373 follows once a reader wires the metric) | src/pkg/forgestate/reconcile.go:58, src/pkg/forgestate/metrics.go:13 |
| Provenance | tracker rows carry fetched_at and source (webhook / reconcile); the overview shows stale · as of HH:MM instead of zeros | operator-reference.md |
Configuration and defaults
| Key | Default | Meaning |
|---|---|---|
governor.eval_interval_s | 300 | legacy polling cadence; the base tick in polling mode |
governor.eval_interval_webhook_s | 900 | the base tick in webhook mode; 0 disables webhook mode (always poll) |
governor.eval_interval_max_s | 1800 | cap on either tick after the API-budget stretch |
governor.webhook_wake_debounce_s | 10 | a delivery and its check fan-out coalesce into one wake cycle |
governor.webhook_wake_min_gap_s | 30 | minimum spacing between webhook-triggered cycles, and between the cycles recorded review verdicts trigger |
governor.ci_failure_webhook_kick | true | a failed check_run/check_suite/status delivery kicks the owning agent (hive-authored PR) or runs a targeted contributor-lane pass (contributor PR) at once; false records the signal for the sweep only |
governor.ci.auto_approve_workflows | true (unset = on) | approve GitHub-gated (action_required) pull_request workflow runs once per trusted PR head, in the sweep right after CI enrichment; the sweep is skipped on cycles where the API-budget decision sets SkipOptional (conserve/critical mode, off-cadence and cached-snapshot cycles). false leaves the gate to a person. See runtime-and-quota.md |
governor.ci.auto_approve_workflow_actors | ['dependabot[bot]', 'renovate[bot]'] | PR authors / triggering actors trusted for workflow approval, besides this hive’s App and heads a person admitted from the dashboard (the automerge label alone does not count); an explicit [] trusts only those two |
governor.ci_failure_kick_cooldown_s | 600 | at most one webhook-triggered CI-failure kick or nudge per PR head in this window |
forgestate.reconcile_interval_s | 900 | ForgeState’s own full-list interval per repo, shortened while drift is high |
forgestate.freshness_bound_s | 60 | maximum item age before merge/close/claim/kick re-reads it |
forgestate.drift_threshold | 0.05 | fraction of items a reconcile may find wrong before the interval shortens |
dashboard.stage_tracker.webhook_refresh_debounce_ms | 2000 | debounce of the single-item re-read a delivery queues |
dashboard.stage_tracker.webhook_refresh_replay_window_s | 1800 | how long a re-read is remembered and re-applied to every status snapshot the governor publishes, so a cycle that enumerated before the close cannot re-list the item |
dashboard.stage_tracker.closed_sweep_interval_s | 300 | how often (driven by status publishes) each repo’s items closed since the last look are listed (one call per repo) and rows still shown are evicted; negative disables |
dashboard.stage_tracker.waiting_push_stale_h | 4 | hours a CI / Review row may go without a push (booked from the pull_request synchronize / push re-read, enumerate as fallback) before its pill’s age takes the muted-amber tone; negative disables |
dashboard.stage_tracker.merged_hold_s | 8 | ceiling (seconds, floor 3) on how long a row stays at “Merging · merge landed” after the payload dropped it before the merged fact arrives; the celebration (toast, bee, odometer) launches as soon as the merge is confirmed, not at the end of the hold (hivecommons/hive#15808) |
| health window | 2 × governor.eval_interval_s (10 min at the defaults) | how recent the newest verified delivery must be for webhooks to count as healthy |
| recovery hysteresis | WebhookRecoveryHealthyWindows = 2 | consecutive healthy windows required before the cadence returns to webhook mode after a degradation |
Defaults: src/pkg/config/config.go:7192 (DefaultEvalIntervalWebhookS,
DefaultWebhookWakeDebounceS, DefaultWebhookWakeMinGapS),
src/pkg/config/forgestate.go:8 (DefaultForgeStateReconcileInterval,
DefaultForgeStateFreshnessBound, DefaultForgeStateDriftThreshold),
src/pkg/github/webhook_cache.go:14 (DefaultWebhookHealthWindow, replaced
by SetWebhookHealthWindow at src/pkg/github/webhook_cache.go:102 once the
governor loop publishes 2 × eval_interval_s), and
src/pkg/github/webhook_mode.go:34 (WebhookRecoveryHealthyWindows). The
effective tick is composed in webhookAwareEvalInterval
(src/cmd/hive/api_budget_decision.go:82): the API-budget stretch still
applies and the larger interval wins.
governor:
eval_interval_s: 300
eval_interval_webhook_s: 900
eval_interval_max_s: 1800
webhook_wake_debounce_s: 10
webhook_wake_min_gap_s: 30
forgestate:
enabled: false
reconcile_interval_s: 900
freshness_bound_s: 60
drift_threshold: 0.05
Fallback when webhooks degrade
Webhook mode is only safe while deliveries actually arrive. The governor
therefore keeps a small cadence mode machine on top of the raw health
window (src/pkg/github/webhook_mode.go:100, observe) and recomputes it
wherever it recomputes the eval interval
(evalIntervalAndModeForConfig, src/cmd/hive/api_budget_decision.go:115),
including the re-check timer that fires as the health window closes
(webhookHealthRecheckDelay, src/cmd/hive/webhook_wake.go:238).
States and transitions:
- polling (boot) — no verified delivery yet. Base tick =
eval_interval_s. The first healthy window engages webhook mode at once; there is no hysteresis before webhooks have ever degraded. - webhook — the base tick is
eval_interval_webhook_s. The mode degrades to polling when either:- the health window lapsed and GitHub was active meanwhile: a cadence
pass listed an item whose
updated_atis newer than the newest delivery (plus a 30 s clock-skew allowance,webhookActivityClockSkew,src/pkg/github/webhook_mode.go:40). The reconcile pass reports that newestupdated_atthroughRecordGitHubActivity(src/pkg/github/webhook_mode.go:183). Silence alone — a quiet repository overnight — is not degradation and keeps the stretched tick; or - deliveries are failing: the receiver saw a delivery (direct or relayed
by the hub) whose signature did not verify or whose payload did not
decode, newer than the last good delivery and inside the window
(
RecordWebhookDeliveryFailure,src/pkg/github/webhook_mode.go:199, called fromsrc/pkg/dashboard/api_github_webhook.go:58). A wrongGITHUB_WEBHOOK_SECRETon a spoke therefore shows up as polling mode withmode_reason: deliveries failing verification, not as a silent stretch to 900 s.
- the health window lapsed and GitHub was active meanwhile: a cadence
pass listed an item whose
- recovering (polling after a degradation) — each observation with a
healthy window and no newer failure extends a streak;
webhooks.healthy_windowscounts it (1 at the first healthy observation, +1 per further window of uninterrupted health). A lapse or a failure resets the streak to zero. - back to webhook once the streak reaches
WebhookRecoveryHealthyWindows(2): at the defaults, one delivery followed by ten more minutes of deliveries.
The transition is logged exactly once each way, at INFO:
webhooks degraded; polling cadence restored (with reason,
effective_interval_s, recovery_healthy_windows) and
webhooks healthy; webhook cadence engaged. The governor’s ticker is reset
on the same observation (eval interval changed, resetting ticker), so the
polling cadence is in force from the next tick, not after the stretched one
would have fired.
The dashboard reads the same machine:
/api/status.webhooks.mode—webhook|polling;/api/status.webhooks.since— when the current mode began;mode_reasonwhile polling after a degradation;healthy_windowswhile recovering (src/pkg/github/webhook_cache.go:307,snapshot).webhooks.healthystays the raw window (a delivery inside 2 ×eval_interval_s), sohealthy: false, mode: webhookreads “quiet but trusted” andhealthy: true, mode: pollingreads “recovering”./api/status.evalMode(webhook|poll) andevalIntervalSare the loop’s view of the same decision, and the governor card’s next-kick estimate uses it (src/pkg/dashboard/status_builder.go:2058).
What stays on the raw window: the per-repo PR-detail cache TTL extension and the “skip the repo’s re-enrichment” shortcut still require a delivery inside the window for that repository, so a quiet-but-trusted repo pays its normal detail refreshes — the fallback only decides the cadence.
Operator checks:
curl -s $SPOKE/api/status?fields=webhooks | jq .webhooks— expectmode: webhookon a busy repo with the App webhook pointed at the hub or the spoke;pollingwith amode_reasonis a configuration fault to fix (secret, App subscription, hub relay target),pollingwithout one means no delivery has ever arrived.grep 'webhooks degraded' hive.log— one line per outage with the reason; the matchingwebhooks healthy; webhook cadence engagedcloses it.- A degradation shortens the cadence back to
eval_interval_s, so API spend returns to the pre-webhook level while it lasts; the budget stretch (conserve/critical) still applies on top.