Oauth grafana

OAuth tokens for Grafana alert webhooks

Grafana fires alerts at the worker via Alertmanager-shaped webhooks (POST /api/webhooks/alertmanager). This guide covers the OAuth-token path — option B from Glad-Labs/poindexter#247.

Why a pre-issued long-TTL JWT, not Grafana’s OAuth contact-point flow

Grafana’s contact-point UI exposes a oauth2 block on newer versions, but in practice it’s brittle:

• The IdP fields drift across Grafana minor versions (the YAML keys in 10.x don’t match the UI labels in 11.x).
• Grafana’s contact-point HTTP client has no way to refresh through a client_credentials grant — it expects an authorization-code style redirect URI, which a headless alerting system can’t satisfy.
• The IdP-side error messages surface as opaque “401 Unauthorized” in the contact-point test page, with no log on the Grafana side.

Pre-issued long-TTL JWTs side-step all of that. The operator mints one JWT per quarter (90-day default), pastes it into the contact-point “Authorization Header” field, and rotates manually. The worker accepts both OAuth JWTs and the legacy static alertmanager_webhook_token during the migration window (Glad-Labs/poindexter#247).

Provision the token

The recommended path is --persist, which stashes the minted JWT encrypted in app_settings.grafana_webhook_oauth_jwt so scripts/start-stack.sh can decrypt and pass it through to Grafana via $GRAFANA_WEBHOOK_TOKEN. No manual paste-into-the-UI step — restart Grafana and the contact point picks up the new credential.

poindexter auth mint-grafana-token --persist --ttl 365d
bash scripts/start-stack.sh up -d --force-recreate --no-deps grafana

This:

1. First call: registers a new OAuth client named grafana-alerts in oauth_clients, persists encrypted credentials to app_settings.grafana_oauth_client_id and app_settings.grafana_oauth_client_secret.
2. Every call: mints a fresh JWT bound to that client with the default scopes api:read api:write and the requested TTL.
3. With --persist: also stores the minted JWT encrypted in app_settings.grafana_webhook_oauth_jwt. scripts/start-stack.sh decrypts it on the host (via scripts/_grafana_webhook_token.py) and exports as $GRAFANA_WEBHOOK_TOKEN before bringing the Grafana container up. infrastructure/grafana/provisioning/alerting/contact-points.yml substitutes that env var into settings.authorization_credentials on the Poindexter Webhook receiver.

Why settings instead of secure_settings

Grafana 13.0.1’s file-provisioning loader does NOT run env-var substitution over secure_settings: — only settings:. Prior attempts to wire the JWT through secure_settings.authorization_credentials silently no-op’d (verified during the 2026-05-19 finding #2 investigation; secureFields stayed null after restart with both camelCase and snake_case keys, and with literal-token values too).

Grafana auto-encrypts plain settings.authorization_credentials at rest on its side, redacts it on read, and (crucially) DOES run env-var substitution over it. Our source of truth stays encrypted at rest in app_settings.grafana_webhook_oauth_jwt via pgcrypto, so the secret isn’t duplicated unprotected anywhere on disk.

Output looks like:

Grafana OAuth client provisioned + token minted.
  client_id:   pdx_a1b2c3d4e5f6...
  scopes:      api:read api:write
  ttl:         90d (7776000s)
  expires_at:  1785000000 (epoch — 2026-08-01 12:00 UTC)
  jti:         3f8b9c1d...

Token (paste into Grafana contact-point Authorization Header):

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwb2luZGV4dGVyIiwic3Vi...

Capture the JWT NOW — it’s not stored anywhere, and re-running the command mints a different JWT (the old one keeps verifying until its own exp elapses).

Paste into Grafana (legacy — only if you ran mint-grafana-token WITHOUT --persist)

If you minted with --persist, skip this section — the JWT is already wired through provisioning. The manual paste workflow below is the fallback for operators who minted with the older flow OR who want to override the auto-provisioned credential.

Grafana’s UI labels move every few releases. The path below is for Grafana 11.x. If your Grafana looks different, search for “contact points” → your contact point → custom HTTP headers.

1. Open Grafana → Alerting → Contact points.
2. Click the contact point that points at the worker webhook (default name: Poindexter Webhook).
3. Edit → Optional Webhook settings → Custom HTTP headers.
4. Click + Add header.
5. Header name: Authorization
6. Header value: Bearer <paste the JWT here>
   • The literal word Bearer, then a single space, then the JWT.
7. Save contact point.
8. Test the contact point (button at top of the edit page). The worker should respond 200 OK with a JSON body counting persisted alerts. If you get 401 Unauthorized, double-check there’s a single space between Bearer and the token and no trailing whitespace.

We deliberately don’t push the token via the Grafana API for the manual workflow — the contact-point JSON contains the bearer in plaintext, and the API push ergonomics (token in argv, token in shell history, token in CI logs) aren’t worth the time saved.

Verify end-to-end

After saving the contact point:

# Watch the worker accept the next test webhook.
docker logs -f poindexter-worker 2>&1 | grep -i "alertmanager webhook"

# In Grafana, hit the "Test" button on the contact point.
# Expected log line: "alertmanager webhook: received=1 persisted=1 paged=0 remediated=0"

If the test fires but the worker logs OAuth JWT rejected: token expired, re-run the mint command — your previous token’s exp already elapsed (unusual on a 90-day TTL, but possible if you provisioned a 60-min token by accident).

Rotation

Set a calendar reminder for ~80 days after each mint (the default 90-day TTL minus 10 days of buffer). When it fires:

poindexter auth mint-grafana-token --ttl 90d

Paste the new JWT into the same Grafana contact-point header, save. The previous JWT keeps verifying until its own exp elapses, so there’s no traffic gap.

To invalidate ALL JWTs for the Grafana client immediately (e.g. after a suspected leak), revoke the underlying OAuth client:

# Get the client_id.
poindexter auth list-clients | grep grafana-alerts

# Revoke. Outstanding JWTs continue verifying until their exp elapses
# — JWT verification is stateless and we trade revocation latency for
# scalability per the #241 design decision.
poindexter auth revoke-client --client-id pdx_<the_id>

# Clear the cached client_id so the next mint provisions a new one.
poindexter settings set grafana_oauth_client_id "" --secret
poindexter settings set grafana_oauth_client_secret "" --secret

# Mint a fresh token bound to the new client.
poindexter auth mint-grafana-token --ttl 90d

The Grafana contact-point header continues using the old JWT until the operator pastes the new one in — same procedure as a normal rotation.

Troubleshooting

Symptom from Grafana	Likely cause
401 Unauthorized on test	Token typo, missing Bearer prefix, the token’s exp elapsed, or the underlying OAuth client was revoked. Re-mint and re-paste.
503 Service Unavailable	app_settings.alertmanager_webhook_token is empty. Run poindexter settings set alertmanager_webhook_token "<value>" --secret.
Test succeeds, real alerts never fire	Check the Grafana alert rule’s Notifications block — the contact point must be selected.

Note: Glad-Labs/poindexter#249 dropped the static-Bearer fallback on the general worker-API middleware, but this webhook route is the exception. verify_alertmanager_token (routes/alertmanager_webhook_routes.py) still accepts the legacy static alertmanager_webhook_token alongside OAuth JWTs, so a correctly-set static token also passes. A header that is neither a valid JWT nor the configured static token gets 401 — or 503 if the static token is unset and the JWT path doesn’t accept (fail-closed).

See also

• docs/operations/secret-rotation.md — full secret-rotation runbook (the Grafana JWT row is one of many).
• Glad-Labs/poindexter#241 — OAuth 2.1 migration umbrella, full design rationale.
• Glad-Labs/poindexter#247 — the issue this work closes.