Monitoring

Health endpoint#

GET /health is the only health endpoint — there is no /ready, /healthz, or /livez. It combines liveness and readiness: it runs SELECT 1 against the framework database and pings Redis, and returns:

• 200 when both dependencies are reachable,
• 503 otherwise, with a per-dependency breakdown so you can tell which leg is down.

{
  "status": "ok",
  "version": "1.4.1",
  "commit": "cdcbcc8",
  "deps": { "framework_db": "ok", "redis": "ok" }
}

On a degraded instance status is "degraded" and the failing entry in deps carries a down: <reason> string. version and commit are the build identity stamped into release images.

Use it everywhere a probe is needed: the Helm chart points both the API liveness and readiness probes at it, the backend image's container healthcheck curls it, and it is the right smoke test after an install:

curl -fsS "http://localhost:8080/api/v1/health"

/health is unauthenticated and keeps working even when the license has expired (the platform goes API-read-only, never health-dark).

Prometheus metrics#

GET /metrics exposes Prometheus metrics (unauthenticated, outside the API schema). The key families:

The complete metric inventory with labels lives in the metrics reference.

Grafana dashboard and recording rules#

The repository ships ready-made observability assets under docs/grafana/ in the product distribution:

• abrq-overview.json — a Grafana dashboard (import via Dashboards → New → Import) with an SLO row plus run-health, CDC, worker-fleet, connector-pool, streams, and HTTP sections.
• recording-rules.yml — Prometheus recording rules and burn-rate alerts for three starting SLOs: API p95 latency, CDC max lag, and 24-hour run success rate. Drop it into your Prometheus rules directory and reload.

The thresholds are starting points — adjust them in the rules file; the dashboard reads the recording rules, so panels follow the thresholds automatically.

Logs#

The backend, workers, and scheduler emit structured JSON logs to stdout — one JSON object per line, with ISO timestamps, level, and the request's user id bound in. There are no application log files; collect logs from container stdout with your platform's log pipeline (the only file ever written is the one-time initial admin password file).

• The level is set with ABRQ_LOG_LEVEL (default INFO).
• A redaction processor runs in every environment: event keys whose names look secret-shaped (password, secret, token, key, credential, and similar — including nested payloads) are replaced with a redaction marker, and Bearer <VALUE> / Authorization: shapes in rendered messages are pattern-scrubbed before the line is emitted.

Run logs and live tails#

Pipeline execution logs are persisted in the framework database, per family: CDC run logs, ETL task run logs, ETL job run logs, and export job run logs. They are browsable per run in the UI, subject to the retention sweep (ABRQ_RUN_LOG_RETENTION_DAYS — see Environment variables).

Live tails are served over Server-Sent Events:

• GET /api/v1/cdc-tables/<ID>/logs/stream — CDC log tail
• GET /api/v1/streams/<ID>/runs/stream — stream run tail

SSE endpoints authenticate with the access token as a ?token= query parameter (browsers cannot set headers on an event stream).

The audit log has its own streaming export — GET /api/v1/audit-log/export (NDJSON or CSV download); see Audit.

Distributed tracing (optional)#

OpenTelemetry tracing is built in and off by default — with a blank endpoint the tracer is a no-op and costs nothing. Three environment variables control it:

When an endpoint is set, spans are batch-exported and FastAPI + SQLAlchemy are auto-instrumented.