Skip to content
ABRQ DATADocs Abrq DIP · latest
Product page Request a trial
On this page

Metrics

Scraping#

The backend exposes Prometheus text exposition at GET /metrics. Every metric name is prefixed abrq_. Label values are deliberately low-cardinality — no run ids, no user ids — with one documented exception (abrq_cdc_table_lag_seconds).

scrape_configs:
  - job_name: abrq-dip
    metrics_path: /metrics
    static_configs:
      - targets: ["192.0.2.20:8000"]

The tables below list every metric that exists. Nothing else is exported.

Build identity#

Metric Type Labels Meaning
abrq_build_info Gauge version, commit Constant 1; the running build's identity is carried in the labels. Join on it to tag other series with the version that produced them, or count by (version) for a fleet view.

License#

These three gauges are published by the hourly license re-check task, so they appear once that task has run at least once — not immediately at boot.

Metric Type Labels Meaning
abrq_license_days_remaining Gauge — Days until the license expires. -1 means it has already expired.
abrq_license_load_failed Gauge — 1 if the most recent re-check failed to load and verify the license file, 0 otherwise. Most useful for alerting on "it used to load and now it doesn't", which happens when a license file is overwritten with a malformed copy.
abrq_license_enforcement_active Gauge — 1 when enforcement is active (ABRQ_ENV=prod), 0 when bypassed in dev or test. Lets you alert on a deployment running with enforcement disabled.

Tip. Alert on abrq_license_days_remaining < 14 to get ahead of a renewal, and on abrq_license_load_failed == 1 to catch a bad file swap before the next restart refuses to boot. See Expiry and clock integrity.

Runs#

Metric Type Labels Meaning
abrq_runs_total Counter job_name, status Total ETL runs by terminal status. status ∈ succeeded, failed, cancelled, degraded. Ad-hoc step runs use the sentinel job_name __adhoc__.
abrq_run_duration_seconds Histogram job_name, status ETL run duration. Buckets: 0.5, 1, 5, 15, 30, 60, 120, 300, 600, 1800 s.
abrq_cdc_runs_total Counter strategy, status CDC sync runs by strategy and terminal status.
abrq_cdc_rows_total Counter op Rows written by CDC syncs, by operation (insert / update / delete).
abrq_cdc_table_lag_seconds Gauge cdc_table_id Seconds since the last successful sync, per CDC table. The one per-entity label in the set.
abrq_stream_runs_total Counter status Stream fetch runs by terminal status.
abrq_stream_rows_total Counter none Rows persisted by stream fetches.
abrq_file_ingestion_runs_total Counter status, file_format File processings by terminal status. status ∈ succeeded, failed, skipped.
abrq_file_ingestion_rows_total Counter file_format Rows persisted by file ingestion runs.
abrq_file_ingestion_parse_errors_total Counter file_format, phase Parse failures by format and cause. phase ∈ read, parse, write.
abrq_runs_zombie_reaped_total Counter kind Rows the reaper marked failed because the worker heartbeat went stale. kind ∈ cdc_runs, stream_runs, etl_task_runs, etl_job_runs, etl_job_group_runs, etl_job_task_runs.
abrq_runs_max_heartbeat_age_seconds Gauge kind Maximum heartbeat age across active runs, updated each reaper tick. A lead indicator — it rises before the reaper actually fires.
abrq_logs_purged_total Counter table Run-log rows deleted by the retention sweep. table ∈ run_logs, etl_task_run_logs, etl_job_run_logs.

Schema drift and rescue#

Metric Type Labels Meaning
abrq_schema_drift_events_total Counter pipeline_type, action Drift events recorded to the registry. pipeline_type ∈ file_ingestion, stream, email_feed, cdc; action ∈ blocked, evolved, rescued, proceeded.
abrq_rescued_rows_total Counter pipeline_type Rows that captured unexpected columns into the _abrq_rescued_data catch-all.

A non-zero rate(abrq_schema_drift_events_total{action="blocked"}[1h]) means a pipeline is stopped waiting for an operator. See Schema drift policies.

Stream fan-out#

Metric Type Labels Meaning
abrq_stream_fanout_deliveries_total Counter destination_kind, outcome Per-destination drain outcomes. outcome ∈ delivered, failed, skipped, dead_lettered.
abrq_stream_fanout_dead_letters_total Counter destination_kind Batches dead-lettered after exhausting the retry budget.
abrq_stream_fanout_circuit_opened_total Counter destination_kind Times a destination's circuit breaker tripped open.

destination_kind is the connector engine — postgres, kafka, s3, ftp and so on. See Stream destinations.

Connector pool#

Metric Type Labels Meaning
abrq_connector_pool_size Gauge none Cached engine handles the pool currently holds.
abrq_connector_pool_acquires_total Counter outcome Acquire calls by cache outcome. outcome ∈ hit, miss.
abrq_connector_pool_evicts_total Counter reason Evictions by trigger. reason ∈ rotated (credentials changed), deleted (connector removed), shutdown.

Hits should dominate in steady state; rate(miss) > rate(hit) means the cache is thrashing.

Notifications#

Metric Type Labels Meaning
abrq_notifications_dispatched_total Counter kind, outcome Notifications dispatched on run completion. outcome ∈ sent, failed.

Scheduler and workers#

Metric Type Labels Meaning
abrq_scheduler_due_rows Gauge scheduler Rows the scheduler considered due on its last tick. scheduler ∈ cdc, stream.
abrq_celery_task_failures_total Counter task Background tasks that exhausted their retries and went terminal-failed. The Redis broker has no native dead-letter queue, so this counter is the signal.

Warehouse modelling#

Metric Type Labels Meaning
abrq_dw_models_generated_total Counter target_engine Warehouse-model generate calls.
abrq_dw_models_saved_total Counter target_engine Warehouse-model configurations persisted.

HTTP#

Metric Type Labels Meaning
abrq_http_requests_total Counter method, route, status Requests served. route is the route template, not the concrete URL, so cardinality is bounded by the route count; unmatched requests are labelled unmatched.
abrq_http_request_duration_seconds Histogram method, route Request latency. Buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 s.

Grafana assets in the repository#

Three files ship under docs/grafana/:

File What it is
abrq-overview.json The overview dashboard — import it into Grafana as-is
recording-rules.yml Prometheus recording rules plus the matching alerts
README.md What the shipped targets mean and how to retune them

The recording rules pre-compute the expressions the dashboard panels read, so a panel and the alert that pages on it evaluate the same expression. One real example, the API latency SLO:

- record: abrq:slo:api_p95_latency_seconds:5m
  expr: |
    histogram_quantile(
      0.95,
      sum by (le) (
        rate(abrq_http_request_duration_seconds_bucket[5m])
      )
    )

Other rules in the file include abrq:slo:cdc_max_lag_seconds, abrq:slo:run_success_rate:24h, the per-route p50/p95/p99 latency rules, a per-route 5xx rate, and abrq:connector_pool:hit_ratio:5m.

Note. The shipped thresholds are calibrated for one internal deployment. Read docs/grafana/README.md and retune them to your own reliability bar — the dashboard follows the rules file automatically.

Tracing#

Metrics answer "how much" and "how often"; distributed tracing answers "where did this request spend its time". Trace export is configured with the OTLP environment variables documented in Environment variables.