Skip to content
ABRQ DATADocs Abrq DIP · latest
Product page Request a trial
On this page

Troubleshooting

How to use this page#

Each section is a symptom → cause → resolution table. Messages are quoted the way the platform emits them, so searching this page for the exact string in your logs is the fastest route to the answer.

If the symptom is not here, go to Where to look and collect the evidence before changing anything.

Startup#

The backend validates its configuration before it serves a single request. A failed validation is a hard exit, not a degraded start.

Symptom Cause Resolution
Container exits immediately with a validation error naming ABRQ_MASTER_KEY The master key is unset or the wrong length. It encrypts every stored connector credential. Generate one and set it. It must stay identical across restarts and across every replica — a changed key makes existing credentials undecryptable.
Container exits with a validation error naming ABRQ_JWT_SECRET The signing secret is unset. Set it to a strong random value, shared by every replica.
ABRQ_JWT_SECRET must be at least 32 characters in prod A secret was set, but it is too short for a production environment. Replace it with at least 32 characters of random material. Rotating it invalidates all live sessions.
ABRQ_INITIAL_ADMIN_PASSWORD is the dev default ('password'). Set it to a strong random value before starting in prod. The production guard refuses to boot with the development seed password. Set ABRQ_INITIAL_ADMIN_PASSWORD to a strong random value. Change it again through the UI after the first sign-in.
Startup refuses with an empty-CORS error in production No allowed origins configured. Production will not fall back to a permissive default. List the exact origins the browser UI is served from.
license not found at startup No license file is mounted where the backend expects it. Mount the license file into the container and confirm the path. Check the volume actually landed with docker compose config.
license signature invalid The file was edited, truncated, or transferred in text mode. Re-copy the original file byte-for-byte. Never edit a license.
license expired The validity window has passed. Contact Abrq for a renewal. Until then the instance runs read-only (see Auth and permissions).
Integrity-manifest mismatch on startup The shipped application files no longer hash to the manifest — a partial upgrade, a modified image layer, or a corrupted pull. Re-pull the image at the pinned tag and restart. Do not patch files inside a running container.
docker compose logs --tail=100 backend

Auth and permissions#

401 — the token itself#

detail Cause Resolution
missing bearer token No Authorization: Bearer … header. SSE endpoints need ?token= instead, because EventSource cannot set headers. Attach the header, or the query parameter for SSE.
token expired Access tokens live 15 minutes. Refresh. Browsers do this automatically via the abrq_refresh cookie.
token invalid Wrong signature — usually a token minted under a different ABRQ_JWT_SECRET, or pointed at a different instance. Sign in again against the instance you are calling.
token revoked The session was revoked. Reusing an already-consumed refresh token revokes the entire chain by design. Sign in again. If it recurs, look for two clients sharing one refresh cookie.

403 — scope, role, membership, license#

detail / message Cause Resolution
api token has read-only scope; this endpoint requires write scope A personal access token with read scope was used on an unsafe method. Scope is enforced centrally, not per route. Mint a token with write scope, or switch the call to a read.
requires role: editor (or higher) The account's role is below what the endpoint needs. Have an administrator raise the role, or perform the action as someone who has it.
permission_denied with reason: not_a_member The account is not a member of the project that owns the resource. Add the account to the project. Note that a project-scoped token cannot reach outside its project at all.
permission_denied with reason: protected_environment The target environment is protected; writes to it need an explicit grant regardless of role. Grant access to the environment, or target a non-protected one.
license_expired The instance is in read-only mode. Every write is refused platform-wide. Renew the license. Reads keep working meanwhile.
A 403 naming a feature that is not licensed The entitlement for that capability is not in the license. Contact Abrq to extend the entitlement. Enabling it in configuration will not bypass the check.

The RBAC 403 body carries reason, required_capability, project_id and environment_id — read those rather than guessing which of the four causes applies. See API.

Deployment#

Symptom Cause Resolution
pull access denied when starting Compose ABRQ_VERSION is unset, so the tag resolved to something that does not exist for your registry credentials. Set ABRQ_VERSION in the environment file and verify the resolved image before starting.
Helm install hangs, migration Job never finishes, database shows a lock wait The pre-install migrator hook Job runs before the in-cluster database is ready, and the two deadlock. Do not enable the pre-install hook Job against an in-cluster database. Run migrations as a post-install/upgrade step, or point at an external database that is already up.
Pods restart in a loop shortly after becoming ready Probes are pointed at an endpoint that does not exist, or their timeouts are shorter than startup takes. Point liveness and readiness at GET /health — it is the only health endpoint; /ready, /readyz and /healthz do not exist. Give the startup probe enough budget for migrations.
One replica works, another rejects every token The replicas do not share ABRQ_JWT_SECRET (and ABRQ_MASTER_KEY). Source both from one secret object mounted into every replica.
docker compose config | grep image:

Pipelines#

Symptom Cause Resolution
First CDC sync fails with strategy 'x' does not support engine 'y' The picker allowed a strategy/engine pairing the platform does not implement. It is validated at run time, not at save time. Re-create the CDC table with a supported pairing — see CDC strategies.
409 when saving a CDC table's destination The destination table is already claimed by another CDC table, or the same source/destination pair already exists. Point at a different destination table, or edit the existing CDC table instead of creating a second one.
422 on a table reference The fully-qualified name is malformed — wrong separator, missing schema, or stray quoting. Supply <schema>.<table> exactly as the source spells it, including case where the engine is case-sensitive.
Run stops with a drift message and a critical alert The pipeline's drift policy is Reject, or a policy deferred to a block_and_alert fallback. Reconcile the destination, then resume. See Schema drift policies.
CDC stops with destructive source schema change (…) — never auto-applied; resolve the mirror manually, then resume A column was dropped or retyped upstream. Destructive drift blocks under every policy for CDC, including Rescue. Apply the change to the mirror yourself, then resume the CDC table.
ETL task fails on environment binding The task's environment has no connector bound for the role the task needs, or the binding points at a deleted connector. Bind a connector for every role the task uses in that environment, then re-run.
ETL fails with an unsupported-engine error before any SQL runs Only the PostgreSQL family (postgres, cockroachdb) and the MySQL family (mysql, singlestore) have ETL adapters. Move the task to a supported engine — see Connectors.
Export fails with source engine 'mariadb' is not a supported export source yet MariaDB appears in the export-source picker but has no export implementation. Use a supported export source: postgres, cockroachdb, mysql or singlestore.

Streams and notifications#

Symptom Cause Resolution
A destination stops receiving; the UI shows the circuit open with a cooldown Five consecutive delivery failures tripped the breaker. It stays open for the 300 s cooldown, then allows one trial batch. Fix the sink, then either wait out the cooldown or use Reset breaker. Resetting into a still-broken sink just re-opens it.
Dead letters accumulating for one destination Batches exhausted the 3-attempt retry budget. The reason class says why: validation, timeout, source, destination or other. Resolve the cause, then replay the dead letters to that destination. Discard is permanent data loss.
Every destination falls behind together The problem is on the source side, not in a sink — the spool cannot stage. Check the source connector and the stream run logs, not the destinations.
Spool grows without bound A destination is enabled and far behind; spool GC only deletes below the minimum high-water mark across enabled destinations. Recover the lagging destination. Disabling it lets GC advance past batches it never received — those are not backfilled on re-enable.
No notifications arrive at all SMTP is not configured. Configure the SMTP host, port and credentials in notification settings, then send a test.
SMTP send fails immediately Both STARTTLS and SSL are enabled. They are mutually exclusive transports. Enable exactly one: STARTTLS on port 587, or implicit SSL on port 465.
Drift on an email feed is never alerted Email feeds have no notification channel. Drift there is recorded and metered, never delivered. Watch abrq_schema_drift_events_total{pipeline_type="email_feed"} instead.

Where to look#

Source What it gives you
Container stdout Structured JSON logs, one object per line, including the request_id that appears in every error response. The platform writes no log files — do not go looking in the container filesystem.
Per-family run-log tables Persisted logs per run for CDC, streams, file ingestions and ETL, visible in the UI and subject to the retention sweep.
SSE tails Live log lines while a run is in progress: the CDC log tail and the stream-runs stream.
Audit feed Who changed which configuration, and when. The first place to check when behaviour changed without a deploy.
GET /health Whether the framework database and Redis are both reachable, plus the running version and commit.
docker compose logs -f backend | grep '"level":"error"'

curl -s https://<ABRQ_HOST>/health

Tip. When you open a support conversation, include the request_id from the error response, the version and commit from GET /health, and the exact message text. Those three make almost every report reproducible.