Background processing

Queue layout#

Tasks are routed to purpose-specific Celery queues so that a long ETL run can never starve short, frequent work such as inbox polls or CDC ticks:

Queue	What runs there
`abrq_dip`	Default queue for anything not routed elsewhere
`abrq_dip.cdc`	CDC sync runs
`abrq_dip.streams`	Stream runs
`abrq_dip.email`	Inbox polling and reminder scans
`abrq_dip.maintenance`	Zombie-run reaper, log purge, license re-check, audit-chain verification and forwarding, auto-backup, file-ingestion and export scheduling ticks, alert evaluation
`abrq_dip.file_ingestion`	Per-feed file-ingestion runs
`abrq_dip.etl`	SQL task runs, job runs, and the job-schedule tick

Workers consume all of these queues by default. To shard, set ABRQ_CELERY_QUEUES to a subset per worker — for example, a dedicated heavy-ETL worker on abrq_dip.etl and a second worker on the remaining queues. A worker listening only on the default queue would leave scheduled work unconsumed, so always keep the full list covered across your fleet.

Runs mode: sync or celery#

ABRQ_RUNS_MODE selects how runs execute:

sync — the API process executes runs inline. This is the code default and suits local evaluation only.
celery — the API enqueues runs to Redis and workers execute them. The Helm chart and the production Compose files both set celery; use it for any real deployment.

Concurrency#

Knob	Default	Effect
`ABRQ_CELERY_CONCURRENCY`	`4`	Concurrent task slots per worker replica; the container entrypoint passes it as Celery's `--concurrency`
Worker replicas	2 (Helm `worker.replicaCount`; Compose `ABRQ_WORKER_REPLICAS`)	Horizontal scale; the Helm chart can also enable an HPA (min 2, max 20)
`worker_prefetch_multiplier`	`1` (fixed)	Each slot prefetches one task, favouring fairness across pipelines over peak throughput
Task time limits	1 hour soft, 1 hour 5 minutes hard (fixed)	A hung task cannot hold a worker slot forever

Tasks are acknowledged late with reject-on-worker-lost, so a task that was mid-flight when a worker died is re-queued rather than silently lost. Runs are looked up by id, not created by the task, which makes such redelivery safe.

The Beat singleton#

Exactly one Beat process must run per installation — Celery Beat has no coordination protocol, and two schedulers would enqueue every periodic task twice. The deployments enforce this:

The Helm chart hardcodes the beat Deployment to replicas: 1 (it is not a template value) with a Recreate update strategy, so the old scheduler is fully stopped before its replacement starts — even during upgrades, two Beats never overlap. Beat has no HPA and no PodDisruptionBudget.
Beat writes its schedule state to /var/abrq-dip/celerybeat-schedule on a persistent volume (a 1 Gi ReadWriteOnce PVC in Helm; a named volume in Compose), so a restarted scheduler resumes where it left off instead of re-firing or skipping ticks.
The production Compose file pins the beat service to one replica.

Beat only enqueues; every enqueued task is executed by the workers.

Built-in periodic schedule#

All times are UTC.

Task	Cadence	Purpose
Zombie-run reaper	Every 2 minutes	Fails runs whose worker disappeared
Run-log purge	Daily	Deletes run-log rows older than `ABRQ_RUN_LOG_RETENTION_DAYS`
Encrypted auto-backup	Daily	No-op unless enabled in Settings; writes a new copy only when configuration changed
Inbox poll	Every minute	Polls all active email inboxes
Email reminder scan	Every 5 minutes	Evaluates email-feed reminders
License re-check	Hourly	Re-reads the local license file; never calls out
Audit-chain verification	Daily	Detects tampering with the hash-chained audit log
Audit SIEM forwarding	Every minute	Forwards audit rows past the cursor to the configured target
File-ingestion poll	Every minute	Triggers due file-ingestion feeds
Job-schedule tick	Every minute	Fires enabled job schedules whose next fire time has passed
Export-schedule tick	Every minute	Fires due scheduled exports
Alert evaluation	Every minute	Evaluates enabled alert rules and delivers on state transitions

Shutdown grace#

ABRQ_RUN_SHUTDOWN_GRACE_SECONDS (default 60) is the drain window a stopping worker gives its in-flight runs. The platform-level stop timeout must be at least as long: the Helm chart sets the worker pods' terminationGracePeriodSeconds to 60 and the production Compose file sets stop_grace_period: 60s. If your longest tasks run close to the limit, raise the platform timeout to at least the longest expected task duration plus the grace value — otherwise the runtime kills the worker mid-drain and the interrupted tasks are re-queued by the late-ack safety net.

Job parallelism cap#

ABRQ_JOB_MAX_PARALLELISM_CAP (default 0, meaning no cap) is a deployment-wide ceiling on any job's configured max_parallelism. When both are set, the smaller value wins. Use it to stop a single wide job from saturating every worker slot in a small deployment.