Architecture overview

Components at a glance#

Abrq DIP is a small set of long-running services around two data stores:

All four backend roles (API, worker, beat, migrator) run from one backend container image; an entrypoint argument selects the process kind. See Components for per-component detail and Background processing for the worker and scheduler model.

Control path#

The control path is how operators and the platform's own metadata move:

• The browser loads the SPA from the frontend and calls the API. On Kubernetes, the ingress routes /api to the API service and everything else to the frontend; in Docker Compose, the frontend's nginx proxies /api/, the SSE log streams, and /metrics to the backend container.
• The API reads and writes only the framework database and Redis. Connector credentials and other secrets are encrypted with the master key before they touch the database, and are decrypted only at the moment a connection is opened.
• With runs mode set to celery (the default in the shipped deployment configurations), the API never executes pipeline work itself — it enqueues runs onto Redis queues for the workers.

Data path#

Pipeline data does not pass through the frontend. The executing process — a Celery worker, or the API process itself when runs mode is sync — connects directly to the configured source and destination systems and moves data between them:

• CDC changes land in mirror tables on a relational destination.
• Stream batches are staged durably, then fanned out to each configured destination independently; a batch is released only after every enabled destination has durably accepted it (or dead-lettered it).
• File-ingestion and email-feed payloads load into relational destination tables.
• SQL tasks read from and write to the engines bound to their environment.

The framework database keeps metadata, run history, and logs; ingested data lands in the destinations you configure. The separate datastore database is a bundled landing area used for CDC mirror tables and file-ingestion landings.

From source to destination#

A typical scheduled run flows like this:

1. Trigger. The Beat scheduler fires a periodic tick (a job schedule, a CDC sync, an inbox or file poll), or a user starts a run from the UI.
2. Enqueue. With ABRQ_RUNS_MODE=celery, the API enqueues the run onto its family-specific Redis queue; with sync, it executes inline.
3. Execute. A worker picks the task up and opens connections to the source and destination using the stored connector definitions.
4. Apply policy. The observed source schema is compared against the schema registry, and any drift is handled by the pipeline's policy — Reject, Evolve, or Rescue. See Schema drift policies.
5. Write. Rows are written to the destination — a mirror table, a warehouse table, a Kafka topic, or objects in a bucket.
6. Record. Run status, logs, and metrics are recorded in the framework database; notifications and alert rules fire according to configuration.

What is stored where#