Skip to content
ABRQ DATADocs Abrq DIP · latest
Product page Request a trial
On this page

What is Abrq DIP?

What Abrq DIP is#

Abrq DIP (Data Integration Platform) is a self-hosted platform for moving and transforming data. It runs entirely on infrastructure you control — a Kubernetes cluster, a single Linux host with Docker Compose, or a fully air-gapped server — and needs no internet egress to operate.

The platform has two halves:

  • Ingestion — continuously bring data in from operational databases, HTTP APIs, files in object storage or on FTP servers, and email inboxes.
  • Transformation — shape that data with SQL pipelines organised into projects, tasks, and jobs, promoted through environments on the way to production.

The web UI is the only user surface. Everything you configure is stored in the platform's own PostgreSQL metadata database (the framework database), with credentials encrypted at rest.

Ingestion families#

  • Change data capture (CDC) — replicates row-level changes from operational databases into mirror tables. Capture is timestamp-based (PostgreSQL, MySQL) or transaction-log-based (PostgreSQL logical decoding, MySQL binlog, SQL Server CDC). See CDC.
  • HTTP streams — ingest records from external HTTP APIs and deliver them to one or more destinations at once: Kafka topics, relational tables, and object storage, with at-least-once delivery, per-destination retries, circuit breakers, and a replayable dead-letter queue. See Streams.
  • File ingestion — picks up files from S3-compatible object storage or FTP/FTPS servers and loads them into relational destinations. See File ingestions.
  • Email feeds — polls IMAP inboxes and loads message attachments into destination tables. See Email feeds.

Note. Streams and email feeds are licensed features (the streams and email entitlements). See Entitlements.

Transform surfaces#

  • SQL pipelines — a project contains SQL tasks (extract-load, transform, or table copy), each scoped to an environment such as dev or prod. Tasks support the load modes full refresh, append, upsert, incremental, and historical upsert (SCD2), and their execution order is derived from the SQL references between them. See SQL pipelines.
  • Jobs — group tasks and run them on a cadence; scheduling lives on jobs, with per-schedule windows and run orchestration. See Jobs.
  • DW generator — generates data-warehouse models targeting PostgreSQL, MySQL, SingleStore, Snowflake, or SQL Server, assisted by an AI model provider that you configure. See DW generator.

Operate surfaces#

  • An operations dashboard, a cross-pipeline executions view, and per-run logs for every pipeline family.
  • Lineage and a data catalog over the configured pipelines.
  • Notifications and alert rules across webhook, email, Slack, Discord, Teams, PagerDuty, and Opsgenie channels.
  • A hash-chained audit log with optional SIEM forwarding, verified daily for tampering.
  • Prometheus metrics at /metrics and a combined health endpoint at /health.

Deployment topologies#

Topology When to use it Guide
Kubernetes with Helm Primary production topology; bundled PostgreSQL and Redis subcharts, or bring your own Kubernetes deployment
Docker Compose Single Linux host — evaluation, development, or small production installs Docker Compose deployment
Air-gapped bundle Offline environments; a versioned tarball ships every container image for docker load, zero egress Air-gapped installation

What Abrq DIP is not#

  • Not SaaS. There is no hosted service and no vendor-side control plane. You deploy and operate every component yourself, and your data never leaves your network.
  • It ships no AI model. AI-assisted features call an AI model provider that you configure yourself; the platform bundles no model and makes no AI calls unless a provider is configured.
  • No telemetry, no phone-home. The platform sends no usage analytics and performs no license heartbeat — license verification is a local file read, and the whole product runs with zero internet egress.

Part of the Abrq Data portfolio#

Abrq DIP is one of the Abrq Data products. The product overview for evaluators lives at Abrq DIP on abrqdata.com; this documentation covers deploying, configuring, and operating the platform itself. For how the pieces fit together technically, continue with the architecture overview.