Data Connectors

Purpose#

The Data Connectors page is where every connection the platform uses is registered once and then reused everywhere. Its subtitle states the intent plainly: "Read-only relational databases that Abrq DIP pulls CDC from. Passwords are encrypted at rest (Fernet) and never returned by the API."

Register a source here and CDC tables, file ingestions, exports, and pipelines all reference it by name. Rotating a password is one edit in one place — never an edit per pipeline.

Before you begin#

• Read — listing connectors and opening one needs any authenticated role.
• Create, edit, delete, and test — all require the editor global role or above.
• Network reachability from the Abrq DIP host to the target: the test runs server-side, so your laptop's access proves nothing.
• Credentials for the target. For source databases, prepare a read-only account before you start; see CDC strategies for the grants each strategy needs.

Warning. Passwords are encrypted at rest with Fernet and are never returned by the API — not to the UI, not to an export, not to an admin. Store them in your own secret manager as well; Abrq DIP is not a place to look a password up later.

Supported engines#

Step 1 of the wizard is a searchable card picker. The label and default port are exactly:

Create a connector#

The wizard has two steps.

Step 1 — Connect data source#

Pick the engine. The picker is searchable, so type post or s3 rather than scanning the grid. Your choice sets the default port and, more importantly, relabels the fields in step 2 so an object store does not ask you for a "database".

Step 2 — Fill in the details#

Step 2 has four sections:

1. Identity — the human name this connector is known by everywhere else.
2. Connection — host, port, and database (or their per-engine equivalents, below).
3. Authentication — username and password (or key and secret).
4. Test connection — run it, read the result, then create.

Step 3 — Test, then create#

Select Test connection. For every engine except Kafka, a passing test is a hard gate on creation: until one succeeds the create button stays disabled and the form says "Run a successful test to enable "Create connector"."

A pass reads "✓ Connection succeeded" and reports the server version. Read the warnings underneath before moving on — a test can pass and still be telling you something important.

Field reference#

Common fields, in the order the form presents them:

Per-engine field relabelling#

The same underlying fields carry different labels — and different meanings — per engine:

Notes that matter in practice:

• S3 adds a Secure (HTTPS) toggle that swaps the port between 443 and 9000 — the second is the usual MinIO plaintext port — and a Buckets list so you can see what the credentials can actually reach.
• Google Cloud Storage wants the whole service-account JSON key pasted into the password field, braces and all, not just the private-key portion.
• FTPS uses port 21: "21 for explicit FTPS (AUTH TLS upgrade). Implicit FTPS on 990 is not currently supported."
• Kafka takes comma-separated bootstrap servers in the host field, for example 192.0.2.10:9092,192.0.2.11:9092, and uses the database field for the SASL mechanism with username and password as the SASL credentials.

Read the test warnings#

Write privileges on a source#

Any successful test against an account that can write reports:

Note: this user has write privileges. Use a read-only role for source databases.

Nothing is blocked — but a source connector with write rights is a standing risk. Swap in a read-only role before you build pipelines on it.

MySQL binlog readiness#

For MySQL, the test additionally probes the binary-log settings that the log_mysql CDC strategy depends on, and reports what would break:

log_bin is OFF — log_mysql CDC strategy will not work
binlog_format is 'STATEMENT'; log_mysql CDC requires 'ROW'
binlog_row_metadata is 'MINIMAL'; log_mysql CDC needs 'FULL' so column names land in the binlog

Fix these on the server before creating CDC tables against it — the connector will still save, but the first sync will fail. The full prerequisite list is in CDC strategies.

Tip. Test results are persisted. Every run appears in Executions with kind conn_test, so "it worked yesterday" is an answerable question rather than an argument.

Edit a connector#

Open the connector and change what you need. The password field behaves differently from the rest:

• Its hint reads "Leave blank to keep the existing one. Type a new value to rotate."
• Its placeholder is (unchanged).
• The current value is never shown, because the API never returns it.

To rotate a credential, type the new secret, re-test, and save. Leaving the field untouched keeps the stored secret exactly as it is.

Browse the schema#

The row action Schema opens a live browser over the connector's tables and columns — it queries the source at that moment rather than replaying a cached snapshot, so it doubles as a permissions check.

If introspection fails you get an error, never a plausible-looking empty list. An empty result means the account genuinely sees no tables; that is a grant problem on the source, not a display quirk.

Delete a connector#

Deleting is a soft delete, and it is designed not to fail:

1. Select Delete on the connector.
2. Type the connector's name to confirm.
3. Confirm.

The row disappears from the list, and dependents keep working until their next sync. There is no "connector is in use" error to fight, because deletion never blocks on references. That trade is deliberate: you can retire a connector immediately, and the pipelines that still reference it surface the problem on their own next run rather than blocking your clean-up.

Warning. Because dependents are not blocked, check Data Lineage or the CDC list for references before you delete. A soft delete is quiet: the first sign of a mistake is usually a failed sync, not a refused delete.

Failure modes#