Lineage

Purpose#

Lineage answers two questions:

Where did this table come from? — trace a destination table back through whatever produced it.
What breaks if I change this? — trace a source system forward to everything downstream of it.

The graph is assembled from the pipelines you have already configured. There is nothing to author and nothing to maintain: if a CDC table, stream, file ingestion, email feed or ETL task exists, it is in the graph.

Lineage lives at /lineage.

Note. Read What lineage does not capture before you rely on this for an audit or an impact assessment. The scope is genuinely useful and genuinely bounded, and the bounds matter.

Before you begin#

Role. Any authenticated user can open Lineage. It is read-only and adds no write capability.
Nothing to enable. There is no indexing step, no scan to schedule, and no "build lineage" button.

What lineage captures#

Granularity: table level#

Lineage is table-level. Nodes are tables, connectors, and the producers between them. The graph tells you that a task reads sales.orders and writes warehouse.fact_orders; it does not tell you which source column fed which destination column.

Warning. There is no column-level lineage in Abrq DIP. If a compliance process needs "which column did this value come from", lineage will not answer it and you should not present it as if it does.

Where the edges come from#

Producer	Source side	Destination side
ETL task (`table_copy`)	The task's declared source schema and table.	The destination table.
ETL task (`sql_extract_load`, `sql_transform`)	Tables parsed out of the task's SQL.	The destination table.
CDC table	The mirrored source table.	The mirror's destination table.
Stream	The API endpoint, as a source node.	The destination table.
File ingestion	The source path, as a file node.	The destination table.
Email feed	The inbox, as a node.	The destination table.

For SQL-based ETL tasks the source tables are obtained by parsing the task's SQL with a SQL parser, in the dialect of the relevant connector. This is why lineage keeps up with a task you edited five minutes ago — and also why unparseable SQL degrades quietly (see below).

Multi-hop chains come for free#

A table is identified by its connector plus its normalised fully-qualified name. That means the same physical table is the same node, no matter which pipeline mentions it.

So a chain like:

source database  →  CDC mirror table  →  ETL task  →  warehouse fact table

appears as a connected path with no wiring on your part. The CDC mirror's destination and the ETL task's source are recognised as one node because they are one table.

Node and edge vocabulary#

Nodes you will see:

Node	Meaning
`connector`	A registered data connector.
`table`, `source_table`, `destination_table`	A physical table, on the read side, the write side, or both.
`source_api`	A stream's HTTP endpoint.
`file_path`	A file ingestion's source path.
`inbox`	An email feed's mailbox.
`etl_task`, `cdc`, `stream`, `file_ingestion`, `email_feed`	The producers.
`project`	The SQL Pipelines project owning a task.

Edges you will see:

Edge	Meaning
`hosts`	A connector hosts a destination table.
`reads`	A connector (or a file path) is read from.
`feeds`	A source feeds a producer.
`produces`	A producer writes a destination table.
`owns`	A project owns an ETL task.

Producer nodes carry their latest run status and a summary of that run — rows read and written, error text, timestamps — so the graph doubles as a health view. For ETL tasks the node also carries the SQL that generated the edges.

Navigate the graph#

The page has four views.

Overview#

The whole graph, filtered. Use the filters to make it legible:

Filter	Use it to
Search	Match producers and destination tables by substring. The fastest way in when you know a table name.
Project	Narrow to one SQL Pipelines project.
Connector	Narrow to one database.
Status	Show only producers whose latest run was `success`, `failed`, `running`, `queued`, `cancelled` or `skipped`.

Filter to failed and the overview becomes a map of what is currently broken and what sits downstream of it.

You can toggle between the graph and a list. The graph reads left to right, sources on the left, destinations on the right; the list is better for copying names and for large result sets.

Destination focus#

Pick a destination table and get its upstream graph — everything that contributes to it, transitively.

This is the "where did this number come from" view. Start here when a figure in a report looks wrong.

Source focus#

Pick a source system — a connector, a stream endpoint, an inbox — and get its downstream graph.

Impact analysis#

The change-management view: name a source and see everything that would be affected if it changed. Use it before dropping a column, renaming a table, retiring a connector, or migrating a database.

Node details#

Clicking any node opens a detail panel with its identity, its metadata, and — for producers — the latest run summary and, for ETL tasks, the SQL. From there you can go on to the owning resource's own page.

The views are deep-linkable: the selected view and node are carried in the URL, so a lineage link can be pasted into an incident channel.

What lineage does not capture#

This section is deliberately blunt. Knowing the edges of the map is what makes the map safe to use.

Not captured at all:

Column-level lineage. Table level only.
Data exports. Export jobs are not in the graph. A table feeding a scheduled export appears to have no consumers. Check Exports separately before retiring a table.
Jobs and schedules. Orchestration is not represented. The graph shows that task A and task B both touch a table; it does not show that a job runs A before B, or that a schedule fires either of them. Read Jobs for ordering.
BI tools, dashboards and anything outside Abrq DIP. The graph ends at the destination table. What reads that table from outside is invisible here.
Masking, retention, and warehouse-model metadata. Not lineage nodes.

Captured imperfectly — read these before trusting a result:

SQL the parser cannot read. If a task's SQL fails to parse — an unsupported construct, an engine whose dialect is not mapped — the task is marked degraded and contributes no source edges. The task still appears; it simply looks like it has no upstreams. A producer with zero inputs is a claim to verify, not a fact.
Tasks with no SQL, or a missing connector. Same outcome: no source edges.
Common table expressions. Table references are extracted from the parsed SQL without filtering out CTE names, so a WITH recent_orders AS (…) alias can appear as a table node that does not exist in any database. Treat unfamiliar single-word tables with suspicion.
Unqualified table names. Matching is on the normalised fully-qualified name. A table referenced without its schema will not join up with the fully-qualified node for the same table, so a chain can appear broken when it is not. Qualify your table names in ETL SQL and this problem disappears.
Deleted connectors. A connector that was deleted but is still referenced shows as a placeholder node with an unknown engine.
Exotic quoting. Fully-qualified-name splitting is straightforward and can misread unusual dotted-and-quoted identifiers.

Tip. The single highest-value habit for accurate lineage: write schema-qualified table names in ETL SQL. It fixes both the unqualified-name mismatch and most of the ambiguity in the parsed output.

Freshness#

The graph is rebuilt from scratch on every request. It is not cached, not persisted, and not refreshed by a background worker.

Consequences:

Lineage is always current as of the moment you loaded the page. Edit a task's SQL, reload, and the edges have moved.
There is no lineage history. You cannot ask what the graph looked like last month.
On a very large deployment the page does real work on each load. Use the project, connector and status filters rather than repeatedly loading the unfiltered overview.

Field reference#

Control	Values	Notes
View	Overview, Source focus, Destination focus, Impact analysis	Carried in the URL.
Search	free text	Substring match on producer labels and destination table names.
Project	one project	Overview only.
Connector	one connector	Overview and the focus views.
Status	`success`, `failed`, `running`, `queued`, `cancelled`, `skipped`	Latest run status of producer nodes.
Display	graph or list	Graph is layered left to right.

Failure modes#

What you see	Cause	Fix
An ETL task with no upstream nodes	Its SQL could not be parsed; the node is marked degraded.	Open the node, read its SQL, and simplify or re-dialect the query. Verify the real inputs by hand.
A table node nobody recognises	A CTE alias was extracted as a table reference.	Ignore it. Naming CTEs distinctly makes this obvious at a glance.
A chain that stops halfway	Two pipelines refer to the same table with different qualification, so they resolved to different nodes.	Use schema-qualified names in ETL SQL.
A table that appears to have no consumers	Exports, BI tools and anything outside Abrq DIP are not in the graph.	Check Exports and your BI layer before retiring the table.
No ordering information between two tasks	Orchestration is not part of lineage.	See Jobs.
A connector node labelled as unknown	The connector was deleted but is still referenced.	Repoint or retire the pipelines that reference it.
The overview is slow or unwieldy	The graph is rebuilt per request and the deployment is large.	Filter by project, connector or status; or use a focus view.