Catalog

Purpose#

The catalog is where you write down what a table means and who is responsible for it.

Lineage can tell you that warehouse.fact_orders is produced by an ETL task reading three source tables. It cannot tell you that the table excludes cancelled orders, that finance owns it, or that it contains personal data. That human knowledge is what the catalog holds.

The catalog lives at /catalog.

Note. The catalog is a deliberate annotation layer, not an automatic inventory. It does not discover your tables. Nothing appears in it until somebody writes an entry. Read What the catalog is not before planning around it.

Before you begin#

• Role.

Creating, editing and deleting entries are audited. The catalog is a statement of record, and changes to it are traceable.

What the catalog indexes#

Exactly one kind of thing: a dataset, identified by its dataset key.

The dataset key is a table's fully-qualified name — the string you would write in SQL. It is unique across the catalog and is the entry's identity.

An entry carries:

Entries also record when they were created and last updated.

Tip. Tag datasets containing personal data with pii. The catalog recognises that tag and badges those entries, which turns "which tables hold personal data?" into a one-click answer.

Add or edit an entry#

1. Go to /catalog.
2. Choose Add — or open an existing entry and choose Edit.
3. Enter the dataset key exactly as the table is qualified in your warehouse.
4. Fill in title, description, owner and tags.
5. Save.

Saving against an existing dataset key updates that entry rather than creating a duplicate — the key is the identity, so there is no way to end up with two entries for one table.

Get the dataset key right#

Because entries are matched by string, consistency in how you qualify names is the difference between a catalog that works and one that half-works. Pick a convention — schema-qualified, lower case — and apply it everywhere.

Tip. Copy the fully-qualified name from a destination node in Lineage rather than typing it. The names there are the ones your pipelines actually use.

Tag suggestions for personal data#

When you are editing an entry, the catalog can suggest which of a list of column names look like personal data, and what category each falls into. It is a naming-pattern heuristic that runs on the column names you give it — it never reads your data and never connects to a database.

Treat it as a prompt for a human decision, not a classification. It will miss a personal-data column called ref_9, and it will flag a customer_name column in a lookup table of fictional test values.

Search and browse#

The search box matches a single term, case-insensitively, against every text field of an entry: dataset key, title, description, owner, and tags. One box, no syntax.

That breadth is what makes it useful — searching finance finds datasets finance owns, datasets with a finance tag, and datasets whose description mentions finance.

Results are ordered by dataset key, so entries from the same schema group naturally.

Once results are on screen you can refine them further:

• Tag chips — click a tag to narrow to datasets carrying it.
• Owner facets — narrow to one owner.
• Sorting — reorder the visible results.

A summary strip shows how many datasets are catalogued, how many have an owner, how many are documented, how many are tagged pii, and how many distinct tags are in use. Read it as a coverage report: it tells you where the catalog is thin.

Each result is a card with the dataset key (copyable in one click), title, description, owner, clickable tags, a personal-data badge where applicable, and when it was last updated.

What the catalog is not#

Being clear about this saves a lot of misplaced expectation.

• There is no discovery, scan or sync. No background job populates the catalog from your connectors, CDC mirrors or ETL tasks. Every entry is written by a person (or by your own automation calling the API).
• Entries are not linked to connectors, CDC mirrors, or ETL tasks. The only connection to the rest of the platform is the dataset key string. There is no reference between a catalog entry and a pipeline, and deleting a pipeline does not touch its catalog entry — nor does deleting a catalog entry affect a pipeline.
• A catalog entry does not mean the table exists. Nothing validates the dataset key against a real database. An entry can outlive the table it describes.
• It is not column-level. Entries describe datasets, not columns. Column detail belongs in the description.
• It is not access control. Tagging a dataset pii documents it; it does not restrict it. For that, see Data masking.
• Search has no field filters and no cursor paging. One search term across all fields, then client-side refinement, with a result cap.

Warning. Because nothing links entries to real tables, a catalog can drift silently: pipelines get retired, tables get renamed, and the entries stay. Review the catalog on the same cadence you review the pipelines.

A workable practice#

The catalog rewards a narrow, consistent habit far more than a big-bang documentation exercise:

1. Catalog destinations, not everything. The tables people consume are the ones that need explaining. Intermediate staging tables usually do not.
2. Write the entry when you build the pipeline, while you still remember the caveats. See SQL pipelines.
3. Always set an owner. An undocumented table with a known owner is more useful than a documented table nobody answers for. The coverage strip makes missing owners visible.
4. Put the exclusions in the description. "Excludes cancelled orders and internal test accounts" prevents more bad reports than any other sentence you can write.
5. Use few tags, consistently. Twelve well-known tags beat two hundred one-off ones. Reserve pii for its specific meaning.

Field reference#

Failure modes#

See also#

• Lineage — the automatic graph the catalog annotates
• Data masking — enforcing what a pii tag documents
• SQL pipelines — where destination tables come from
• API reference