Retention
Two different retentions#
Abrq DIP has two retention systems that share a name and nothing else:
- Log retention — how long the platform keeps its own run logs. Environment-configured, swept automatically every day.
- Dataset retention policies — deleting your aged data rows from destination tables. Admin-defined per table, never automatic beyond what you explicitly execute.
Log retention (environment variables)#
Three windows, all env vars — restart to change:
| Variable | Default | What it governs |
|---|---|---|
ABRQ_RUN_LOG_RETENTION_DAYS |
30 |
Run logs across CDC, ETL tasks, jobs, and exports older than this are deleted by the daily sweep; 0 disables |
ABRQ_KEEP_SUCCESS_RUN_LOGS_HOURS |
24 |
Detail logs of successful runs are pruned after this many hours (failure logs stay for the full window); 0 or less disables |
ABRQ_AUDIT_LOG_RETENTION_DAYS |
365 |
Reserved — no sweep consumes this in this version. The audit trail is not purged by the retention sweeper |
The sweep is a daily background task
(abrq_dip.maintenance.purge_logs) covering the run-log tables for CDC
runs, ETL task runs, job runs, and export runs. Audit logs are
deliberately out of its scope.
The current windows are visible in the UI on the read-only Log Retention card under Settings → License & Backup — the card displays the values and points you at the deployment's environment file to change them; there is no in-app edit.
Note. Success-run detail-log pruning exists because verbose per-row logs of a run that succeeded lose their value within a day, while the run's summary row (status, counts, timings) is kept for the full retention window.
Dataset retention policies (Settings → Retention)#
Retention policies delete aged rows from destination tables you
choose — typically tables fed by CDC or ETL pipelines. Policies are
managed at Settings → Retention (/api/v1/retention-policies),
admin-only and audited.
Each policy defines:
| Field | Meaning |
|---|---|
| Name | Unique label — also the confirmation token for execution |
| Connector | The data connector hosting the target table |
| Schema (optional) and table | The table rows are deleted from |
| Date column | The timestamp or date column rows age against |
| Retention days | Rows whose date column is older than this are eligible for deletion |
| Enabled | Whether the policy is active |
Dry-run first#
POST /api/v1/retention-policies/<POLICY_ID>/dry-run (or the dry-run
button on the policy) counts the rows currently older than the
window — read-only, no deletion, safe to run any time. Always dry-run
before executing so you know what a policy will remove.
Executing a policy#
Execution is deliberately friction-heavy:
- Admin-only.
- Guarded by a confirmation token: the request must repeat the
policy's name exactly (
confirm_name); a mismatch is rejected and nothing is deleted. - The response reports the actual number of rows removed, and the execution is written to the audit log.
Warning. Execution permanently deletes rows from your destination table. There is no undo on the platform side — recovery means restoring from your own database backups. Dry-run first, check the count, then confirm.