Upgrade & Rollback

This page walks the upgrade path for the Compose/release-bundle deployment (the shape the vendor's UPGRADE.md ships for), then covers rollback, license renewals, and the backup essentials every path shares. Read the whole page before starting: upgrades are one-way most of the time — the rollback story works, but it is more operationally involved.

Release classes#

Vendor releases come with release notes. Check the class before applying:

Pre-upgrade checks#

Confirm the running version and that no runs are in flight:

docker compose exec backend cat /app/VERSION 2>/dev/null \
  || cat VERSION  # the file in the deployment directory
docker compose exec framework_db psql -U abrq_dip -d abrq_dip_framework \
  -c "SELECT
        (SELECT count(*) FROM etl_task_runs WHERE status = 'running') AS etl_running,
        (SELECT count(*) FROM cdc_runs      WHERE status = 'running') AS cdc_running;"

If either count is above zero, wait for the runs to finish or cancel them through the UI first. A mid-upgrade interruption can leave runs stuck in running until the next reaper pass (default 5 minute grace).

Take a backup — always#

No exceptions, even for patch releases.

docker compose exec backend uv run python scripts/backup_metadata.py \
  dump --out /var/abrq-dip/pre-upgrade-$(date +%Y%m%d-%H%M%S).json

docker compose cp backend:/var/abrq-dip/pre-upgrade-*.json ./

Stash the file outside the deployment directory — ideally on a different disk or host. It is small (typically a few MB).

Stop the stack — without -v#

docker compose down

This stops everything but preserves the named volumes (Postgres data, Redis AOF, beat schedule state).

Warning. Never pass -v here — it wipes those volumes, including your database.

Verify and stage the new release#

shasum -a 256 abrq-dip-<NEW_VERSION>.tar.gz

A mismatch against the vendor-sent .sha256 means corruption in transit — stop and request a re-send. Extract to a sibling directory (do not overwrite the running deployment):

tar -xzf abrq-dip-<NEW_VERSION>.tar.gz

Carry your state forward#

The .env and the license are customer state that must not change just because the version did:

cp abrq-dip-<OLD_VERSION>/.env       abrq-dip-<NEW_VERSION>/.env
cp abrq-dip-<OLD_VERSION>/license/license.json \
   abrq-dip-<NEW_VERSION>/license/license.json

Then update ABRQ_VERSION in the carried-forward .env to the new version (the new release directory's VERSION file has the exact value):

NEW_VERSION=$(cat abrq-dip-<NEW_VERSION>/VERSION)
sed -i "s/^ABRQ_VERSION=.*/ABRQ_VERSION=${NEW_VERSION}/" \
       abrq-dip-<NEW_VERSION>/.env
cd abrq-dip-<NEW_VERSION>/

Load the new images and start#

docker load -i images/abrq-dip-backend.tar
docker load -i images/abrq-dip-frontend.tar
# postgres + redis are usually unchanged across releases; loading is a
# fast no-op when they are.
docker load -i images/postgres-15-alpine.tar
docker load -i images/redis-7-alpine.tar

docker compose up -d

The migrator service runs once and applies any new migrations before the backend starts. Watch it and confirm it exited cleanly:

docker compose logs -f migrator
docker compose ps migrator
# State should be "exited (0)"

Then verify the backend:

docker compose logs --tail 50 backend
curl -fsS http://localhost:8080/api/v1/health | jq

Verify the upgrade#

In the web UI, confirm you can log in with your existing admin password, existing data connectors still appear, and recent run history is intact. Then verify the tamper-evident audit chain survived the migration — an offline-friendly integrity check that needs no browser:

docker compose exec backend uv run python scripts/verify_audit_chain.py
# Exit 0 = chain intact; 1 = a break was found (report the row id to the
# vendor before continuing); 2 = could-not-verify (e.g. DB unreachable).

A broken chain after an upgrade is not expected — migrations never rewrite audit history. If the check exits non-zero, stop and use the rollback paths below; do not clear the finding.

Rollback#

You took a backup. Recovery has two shapes.

Image-only revert: the new backend will not start, schema is intact#

The easy case — the new image has a bug, the old image's schema works. Do not downgrade the schema; just run the old image again:

cd ../abrq-dip-<OLD_VERSION>/
# Restore the .env that lived here before you copied it forward.
docker compose up -d

Tell the vendor what went wrong so the next release carries the fix.

Schema corruption: migration partially applied#

The framework DB is in a state the migration tooling does not recognise. Restore from the pre-upgrade backup:

# 1. Stop everything
docker compose down

# 2. Wipe the framework DB volume (DESTRUCTIVE)
docker volume rm abrq-dip-<NEW_VERSION>_framework_db_data

# 3. Start fresh — the migrator runs against an empty DB and applies
#    every migration cleanly through to the new head.
docker compose up -d framework_db migrator
docker compose ps migrator  # confirm exited (0)

# 4. Restore metadata from the pre-upgrade backup.
docker compose up -d backend
docker compose cp pre-upgrade-<TIMESTAMP>.json backend:/tmp/restore.json
docker compose exec backend uv run python scripts/backup_metadata.py \
  restore --in /tmp/restore.json --force

# 5. Bring the rest of the stack up.
docker compose up -d

Warning. restore --force truncates each table before inserting from the backup. Without --force the restore aborts if any table is non-empty — the right default against an accidental restore, the wrong one during a deliberate recovery. Be certain you are on the host and deployment you think you are.

License renewals#

Roughly 30 days before your license's expires_at, the backend logs a license_expiry_approaching warning on every startup (and the UI shows a non-dismissible banner as expiry nears). Get a renewed license.json from the vendor, replace the file, and restart the backend processes — no image change needed:

cp /path/to/new/license.json license/license.json
docker compose restart backend worker beat

Occasionally the vendor rotates the license-signing keypair. That arrives as a normal release upgrade (new embedded public key in the backend image) plus a new license.json signed with the new key — apply the upgrade exactly as above with both in hand. Upgrading the image without the matching new license fails startup with a signature error; keep the old release directory until the new one boots cleanly. More: installing a license and expiry and clock behaviour.

Kubernetes upgrades#

On the Helm path the same discipline applies — back up first, verify after:

• Take a backup_metadata.py dump (exec into an api pod) and copy it off the cluster before upgrading.
• helm upgrade with the new image tags. Migrations run automatically: the advisory-lock init container on each new pod applies alembic upgrade head exactly once while replicas of the old ReplicaSet keep serving. The optional pre-upgrade hook Job stays off by default (see Kubernetes deployment).
• Image-only rollback is helm rollback — safe when no migration was applied, or when the old code runs against the new schema. Schema corruption follows the same restore shape as Compose: fresh schema via the migrator, then backup_metadata.py restore --force.

Backup and restore essentials#

• backup_metadata.py dump exports the platform metadata — connectors, projects, tasks, jobs, schedules, users, and the encrypted secrets — as a Fernet-encrypted envelope (always encrypted; restoring it needs the same ABRQ_MASTER_KEY, one more reason that key must be backed up separately).
• The auto-backup feature (ADR 0040) runs a daily scheduled dump and an on-enable/run-now copy, surfaced in Settings → License & Backup. Where those files land — and whether they are durable — depends on your deployment: a host bind-mount on the Compose path, an RWX PVC on the Helm path (ok_ephemeral status when no durable storage is mounted), and container-local (not durable) in the shipped release bundle.
• Backups cover metadata, not your data warehouse contents: source and destination databases are yours and are backed up by your own DBA tooling.

Full operational detail: backup and restore.