Skip to content

NIP: Spaces archive failed

Alerts: NipSpacesArchiveFailed, NipSpacesArchiveMetricsMissing · Severity: Critical / Warning · Host: monitoring-prod

What this means

The nightly job nip-spaces-archive-export.sh (cron 02:00 UTC) exports yesterday's Postgres audit rows + Loki logs to DO Spaces bucket pup-nip-archive-prod. This alert means the last run failed or Prometheus is not scraping archive metrics.

Business impact

  • 7-year cold retention gap (CTO §5.4) — compliance / dispute evidence may be missing for that day.
  • Hot Loki (90d) and Postgres still hold data; this is not an immediate transfer outage.
  • Must be fixed before the next nightly run.

Confirm

On monitoring-prod:

cat /var/lib/node_exporter/textfile_collector/nip_archive.prom
tail -100 /var/log/nip-spaces-archive.log

cd /opt/monitoring-staging
set -a && . /etc/pup/nip-spaces-archive.env && set +a
./scripts/spaces-archive/nip-spaces-archive-alerts-verify.sh
Metric Good Bad
nip_archive_last_run_success 1 0
nip_archive_last_error_info absent present

Manual re-run:

set -a && . /etc/pup/nip-spaces-archive.env && set +a
./scripts/spaces-archive/nip-spaces-archive-export.sh --date YYYY-MM-DD

Common causes & fixes

Cause Fix
Bad Spaces credentials Fix /etc/pup/nip-spaces-archive.env; bash -n for syntax
Wrong endpoint (bucket. URL) Use https://lon1.digitaloceanspaces.com
Postgres unreachable Check ARCHIVE_PG_* vars; test psql from monitoring-prod
Loki query error Fix LOKI_QUERY in env (no extra })
node-exporter-monitoring down docker compose up -d node-exporter-monitoring; reload Prometheus
Cron not running Check /etc/cron.d/nip-spaces-archive

Verify recovery

aws s3 ls s3://pup-nip-archive-prod/nip/transactions/ --recursive \
  --endpoint-url https://lon1.digitaloceanspaces.com | tail

./scripts/spaces-archive/nip-spaces-archive-alerts-verify.sh

Escalate if

  • Export fails twice in a row with unclear error in log.
  • Bucket access denied — may need admin key for lifecycle, not archival key.