Skip to content

NIP: host disk critical

Alert: NipHostDiskCritical · Severity: Critical (pages)

What this means

Disk usage on the host has crossed the critical threshold. Left unaddressed, this leads to a hard outage: the database can't write, logs fail, containers crash on write errors.

Business impact

  • Not an outage yet, but a near-certain outage within a short window if disk fills completely. Treat as urgent, not "watch it."

Servers

Server Role Tailscale IP
BaaSey (inward) baasey-nip-server 100.94.107.110
EasyPay (outward) easypay-nip-integration-prod 100.65.109.7

Confirm

  • SSH: df -h — identify which mount is filling up (usually /var/lib/docker)
  • docker system df — check overlay2 layer / image / volume usage
  • du -sh /var/lib/docker/containers/*/*.log | sort -h | tail -10 — check for a runaway container log

Fix

  1. Identify the largest consumer first — don't blind-prune on a live payment server.
  2. Most common cause on this stack: accumulated Docker overlay2 layers from repeated image pulls/builds.
  3. docker system prune -af --volumes=false (confirm you're not pruning volumes you need — leave --volumes off unless you've checked).
  4. If a container log file is the culprit: rotate or truncate it (truncate -s 0 <logfile>), then fix logging/rotation config so it doesn't recur (this has happened before on this infra — set up a recurring prune cron if one isn't already in place).
  5. Re-check df -h — confirm usage is back under the warning threshold, not just under critical.

Escalate if

  • Freeing obvious Docker bloat doesn't bring usage down meaningfully (may mean DB data growth, not container cruft — different problem, different owner).
  • You're unsure whether a directory is safe to delete on a production payment host.
  • Grafana → NIP → NIP Servers — Infrastructure (disk panel)

Prevention

  • A recurring docker system prune cron was previously recommended for this exact failure mode — confirm it's actually deployed on both BaaSey and EasyPay,