Skip to content

NIP: cAdvisor down

Alert: NipCadvisorDown · Severity: Critical (pages) · Job: nip-cadvisors

What this means

Prometheus can't scrape per-container metrics (CPU/memory/restarts per container) from this host. Container metrics visibility only — transfers are not directly affected.

Business impact

  • You lose the ability to see container-level resource usage and can't detect container OOM/restart patterns until this is fixed.
  • Downstream effect: NipContainerOOMKilled and NipContainerRestartLoop may not fire correctly while cAdvisor is down — you're doubly blind.

Servers

Server Role
BaaSey (inward) baasey-nip-server
EasyPay (outward) easypay-nip-integration-prod

Confirm

  • SSH: docker ps | grep cadvisor
  • Check bind config: CADVISOR_BIND=0.0.0.0:8080 in /opt/monitoring-agents/.env
  • Prometheus target: nip-cadvisors job → DOWN for the affected droplet

Fix

  1. SSH to the affected server.
  2. cd /opt/monitoring-agents && docker ps -a | grep cadvisor
  3. If stopped: docker start cadvisor
  4. If missing: docker compose up -d cadvisor
  5. Verify: curl -s localhost:8080/metrics | head -5
  6. Wait ~2 minutes, confirm alert clears in Slack.

Escalate if

  • cAdvisor won't stay up (repeated crash) — check host resource pressure first, this can be a symptom of a bigger host problem, not just a dead container.
  • Grafana → NIP → NIP Servers — Infrastructure