Skip to content

NIP: node_exporter down

Alert: NipNodeExporterDown · Severity: Critical (pages) · Job: nip-node-exporters

What this means

Prometheus can no longer scrape host-level metrics (CPU, memory, disk) from this server. This is a monitoring blind spot, not necessarily a transfer outage — node-exporter is a separate container from the transfer/web containers.

Business impact

  • Transfers may still be completing normally.
  • You are blind to CPU/memory/disk conditions on this box until resolved.
  • If this fires at the same time as other alerts, treat it as reduced visibility during a possible real incident — do not assume "safe to ignore."

Servers

Server Role Tailscale IP
BaaSey (inward) baasey-nip-server 100.94.107.110
EasyPay (outward) easypay-nip-integration-prod 100.65.109.7

Confirm

  • Prometheus targets: nip-node-exporters job → target for the affected droplet label is DOWN
  • Grafana Explore query: up{job="nip-node-exporters", droplet="<server>"}
  • SSH check: docker ps | grep node-exporter

Fix

  1. SSH to the affected server over Tailscale (use the monitoring user, not root).
  2. cd /opt/monitoring-agents && docker ps -a | grep node-exporter
  3. If stopped: docker start node-exporter
  4. If missing entirely: docker compose up -d node-exporter
  5. Verify locally: curl -s localhost:9100/metrics | head -3
  6. Wait ~2 minutes for Prometheus's next scrape — alert should auto-resolve in Slack.

Escalate if

  • Server unreachable over Tailscale (possible network/host-level problem, escalate immediately — this is bigger than a dead exporter).
  • Container won't start or keeps exiting after 2 attempts.
  • Metrics still missing after 10 minutes.
  • Grafana → NIP → NIP Servers — Infrastructure

Notes

  • This is the alert you've already proven end-to-end in staging — safe to test by briefly stopping node-exporter (no impact on transfers).