NIP: Prometheus scrape target down

Alert: NipPrometheusTargetDown · Severity: Warning · Job: any nip-* except nip-app-metrics

What this means

Prometheus on monitoring-prod cannot scrape a NIP exporter target for 5+ minutes. Labels show which job and droplet.

Business impact

• Monitoring blind spot for that metric type — not always a transfer outage.
• Map job → impact:

Job	If down, you lose
nip-node-exporters	Host CPU/RAM/disk
nip-cadvisors	Per-container metrics, restart/OOM alerts
nip-postgres-exporters	DB connection metrics
nip-redis-exporters	Redis metrics
nip-blackbox-health	External /health visibility

Confirm

Grafana → Explore → Prometheus:

up{job=~"nip-.*", droplet="<from alert>"}

Or: http://127.0.0.1:9091/targets → filter by job.

Fix

1. SSH to the app server (or monitoring-prod for archive metrics).
2. cd /opt/monitoring-agents && docker compose ps
3. Start the missing exporter: docker compose up -d <service>
4. Verify locally: curl localhost:9100/metrics (node) or :8080/metrics (cadvisor).
5. Wait 2–5 min for scrape recovery.

Escalate if

• Target down because host unreachable (Tailscale/VPC).
• Multiple jobs down on same droplet — likely server or docker daemon issue.

Related runbooks

Use the specific runbook for the job (e.g. node-exporter down).