Skip to content

NIP: /health probe failing

Alert: NipHealthProbeDown · Severity: Critical (pages) · Job: nip-blackbox-health

What this means

Monitoring-prod probes http://100.94.107.110:8009/health/ (BaaSey inward) every 15 seconds via blackbox_exporter. The probe expects HTTP 200 and JSON body "status": "ok". Failure for 2+ minutes triggers this alert.

This is the same failure mode as the July 2026 incident: gunicorn workers wedged — container shows "Up" but /health/ times out or returns bad data.

Business impact

  • Inward NIP transfers likely failing or severely degraded on BaaSey.
  • Escalation emails at T+10m / T+30m are expected if unresolved.
  • Not a monitoring-only issue — treat as production incident.

Confirm

From monitoring-prod:

cd /opt/monitoring-staging
./scripts/validate-nip-health-probe.sh

curl -sG 'http://127.0.0.1:9091/api/v1/query' \
  --data-urlencode 'query=probe_success{job="nip-blackbox-health",droplet="baasey-nip-server"}'

curl -sG 'http://127.0.0.1:9091/api/v1/query' \
  --data-urlencode 'query=probe_http_status_code{job="nip-blackbox-health",droplet="baasey-nip-server"}'
probe_http_status_code Meaning
0 Timeout / connection refused — app down or network
502 / 504 nginx-proxy / upstream dead
200 but probe still 0 Body missing "status":"ok" — wedged workers

On BaaSey:

curl -i http://10.16.0.5:8009/health/
docker ps | grep web
docker inspect inward_dev-web-1 --format 'Health={{.State.Health.Status}}'
docker logs inward_dev-web-1 --tail 80

Fix

  1. Check Docker health (not just docker ps Up): bash docker inspect inward_dev-web-1 --format '{{.State.Health.Status}}'
  2. If unhealthy or hung >2 min, restart web: bash cd /home/inward_dev docker compose -f docker-compose.deploy.yml restart web
  3. Wait for healthy (~1–3 min). autoheal may also restart if healthcheck fails — check: bash docker logs inward_dev-autoheal-1 --since 10m
  4. Verify: bash curl -fsS http://10.16.0.5:8009/health/
  5. Re-run probe validation from monitoring-prod; alert should clear in ~2 min.

Escalate if

  • /health/ still fails after restart.
  • Restarts loop (see container restart loop).
  • Postgres/Redis down — fix dependencies first.
  • Suspected CBS/SSM outage with workers blocked — escalate to app + integration owners.
  • NIP Service Health — probe stat, stuck CBS_SENT table
  • Transaction Explorer — stuck/failed transfers