Skip to content

NIP: web container down

Alert: NipWebContainerDown · Severity: Critical (pages) · Source: cAdvisor

What this means

Prometheus no longer sees a web container running on this NIP app server. The inward/outward HTTP service that handles NIP transfers is missing from the host — not just slow or unhealthy.

Business impact

  • New NIP transfers cannot be processed on the affected server until the web container is back.
  • Existing in-flight requests may fail or hang depending on how the container stopped.
  • This is a customer-facing outage for the direction served by that box (inward on BaaSey, outward on EasyPay).

Servers

Server Droplet label Web container Health URL
BaaSey (inward) baasey-nip-server inward_dev-web-1 http://10.16.0.5:8009/health/
EasyPay (outward) easypay-nip-integration-prod transfer_bridge-web (verify name) Check compose on host

Confirm (2 minutes)

From monitoring-prod:

# cAdvisor should show zero web containers
curl -sG 'http://127.0.0.1:9091/api/v1/query' \
  --data-urlencode 'query=count(container_memory_usage_bytes{job="nip-cadvisors",droplet="baasey-nip-server",container=~".*web.*"})'

On the app server (SSH via Tailscale):

cd /home/inward_dev   # BaaSey inward
docker ps -a | grep -E 'web|NAME'
docker compose -f docker-compose.deploy.yml ps web
curl -fsS http://10.16.0.5:8009/health/ || echo "health FAILED"

Fix

  1. SSH to the affected server (Tailscale IP from alert labels).
  2. Check why web is gone: bash docker logs inward_dev-web-1 --tail 100 docker inspect inward_dev-web-1 --format 'Status={{.State.Status}} ExitCode={{.State.ExitCode}} OOM={{.State.OOMKilled}}'
  3. Start web: bash cd /home/inward_dev docker compose -f docker-compose.deploy.yml up -d web
  4. Wait for healthy (healthcheck ~40s start period + up to 90s): bash watch -n 5 'docker inspect inward_dev-web-1 --format "{{.State.Health.Status}}"'
  5. Verify externally: bash curl -fsS http://10.16.0.5:8009/health/
  6. Confirm alert clears in Slack within ~2–5 minutes.

If web keeps crashing

Exit code Likely cause Next step
137 OOM killed See container OOM runbook; check memory limits
1 App crash on boot Read docker logs; escalate to app team
0 (restarting loop) Dependency down (DB/Redis) Check db, redis containers; see restart-loop runbook

Escalate if

  • Web will not stay up after two restart attempts.
  • /health/ returns 200 but NIBSS/partners still report failures (may be nginx-proxy or network).
  • You see stuck CBS_SENT rows in Postgres after recovery — notify reconciliation owner.
  • Grafana → NIP Service Health — probe status, stuck transactions
  • Grafana → NIP Servers — Infrastructure — container list

Prevention

  • restart: unless-stopped on all compose services
  • autoheal sidecar restarts unhealthy web (not exited containers) — see compose hardening FO #23–#24