Skip to content

NIP: all workers busy

Alert: NipAllWorkersBusy · Severity: Critical (pages) · Requires app /metrics

Status: not live yet. This alert depends on django-prometheus (or equivalent) being shipped in the app so worker-pool state is exposed. Until then, this rule exists in nip-alerts.yml but has nothing to scrape.

What this means

All application worker threads/processes (e.g. gthread workers) are occupied and the app can no longer accept new requests promptly. This is the exact failure mode behind the July 2026 incident, where all gthread workers wedged and transfers stopped despite the container technically staying "Up."

Business impact

  • Transfers queue or fail while appearing "up" at the infrastructure level — this is why infra-only monitoring (CPU/memory/container-up) missed the July 2026 incident. This alert is the direct fix for that blind spot.

Servers

Server App
BaaSey (inward) inward_dev-web-1
EasyPay (outward) <TODO>

Confirm

  • Grafana: worker pool utilization panel (once /metrics ships) — worker_busy_count / worker_total_count
  • App-level: check request queue depth / response latency spike with container CPU/memory looking normal (that mismatch is the signature of this failure mode)

Fix

  1. Confirm workers are actually wedged, not just under legitimate high load — check request latency trend, not just worker count.
  2. If wedged: restart the app container/process to clear stuck workers. docker restart <container> (this is a real-traffic-impacting action — same access-control caution as web-container-down.md).
  3. After restart, watch worker utilization return to normal and confirm transfer request rate recovers.
  4. File this as an incident even if resolved quickly — repeated worker wedging needs root-cause work (thread pool sizing, a blocking call, a slow dependency), not just repeated restarts.

Escalate if

  • Workers wedge again shortly after restart (root cause not fixed).
  • You can't tell whether it's "wedged" vs "genuinely busy" — this needs someone who understands the app's worker model, not just infra.
  • Grafana → NIP → business/app metrics dashboard (once /metrics ships)

Blocking dependency

This runbook is not fully actionable until app /metrics is deployed on BaaSey + EasyPay. Ship that first — this is the highest business-value gap called out in the gap assessment.