Skip to content

NIP: container OOM-killed

Alert: NipContainerOOMKilled · Severity: Critical (pages)

What this means

The kernel killed a container because it exceeded its memory limit (or the host ran out of memory). If this is a transfer-handling container, requests in flight at the moment of the kill are dropped.

Business impact

  • Any transfer mid-flight through the killed container may be lost or left in an inconsistent state — this needs reconciliation, not just "restart and move on."

Servers & containers

Server Container(s) at risk
BaaSey (inward) inward_dev-web-1, inward_dev-schedule_task-1, inward_dev-always_on_task-1
EasyPay (outward) <TODO: container names>

Confirm

  • docker ps -a — look for Exited (137) status (137 = OOM kill signature)
  • dmesg | grep -i "killed process" or journalctl -k | grep -i oom
  • Grafana: memory panel for the container showing a spike right up to the limit before the drop to zero

Fix

  1. Confirm it's actually OOM (exit code 137 / kernel log), not a normal crash.
  2. Restart the container: docker start <container> or docker compose up -d <container>.
  3. Before closing this out, check whether any transfers were in flight at the kill timestamp — cross-reference app logs / transfer status table for that window. This is the step that's easy to skip and shouldn't be.
  4. If this is a recurring container, consider whether its memory limit is too low for real traffic, or whether there's a leak — a repeat OOM on the same container within days is a code/config issue, not a one-off.

Escalate if

  • Any transfer appears to have been dropped or left in an ambiguous state — this goes to whoever owns transfer reconciliation, not just infra.
  • The same container OOMs more than once in a short window (memory leak candidate).
  • Grafana → NIP → NIP Servers — Infrastructure (per-container memory panel)