NIP runbooks
Operational playbooks for alerts routed to #nip-ops (platform=nip-bridge).
Wiki base URL: https://wiki.pup.finance/runbooks/nip/
Source of truth: docs/runbooks/nip/ in the monitoring-staging git repo — edit there, run mkdocs build, deploy site/ to the wiki server.
Servers
| Droplet | Role | Tailscale | Notes |
|---|---|---|---|
baasey-nip-server |
NIP inward | 100.94.107.110 |
/home/inward_dev, port 8009 |
easypay-nip-integration-prod |
NIP outward | 100.65.109.7 |
Outward bridge |
monitoring-prod |
Prometheus, Grafana, Loki, archive | 100.115.57.112 |
/opt/monitoring-staging |
Runbook index
Scrape & reachability
| Alert | Severity | Runbook |
|---|---|---|
NipNodeExporterDown |
Critical | node-exporter down |
NipCadvisorDown |
Critical | cAdvisor down |
NipHealthProbeDown |
Critical | /health probe down |
NipAppMetricsDown |
Warning | app /metrics down |
NipPrometheusTargetDown |
Warning | scrape target down |
Host (node_exporter)
| Alert | Severity | Runbook |
|---|---|---|
NipHostCPUHigh |
Warning | host CPU high |
NipHostMemoryHigh |
Warning | host memory high |
NipHostDiskWarning |
Warning | host disk warning |
NipHostDiskCritical |
Critical | host disk critical |
Containers (cAdvisor)
| Alert | Severity | Runbook |
|---|---|---|
NipWebContainerDown |
Critical | web container down |
NipContainerOOMKilled |
Critical | container OOM |
NipContainerRestartLoop |
Warning | container restart loop |
Postgres / Redis
| Alert | Runbook |
|---|---|
NipPostgresExporterDown |
postgres exporter down |
NipPostgresConnectionsHigh |
postgres connections high |
NipPostgresSlowQueries |
postgres slow queries |
NipPostgresDeadlocks |
postgres deadlocks |
NipRedisExporterDown |
redis exporter down |
NipRedisMemoryHigh |
redis memory high |
NipRedisEvictions |
redis evictions |
NipRedisHitRateLow |
redis hit rate low |
Business metrics (requires app /metrics)
| Alert | Runbook |
|---|---|
NipAllWorkersBusy |
all workers busy |
NipTransferFailureSpike |
transfer failure spike |
NipDependencySlow |
dependency slow |
Spaces archival (monitoring-prod)
| Alert | Runbook |
|---|---|
NipSpacesArchiveFailed |
spaces archive failed |
NipSpacesArchiveStale |
spaces archive stale |
NipSpacesArchiveMetricsMissing |
spaces archive failed |
After editing runbooks
- Commit changes in
monitoring-staging. - On wiki host:
mkdocs build→ publishsite/towiki.pup.finance. - On monitoring-prod:
curl -X POST http://127.0.0.1:9091/-/reload(if alert rules changed).