Skip to content

NIP: app /metrics down

Alert: NipAppMetricsDown · Severity: Warning · Job: nip-app-metrics

What this means

Prometheus cannot scrape the Django/application metrics endpoint (e.g. 100.94.107.110:8009/metrics). Business dashboards and alerts (NipAllWorkersBusy, failure rate, dependency latency) have no data.

Business impact

  • Transfers may still work — this is observability gap, not necessarily outage.
  • You lose early warning for worker wedging until /health probe also fails.

Confirm

# From monitoring-prod
curl -s -o /dev/null -w '%{http_code}' http://100.94.107.110:8009/metrics

curl -sG 'http://127.0.0.1:9091/api/v1/query' \
  --data-urlencode 'query=up{job="nip-app-metrics"}'

On app server:

curl -s http://127.0.0.1:8000/metrics | head
docker logs inward_dev-web-1 --tail 50 | grep -i metric

Fix

  1. Expected until app team ships metrics — acknowledge if rollout not done.
  2. If metrics should be live:
  3. Confirm django-prometheus (or equivalent) in deployed image.
  4. Confirm URL path and port in prometheus/file_sd/nip-app-metrics.json.
  5. Restart web after deploy.
  6. Reload Prometheus: curl -X POST http://127.0.0.1:9091/-/reload

Escalate if

  • /metrics returns 401/403 — auth middleware blocking scrape; app team must allow monitoring IP or internal path.

Blocking dependency

Health dashboard business panels (traffic, workers, SSM/CBS) require this endpoint.