Write a message to start your investigation

↵ Enter to investigate

Actively triaging across 5 services · Last sweep 16s ago

Active investigations (2)

search, e.g. severity:critical is:resolved

INV-2026-0704-3F2A checkout-svc — 5xx error spike Critical Remediation ◇ Agent working

INV-2026-0701-9B10 ingest-worker — queue backlog growing Major Hypotheses You

Issues (5)

payment-gateway · Retry storm detected on payment-gateway, correlates with checkout-svc 5xx spike related ▸ INV-2026-0704-3F2A

notifications-svc · Elevated send latency, no linked investigation yet

analytics-svc · Consumer lag on partition 3 matches ingest-worker backlog pattern related ▸ INV-2026-0701-9B10

orders-db · Connection pool at 92% utilization, trending toward exhaustion

Pre-alert 78% orders-db pool exhaustion, checkout writes start failing, ~20m to impact

notifications-svc · Send queue depth growing steadily, no alert threshold crossed yet

Pre-alert 41% delayed notification delivery, no failed sends yet, ~55m to impact

Live signals

Streaming

Service: All ▾Cluster: All ▾Type: All ▾

434 new · Resume

MEM analytics-svc consumer lag increasing on partition 3 00:01:07

LOG notifications-svc delivered 214 notifications in the last hour 00:01:06

NET payment-gateway request queue depth exceeded 500 00:01:04

CPU ingest-worker CPU utilization at 54% on pod ingest-worker-7d4f 00:01:03

TRACE checkout-svc request completed in 142ms 00:01:01

MEM analytics-svc consumer lag increasing on partition 3 00:01:00

LOG notifications-svc delivered 214 notifications in the last hour 00:00:58

NET payment-gateway request queue depth exceeded 500 00:00:57

CPU ingest-worker CPU utilization at 54% on pod ingest-worker-7d4f 00:00:55

TRACE checkout-svc request completed in 142ms 00:00:54

MEM analytics-svc consumer lag increasing on partition 3 00:00:52

LOG notifications-svc delivered 214 notifications in the last hour 00:00:51

NET payment-gateway request queue depth exceeded 500 00:00:49

CPU ingest-worker CPU utilization at 54% on pod ingest-worker-7d4f 00:00:48

Services

6 services · prod-us-east-1

Services

6

monitored

Healthy

3

nominal

Degraded

2

elevated errors

Down

1

critical

Avg p95

214ms

across services

Service health

filter services

Service	Status	Error rate	p95	RPS	Owner
checkout-svc	Down	4.8%	980ms	1.2k	Payments
payment-gateway	Degraded	1.1%	340ms	860	Payments
ingest-worker	Degraded	0.7%	210ms	2.4k	Data
analytics-svc	Healthy	0.1%	120ms	540	Data
notifications-svc	Healthy	0.2%	180ms	410	Growth
orders-db	Healthy	0.0%	22ms	3.1k	Platform

Reliability

SLOs · error budgets · last 14 days

Availability

99.94%

target 99.9% · met

Error budget

62%

remaining this month

MTTD

2.4m

mean time to detect

MTTR

18m

mean time to resolve

Incidents

7

last 14 days

Incidents · last 14 days

Jun 26Jul 3Jul 10

Error budget by service

checkout-svc12%burning fast

payment-gateway48%watch

ingest-worker71%healthy

analytics-svc94%healthy

Back to feed

INV-2026-0704-3F2A

checkout-svc — 5xx error spike

Critical Remediation ◇ Agent working

Investigation timeline

Live

Detected09:41:02

checkout-svc 5xx rate crossed 5% (baseline 0.3%). Auto-opened investigation and paged the on-call agent.

Gathering evidence09:41:20

Pulled 5xx logs, distributed traces, and the last 3 deploys. Correlated a payment-gateway retry storm starting 40s before the spike.

Hypotheses09:42:05

Connection pool exhaustion from retry storm86%

Upstream payment-gateway deploy v482 regression64%

Node-level network saturation18%

Root causeConfirmed

payment-gateway v482 shipped an aggressive client-side retry (3x, no backoff). The retry storm exhausted checkout-svc's DB connection pool (max 40), so new checkout writes failed with 5xx.

Remediation09:43:10

Proposed: roll back payment-gateway to v481 and raise checkout-svc pool to 80 as a guard. Impact: ~30s of elevated latency during rollback.

Coming soon

This screen is outside the current redesign scope. Feed, Services, Reliability, and investigation detail are built to the brand system.