Replay last night’s incident on your laptop.

Event log

1. t+00:00 deploy release 4.19.0 rolled to 8% of checkout-api
2. t+01:35 queue orders.enqueue 41/s, within the hourly average
3. t+03:30 db connection pool resized 40 → 64 by the autoscaler
4. t+05:30 deploy release 4.19.0 rolled to 50% of checkout-api
5. t+06:42 db slow query orders_by_customer: 1.4 s, no index on tenant_id
6. t+07:50 queue enqueue outruns drain for the first time
7. t+09:05 worker worker-3 misses its 2 s deadline and is recycled
8. t+10:10 queue queue depth crosses the 200 alarm line
9. t+11:45 human on-call paged: checkout-api queue depth
10. t+13:20 worker worker-6 recycled on the same deadline
11. t+15:05 edge edge starts shedding 3% of POST /orders
12. t+16:50 human on-call opens last Tuesday’s replay beside this one
13. t+18:00 db orders_by_customer now 2.9 s at p95
14. t+20:15 queue depth 426: drain 12/s against 47/s in
15. t+23:20 worker two workers recycled inside one minute
16. t+27:00 human bisect names the cause: tenant_id index missing on the new path
17. t+30:00 deploy release 4.19.0 held at 50%, rollback drafted
18. t+34:00 db index orders_tenant_idx created concurrently
19. t+38:00 queue drain overtakes enqueue for the first time
20. t+40:50 worker worker-3 and worker-6 back in rotation
21. t+44:00 edge card network settlement re-checked (re-fetched, not replayed) · not replayed
22. t+48:00 queue depth back under the alarm line
23. t+52:00 db p95 settles at 214 ms, 24 ms above the pre-release figure
24. t+55:00 deploy release 4.19.1 ships with the index in its migration
25. t+58:00 human incident closed against the recording, not against memory
26. t+59:45 queue enqueue 39/s: the hour ends where it began

State at the head

Queue depth

426 +210

Workers up

6

p95 latency

1180 ms

Failed orders

44

Release stage

degraded

Folded from 14 of 26 events. Drag back and the state is refolded from 02:00, so it always matches.