Bisecting a race condition that only happened on Tuesdays

Some bugs refuse to reproduce. This one double-charged roughly one order in four thousand, only between 02:00 and 03:00, and only on Tuesdays. Here is how it was found in an afternoon.

Pick a good hour and a bad hour

Tuesday’s batch export runs at 02:00. Wednesday’s hour has the same traffic without the export. Both were already recorded, so there was nothing to reproduce:

fenwright pull checkout-api --window 2026-08-25T02:00/1h --out bad.ndjson
fenwright pull checkout-api --window 2026-08-26T02:00/1h --out good.ndjson

Diff them

A diff aligns the two windows by request key and prints the first place their state differs:

fenwright diff good.ndjson bad.ndjson --key order_id

Most differences are noise — different customers, different baskets. The diff ignores fields you mark as volatile, which left eleven orders whose state diverged.

Bisect to the first event

Bisect then halves the bad window until it finds the earliest event after which the state can no longer be explained by the good hour:

fenwright bisect bad.ndjson --against good.ndjson --key order_id

It stopped on a retry from the payment worker arriving 40 ms after the export job had taken a row lock. The retry did not see the lock, so it charged again.

Prove the fix against the recording

The fix was an idempotency key on the charge call. Replaying the bad hour locally against the patched worker produced zero double charges across all eleven orders — the same hour, the same events, a different outcome. That replay is attached to the pull request, so the reviewer could run it too.