Debugging a Production Bug I Cannot Reproduce Locally
- Reconstruct the timeline from prod logs before touching code
- Bisect by deploy to find the exact commit that broke
- Add targeted instrumentation, test one hypothesis at a time
- Verify the fix in the real environment, not localhost
A checkout endpoint failed for 3 percent of orders. It worked every time on my machine. I fixed it in 90 minutes using a five-step process I now run every time prod breaks and localhost stays green. Here is the exact method, with the real numbers from that bug.
Step One: Reconstruct The Timeline From Logs
The first instinct is to open the code and start reading. That is the slowest path. When a bug only shows up in production, the production logs already contain the answer. My job is to read the timeline, not guess at it.
I started by pulling every log line tagged to the failing endpoint for the last 24 hours. That was around 4,200 lines. Too many to read one by one. So I filtered to only the error-level entries and the requests that returned a 500 status. That dropped it to 61 lines. Now I could actually see a pattern.
The pattern was ugly at first glance. The failures were not evenly spread. They clustered. Between 14:00 and 14:40 there were 44 of the 61 errors, then it quieted down and trickled the rest of the day. That spike told me something happened at a moment in time, not something baked into the code from the start.
I write the timeline out as plain text. Timestamp, request ID, error message, and what the user was doing. When you line these up you often see the thing that connects them. In this case the connecting thread was the payment provider response. Every failed request had a slightly different response shape from the provider than the successful ones did.
The lesson I keep relearning: the logs are a crime scene, and the timestamps are the witnesses. Do not touch anything until you have read them in order. If your logs are too thin to reconstruct a timeline, that is itself the first bug to fix, and I cover the wider setup discipline in Claude Blueprint.
One practical tip. I copy the raw log block into a scratch file and ask Claude to group the errors by shared fields. It found that all 44 spike errors shared one field value I had skimmed right past. That saved me maybe 20 minutes of manual squinting. The timeline is step one because everything after it is only as good as the story you build here.
Step Two: Bisect By Deploy
Once I had a timeline, I had a window. Something changed around 14:00. The fastest way to find what changed is to look at what I shipped.
I keep a deploy log. Every push to production gets a line: timestamp, commit hash, one sentence of what it did. This is the single most useful habit for this kind of bug. Without it you are searching a haystack. With it you have a short list of suspects ordered by time.
My deploy at 13:52 was a change to how I parsed the payment provider webhook. That is eight minutes before the error spike started. When a deploy and a spike line up that closely, you have your suspect. It is not proof, but it narrows the search from the entire codebase to one 40-line diff.
Bisecting by deploy is faster than bisecting by commit in most cases, because a deploy is a real event in the real environment. A commit that never shipped cannot cause a production bug. I only drop down to commit-level bisection when a single deploy bundled many commits and I need to know which one. In this case the deploy was two commits, so I read both.
Here is the trap. The diff looked correct. It parsed the webhook fine on every test payload I had. That is exactly why I could not reproduce it locally: my test payloads were all the old response shape. The provider had quietly started sending a new field ordering to a subset of accounts, and my new parser assumed a fixed order.
So the deploy did not create the bug on its own. It created a fragility that the provider's own change then triggered. Two changes, one mine and one theirs, colliding at 14:00.
If your deploy log does not exist yet, start it today. One line per push. I wrote about the workflow around shipping small and often in the broader build notes, and it pays off most on days like this one. Bisecting by deploy turned a whole-codebase mystery into a two-commit inspection in about ten minutes.
Step Three: Add Targeted Instrumentation
I had a strong suspect and a strong theory. What I did not have was proof, and I refuse to ship a fix based on a theory alone. The next step is to make the invisible visible.
I did not try to reproduce the exact bad payload locally. That would have meant guessing the provider's new format, and guessing is how you waste an afternoon. Instead I added instrumentation to production that would capture the truth the next time it happened.
Targeted is the key word. I did not turn on verbose logging everywhere, which would have buried the signal in noise and slowed the server down. I added exactly three log lines inside the webhook parser. One logged the raw incoming payload before parsing. One logged the parsed result. One logged which branch of the parsing logic ran. That is enough to see input, output, and path.
I shipped that instrumentation as its own tiny deploy at 14:20 in my working session. Then I waited. Because the failures were only hitting a subset of accounts, I did not have to wait long. Within 12 minutes a real failing request came through and my three log lines captured the whole thing.
The raw payload showed the new field ordering exactly as I suspected. The parsed result showed a null where an amount should be. The branch log confirmed the parser took the wrong path. Theory confirmed with evidence, not assumption.
There is a discipline here that is easy to skip. Instrumentation is temporary. I tag every debug log line with a marker so I can find and remove all of them after. Leaving debug noise in production is how logs become useless for the next incident, which loops right back to why step one worked so well this time.
If reproducing locally is impossible, instrument production and let the bug reproduce itself in front of you. This is far cheaper than building an elaborate local mock of a third-party service you do not fully understand. I let the real environment show me the real input, then I went and built a local test from that captured payload, which is the reverse of the usual order and much more reliable.
Step Four: One Hypothesis At A Time
Now I had proof and I was tempted to fix five things at once. The parser was fragile, the error handling was thin, the logging had been too quiet, and the provider integration had no schema validation. All true. All tempting to fix in one big commit.
That is the mistake that turns a 90-minute fix into a three-hour one. When you change five things and the bug goes away, you do not know which change did it. Worse, if one of your five changes introduces a new bug, you now have two mysteries instead of one.
So I wrote down every hypothesis, ranked them, and committed to testing exactly one. Hypothesis one: the parser fails because it assumes field order. Fix: parse by field name instead of position. That is the smallest change that addresses the proven root cause.
I made that single change. Nothing else. Not the error handling, not the schema validation, not the logging improvements. Those went on a list for later, and later they got their own commits with their own verification. The list matters because a real bug always exposes three other weaknesses, and if you do not write them down you either forget them or you fix them all at once and lose the thread.
Testing one hypothesis at a time also keeps your revert clean. If parsing by field name had not fixed it, I could revert one two-line change and be exactly back where I started, with my understanding intact. Compare that to reverting a sprawling commit that touched five files and left you unsure what state you are in.
This is the same principle behind shipping small deploys, and it compounds. Small change, clear test, clean revert path. I applied the fix, and the captured bad payload now parsed correctly in my local test built from the real data. One hypothesis, one change, one confirmation. Only then did I move to the next item on the list, and each of those got the same treatment on its own.
Bottom Line
Production bugs that vanish on localhost are not magic. They are a gap between your test environment and the real one, and the five steps close that gap in order. Reconstruct the timeline from logs so you know when it broke. Bisect by deploy so you know what changed. Add targeted instrumentation so the bug proves itself instead of you guessing. Test one hypothesis at a time so your fix is clean and your revert is clean. Then verify in the real environment, because a fix that only works on your machine is the same trap that started this.
That last verification is not optional. I watched the failing endpoint for a full hour after shipping the fix. Zero errors across roughly 900 requests. That is when I called it done, not when the code looked right.
If you want the full workflow I run around shipping, logging, and testing as a solo operator, start with Claude Blueprint. It ties these habits together so the next unreproducible bug takes 90 minutes instead of a lost day.
Back to all articles
