Hoof-beats

"When you hear hoof-beats behind you, don't expect to see a zebra." Is a common charge used in medicinal circles (or so I'm told). It applies equally well to debugging. For several weeks I was looking for a bug. This one bug was blocking the customer's in house testing from continuing, so on several occasions I went looking for it. We tested and poked and pondered, but no cause for the bug was apparent. To make it worse we could not reliably reproduce the problem. I was racking my brain for ways this might happen, and the scenarios got more and more far fetched. We verified hardware, checked versions everything was matching between the customer's environment and ours. Well we finally got the customer to send us the piece of hardware they were using, setup in the way they were using it. Low and behold the problem became apparent. The hardware was using a 3rd party service that the hardware had to be set up with. Well the test account they were using and the one we were using were setup different. I could have kicked myself. So when a bug is entrenched, seems to be growing horns and taunting you remember its probably a horse not a zebra.