Thursday, June 16, 2005

User Profiles and Exploratory Testing

"When the tester moved on to other things, I spent time in their work environment to see if there was something else I was missing. I found that their test device had connections that weren't fully seated, and they had stacked the embedded device on both a router and a power supply so the device gently rocked when you typed. I went back to my desk, unseated the cables so they barely made a connection, and while installing a new firmware build, tapped my desk with my knee to simulate the rocking, and presto! Every time I did this on a build that this tester had been using the bug appeared. I collaborated with a developer and he went from saying 'that can't happen' to 'uh oh, I didn't test if the system time is back in time, *and* that the connection to the device is down during installation to trap the error.' The time offset and the flakey connection were causing two related unrepeatable bugs. This sounds simple, but it wasn't from the perspective of the source code. These areas of code were completely unrelated and weren't obvious when testing at the code level."