Story of how a hang becomes a crash (because testers love reporting crashes)
Now, I am going to tell you a story that I had been hiding for quite a while. Not intentionally hiding but was planning to blog about it and the time is right now :)
I provide many applications to test in my workshop. At the end of a testing session, I check with the participants if they found any crashes with the application. Not to be surprised, a room full of testers, they do report crashes.
I ask them to reproduce it with me watching what they do and here is a shocking information; most of what they report as crash weren't crashes at all. What you are about to read is also a great example of confirmation bias that I have witnessed.
Here is what they are doing: They are performing an input constraint attack or providing a large set of characters as an input to a field and hits the "Submit" button. They wait for 10 seconds to see what the application does and on seeing the application in not responding state, kill the application and report that as a crash.
Awesome! Isn't it?
Now, I got conscious of the fact that testers seem to be reporting a hang as a crash and was keen in looking at live projects during my consulting assignments. I get access to the bug tracking system during my consulting (wow) and I see patterns of such reports.
I go filter out some of the crash reports from the bug tracking system and try to attempt what the tester did to report that type of a crash and bang, that's a hang.
I have made this point to testers who attend my workshop in order to help them be more conscious of what they are seeing versus what they are reporting versus what they are eager to report versus what they should be reporting.
It seems to me that testers have an anxiety to report crashes. There's nothing wrong in it but it goes horribly wrong when you report a crash by fooling yourselves and the people around you. I recently blogged about the Obsessive Checking if being mentioned disorder that I was suffering from and here is a relevant excerpt from it, "Months later, I started replacing every word by my name till I saw my name on others post. So, I may have read a few posts without actually learning anything from it because all I saw is "Pradeep" & "Tester Tested" on those posts."
Now, why is that relevant to this post? I see that the testers I have witnessed who report hangs as crashes also appear to have a similar problem of obsession towards reporting crashes that they appear to not see a hang but see a crash. I would love if you ask if these testers ever report a hang? Yes, they do. If an application recovers before they kill, that's a hang.
I understand that an application might have hanged because it has crashed but how do you know? Oh yeah, allowing the GUI to fool you and me?
Here is an excerpt ( and tweaked to hide confidential information ) from an issue that I reported on Sep 30, 2010 @ 12: 13 PM IST (thank you Jira) while testing a new OS that is soon to be launched. No, not Chrome OS.
"I opened Media Player application and made an attempt to subscribe to a podcast.
The application appeared to hang and it also did not allow me to close using the X mark on top right corner.
In an attempt to kill it, I navigated to XXXXX (product name masked) and tried closing it from there. However, the XXXX is open and Media player still seems to be remaining in the hung state.
From that point, I might be forced to reboot the system to get out of it or can escape by waiting for a longer time that I waited for Media Player to recover (>5 mins). No user to my knowledge would wait for more than 5 mins to see the Media Player function. Even a reboot appears to be a faster option"
Now, if that was the Description, here is the summary :
"Unable to kill a XXXX when the application in it appears to hang or is busy"
- Why I posted the above? Is it because I wanted people to note the usage of the word "appears"?
- Why did I make a few words in it bold? Is it to highlight it?
Long ago, well, not so long ago, I blogged about how teaching testing is helping me test better. The above post and the bug report I posted, is another example of how teaching testers has helped me do it a lil better.
Now if you are going to be sitting at my workshop and be worried if you are going to make mistakes that I am going to highlight, please be informed that you are paying to my workshop to get into the safest environment to fail. No managers watching you. No clients frowning at you. No appraisals being done. Just your own dream of personal excellence aching you for not being there yet but being happy that you know why you are not there. Its a wonderful feeling.
