Traditional Testing

Traditional Software Testing Process - The cash cow demystified

So, I don't need to tell you that most of the software testing in the world is outsourced. I live in Bangalore and I assume a considerable amount of the world's outsourced testing projects are here. I meet testers from most of these organizations and hear the same process of testing software although called by different names their organization executives named them.

Most testers I meet lament about the process (or sucked within it for survival ) and approach they are forced to follow and try talking to their management about better ideas and better ways to test. Those executives don't seem to listen much nor willing to change.

Wondering why they wouldn't?

They have hit a cash cow.

Mark Crowther and I were discussing about Code Coverage & Requirements Based Testing in Test Republic and some how drove me to write about the way most businessmen are fooling their customers through the process they love to follow.

So, here it goes with editing and expansion. Assuming Mark from UK or maybe even you outsourced a testing project to me:

• I would spend a couple of days analyzing your requirement document and bill you for X hours per person involved in my team.
• I would spend a couple of days writing a test plan document ( but not refer to it ) and bill you for 2X hours per person involved in my team for preparing it.
• I would spend a month or two writing test case document ( and refer only to it ) and bill you for 10X hours per person involved in my team for preparing it.
• I would then again create a traceability matrix ( just to fool you and your boss about our coverage ) and bill you for 5X hours per person involved in my team for preparing it.
• So far, total of 18X hours per person involved in the team is the billing.
• Assuming X is 50 hours and there are about 10 members in my team, that's 18 * 50 * 10 = 9000 hours of billing with no single bug found yet. ( Mark wrote about it, too )
  
• If you are paying $20 an hour per person on an average, you would have actually given me a business of $180,000 without me or my team finding any bug yet.
• So after investing $1,80,000 on me and my team, you would want some benefits of that. So, you wouldn't pull the project out or move it to another vendor because more or less he would do the same and you would end up paying another $180,000
  
• Then comes the test case execution cycles for our documented 10,000 tests out of a possible hundred million tests
  
• For every new bug that you find out of the releases I make, my team would spend documenting the new test case, getting it reviewed and resulting in slower testing for you and more money for me.
• So assuming running 10,000 tests take 2 weeks for a team of 10 members to execute. Also assuming least 50 cycles of testing, you would have paid me about $140,000 for a coverage whose value might be not worth of the money.
• Of course there is additional documentation of missed test cases and other template filling activities that will be billed to you.
• To fool you further, I would instruct my team to use some expensive license based tools ( what else will I do with the money you are pumping in ) to give you a sense of faster testing ( by foolishly comparing it with human speed of testing ) and call it "Automation Testing". It turns out that these tools could have helped me find bugs that are of not a great value to you but hey, we want to see test case pass more than fail.
• Your coverage isn't improving much because we have converted manual tests to automated tests ( although its not the same test ) but to show you the speed of our tools.
  
• So think about adding another $25,000 and giving you an illusion of an ROI of $100,000 while pumping multiples of $100,000 from your bank account to mine.
  
• The CMM, TMM, Six Sigma, ISTQB scams are built around this eco-system to enable more money flow for hardly any value. Who knows there could be a cut for the people who know all this and yet do it.
  

Why wouldn't a businessman be glad about the traditional approaches to test software?

Are you asking about what happens to the users of the product?

• Lets bother about the users of your product later during our maintenance billing phase. Don't you know SDLC ends in Maintenance Phase? If we do everything right in the previous stages then how do we prove we follow SDLC when there is hardly any work in maintenance phase?
  

While you are reading this, you shouldn't be thinking of this happening only in India but in most parts of the world and even within places like United States and Europe. There are smart businessmen everywhere. At one end they pay us but at the other end use us to make more money. We need money and they need us to make that. Don't make smart businessmen exclusive to India and leave your own country out of it.

I hope those who outsource start pushing for services that doesn't fool them. Who would actually listen to this argument is testers turned businessmen and testers turned outsourcing heads and testers turned business leaders.

Whenever I visit a testing services company and see a testimonial of a customer who talks about the great ROI they got and faster testing, I wonder what a heavy price they paid to believe so.

Exploratory Testing ++ , Context Driven Testing ++ , Rapid Software Testing ++ or else YourMoney --

Don't want to be fooled by outsourcing software testing? One of those who could be of help to you among many folks I know, is myself.

Read more »

What all scenarios can be taken up for Login functionality testing?

A) As a user I want to log into the system so that I can get access to restricted functionality
•There should be a login form on the home page
•User enters email and password and clicks "Login"
•If the details are correct, they are logged in and the home page is redisplayed without the login form
•Their email address should then be shown at the top of every page "Logged in as ..."
•If either of email/password are missing or incorrect, and error message should be shown "Invalid email or password." (the same error message is used to avoid revealing information about which users exist in the database).
•If a login fails, the password field should be cleared but the email field left.
•Once logged in, the session should timeout after 30 minutes of inactivity
•Users who are either deactivated, rejected, or pending approval should not be able to log in (covered in other stories below also).
•If an account is deactivated or locked, logging in with an invalid password should display "Invalid email or password" (NOT "your account is not active")
•Attempting to access any page that requires login (while not logged in) should redirect the user to the login page. After successfully logging in they should be redirected to the page they originally requested.
(B) As a user I want to apply for access so that I can login into the system
Users who do not yet have access need to be able to request access. Allowing them to request access themselves has the benefit that they get to choose their own password so administrators do not have to deal with secure password distribution.

•There should be a "Sign Up" link shown on the home page when a user is not logged in
•It should take the user to a Sign Up form where they must enter:
•Email (must a valid email address, which will be used as their login id)
•Password (at least 6 characters, max 20, must include at least one of each of upper/lower/number/symbol)
•Password confirmation - must match password exactly
•First name
•Last name
•All are mandatory fields. Email/first/last should be limited to 255 characters, both in the form and on the server side. Password should be limited to 20.
•After correctly entering all fields and clicking "Submit Request", they should be taken back to the home page with a message "Thanks for requesting an account. You will receive an email when your request has been approved."
•At this point they should not yet be able to log in. If they try to log in they should see "Your account is not active."
•If there's any validation errors on the form, it is redisplayed with informative error messages and the account is not created. The two password fields should be cleared out each time this happens.

Read more »

Ants solve problems that testers struggle to!

Ants solve problems that testers struggle to!

Ants can go into corners of tables and chair, which we humans might not be able to.
Ants can travel to any country without a passport or Visa (if they manage to slip into someone's baggage).
Ants can live anywhere and yet not pay rent or property tax.
Ants can eat junk food forever and live happily.

Doesn't all of them sound so obvious?

So, here is something that might not be so obvious about ants... Little creatures that we commonly refer to as ants have solved a problem that many testers today struggle to.

Have you heard a tester say, "I haven't tested this before / this is too complex / I don't know how to start testing it".

The important thing to note is that we might have heard it from our own mouth or heart and sometimes from others, too.

Here are some experiments that I did with ants to see how they respond to complexity:

Experiment 1
I took an ounce of sugar, placed it near a colony of ants and observed what happened to the sugar I placed after a while.

My experiment resulted in each ant carrying a crystal of sugar and moved them to their nest. Sugar is all gone!

Experiment 2

I placed a cake of size approximately 1000 times bigger than an ant in the same place and observed what happened to the cake.

This experiment resulted in each ant breaking the huge cake into smaller fragments and moved it to their nest bit by bit. Cake is all gone!
Experiment 3

I put a piece of solidified sugar candy near the same colony, which was hard to break for an ant.

This experiment resulted in ants trying to break the candy to smaller segments, but probably the smart ants realized it to be complex than their earlier two assignments I gave them and then they all co-ordinated together and carried the candy to their nest. Candy is all gone!

I bet I could place a mammoth of food for them and they, without bothering about the complexity of it, would try moving it to their nest or create more nests and colonies to accomodate it or work a strategy to get it to their nest or much it as much as they can till the mammoth lies there...

Ben Simo, a wonderful tester you might already know, recently wrote a post titled Solving Intractable Problems and ended it beautifully by saying, "Start with what you recognize" - which I now think is what ants do instead of saying, "That food(work) is not for us, it is for bigger (skilled) ants"

After having done these experiments I was ashamed of my behavior of hesitating to test a product (which I did a couple of years ago) just because something appeared to be too complex or I thought I didn't know about the product or technology.

No, I said, *I* was ashamed and I am not going to ask if you were ashamed because you might have not done that.

After all they are ants and they don't have the super brains we have and lets ignore them folks and continue to live as happy as we are, complaining that things are complex.

Silly ants, they don't even know they are dealing with complexity. I think they should be ashamed of it.

Read more »

Interview Tips

I interviewed few candidates for my company. A few tips.

Usually interviewers form an impression on whether the candidate is good or not within 5 minutes of seeing and talking to him/her.Rest of the time, they try to validate whether their impression is correct or wrong(atleast this is what I do).So its important you make a good impression in the first few minutes.

Dress code

Well dress you wear for the interview is not everything, but you dress well, it will help you to make a initial good impression. The color of the dress gives certain impression, it seems(I don't remember paying attention to the dress, the candidate wore, but it seems it would have affected my impression at a subconscious level.But what you loose dressing up well)

• Red:You wear it to attract other people's attention.If you are presenting something or you want others to pay attention to you, dress in red. This is not the best color to wear for interview.
• Blue:If you want other people to like you, dress in blue.
• White:It gives the impression that you are hardworking.Wear this. I mean not entirely in white but the predominant color should be white.Like a black trouser and white shirt should do.

Resume

• Whatever you mention in the resume, you must try to be thorough in it.
• Be honest of what you put in it.
• Try to keep it small,it should not be longer than 2 to 3 pages.
• One of the main reason of rejection is, something is mentioned in the resume and when asked questions about it, not able to answer them and in few cases saying that "I have done that 'something' long back and I don't remember anything".In many cases its true that you have used a tool or something a year back and you don't remember much about it, in that case before attending the interview, brush up atleast some basics regarding it, do some home work and prepare yourself to talk about it or don't mention about it in the resume, if it is not that important.

Topics to prepare, for a entry level testing job.

Testing fundamentals: Testing terminology, definitions etc.

Testing Processes: The testing processes used in your current/previous company.Knowledge of life cycle testing etc.

Testing tools: Given a problem, how to solve them using testing tools.

Testing deliverables: Able to write a test plan or test case for a given product or requirements

Analytical skills: Solving Puzzles

Programming skills(optional): Able to write pseudo code for a given problem.

Few more tips

Good Communication skills: obvious isn't it

Good Listening skills or understanding skills: Able to understand the questions posed by interviewer. Try your best to understand fully what the interviewer is blabbering in the first say.But if what he says is not clear, better to ask or try to confirm what you heard is correct before answering.

Confidence level: Watch your body language, keep eye contact, don't be over confident just be confident.

Two most important tips: You should have the right skills for the job.Also equally important is you SHOW or DISPLAY in front of the interviewer that you have the right skills.

Read more »

Exploratory testing

Exploratory testing is not defined by any specific example of exploratory testing.

Just as tap dancing does not characterize ballroom dancing, you can’t take any one example of exploratory testing and treat that as representative of the entire concept of ET.

If you were to hear me singing an aria by Mozart, that would be an example of opera singing. It would be an example of BAD opera singing, but it would truly be an example of the style. Similarly, I regularly talk to testers who go “oh yeah I’ve seen that exploratory testing stuff but it’s not structured… not documented… not x… not y… not whatever.” And my reply is “you probably haven’t seen skilled exploratory testing. Would you like to hear me sing an opera now? OR, I could show you a good example of ET in practice.”

Exploratory testing can be done in an unskilled, slapdash, silly way. Just as a unskilled driver behind the wheel of a car is still a driver who is driving a car, a poor tester can still be doing ET– albeit probably not very well.

The cool thing about ET is that, even done badly, it’s still a great way to find some bugs. Michael and I try to help you do much, much better than that.

The core idea of ET remains as it always has been. It’s been expressed in many different ways, but boils down to this: test design and test execution and learning mixed together in a mutually supportive way. Whenever you see that, and to the degree that you see that, you are seeing exploratory testing.

Read more »

Boundary Value Analysis(BVA)

Boundary Value Testing is the most well-known and simple technique of test design, which helps the tester choose the most effective values ​​from the ranges of values. This technique is applicable at all levels of testing – unit, integration, system, and system-integration test levels.

We consider the steps of using of the equivalence classes technique:
1.Determining the range of values ​​(usually, the equivalence class).
2.Determination of the boundary range.
3.Creating three test cases for each boundary – one that checks the border value; second that checks the value below boundary; and the third that checks the value above boundary.

An example of the ‘upper’ and ‘lower’ boundary values. If the boundary is $ 3, the “lower” boundary will be $ 2, and the “upper” – $ 4. If the boundary is $ 2.00, the “lower” boundary will be $ 1.99, while the “upper” $ 02.01, etc.
Let’s discuss this technique on an example.
Returning to our pencils (see Equivalence Class Testing), the pens vary in price depending on the ordered quantity:
1 – 100 – $ 10. the pens;
101 – 200 – $ 9. the pens;
201 – 300 – $ 8 on the pens, etc.
With each one hundred, the price is reduced by $ 1.

The maximum available amount is 500 pcs. Boundary value analysis are very interested in the boundaries of the values intervals, because it is “analysis of the boundary.” Using the method of equivalent partitioning, the following classes were identified:
1. Valid values ​​are from 1 to 100;
2. Valid value: 101 to 200;
3. Valid value: 201 to 300;
4. Valid value: 301 to 400;
5. Valid value: from 401 to 500;
6. Invalid value:> 500 pieces;
7. Invalid value <=0;
8. Invalid value: a float from 0 to 500;
9. Invalid value: a negative number;
10. Invalid value: a collection of letters;
11. Invalid value is an empty string.
It’s time to take on boundary values ​​of these classes. You should take not only the limiting value, but also retreat one step up / down (on the tiniest step as possible). We obtain for the above written classes:
1. 1; 0; -1;
2. 99; 100; 101;
3. 199; 200; 201;
4. 299; 300; 301;
5. 399; 400; 401;
6. 499; 500; 501;
Thus, we get to 3 values ​​for the boundaries and take one value from “bodies” of the equivalence classes for a total of 21 + 11 = we take 32 values for verifying. M-m-m-m-m it is more than 11, but much less than 500. Yes, and we check not only the valid values, but also invalid! But it’s only two techniques of … to be continued …

Read more »

Letter to myself

I have been with you all this while and shall continue to be with you. I am enjoying each moment you enjoy as a tester and also each moment that you don't enjoy as a tester. I am sorry for enjoying those moments which you think you haven't been enjoying. I am sure you'd be interested to know why and how I have been enjoying those moments that you haven't been.

First, let me list the thing that you haven't been enjoying:

• Whenever and wherever you write about test automation, a handful of readers tend to think that you reject the idea of test automation and they write harsh e-mails to you.
• Whenever and wherever you write about certification, the certified tester community attacks you over e-mails/phone calls that you and I are spoiling the craft.
• Whenever and wherever you write about programming skill for testers, another handful readers think that you are suggesting testers not to learn programming.
• Whenever and wherever you write about tools, most testers think you are referring to test automation tool.
• Whenever and wherever you write about yourself or your experience, a set of people think you lack humility.
• Whenever and wherever you write about exploratory testing as a skilled activity, a set of people think "no tester would do be able to do that".
• Whenever and wherever you write about ideas to solve a problem than giving a one line answer, a set of people think you don't know how to solve it and is faking what you know.
• Whenever and wherever you are writing about testing being an excellent thinking job, a few people think you are trying to paint a picture that does not exist.
• Whenever and wherever you - do bad testing, fail in testing course like BBST, you feel intimidated by more skilled people than you, you feel bad about not having learned or practiced those things that helps you become a better tester, you fail to give enough respect to expert testers time, etc...

In this context, I'd like to remind you of a learning you had from Michael Bolton: There are some things under your control and there are other things that are not under your control. Taking advantage of things under your control, as a tester, is essential to clear traps and it might also lead to gaining more control. To take advantage of things under your control, you first should realize what are the things you control.

I also remember that you had made a note in your Moleskine of Saurav Ganguly's television interview where he was asked: How were you able to make a great comeback to the world cup cricket squad after being axed for poor performance? His answer: I didn't worry about things that are not under my control ( media critique, jokes on bad batting performance, e-mail forwards about my performance, people gossiping about it ) but focussed on things that are under my control ( Practice, skill enhancement, consistent batting record in Ranji trophy)

Similarly, you don't have a control over the thoughts of people thinking whatever they think after reading whatever you have written. You have a control over what you write and you have a control over the way you write it.

Your testing has been influenced by a lot of experts but not all have similar influence. They haven't seen great testing to appreciate the things you are sharing and I doubt if all those who witness it would be influenced by it because it's hard work and high skill demanding.

• Not all testers want to do great testing
• Not all testers know they are doing bad testing.
• Not all testers want to know they are doing bad testing.
• Not all testers want to know more about testing.
• Not all testers know what skills to gain and practice.
• Not all testers agree to be context driven.

Here are three questions ( like the Monty Python and the Holy grail bridge of death piece you enjoy )


1.Whom are you serving through your writing?
I am sure your ongoing struggle is in understanding that. Let me help you with what I think about whom you are serving - You are serving those testers who look for better thought process and those who enjoy the better thought process and those who think you have a better thought process.
2. Who asked you to serve them?
I asked you to do that!
3. Why haven't you been enjoying some moments that I have been?
You want all testers to do great testing although you know its not possible. Some people question your idea of "great testing" because they already have an idea of "great testing" and it conflicts with the idea you have. You are able to demonstrate that their idea of "great testing" lacks critical thought as your idea of "great testing".
By the way, your idea of "great testing", is not yours but of those people who have influenced you. You have just subscribe to those ideas and are contributing to it in different forms. I have occasionally witnessed you doing bad testing and I am sure I would see that in future, too. Do not forget that you are a human and your ideas are fallible. I know bad testing and bad thought process irritates you, even if you are the one who is doing it.
I would love to see you doing things that are under your control - learning, reading, writing, bettering your skills, helping those testers who enjoy the thought process that you enjoy, speaking, coaching and mentoring.
Your power to influence testers is limited. Limited to the ones who don't want to limit themselves. So do unlimited things under limited time that you and I will be here in this world.
I will be with you forever, enjoying everything you do from great things to not so great things. Anything you do is great to me.
I will write to you whenever I feel a need for it. This letter is personal, just between you and me.

"Here is a way to test if your mission on earth is complete - if you are alive, it isn't" -- Richard Bach

Yours truly,



Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com



"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it."

Read more »

Performance testing tool

This is in fact most crucial test for time critical applications such as stock management systems which gets refreshed every minute creating huge load on web site as there are thousands of users accessing the web site at the same moment.

For such applications stress testing with more than 10,000 users is a basic test. The WAPT Pro comes with default “Load Agents” functionality to test average load on any web site. But for high capacity test with more than 10,000 users we can now leverage the power of x64 Load Engine extension.

x64 Load Engine is similar to Load Agents feature on WAPT Pro installation. The main difference is in the ability of load engine to generate huge load using 64 bit Windows systems. Using one 64 bit server you can dramatically increase the load testing capacity on web site under test. The x64 Load Engine can be configured in such a way that using slightly high end hardware you can easily generate more than 100k virtual users load on web site.

Assume that you want to stress test your web site with 100,000 concurrent virtual users. You can achieve this using 4 servers each with 25,000 virtual users. Setup this load engine on 4 systems and use them concurrently to achieve desired web site load. This powerful x64 Load Engine effectively utilizes the available memory resources on 64 bit system architecture.

WAPT Pro x64 Load Engine Installation:
- You can install x64 Load Engine on 64 bit version of Windows XP/2003/Vista/2008/Win7 OS.
- x64 Load engine works best on following hardware configuration:
Core i5/Phenom or CPU better than this, 8+ GB RAM and Gigabit Ethernet

How to install:    
x64 Load Engine is a WAPT Pro extension so it runs on WAPT Pro tool similar to Load Agents. To use this load engine you must have WAPT Pro installed first. You can download and install x64 Load Engine from below mentioned link. After installation you can start using load engine using the WAPT pro workplace itself.
Download WAPT Pro x64 Engine .

x64 Load Engine Pros:
- You can generate almost unlimited virtual users (test load) on Windows 64 system.
- Easy to install, learn and configure
Check out below screenshot to know the simplicity of x64 Load Engine Manager:

This is in fact most crucial test for time critical applications such as stock management systems which gets refreshed every minute creating huge load on web site as there are thousands of users accessing the web site at the same moment.

For such applications stress testing with more than 10,000 users is a basic test. The WAPT Pro comes with default “Load Agents” functionality to test average load on any web site. But for high capacity test with more than 10,000 users we can now leverage the power of x64 Load Engine extension.

x64 Load Engine is similar to Load Agents feature on WAPT Pro installation. The main difference is in the ability of load engine to generate huge load using 64 bit Windows systems. Using one 64 bit server you can dramatically increase the load testing capacity on web site under test. The x64 Load Engine can be configured in such a way that using slightly high end hardware you can easily generate more than 100k virtual users load on web site.

Assume that you want to stress test your web site with 100,000 concurrent virtual users. You can achieve this using 4 servers each with 25,000 virtual users. Setup this load engine on 4 systems and use them concurrently to achieve desired web site load. This powerful x64 Load Engine effectively utilizes the available memory resources on 64 bit system architecture.

WAPT Pro x64 Load Engine Installation:
- You can install x64 Load Engine on 64 bit version of Windows XP/2003/Vista/2008/Win7 OS.
- x64 Load engine works best on following hardware configuration:
Core i5/Phenom or CPU better than this, 8+ GB RAM and Gigabit Ethernet

How to install:    
x64 Load Engine is a WAPT Pro extension so it runs on WAPT Pro tool similar to Load Agents. To use this load engine you must have WAPT Pro installed first. You can download and install x64 Load Engine from below mentioned link. After installation you can start using load engine using the WAPT pro workplace itself.
Download WAPT Pro x64 Engine.

x64 Load Engine Pros:
- You can generate almost unlimited virtual users (test load) on Windows 64 system.
- Easy to install, learn and configure
Check out below screenshot to know the simplicity of x64 Load Engine Manager:

- No restrictions on the size of used virtual memory

x64 Load Engine Cons:
- No evaluation period available
- Works with WAPT Pro only

As said earlier in my comment, this tool is very useful for performance, stress & load testers. Other testers can get a glimpse of the performance testing scope which is very interesting to work on!

- No restrictions on the size of used virtual memory

x64 Load Engine Cons:
- No evaluation period available
- Works with WAPT Pro only

As said earlier in my comment, this tool is very useful for performance, stress & load testers. Other testers can get a glimpse of the performance testing scope which is very interesting to work on!

Read more »

Story of how a hang becomes a crash (because testers love reporting crashes)

I am doing my first exploratory testing workshop for 2011 and first from Moolya Software Testing Pvt Ltd. and I am excited about it. You can find the details by clicking on this link . If you are in India and plan to attend it, please do it quickly. I usually don't take a lot of participants although I know I'd gain more money out of more participants. Ever since I announced it yesterday, there are 4 seats that got filled within 5 hours.


Now, I am going to tell you a story that I had been hiding for quite a while. Not intentionally hiding but was planning to blog about it and the time is right now :)


I provide many applications to test in my workshop. At the end of a testing session, I check with the participants if they found any crashes with the application. Not to be surprised, a room full of testers, they do report crashes.


I ask them to reproduce it with me watching what they do and here is a shocking information; most of what they report as crash weren't crashes at all. What you are about to read is also a great example of confirmation bias that I have witnessed.


Here is what they are doing: They are performing an input constraint attack or providing a large set of characters as an input to a field and hits the "Submit" button. They wait for 10 seconds to see what the application does and on seeing the application in not responding state, kill the application and report that as a crash. 


Awesome! Isn't it?


Now, I got conscious of the fact that testers seem to be reporting a hang as a crash and was keen in looking at live projects during my consulting assignments. I get access to the bug tracking system during my consulting (wow) and I see patterns of such reports. 


I go filter out some of the crash reports from the bug tracking system and try to attempt what the tester did to report that type of a crash and bang, that's a hang.


I have made this point to testers who attend my workshop in order to help them be more conscious of what they are seeing versus what they are reporting versus what they are eager to report versus what they should be reporting. 


It seems to me that testers have an anxiety to report crashes. There's nothing wrong in it but it goes horribly wrong when you report a crash by fooling yourselves and the people around you. I recently blogged about the Obsessive Checking if being mentioned disorder that I was suffering from and here is a relevant excerpt from it, "Months later, I started replacing every word by my name till I saw my name on others post. So, I may have read a few posts without actually learning anything from it because all I saw is "Pradeep" & "Tester Tested" on those posts."


Now, why is that relevant to this post? I see that the testers I have witnessed who report hangs as crashes also appear to have a similar problem of obsession towards reporting crashes that they appear to not see a hang but see a crash. I would love if you ask if these testers ever report a hang? Yes, they do. If an application recovers before they kill, that's a hang.


I understand that an application might have hanged because it has crashed but how do you know? Oh yeah, allowing the GUI to fool you and me?


Here is an excerpt ( and tweaked to hide confidential information ) from an issue that I reported on Sep 30, 2010 @ 12: 13 PM IST (thank you Jira) while testing a new OS that is soon to be launched. No, not Chrome OS.


"I opened Media Player application and made an attempt to subscribe to a podcast. 
The application appeared to hang and it also did not allow me to close using the X mark on top right corner.


In an attempt to kill it, I navigated to XXXXX (product name masked) and tried closing it from there. However, the XXXX is open and Media player still seems to be remaining in the hung state.


From that point, I might be forced to reboot the system to get out of it or can escape by waiting for a longer time that I waited for Media Player to recover (>5 mins). No user to my knowledge would wait for more than 5 mins to see the Media Player function. Even a reboot appears to be a faster option" 

Now, if that was the Description, here is the summary :

"Unable to kill a XXXX when the application in it appears to hang or is busy"

• Why I posted the above? Is it because I wanted people to note the usage of the word "appears"?
• Why did I make a few words in it bold? Is it to highlight it?

Long ago, well, not so long ago, I blogged about how teaching testing is helping me test better. The above post and the bug report I posted, is another example of how teaching testers has helped me do it a lil better.


Now if you are going to be sitting at my workshop and be worried if you are going to make mistakes that I am going to highlight, please be informed that you are paying to my workshop to get into the safest environment to fail. No managers watching you. No clients frowning at you. No appraisals being done. Just your own dream of personal excellence aching you for not being there yet but being happy that you know why you are not there. Its a wonderful feeling.

Read more »

Technology that failed in India ( Testing Bowl of the world )

Once while at work I was getting my test cases reviewed about a project/domain which I was new to and one of my colleague gave a strong comment "Pradeep , please think of more cases".

I was a bit dissapointed because already there were enough test cases that it will take 3 days to complete a cycle and I thought "should I further shoot at my own legs ?" ( the more cases I write , more I have to test , its painful , isnt it ? ) .

I tried to defend that testing would become complex if more cases are there than beyond the objective of the release/product requirements.

Ha ! thanks to what I told , he tshared with me a great story which happened in his previous company and it is one of the best lessons I have had as a tester !

__ This (could be a) is True Story that happened in some company ___

A digital camera was released in the market and after sometime a customer came back complaining that .......


Problem :The camera's software was crashing continously while he tried to capture images.

FIR : They first asked some basic information from the customer and made an attempt to reproduce the issue but it was not reproducible.

Analysis :

1) Software is crashing - so they listed out all possible scenarios where the software could crash.
2) Tried putting the camera to all listed possibilities.
3) Giving a report to the management that it was unable to be reproduced.

Further Analysis :

Taking this as a challenge , a great tester had the following concern -

1) He was concerened what festival was it ?
2) He was concerned what was the customer was trying to capture ?
3) He was concerned what is the frequency of the crash the customer noticed ?
4) He was concerened about all other surrounding factors like temperuature , place where the festival was celebrated ... etc ?

Outcome of Analysis :

1) The festival was Diwali or Deepavali !
2) The customer was trying to capture fire crackers , explosion , fireworks display in an open ground at around 22 00 hours ( night ).

What a fantastic tester was he !

He identified the problem and told the developers

"The camera software is crashing when the viewfinder is black ( because of dark night ) and a sudden gush of light through the firecracker explosion is putting the DSP ( digital signal processor ) to a load/stress beyond its boundaries to display the image. And at this point of time when the user tries to capture it simply crashes"


What he said ?

was right !

Finally they fixed it and now today its a robust digital camera !

____ End of a probable true story _____

Summary of lessons -

1) Collect real time data for testing a product/application.
2) When you are unable to reproduce an issue , think out of the box , think out of the black box.
3) When you are unable to reproduce an issue call someone who can dig deeper than you.
4) Dont assign test case draft to the person who is going to test it, he may omit complicated cases to bring down his/her complexity in executing it.
5) Review the cases with a non team member/customers/domain specailist/tester network.
6) When you release a product : aniticpate and be prepared to face such situation.
7) Hate the product while testing and love the product after its release - emotional testing ( ha new concept introduced by me , i do it , to be frank )
8) Good Testers not only report bugs , they suggest a solution ! ( more to come abt this )
9) Keep learning to be a good tester !
10) Test all your products in India , its Testing Bowl of the world !
11) Dont terminate an issue just because you are unable to reproduce , it could kill your company !

Read more »

TASTRO – Tester’s Astro – What do your test signs foretell?

TASTRO – Tester’s Astro – What do your test signs foretell?

Aries		Your stars look good for coming week however you might face an environment downtime. Why not make a quick checklist on how to set it up?
Taurus		Avoid calls during Rahu Kaal. Those who have calls with your on-site coordinators during this time, Beware!!
Gemini		The planetary movements suggest that the build scheduled on week day will be delivered to the Testing team on Friday after sunset. You have plenty of time to read Rapid Software Testing Appendix and practice new testing ideas.
Cancer		Though you think you know it, you have no certainty until you try. You might see a surprise appreciation for your smart work. Smart work could mean, you use oracles to test.
Leo		You are worried with unplanned work load. Read the book – Lessons Learned in Software Testing at home and you may see the change in office.
Virgo		Worried why you your productivity appears to have come down? How long has it been since you took a break? Quick breaks between test sessions are important.
Libra		In the busy times, be prepared to work late, eat pizza for dinner at work, and work for some weekends. Don't wanna do that? Go beyond test cases, you will find more bugs.
Scorpio		There is a chance that your relationship with developers would go sour in the coming week. So treat them with chocolates. Developers are rich source of information for a tester.
Sagittarius		Your customers would be under the influence of aggressive Mars. You would be forced to test whatever is thrown at you. Check for the mission to be achieved to avoid falling into traps
Capricorn		You would be trying to achieve the stars by clicking here and there with your monkey paws. Stop doing that and your career could get better.
Aquarius		Your managers would somehow have a strong notion that you have just been marking those test cases as “pass” without executing them. Honesty is important for a good tester.
Pisces		When you have crashed the software and waiting for the system to boot, prepare your own test idea cheat sheet. For those who do, future has been bright.

Isn't this awesome? I feel testers like Rrajesh Barde are a huge boon to our industry. The beauty of my consulting was, I felt there wasn't just one Rrajesh Barde I met but many. I may cover about others in future posts.

A couple of years ago, I used to go to a consulting assignment as though I am superior and I consult people because they were inferior. These days, I go to consulting to get humbled by people like the ones I met.

Please, everybody, stretch out your creativity, you would find an Andy Glover or Rrajesh Barde in you. For those who want to follow Rrajesh's blog, here is the link. He came out with another concept called Bug Burji (Burji is a dish made out of Egg and we call it Egg Burji, Rrajesh made a Bug Burji out of it). Rrajesh, you inspire me. I hope after reading your work, a couple of others may join me in admiring your work and contribution.

I am telling myself that I was born to witness this beginning of the golden era of software testing. Don't know if you can even see what I am experiencing.

Read more »

Does Quality Assurance Remove Need for Quality Control?

“If QA (Quality Assurance) is done then why do we need to perform QC (Quality Control)?”, this thought may come to our mind some times and looks a valid point too.  This means if we have followed all the pre-defined processes, policies and standards correctly and completely then why do we need to perform a round of QC?

In my opinion QC is required after QA is done. While in ‘QA’ we define the processes, policies, strategies, establish standards, developing checklists etc. to be used and followed through out the life cycle of a project. And while in QC we follow all those defined processes, standards and policies to make sure that the project has been developed with high quality and at least meets customer’s expectations.

QA does not assure quality, rather it creates and ensures the processes are being followed to assure quality. QC does not control quality, rather it measures quality. QC measurement results can be utilized to correct/modify QA processes which can be successfully implemented in new projects as well.

Quality control activities are focused on the deliverable itself. Quality assurance activities are focused on the processes used to create the deliverable. 

QA and QC are both powerful techniques which can be used to ensure that the deliverables meet high quality expectations of customers.

E.g.: we have to use an Issue tracking system to log the bugs during testing a web application. QA would include defining the standard for adding a bug and what all details should be there in a bug, like summary of the issue, where it is observed, steps to reproduce the bugs, screenshots etc. This is a process to create deliverable ‘bug–report’. When a bug is actually added in issue tracking system based on these standards then that bug report is our deliverable.

Now, suppose some time at later stage of project we realize that adding ‘probable root cause’ to the bug based on tester’s analysis would provide some more insight to the Dev team, then we will update our pre-defined process and finally it will be reflected in our bug reports as well. This is how QC gives inputs to QA to further improve the QA.

Read more »

Google's Record Playback Framework

At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.

Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.

We were wrong, but in a good way!

We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?

During that first week of trying out RPF, Wensi found:

• 77% of the features in Webstore were testable by RPF
• Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
• The RPF scripts caught 6 functional regressions and many more intermittent server failures.
• Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
• RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
• Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
• Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.

So, we knew we were onto something interesting and continued development. In the near term though, chrome web store testing went back to using Selenium because that final 23% of features required some local Java code to handle file upload and secure checkout scenarios. In hindsight, a little testability work on the server could have solved this with some AJAX calls from the client.

We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.

Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.

RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.

Read more »