Traditional Testing

Traditional Software Testing Process - The cash cow demystified

So, I don't need to tell you that most of the software testing in the world is outsourced. I live in Bangalore and I assume a considerable amount of the world's outsourced testing projects are here. I meet testers from most of these organizations and hear the same process of testing software although called by different names their organization executives named them.

Most testers I meet lament about the process (or sucked within it for survival ) and approach they are forced to follow and try talking to their management about better ideas and better ways to test. Those executives don't seem to listen much nor willing to change.

Wondering why they wouldn't?

They have hit a cash cow.

Mark Crowther and I were discussing about Code Coverage & Requirements Based Testing in Test Republic and some how drove me to write about the way most businessmen are fooling their customers through the process they love to follow.

So, here it goes with editing and expansion. Assuming Mark from UK or maybe even you outsourced a testing project to me:

• I would spend a couple of days analyzing your requirement document and bill you for X hours per person involved in my team.
• I would spend a couple of days writing a test plan document ( but not refer to it ) and bill you for 2X hours per person involved in my team for preparing it.
• I would spend a month or two writing test case document ( and refer only to it ) and bill you for 10X hours per person involved in my team for preparing it.
• I would then again create a traceability matrix ( just to fool you and your boss about our coverage ) and bill you for 5X hours per person involved in my team for preparing it.
• So far, total of 18X hours per person involved in the team is the billing.
• Assuming X is 50 hours and there are about 10 members in my team, that's 18 * 50 * 10 = 9000 hours of billing with no single bug found yet. ( Mark wrote about it, too )
  
• If you are paying $20 an hour per person on an average, you would have actually given me a business of $180,000 without me or my team finding any bug yet.
• So after investing $1,80,000 on me and my team, you would want some benefits of that. So, you wouldn't pull the project out or move it to another vendor because more or less he would do the same and you would end up paying another $180,000
  
• Then comes the test case execution cycles for our documented 10,000 tests out of a possible hundred million tests
  
• For every new bug that you find out of the releases I make, my team would spend documenting the new test case, getting it reviewed and resulting in slower testing for you and more money for me.
• So assuming running 10,000 tests take 2 weeks for a team of 10 members to execute. Also assuming least 50 cycles of testing, you would have paid me about $140,000 for a coverage whose value might be not worth of the money.
• Of course there is additional documentation of missed test cases and other template filling activities that will be billed to you.
• To fool you further, I would instruct my team to use some expensive license based tools ( what else will I do with the money you are pumping in ) to give you a sense of faster testing ( by foolishly comparing it with human speed of testing ) and call it "Automation Testing". It turns out that these tools could have helped me find bugs that are of not a great value to you but hey, we want to see test case pass more than fail.
• Your coverage isn't improving much because we have converted manual tests to automated tests ( although its not the same test ) but to show you the speed of our tools.
  
• So think about adding another $25,000 and giving you an illusion of an ROI of $100,000 while pumping multiples of $100,000 from your bank account to mine.
  
• The CMM, TMM, Six Sigma, ISTQB scams are built around this eco-system to enable more money flow for hardly any value. Who knows there could be a cut for the people who know all this and yet do it.
  

Why wouldn't a businessman be glad about the traditional approaches to test software?

Are you asking about what happens to the users of the product?

• Lets bother about the users of your product later during our maintenance billing phase. Don't you know SDLC ends in Maintenance Phase? If we do everything right in the previous stages then how do we prove we follow SDLC when there is hardly any work in maintenance phase?
  

While you are reading this, you shouldn't be thinking of this happening only in India but in most parts of the world and even within places like United States and Europe. There are smart businessmen everywhere. At one end they pay us but at the other end use us to make more money. We need money and they need us to make that. Don't make smart businessmen exclusive to India and leave your own country out of it.

I hope those who outsource start pushing for services that doesn't fool them. Who would actually listen to this argument is testers turned businessmen and testers turned outsourcing heads and testers turned business leaders.

Whenever I visit a testing services company and see a testimonial of a customer who talks about the great ROI they got and faster testing, I wonder what a heavy price they paid to believe so.

Exploratory Testing ++ , Context Driven Testing ++ , Rapid Software Testing ++ or else YourMoney --

Don't want to be fooled by outsourcing software testing? One of those who could be of help to you among many folks I know, is myself.

Read more »

What all scenarios can be taken up for Login functionality testing?

A) As a user I want to log into the system so that I can get access to restricted functionality
•There should be a login form on the home page
•User enters email and password and clicks "Login"
•If the details are correct, they are logged in and the home page is redisplayed without the login form
•Their email address should then be shown at the top of every page "Logged in as ..."
•If either of email/password are missing or incorrect, and error message should be shown "Invalid email or password." (the same error message is used to avoid revealing information about which users exist in the database).
•If a login fails, the password field should be cleared but the email field left.
•Once logged in, the session should timeout after 30 minutes of inactivity
•Users who are either deactivated, rejected, or pending approval should not be able to log in (covered in other stories below also).
•If an account is deactivated or locked, logging in with an invalid password should display "Invalid email or password" (NOT "your account is not active")
•Attempting to access any page that requires login (while not logged in) should redirect the user to the login page. After successfully logging in they should be redirected to the page they originally requested.
(B) As a user I want to apply for access so that I can login into the system
Users who do not yet have access need to be able to request access. Allowing them to request access themselves has the benefit that they get to choose their own password so administrators do not have to deal with secure password distribution.

•There should be a "Sign Up" link shown on the home page when a user is not logged in
•It should take the user to a Sign Up form where they must enter:
•Email (must a valid email address, which will be used as their login id)
•Password (at least 6 characters, max 20, must include at least one of each of upper/lower/number/symbol)
•Password confirmation - must match password exactly
•First name
•Last name
•All are mandatory fields. Email/first/last should be limited to 255 characters, both in the form and on the server side. Password should be limited to 20.
•After correctly entering all fields and clicking "Submit Request", they should be taken back to the home page with a message "Thanks for requesting an account. You will receive an email when your request has been approved."
•At this point they should not yet be able to log in. If they try to log in they should see "Your account is not active."
•If there's any validation errors on the form, it is redisplayed with informative error messages and the account is not created. The two password fields should be cleared out each time this happens.

Read more »

Ants solve problems that testers struggle to!

Ants solve problems that testers struggle to!

Ants can go into corners of tables and chair, which we humans might not be able to.
Ants can travel to any country without a passport or Visa (if they manage to slip into someone's baggage).
Ants can live anywhere and yet not pay rent or property tax.
Ants can eat junk food forever and live happily.

Doesn't all of them sound so obvious?

So, here is something that might not be so obvious about ants... Little creatures that we commonly refer to as ants have solved a problem that many testers today struggle to.

Have you heard a tester say, "I haven't tested this before / this is too complex / I don't know how to start testing it".

The important thing to note is that we might have heard it from our own mouth or heart and sometimes from others, too.

Here are some experiments that I did with ants to see how they respond to complexity:

Experiment 1
I took an ounce of sugar, placed it near a colony of ants and observed what happened to the sugar I placed after a while.

My experiment resulted in each ant carrying a crystal of sugar and moved them to their nest. Sugar is all gone!

Experiment 2

I placed a cake of size approximately 1000 times bigger than an ant in the same place and observed what happened to the cake.

This experiment resulted in each ant breaking the huge cake into smaller fragments and moved it to their nest bit by bit. Cake is all gone!
Experiment 3

I put a piece of solidified sugar candy near the same colony, which was hard to break for an ant.

This experiment resulted in ants trying to break the candy to smaller segments, but probably the smart ants realized it to be complex than their earlier two assignments I gave them and then they all co-ordinated together and carried the candy to their nest. Candy is all gone!

I bet I could place a mammoth of food for them and they, without bothering about the complexity of it, would try moving it to their nest or create more nests and colonies to accomodate it or work a strategy to get it to their nest or much it as much as they can till the mammoth lies there...

Ben Simo, a wonderful tester you might already know, recently wrote a post titled Solving Intractable Problems and ended it beautifully by saying, "Start with what you recognize" - which I now think is what ants do instead of saying, "That food(work) is not for us, it is for bigger (skilled) ants"

After having done these experiments I was ashamed of my behavior of hesitating to test a product (which I did a couple of years ago) just because something appeared to be too complex or I thought I didn't know about the product or technology.

No, I said, *I* was ashamed and I am not going to ask if you were ashamed because you might have not done that.

After all they are ants and they don't have the super brains we have and lets ignore them folks and continue to live as happy as we are, complaining that things are complex.

Silly ants, they don't even know they are dealing with complexity. I think they should be ashamed of it.

Read more »

Interview Tips

I interviewed few candidates for my company. A few tips.

Usually interviewers form an impression on whether the candidate is good or not within 5 minutes of seeing and talking to him/her.Rest of the time, they try to validate whether their impression is correct or wrong(atleast this is what I do).So its important you make a good impression in the first few minutes.

Dress code

Well dress you wear for the interview is not everything, but you dress well, it will help you to make a initial good impression. The color of the dress gives certain impression, it seems(I don't remember paying attention to the dress, the candidate wore, but it seems it would have affected my impression at a subconscious level.But what you loose dressing up well)

• Red:You wear it to attract other people's attention.If you are presenting something or you want others to pay attention to you, dress in red. This is not the best color to wear for interview.
• Blue:If you want other people to like you, dress in blue.
• White:It gives the impression that you are hardworking.Wear this. I mean not entirely in white but the predominant color should be white.Like a black trouser and white shirt should do.

Resume

• Whatever you mention in the resume, you must try to be thorough in it.
• Be honest of what you put in it.
• Try to keep it small,it should not be longer than 2 to 3 pages.
• One of the main reason of rejection is, something is mentioned in the resume and when asked questions about it, not able to answer them and in few cases saying that "I have done that 'something' long back and I don't remember anything".In many cases its true that you have used a tool or something a year back and you don't remember much about it, in that case before attending the interview, brush up atleast some basics regarding it, do some home work and prepare yourself to talk about it or don't mention about it in the resume, if it is not that important.

Topics to prepare, for a entry level testing job.

Testing fundamentals: Testing terminology, definitions etc.

Testing Processes: The testing processes used in your current/previous company.Knowledge of life cycle testing etc.

Testing tools: Given a problem, how to solve them using testing tools.

Testing deliverables: Able to write a test plan or test case for a given product or requirements

Analytical skills: Solving Puzzles

Programming skills(optional): Able to write pseudo code for a given problem.

Few more tips

Good Communication skills: obvious isn't it

Good Listening skills or understanding skills: Able to understand the questions posed by interviewer. Try your best to understand fully what the interviewer is blabbering in the first say.But if what he says is not clear, better to ask or try to confirm what you heard is correct before answering.

Confidence level: Watch your body language, keep eye contact, don't be over confident just be confident.

Two most important tips: You should have the right skills for the job.Also equally important is you SHOW or DISPLAY in front of the interviewer that you have the right skills.

Read more »

Exploratory testing

Exploratory testing is not defined by any specific example of exploratory testing.

Just as tap dancing does not characterize ballroom dancing, you can’t take any one example of exploratory testing and treat that as representative of the entire concept of ET.

If you were to hear me singing an aria by Mozart, that would be an example of opera singing. It would be an example of BAD opera singing, but it would truly be an example of the style. Similarly, I regularly talk to testers who go “oh yeah I’ve seen that exploratory testing stuff but it’s not structured… not documented… not x… not y… not whatever.” And my reply is “you probably haven’t seen skilled exploratory testing. Would you like to hear me sing an opera now? OR, I could show you a good example of ET in practice.”

Exploratory testing can be done in an unskilled, slapdash, silly way. Just as a unskilled driver behind the wheel of a car is still a driver who is driving a car, a poor tester can still be doing ET– albeit probably not very well.

The cool thing about ET is that, even done badly, it’s still a great way to find some bugs. Michael and I try to help you do much, much better than that.

The core idea of ET remains as it always has been. It’s been expressed in many different ways, but boils down to this: test design and test execution and learning mixed together in a mutually supportive way. Whenever you see that, and to the degree that you see that, you are seeing exploratory testing.

Read more »

Boundary Value Analysis(BVA)

Boundary Value Testing is the most well-known and simple technique of test design, which helps the tester choose the most effective values ​​from the ranges of values. This technique is applicable at all levels of testing – unit, integration, system, and system-integration test levels.

We consider the steps of using of the equivalence classes technique:
1.Determining the range of values ​​(usually, the equivalence class).
2.Determination of the boundary range.
3.Creating three test cases for each boundary – one that checks the border value; second that checks the value below boundary; and the third that checks the value above boundary.

An example of the ‘upper’ and ‘lower’ boundary values. If the boundary is $ 3, the “lower” boundary will be $ 2, and the “upper” – $ 4. If the boundary is $ 2.00, the “lower” boundary will be $ 1.99, while the “upper” $ 02.01, etc.
Let’s discuss this technique on an example.
Returning to our pencils (see Equivalence Class Testing), the pens vary in price depending on the ordered quantity:
1 – 100 – $ 10. the pens;
101 – 200 – $ 9. the pens;
201 – 300 – $ 8 on the pens, etc.
With each one hundred, the price is reduced by $ 1.

The maximum available amount is 500 pcs. Boundary value analysis are very interested in the boundaries of the values intervals, because it is “analysis of the boundary.” Using the method of equivalent partitioning, the following classes were identified:
1. Valid values ​​are from 1 to 100;
2. Valid value: 101 to 200;
3. Valid value: 201 to 300;
4. Valid value: 301 to 400;
5. Valid value: from 401 to 500;
6. Invalid value:> 500 pieces;
7. Invalid value <=0;
8. Invalid value: a float from 0 to 500;
9. Invalid value: a negative number;
10. Invalid value: a collection of letters;
11. Invalid value is an empty string.
It’s time to take on boundary values ​​of these classes. You should take not only the limiting value, but also retreat one step up / down (on the tiniest step as possible). We obtain for the above written classes:
1. 1; 0; -1;
2. 99; 100; 101;
3. 199; 200; 201;
4. 299; 300; 301;
5. 399; 400; 401;
6. 499; 500; 501;
Thus, we get to 3 values ​​for the boundaries and take one value from “bodies” of the equivalence classes for a total of 21 + 11 = we take 32 values for verifying. M-m-m-m-m it is more than 11, but much less than 500. Yes, and we check not only the valid values, but also invalid! But it’s only two techniques of … to be continued …

Read more »

Letter to myself

I have been with you all this while and shall continue to be with you. I am enjoying each moment you enjoy as a tester and also each moment that you don't enjoy as a tester. I am sorry for enjoying those moments which you think you haven't been enjoying. I am sure you'd be interested to know why and how I have been enjoying those moments that you haven't been.

First, let me list the thing that you haven't been enjoying:

• Whenever and wherever you write about test automation, a handful of readers tend to think that you reject the idea of test automation and they write harsh e-mails to you.
• Whenever and wherever you write about certification, the certified tester community attacks you over e-mails/phone calls that you and I are spoiling the craft.
• Whenever and wherever you write about programming skill for testers, another handful readers think that you are suggesting testers not to learn programming.
• Whenever and wherever you write about tools, most testers think you are referring to test automation tool.
• Whenever and wherever you write about yourself or your experience, a set of people think you lack humility.
• Whenever and wherever you write about exploratory testing as a skilled activity, a set of people think "no tester would do be able to do that".
• Whenever and wherever you write about ideas to solve a problem than giving a one line answer, a set of people think you don't know how to solve it and is faking what you know.
• Whenever and wherever you are writing about testing being an excellent thinking job, a few people think you are trying to paint a picture that does not exist.
• Whenever and wherever you - do bad testing, fail in testing course like BBST, you feel intimidated by more skilled people than you, you feel bad about not having learned or practiced those things that helps you become a better tester, you fail to give enough respect to expert testers time, etc...

In this context, I'd like to remind you of a learning you had from Michael Bolton: There are some things under your control and there are other things that are not under your control. Taking advantage of things under your control, as a tester, is essential to clear traps and it might also lead to gaining more control. To take advantage of things under your control, you first should realize what are the things you control.

I also remember that you had made a note in your Moleskine of Saurav Ganguly's television interview where he was asked: How were you able to make a great comeback to the world cup cricket squad after being axed for poor performance? His answer: I didn't worry about things that are not under my control ( media critique, jokes on bad batting performance, e-mail forwards about my performance, people gossiping about it ) but focussed on things that are under my control ( Practice, skill enhancement, consistent batting record in Ranji trophy)

Similarly, you don't have a control over the thoughts of people thinking whatever they think after reading whatever you have written. You have a control over what you write and you have a control over the way you write it.

Your testing has been influenced by a lot of experts but not all have similar influence. They haven't seen great testing to appreciate the things you are sharing and I doubt if all those who witness it would be influenced by it because it's hard work and high skill demanding.

• Not all testers want to do great testing
• Not all testers know they are doing bad testing.
• Not all testers want to know they are doing bad testing.
• Not all testers want to know more about testing.
• Not all testers know what skills to gain and practice.
• Not all testers agree to be context driven.

Here are three questions ( like the Monty Python and the Holy grail bridge of death piece you enjoy )


1.Whom are you serving through your writing?
I am sure your ongoing struggle is in understanding that. Let me help you with what I think about whom you are serving - You are serving those testers who look for better thought process and those who enjoy the better thought process and those who think you have a better thought process.
2. Who asked you to serve them?
I asked you to do that!
3. Why haven't you been enjoying some moments that I have been?
You want all testers to do great testing although you know its not possible. Some people question your idea of "great testing" because they already have an idea of "great testing" and it conflicts with the idea you have. You are able to demonstrate that their idea of "great testing" lacks critical thought as your idea of "great testing".
By the way, your idea of "great testing", is not yours but of those people who have influenced you. You have just subscribe to those ideas and are contributing to it in different forms. I have occasionally witnessed you doing bad testing and I am sure I would see that in future, too. Do not forget that you are a human and your ideas are fallible. I know bad testing and bad thought process irritates you, even if you are the one who is doing it.
I would love to see you doing things that are under your control - learning, reading, writing, bettering your skills, helping those testers who enjoy the thought process that you enjoy, speaking, coaching and mentoring.
Your power to influence testers is limited. Limited to the ones who don't want to limit themselves. So do unlimited things under limited time that you and I will be here in this world.
I will be with you forever, enjoying everything you do from great things to not so great things. Anything you do is great to me.
I will write to you whenever I feel a need for it. This letter is personal, just between you and me.

"Here is a way to test if your mission on earth is complete - if you are alive, it isn't" -- Richard Bach

Yours truly,



Pradeep Soundararajan - http://testertested.blogspot.com - +91-98451-76817 - pradeep.srajan@gmail.com



"The test doesn't find the bug. A human finds the bug, and the test plays a role in helping the human find it."

Read more »