Performance testing tool

This is in fact most crucial test for time critical applications such as stock management systems which gets refreshed every minute creating huge load on web site as there are thousands of users accessing the web site at the same moment.

For such applications stress testing with more than 10,000 users is a basic test. The WAPT Pro comes with default “Load Agents” functionality to test average load on any web site. But for high capacity test with more than 10,000 users we can now leverage the power of x64 Load Engine extension.

x64 Load Engine is similar to Load Agents feature on WAPT Pro installation. The main difference is in the ability of load engine to generate huge load using 64 bit Windows systems. Using one 64 bit server you can dramatically increase the load testing capacity on web site under test. The x64 Load Engine can be configured in such a way that using slightly high end hardware you can easily generate more than 100k virtual users load on web site.

Assume that you want to stress test your web site with 100,000 concurrent virtual users. You can achieve this using 4 servers each with 25,000 virtual users. Setup this load engine on 4 systems and use them concurrently to achieve desired web site load. This powerful x64 Load Engine effectively utilizes the available memory resources on 64 bit system architecture.

WAPT Pro x64 Load Engine Installation:
- You can install x64 Load Engine on 64 bit version of Windows XP/2003/Vista/2008/Win7 OS.
- x64 Load engine works best on following hardware configuration:
Core i5/Phenom or CPU better than this, 8+ GB RAM and Gigabit Ethernet

How to install:    
x64 Load Engine is a WAPT Pro extension so it runs on WAPT Pro tool similar to Load Agents. To use this load engine you must have WAPT Pro installed first. You can download and install x64 Load Engine from below mentioned link. After installation you can start using load engine using the WAPT pro workplace itself.
Download WAPT Pro x64 Engine .

x64 Load Engine Pros:
- You can generate almost unlimited virtual users (test load) on Windows 64 system.
- Easy to install, learn and configure
Check out below screenshot to know the simplicity of x64 Load Engine Manager:

This is in fact most crucial test for time critical applications such as stock management systems which gets refreshed every minute creating huge load on web site as there are thousands of users accessing the web site at the same moment.

For such applications stress testing with more than 10,000 users is a basic test. The WAPT Pro comes with default “Load Agents” functionality to test average load on any web site. But for high capacity test with more than 10,000 users we can now leverage the power of x64 Load Engine extension.

x64 Load Engine is similar to Load Agents feature on WAPT Pro installation. The main difference is in the ability of load engine to generate huge load using 64 bit Windows systems. Using one 64 bit server you can dramatically increase the load testing capacity on web site under test. The x64 Load Engine can be configured in such a way that using slightly high end hardware you can easily generate more than 100k virtual users load on web site.

Assume that you want to stress test your web site with 100,000 concurrent virtual users. You can achieve this using 4 servers each with 25,000 virtual users. Setup this load engine on 4 systems and use them concurrently to achieve desired web site load. This powerful x64 Load Engine effectively utilizes the available memory resources on 64 bit system architecture.

WAPT Pro x64 Load Engine Installation:
- You can install x64 Load Engine on 64 bit version of Windows XP/2003/Vista/2008/Win7 OS.
- x64 Load engine works best on following hardware configuration:
Core i5/Phenom or CPU better than this, 8+ GB RAM and Gigabit Ethernet

How to install:    
x64 Load Engine is a WAPT Pro extension so it runs on WAPT Pro tool similar to Load Agents. To use this load engine you must have WAPT Pro installed first. You can download and install x64 Load Engine from below mentioned link. After installation you can start using load engine using the WAPT pro workplace itself.
Download WAPT Pro x64 Engine.

x64 Load Engine Pros:
- You can generate almost unlimited virtual users (test load) on Windows 64 system.
- Easy to install, learn and configure
Check out below screenshot to know the simplicity of x64 Load Engine Manager:

- No restrictions on the size of used virtual memory

x64 Load Engine Cons:
- No evaluation period available
- Works with WAPT Pro only

As said earlier in my comment, this tool is very useful for performance, stress & load testers. Other testers can get a glimpse of the performance testing scope which is very interesting to work on!

- No restrictions on the size of used virtual memory

x64 Load Engine Cons:
- No evaluation period available
- Works with WAPT Pro only

As said earlier in my comment, this tool is very useful for performance, stress & load testers. Other testers can get a glimpse of the performance testing scope which is very interesting to work on!

Read more »

Story of how a hang becomes a crash (because testers love reporting crashes)

I am doing my first exploratory testing workshop for 2011 and first from Moolya Software Testing Pvt Ltd. and I am excited about it. You can find the details by clicking on this link . If you are in India and plan to attend it, please do it quickly. I usually don't take a lot of participants although I know I'd gain more money out of more participants. Ever since I announced it yesterday, there are 4 seats that got filled within 5 hours.


Now, I am going to tell you a story that I had been hiding for quite a while. Not intentionally hiding but was planning to blog about it and the time is right now :)


I provide many applications to test in my workshop. At the end of a testing session, I check with the participants if they found any crashes with the application. Not to be surprised, a room full of testers, they do report crashes.


I ask them to reproduce it with me watching what they do and here is a shocking information; most of what they report as crash weren't crashes at all. What you are about to read is also a great example of confirmation bias that I have witnessed.


Here is what they are doing: They are performing an input constraint attack or providing a large set of characters as an input to a field and hits the "Submit" button. They wait for 10 seconds to see what the application does and on seeing the application in not responding state, kill the application and report that as a crash. 


Awesome! Isn't it?


Now, I got conscious of the fact that testers seem to be reporting a hang as a crash and was keen in looking at live projects during my consulting assignments. I get access to the bug tracking system during my consulting (wow) and I see patterns of such reports. 


I go filter out some of the crash reports from the bug tracking system and try to attempt what the tester did to report that type of a crash and bang, that's a hang.


I have made this point to testers who attend my workshop in order to help them be more conscious of what they are seeing versus what they are reporting versus what they are eager to report versus what they should be reporting. 


It seems to me that testers have an anxiety to report crashes. There's nothing wrong in it but it goes horribly wrong when you report a crash by fooling yourselves and the people around you. I recently blogged about the Obsessive Checking if being mentioned disorder that I was suffering from and here is a relevant excerpt from it, "Months later, I started replacing every word by my name till I saw my name on others post. So, I may have read a few posts without actually learning anything from it because all I saw is "Pradeep" & "Tester Tested" on those posts."


Now, why is that relevant to this post? I see that the testers I have witnessed who report hangs as crashes also appear to have a similar problem of obsession towards reporting crashes that they appear to not see a hang but see a crash. I would love if you ask if these testers ever report a hang? Yes, they do. If an application recovers before they kill, that's a hang.


I understand that an application might have hanged because it has crashed but how do you know? Oh yeah, allowing the GUI to fool you and me?


Here is an excerpt ( and tweaked to hide confidential information ) from an issue that I reported on Sep 30, 2010 @ 12: 13 PM IST (thank you Jira) while testing a new OS that is soon to be launched. No, not Chrome OS.


"I opened Media Player application and made an attempt to subscribe to a podcast. 
The application appeared to hang and it also did not allow me to close using the X mark on top right corner.


In an attempt to kill it, I navigated to XXXXX (product name masked) and tried closing it from there. However, the XXXX is open and Media player still seems to be remaining in the hung state.


From that point, I might be forced to reboot the system to get out of it or can escape by waiting for a longer time that I waited for Media Player to recover (>5 mins). No user to my knowledge would wait for more than 5 mins to see the Media Player function. Even a reboot appears to be a faster option" 

Now, if that was the Description, here is the summary :

"Unable to kill a XXXX when the application in it appears to hang or is busy"

• Why I posted the above? Is it because I wanted people to note the usage of the word "appears"?
• Why did I make a few words in it bold? Is it to highlight it?

Long ago, well, not so long ago, I blogged about how teaching testing is helping me test better. The above post and the bug report I posted, is another example of how teaching testers has helped me do it a lil better.


Now if you are going to be sitting at my workshop and be worried if you are going to make mistakes that I am going to highlight, please be informed that you are paying to my workshop to get into the safest environment to fail. No managers watching you. No clients frowning at you. No appraisals being done. Just your own dream of personal excellence aching you for not being there yet but being happy that you know why you are not there. Its a wonderful feeling.

Read more »

Technology that failed in India ( Testing Bowl of the world )

Once while at work I was getting my test cases reviewed about a project/domain which I was new to and one of my colleague gave a strong comment "Pradeep , please think of more cases".

I was a bit dissapointed because already there were enough test cases that it will take 3 days to complete a cycle and I thought "should I further shoot at my own legs ?" ( the more cases I write , more I have to test , its painful , isnt it ? ) .

I tried to defend that testing would become complex if more cases are there than beyond the objective of the release/product requirements.

Ha ! thanks to what I told , he tshared with me a great story which happened in his previous company and it is one of the best lessons I have had as a tester !

__ This (could be a) is True Story that happened in some company ___

A digital camera was released in the market and after sometime a customer came back complaining that .......


Problem :The camera's software was crashing continously while he tried to capture images.

FIR : They first asked some basic information from the customer and made an attempt to reproduce the issue but it was not reproducible.

Analysis :

1) Software is crashing - so they listed out all possible scenarios where the software could crash.
2) Tried putting the camera to all listed possibilities.
3) Giving a report to the management that it was unable to be reproduced.

Further Analysis :

Taking this as a challenge , a great tester had the following concern -

1) He was concerened what festival was it ?
2) He was concerned what was the customer was trying to capture ?
3) He was concerned what is the frequency of the crash the customer noticed ?
4) He was concerened about all other surrounding factors like temperuature , place where the festival was celebrated ... etc ?

Outcome of Analysis :

1) The festival was Diwali or Deepavali !
2) The customer was trying to capture fire crackers , explosion , fireworks display in an open ground at around 22 00 hours ( night ).

What a fantastic tester was he !

He identified the problem and told the developers

"The camera software is crashing when the viewfinder is black ( because of dark night ) and a sudden gush of light through the firecracker explosion is putting the DSP ( digital signal processor ) to a load/stress beyond its boundaries to display the image. And at this point of time when the user tries to capture it simply crashes"


What he said ?

was right !

Finally they fixed it and now today its a robust digital camera !

____ End of a probable true story _____

Summary of lessons -

1) Collect real time data for testing a product/application.
2) When you are unable to reproduce an issue , think out of the box , think out of the black box.
3) When you are unable to reproduce an issue call someone who can dig deeper than you.
4) Dont assign test case draft to the person who is going to test it, he may omit complicated cases to bring down his/her complexity in executing it.
5) Review the cases with a non team member/customers/domain specailist/tester network.
6) When you release a product : aniticpate and be prepared to face such situation.
7) Hate the product while testing and love the product after its release - emotional testing ( ha new concept introduced by me , i do it , to be frank )
8) Good Testers not only report bugs , they suggest a solution ! ( more to come abt this )
9) Keep learning to be a good tester !
10) Test all your products in India , its Testing Bowl of the world !
11) Dont terminate an issue just because you are unable to reproduce , it could kill your company !

Read more »

TASTRO – Tester’s Astro – What do your test signs foretell?

TASTRO – Tester’s Astro – What do your test signs foretell?

Aries		Your stars look good for coming week however you might face an environment downtime. Why not make a quick checklist on how to set it up?
Taurus		Avoid calls during Rahu Kaal. Those who have calls with your on-site coordinators during this time, Beware!!
Gemini		The planetary movements suggest that the build scheduled on week day will be delivered to the Testing team on Friday after sunset. You have plenty of time to read Rapid Software Testing Appendix and practice new testing ideas.
Cancer		Though you think you know it, you have no certainty until you try. You might see a surprise appreciation for your smart work. Smart work could mean, you use oracles to test.
Leo		You are worried with unplanned work load. Read the book – Lessons Learned in Software Testing at home and you may see the change in office.
Virgo		Worried why you your productivity appears to have come down? How long has it been since you took a break? Quick breaks between test sessions are important.
Libra		In the busy times, be prepared to work late, eat pizza for dinner at work, and work for some weekends. Don't wanna do that? Go beyond test cases, you will find more bugs.
Scorpio		There is a chance that your relationship with developers would go sour in the coming week. So treat them with chocolates. Developers are rich source of information for a tester.
Sagittarius		Your customers would be under the influence of aggressive Mars. You would be forced to test whatever is thrown at you. Check for the mission to be achieved to avoid falling into traps
Capricorn		You would be trying to achieve the stars by clicking here and there with your monkey paws. Stop doing that and your career could get better.
Aquarius		Your managers would somehow have a strong notion that you have just been marking those test cases as “pass” without executing them. Honesty is important for a good tester.
Pisces		When you have crashed the software and waiting for the system to boot, prepare your own test idea cheat sheet. For those who do, future has been bright.

Isn't this awesome? I feel testers like Rrajesh Barde are a huge boon to our industry. The beauty of my consulting was, I felt there wasn't just one Rrajesh Barde I met but many. I may cover about others in future posts.

A couple of years ago, I used to go to a consulting assignment as though I am superior and I consult people because they were inferior. These days, I go to consulting to get humbled by people like the ones I met.

Please, everybody, stretch out your creativity, you would find an Andy Glover or Rrajesh Barde in you. For those who want to follow Rrajesh's blog, here is the link. He came out with another concept called Bug Burji (Burji is a dish made out of Egg and we call it Egg Burji, Rrajesh made a Bug Burji out of it). Rrajesh, you inspire me. I hope after reading your work, a couple of others may join me in admiring your work and contribution.

I am telling myself that I was born to witness this beginning of the golden era of software testing. Don't know if you can even see what I am experiencing.

Read more »

Does Quality Assurance Remove Need for Quality Control?

“If QA (Quality Assurance) is done then why do we need to perform QC (Quality Control)?”, this thought may come to our mind some times and looks a valid point too.  This means if we have followed all the pre-defined processes, policies and standards correctly and completely then why do we need to perform a round of QC?

In my opinion QC is required after QA is done. While in ‘QA’ we define the processes, policies, strategies, establish standards, developing checklists etc. to be used and followed through out the life cycle of a project. And while in QC we follow all those defined processes, standards and policies to make sure that the project has been developed with high quality and at least meets customer’s expectations.

QA does not assure quality, rather it creates and ensures the processes are being followed to assure quality. QC does not control quality, rather it measures quality. QC measurement results can be utilized to correct/modify QA processes which can be successfully implemented in new projects as well.

Quality control activities are focused on the deliverable itself. Quality assurance activities are focused on the processes used to create the deliverable. 

QA and QC are both powerful techniques which can be used to ensure that the deliverables meet high quality expectations of customers.

E.g.: we have to use an Issue tracking system to log the bugs during testing a web application. QA would include defining the standard for adding a bug and what all details should be there in a bug, like summary of the issue, where it is observed, steps to reproduce the bugs, screenshots etc. This is a process to create deliverable ‘bug–report’. When a bug is actually added in issue tracking system based on these standards then that bug report is our deliverable.

Now, suppose some time at later stage of project we realize that adding ‘probable root cause’ to the bug based on tester’s analysis would provide some more insight to the Dev team, then we will update our pre-defined process and finally it will be reflected in our bug reports as well. This is how QC gives inputs to QA to further improve the QA.

Read more »

Google's Record Playback Framework

At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.

Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.

We were wrong, but in a good way!

We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?

During that first week of trying out RPF, Wensi found:

• 77% of the features in Webstore were testable by RPF
• Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
• The RPF scripts caught 6 functional regressions and many more intermittent server failures.
• Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
• RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
• Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
• Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.

So, we knew we were onto something interesting and continued development. In the near term though, chrome web store testing went back to using Selenium because that final 23% of features required some local Java code to handle file upload and secure checkout scenarios. In hindsight, a little testability work on the server could have solved this with some AJAX calls from the client.

We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.

Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.

RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.

Read more »

From the mailbox: selecting test automation tools

A long time ago, all the way back in 1999, I wrote an article on selecting GUI test automation tools. Someone recently found it and wrote me an email to ask about getting help with evaluating tools. I decided my response might be useful for other people trying to choose tools, so I turned it into a blog post.

By the way, so much has changed since my article on GUI testing tools was published back in 1999 that my approach is a little different these days. There are so many options available now that weren’t 12 years ago, and there are new options appearing nearly every day it seems.

Back in 1999 I advocated a heavy-weight evaluation process. I helped companies evaluate commercial tools, and at the time it made sense to spend lots of time and money on the evaluation process. The cost of making a mistake in tool selection was too high.

After all, once we chose a tool we would have to pay for it, and that licensing fee became a sunk cost. Further, the cost of switching between tools was exorbitant. Tests were tool-specific and could not move from one tool to another. Thus we’d have to throw away anything we created in Tool A if we later decided to adopt Tool B. Further, any new tool would cost even more money in licensing fees. So spending a month evaluating tools before making a 6-figure investment made sense.

But now the market has changed. Open source tools are surpassing commercial tools, so the license fee is less of an issue. There are still commercial tools, but I always recommend looking at the open source tools first to see if there’s anything that fits before diving into commercial tool evaluations.

So here’s my quick and dirty guide to test tool selection.

If you want a tool to do functional test automation (as opposed to unit testing), you will probably need both a framework and a driver.

• The framework is responsible for defining the format of the tests, making the connection between the tests and test automation code, executing the tests, and reporting results.
• The driver is responsible for manipulating the interface.

So, for example, on my side project entaggle.com, I use Cucumber (framework) with Capybara (driver).

To decide what combination of framework(s) and driver(s) are right for your context…

Step 1. Identify possible frameworks…

Consideration #1: Test Format

The first thing to consider is if you need a framework that supports expressing tests in a natural language (e.g. English), or in code.

This is a question for the whole team, not just the testers or programmers. Everyone on the project must be able to at least read the functional tests. Done well, the tests can become executable requirements. So the functional testing framework needs to support test formats that work for collaboration across the whole team.

Instead of assuming what the various stakeholders want to see, ask them.

In particular, if you are contemplating expressing tests in code, make very sure to ask the business stakeholders how they feel about that. And I don’t mean ask them like, “Hey, you don’t mind the occasional semi-colon, right? It’s no big deal, right? I mean, you’re SMART ENOUGH to read CODE, right?” That kind of questioning backs the business stakeholders into a corner. They might say, “OK,” but it’s only because they’ve been bullied.

I mean mock up some samples and ask like this: “Hey, here’s an example of some tests for our system written in a framework we’re considering using. Can you read this? What do you think it’s testing?” If they are comfortable with the tests, the format is probably going to work. If not, consider other frameworks.

Note that the reason that it’s useful to express expectations in English isn’t to dumb down the tests. This isn’t about making it possible for non-technical people to do all the automation.

Even with frameworks that express tests in natural language, There is still programming involved. Test automation is still inherently about programming.

But by separating the essence of the tests from the test support code, we’re able to separate the concerns in a way that makes it easier to collaborate on the tests, and further the tests become more maintainable and reusable.

When I explain all that, people sometimes ask me, “OK, that’s fine, but what’s the EASIEST test automation tool to learn?” Usually they’re thinking that “easy” is synonymous with “record and playback.”

Such kinds of easy paths may look inviting, but it’s a trap leads into a deep dark swamp from which there may be no escape. None of the tools I’ve talked about do record and playback. Yes, there is a Selenium recorder. I do not recommend using it except as a way to learn.

So natural language tests facilitate collaboration. But I’ve seen organizations write acceptance tests in Java with JUnit using Selenium as the driver and still get a high degree of collaboration. The important thing is the collaboration, not the test format.

In fact, there are advantages to expressing tests in code.

Using the same unit testing framework for the functional tests and the code-facing tests removes one layer of abstraction. That can reduce the complexity of the tests and make it easier for the technical folks to create and update the tests.

But the times I have seen this work well for the organization is when the business people were all technology savvy so they were able to read the tests just fine even when expressed in Java rather than English.

Consideration #2: Programming Language

The next consideration is the production code language.

If your production code is written in…	And you want to express expectations in natural language, consider…	Or you want to express expectations in code, consider…
Java	Robot Framework, JBehave, Fitnesse, Concordion	JUnit, TestNG
Ruby	Cucumber	Test::Unit, RSpec
.NET	Specflow	NUnit

By the way, the tools I’ve mentioned so far are not even remotely close to a comprehensive list. There are lots more tools listed on the AA-FTT spreadsheet. (The AA-FTT is the Agile Alliance Functional Testing Tools group. It’s a program of the Agile Alliance. The spreadsheet came out of work that the AA-FTT community did. If you need help interpreting the spreadsheet, you can ask questions about it on the AA-FTT mail list.)

So, why consider the language that the production code is written in? I advocate choosing a tool that will allow you to write the test automation code in the same language (or at least one of the same languages if there are several) as the production code for a number of reasons:

1. The programmers will already know the language. This is a huge boon for getting the programmers to collaborate on functional test automation.
2. It’s probably a real programming language with a real IDE that supports automated refactoring and other kinds of good programming groovy-ness. It’s critical to treat test automation code with the same level of care as production code. Test automation code should be well factored to increase maintainability, remove duplication, and exhibit SOLID principles.
3. It increases the probability that you’ll be able to bypass the GUI for setting up conditions and data. You may even be able to leverage test helper code from the unit tests. For example, on entaggle.com, I have some data generation code that is shared between the unit tests and the acceptance tests. Such reuse drastically cuts down on the cost of creating and maintaining automated tests.

Consideration #3: The Ecosystem

Finally, as you are considering frameworks, consider also the ecosystem in which that framework will live. I personally dismiss any test framework that does not play nicely with both the source control system and the automated build process or continuous integration server. That means at a bare minimum:

• All assets must be flat files, no binaries. So no assets stored in databases, and no XLS spreadsheets (though comma separated values or .CSV files can be OK). In short, if you can’t read all the assets in a plain old text editor like Notepad, you’re going to run into problems with versioning.
• It can execute from a command line and return an exit code of 0 if everything passes or some other number if there’s a failure. (You may need more than this to kick off the tests from the automated build and report results, but the exit code criteria is absolutely critical.)

Step 2. Choose your driver(s)…

A driver is just a library that knows how to manipulate the interface you’re testing against. You may actually need more than one driver depending on the interfaces in the system you’re testing. You might need one driver to handle web stuff while another driver can manipulate Windows apps.

Note that the awesome thing about the way test tools work these days is that you can use multiple drivers with any given functional testing framework. In fact, you can use multiple drivers all in a single test. Or you can have a test that executes against multiple interfaces. Not a copy of the test, but actually the same test. By separating concerns, separating the framework from the driver, we make it possible for tests to be completely driver agnostic.

Choosing drivers is often a matter of just finding the most popular driver for your particular technical context. It’s hard for me to offer advice on which drivers are good because there are so many more drivers available than I know about. Most of the work I do these days is web-based. So I use Selenium / WebDriver.

To find a specific driver for a specific kind of interface, look at the tools spreadsheet or ask on the AA-FTT mail list.

Step 3. Experiment

Don’t worry about choosing The One Right tool. Choose something that fits your basic criteria and see how it works in practice. These days it’s so much less costly to experiment and see how things go working with the tool on real stuff than to do an extensive tool evaluation.

How can this possibly be? First, lots of organizations are figuring out that the licensing costs are no longer an issue. Open source tools rule. Better yet, if you go with a tool that lets you express tests in natural language it’s really not that hard to convert tests from one framework to another. I converted a small set of Robot Framework tests to Cucumber and it took me almost no time to convert the tests themselves. The formats were remarkably similar. The test automation code took a little longer, but there was less of it.

Given that the cost of making a mistake on tool choice is so low, I recommend experimenting freely. Try a tool for a couple weeks on real tests for your real project. If it works well for the team, awesome. If not, try a different one.

But whatever you do, don’t spend a month (or more) in meetings speculating about what tools will work. Just pick something to start with so you can try and see right away. (As you all know, empirical evidence trumps speculation. 

Eventually, if you are in a larger organization, you might find that a proliferation of testing frameworks becomes a problem. It may be necessary to reduce the number of technologies that have to be supported and make reporting consistent across teams.

But beware premature standardization. Back in 1999, choosing a single tool gave large organizations an economy of scale. They could negotiate better deals on licenses and run everyone through the same training classes. Such economies of scale are evaporating in the open source world where license deals are irrelevant and training is much more likely to be informal and community-based.

So even in a large organization I advocate experimenting extensively before standardizing.

Also, it’s worth noting that while I can see a need to standardize on a testing framework, I see much less need to standardize on drivers. So be careful about what aspects of the test automation ecosystem you standardize on.

Read more »

Static Methods

Static Methods

Static methods are procedural in nature and they have no place in OO world. I can already hear the screams, so let me explain why, but first we need to agree that global variables and state is evil. If you agree with previous statement than for a static method to do something interesting it needs to have some arguments, otherwise it will always return a constant. Call to a staticMethod() must always return the same thing, if there is no global state. (Time and random, has global state, so that does not count and object instantiation may have different instance but the object graph will be wired the same way.)

This means that for a static method to do something interesting it needs to have arguments. But in that case I will argue that the method simply belongs on one of its arguments. Example: Math.abs(-3) should really be -3.abs(). Now that does not imply that -3 needs to be object, only that the compiler needs to do the magic on my behalf, which BTW, Ruby got right. If you have multiple arguments you should choose the argument with which method interacts the most.

But most justifications for static methods argue that they are "utility methods". Let's say that you want to have toCamelCase() method to convert string "my_workspace" to "myWorkspace". Most developers will solve this as StringUtil.toCamelCase("my_workspace"). But, again, I am going to argue that the method simply belongs to the String class and should be "my_workspace".toCamelCase(). But we can't extend the String class in Java, so we are stuck, but in many other OO languages you can add methods to existing classes.

In the end I am sometimes (handful of times per year) forced to write static methods due to limitation of the language. But that is a rare event since static methods are death to testability. What I do find, is that in most projects static methods are rampant.

Instance Methods

So you got rid of all of your static methods but your codes still is procedural. OO says that code and data live together. So when one looks at code one can judge how OO it is without understanding what the code does, simply by looking at the relationship of data and code.

class Database {
 // some fields declared here
 boolean isDirty(Cache cache, Object obj) {
   for (Object cachedObj : cache.getObjects) {
     if (cachedObj.equals(obj))
       return false;
   }
   return true;
 }
}

The problem here is the method may as well be static! It is in the wrong place, and you can tell this because it does not interact with any of the data in the Database, instead it interacts with the data in cache which it fetches by calling the getObjects() method. My guess is that this method belongs to one of its arguments most likely Cache. If you move it to Cache you well notice that the Cache will no longer need the getObjects() method since the for loop can access the internal state of the Cache directly. Hey, we simplified the code (moved one method, deleted one method) and we have made Demeter happy.

The funny thing about the getter methods is that it usually means that the code where the data is processed is outside of the class which has the data. In other words the code and data are not together.

class Authenticator {
 Ldap ldap;
 Cookie login(User user) {
   if (user.isSuperUser()) {
     if ( ldap.auth(user.getUser(),
            user.getPassword()) )
       return new Cookie(user.getActingAsUser());
   } else (user.isAgent) {
       return new Cookie(user.getActingAsUser());
   } else {
     if ( ldap.auth(user.getUser(),
            user.getPassword()) )
       return new Cookie(user.getUser());
   }
   return null;
 }
}

Now I don't know if this code is well written or not, but I do know that the login() method has a very high affinity to user. It interacts with the user a lot more than it interacts with its own state. Except it does not interact with user, it uses it as a dumb storage for data. Again, code lives with data is being violated. I believe that the method should be on the object with which it interacts the most, in this case on User. So lets have a look:

class User {
 String user;
 String password;
 boolean isAgent;
 boolean isSuperUser;
 String actingAsUser;

 Cookie login(Ldap ldap) {
   if (isSuperUser) {
     if ( ldap.auth(user, password) )
       return new Cookie(actingAsUser);
   } else (user.isAgent) {
       return new Cookie(actingAsUser);
   } else {
     if ( ldap.auth(user, password) )
       return new Cookie(user);
   }
   return null;
 }
}

Ok we are making progress, notice how the need for all of the getters has disappeared, (and in this simplified example the need for the Authenticator class disappears) but there is still something wrong. The ifs branch on internal state of the object. My guess is that this code-base is riddled with if (user.isSuperUser()). The issue is that if you add a new flag you have to remember to change all of the ifs which are dispersed all over the code-base. Whenever I see If or switch on a flag I can almost always know that polymorphism is in order.

class User {
 String user;
 String password;

 Cookie login(Ldap ldap) {
   if ( ldap.auth(user, password) )
     return new Cookie(user);
   return null;
 }
}

class SuperUser extends User {
 String actingAsUser;

 Cookie login(Ldap ldap) {
   if ( ldap.auth(user, password) )
     return new Cookie(actingAsUser);
   return null;
 }
}

class AgentUser extends User {
 String actingAsUser;

 Cookie login(Ldap ldap) {
   return new Cookie(actingAsUser);
 }
}

Now that we took advantage of polymorphism, each different kind of user knows how to log in and we can easily add new kind of user type to the system. Also notice how the user no longer has all of the flag fields which were controlling the ifs to give the user different behavior. The ifs and flags have disappeared.

Now this begs the question: should the User know about the Ldap? There are actually two questions in there. 1) should User have a field reference to Ldap? and 2) should User have compile time dependency on Ldap?

Should User have a field reference to Ldap? The answer is no, because you may want to serialize the user to database but you don't want to serialize the Ldap. See here.

Should User have compile time dependency on Ldap? This is more complicated, but in general the answer depends on weather or not you are planning on reusing the User on a different project, since compile time dependencies are transitive in strongly typed languages. My experience is that everyone always writes code that one day they will reuse it, but that day never comes, and when it does, usually the code is entangled in other ways anyway, so code reuse after the fact just does not happen. (developing a library is different since code reuse is an explicit goal.) My point is that a lot of people pay the price of "what if" but never get any benefit out of it. Therefore don't worry abut it and make the User depend on Ldap. Permalink | Links to this post | 19 comments  

The Advantages of Unit Testing Early

Wednesday, July 15, 2009 8:33 PM

by Shyam Seshadri

Nowadays, when I talk with (read: rant at) anyone about why they should do test driven development or write unit tests, my spiel has gotten extremely similar and redundant to the point that I don't have to think about it anymore. But even when I do pairing with skeptics, even as I cajole and coax testable code or some specific refactorings out of them, I wonder, why is it that I have to convince you of the worth of testing ? Shouldn't it be obvious ?

And sadly, it isn't. Not to many people. To many people, I come advocating the rise of the devil itself. To others, it is this redundant, totally useless thing that is covered by the manual testers anyway. The general opinion seems to be, "I'm a software engineer. It is my job to write software. Nowhere in the job description does it say that I have to write these unit tests." Well, to be fair, I haven't heard that too many times, but they might as well be thinking it, given their investment in writing unit tests. And last time I checked, an engineer's role is to deliver a working software. How do you even prove that your software works without having some unit tests to back you up ? Do you pull it up and go through it step by step, and start cursing when it breaks ? Because without unit tests, the odds are that it will.

But writing unit tests as you develop isn't just to prove that your code works (though that is a great portion of it). There are so many more benefits to writing unit tests. Lets talk in depth about a few of these below.

Instantaneous Gratification

The biggest and most obvious reason for writing unit tests (either as you go along, or before you even write code) is instantaneous gratification. When I write code (write, not spike. That is a whole different ball game that I won't get into now), I love to know that it works and does what it should do. If you are writing a smaller component of a bigger app (especially one that isn't complete yet), how are you even supposed to know if what you just painstakingly wrote even works or not ? Even the best engineers make mistakes.

Whereas with unit tests, I can write my code. Then just hit my shortcut keys to run my tests, and voila, within a second or two, I have the results, telling me that everything passed (in the ideal case) or what failed and at which line, so I know exactly what I need to work on. It just gives you a safety net to fall back on, so you don't have to remember all the ways it is supposed to work in. Something tells you if it is or not.

Also, doing Test Driven Development when developing is one of the best ways to keep track of what you are working on. I have times when I am churning out code and tests, one after the other, before I need to take a break. The concept of TDD is that I write a failing test, and then I write just enough code to pass that test. So when I take a break, I make it a point to leave at a failing test, so that when I come back, I can jump right back into writing the code to get it to pass. I don't have to spend 15 - 20 minutes reading through the code to figure out where I left off. My asserts usually tell me exactly what I need to do.

Imposing Modularity / Reusability

The very first rule of reusable code is that you have to be able to instantiate an instance of the class before you can use it. And guess what ? With unit tests, you almost always have to instantiate an instance of the class under test. Therefore, writing a unit test is always a first great step in making code reusable. And the minute you start writing unit tests, most likely, you will start running into the common pain points of not having injectable dependencies (Unless of course, you are one of the converts, in which case, good for you!).

Which brings me to the next point. Once you start having to jump through fiery hoops to set up your class just right to test it, you will start to realize when a class is getting bloated, or when a certain component belongs in its own class. For instance, why test the House when what you really want to test is the Kitchen it contains. So if the Kitchen class was initially part of the House, when you start writing unit tests, it becomes obvious enough that it belongs separately. Before long, you have modular classes which are small and self contained and can be tested independently without effort. And it definitely helps keep the code base cleaner and more comprehensible.

Refactoring Safety Net

Any project, no matter what you do, usually ends up at a juncture where the requirements change on you. And you are left with the option of refactoring your codebase to add / change it, or rewrite from scratch. One, never rewrite from scratch, always refactor. Its always faster when you refactor, no matter what you may think. Two, what do you do when you have to refactor and you don't have unit tests ? How do you know you haven't horribly broken something in that refactor ? Granted, IDE's such as Eclipse and IntelliJ have made refactoring much more convenient, but adding new functionality or editing existing features is never simple.

More often than not, we end up changing some undocumented way the existing code behaved, and blow up 10 different things (it takes skill to blow up more, believe me, I have tried). And its often something as simple as changing the way a variable is set or unset. In those cases, having unittests (remember those things you were supposed to have written?) to confirm that your refactoring broke nothing is godsend. I can't tell you the amount of times I have had to refactor a legacy code base without this safety net. The only way to ensure I did it correct was to write these large integration tests (because again, no unit tests usually tends to increase the coupling and reduce modularity, even in the most well designed code bases) which verified things at a higher level and pray fervently that I broke nothing. Then I would spend a few minutes bringing up the app everytime, and clicking on random things to make sure nothing blew up. A complete waste of my time when I could have known the same thing by just running my unit tests.

Documentation

Finally, one of my favorite advantages to doing TDD or writing unit tests as I code. I have a short memory for code I have written. I could look back at the code I wrote two days ago, and have no clue what I was thinking. In those cases, all I have to do is go look at the test for a particular method, and that almost always will tell me what that method takes in as parameters, and what all it should be doing. A well constructed set of tests tell you about valid and invalid inputs, state that it should modify and output that it may return.

Now this is useful for people like me with short memory spans. But it is also useful, say, when you have a new person joining the team. We had this cushion the last time someone joined our team for a short period of time, and when we asked him to add a particular check to a method, we just pointed him to the tests for that method, which basically told him what the method does. He was able to understand the requirements, and go ahead and add the check with minimal handholding. And the tests give a safety net so he doesn't break anything else while he was at it.

Also useful is the fact that later, when someone comes marching through your door, demanding you fix this bug, you can always make sure whether it was a a bug (in which case, you are obviously missing a test case) or if it was a feature that they have now changed the requirements on (in which case you already have a test which proves it was your intent to do it, and thus not a bug).
Permalink | Links to this post | 13 comments  

Software Testing Categorization

Tuesday, July 14, 2009 12:29 PM

by Miško Hevery


You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? Here is how I think about tests.

Unit/Small

Lets start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don't need debugger to figure out what is wrong. Now this has some implications. Under 1 ms means that your test cannot do any I/O. The reason this is important is that you want to run ALL (thousands) of your unit-tests every time you modify anything, preferably on each save. My patience is two seconds max. In two seconds I want to make sure that all of my unit tests ran and nothing broke. This is a great world to be in, since if tests go red you just hit Ctrl-Z few times to undo what you have done and try again. The immediate feedback is addictive. Not needing a debugger implies that the test is localized (hence the word unit, as in single class).

The purpose of the unit-test is to check the conditional logic in your code, your 'ifs' and 'loops'. This is where the majority of your bugs come from (see theory of bugs). Which is why if you do no other testing, unit tests are the best bang for your buck! Unit tests, also make sure that you have testable code. If you have unit-testable code than all other testing levels will be testable as well.

A KeyedMultiStackTest.java is what I would consider great unit test example from Testability Explorer. Notice how each test tells a story. It is not testMethodA, testMethodB, etc, rather each test is a scenario. Notice how at the beginning the test are normal operations you would expect but as you get to the bottom of the file the test become little stranger. It is because those are weird corner cases which I have discovered later. Now the funny thing about KeyedMultiStack.java is that I had to rewrite this class three times. Since I could not get it to work under all of the test cases. One of the test was always failing, until I realized that my algorithm was fundamentally flawed. By this time I had most of the project working and this is a key class for byte-code analysis process. How would you feel about ripping out something so fundamental out of your system and rewriting it from scratch? It took me two days to rewrite it until all of my test passed again. After the rewrite the overall application still worked. This is where you have an AHa! moment, when you realize just how amazing unit-tests are.

Does each class need a unit test? A qualified no. Many classes get tested indirectly when testing something else. Usually simple value objects do not have tests of their own. But don't confuse not having tests and not having test coverage. All classes/methods should have test coverage. If you TDD, than this is automatic.

Medium/Functional

So you proved that each class works individually, but how do you know that they work together? For this we need to wire related classes together just as they would be in production and exercise some basic execution paths through it. The question here we are trying to answer is not if the 'ifs' and 'loops' work, (we have already answered that,) but whether the interfaces between classes abide by their contracts. Great example of functional test is MetricComputerTest.java. Notice how the input of each test is an inner class in the test file and the output is ClassCost.java. To get the output several classes have to collaborate together to: parse byte-codes, analyze code paths, and compute costs until the final cost numbers are asserted.

Many of the classes are tested twice. Once directly throughout unit-test as described above, and once indirectly through the functional-tests. If you would remove the unit tests I would still have high confidence that the functional tests would catch most changes which would break things, but I would have no idea where to go to look for a fix, since the mistake can be in any class involved in the execution path. The no debugger needed rule is broken here. When a functional test fails, (and there are no unit tests failing) I am forced to take out my debugger. When I find the problem, I add a unit test retroactively to my unit test to 1) prove to myself that I understand the bug and 2) prevent this bug from happening again. The retroactive unit test is the reason why the unit tests at the end of KeyedMultiStackTest.java file are "strange" for a lack of a better world. They are things which I did not think of when i wrote the unit-test, but discovered when I wrote functional tests, and through lot of hours behind debugger track down to KeyedMultiStack.java class as the culprit.

Now computing metrics is just a small part of what testability explorer does, (it also does reports, and suggestions) but those are not tested in this functional test (there are other functional tests for that). You can think of functional-tests as a set of related classes which form a cohesive functional unit for the overall application. Here are some of the functional areas in testability explorer: java byte-code parsing, java source parsing, c++ parsing, cost analysis, 3 different kinds of reports, and suggestion engine. All of these have unique set of related classes which work together and need to be tested together, but for the most part are independent.

Large/End-to-End/Scenario

We have proved that: 'ifs' and 'loops' work; and that the contracts are compatible, what else can we test? There is still one class of mistake we can make. You can wire the whole thing wrong. For example, passing in null instead of report, not configuring the location of the jar file for parsing, and so on. These are not logical bugs, but wiring bugs. Luckily, wiring bugs have this nice property that they fail consistently and usually spectacularly with an exception. Here is an example of end-to-end test: TestabilityRunnerTest.java. Notice how these tests exercises the whole application, and do not assert much. What is there to assert? We have already proven that everything works, we just want to make sure that it is wired properly.
Permalink | Links to this post | 8 comments  

Why are we embarrassed to admit that we don't know how to write tests?

Wednesday, July 08, 2009 9:49 AM

Take your average developer and ask "do you know language/technology X?" None of us will feel any shame in admitting that we do not know X. After all there are so many languages, frameworks and technologies, how could you know them all? But what if X is writing testable code? Somehow we have trouble answering the question "do you know how to write tests?" Everyone says yes, whether or not we actually know it. It is as if there is some shame in admitting that you don't know how to write tests.
Now I am not suggesting that people knowingly lie here, it is just that they think there is nothing to it. We think: I know how to write code, I think my code is pretty good, therefore my code is testable!
I personally think that we would do a lot better if we would recognize testability as a skill in its own right. And as such skills are not innate and take years of practice to develop. We could than treat it as any other skill and freely admit that we don't know it. We could than do something about it. We could offer classes, or other materials to grow our developers, but instead we treat it like breathing. We think that any developer can write testable code.
It took me two years of writing tests first, where I had as much tests as production code, before I started to understand what is the difference between testable and hard to test code. Ask yourself, how long have you been writing tests? What percentage of the code you write is tests?
Here is a question which you can ask to prove my point: "How do you write hard to test code?" I like to ask this question in interviews and most of the time I get silence. Sometimes I get people to say, make things private. Well if visibility is your only problem, I have a RegExp for you which will solve all of your problems. The truth is a lot more complicated, the code is hard to test doe to its structure, not doe to its naming conventions or visibility. Do you know the answer?
We all start at the same place. When I first heard about testing I immediately thought about writing a framework which will pretend to be a user so that I can put the app through its paces. It is only natural to thing this way. This kind of tests are called end-to-end-tests (or scenario or large tests), and they should be the last kind of tests which you write not the first thing you think of. End-to-end-tests are great for locating wiring bugs but are pretty bad at locating logical bugs. And most of your mistakes are in logical bugs, those are the hard ones to find. I find it a bit amusing that to fight buggy code we write even more complex framework which will pretends to be the user, so now we have even more code to test.
Everyone is in search of some magic test framework, technology, the know-how, which will solve the testing woes. Well I have news for you: there is no such thing. The secret in tests is in writing testable code, not in knowing some magic on testing side. And it certainly is not in some company which will sell you some test automation framework. Let me make this super clear: The secret in testing is in writing testable-code! You need to go after your developers not your test-organization.
Now lets think about this. Most organizations have developers which write code and than a test organization to test it. So let me make sure I understand. There is a group of people which write untestable code and a group which desperately tries to put tests around the untestable code. (Oh and test-group is not allowed to change the production code.) The developers are where the mistakes are made, and testers are the ones who feel the pain. Do you think that the developers have any incentive to change their behavior if they don't feel the pain of their mistakes? Can the test-organization be effective if they can't change the production code?
It is so easy to hide behind a "framework" which needs to be built/bought and things will be better. But the root cause is the untestable code, and until we learn to admit that we don't know how to write testable code, nothing is going to change...

Read more »