How can I best identify variation in the system I'm testing?
When test designers struggle to use Hexawise, the most common reason is that they struggle to identify what test inputs they should include in Hexawise's "Define Inputs" screen. To help address this challenge, this lesson describes several categories of variables that are often useful to include in software tests. At the risk of stating the obvious, you'll need to use your judgement here; not all of these categories will be necessary (or even useful!) to include in all of your sets of tests.
An overview of variation ideas
Variables for software tests can be described in these general categories. It is important to remember that when you're entering variables into the "Define Inputs" screen, you should include test INPUTS only. You should not include outcomes or expected results in the variables you include on the "Define Inputs" screen.
Environmental variations relate to different potential combinations of hardware and software that people might use, as well as different locations of data that might be pulled from when a given transaction is executed. Imagine you're creating some end-to-end tests for a flight reservation system, like hipmunk.com, a site which has been widely praised for its exceptional User Interface. As a web application, you would probably want to include multiple combinations of hardware and software configurations.
Hardware configuration inputs could include:
Manufacturer: Dell, Lenovo, Apple
Processor: Intel, AMD
Age: manufactured within last 2 years, manufactured 2 or more years ago
Software configuration inputs could include:
Operating System: Windows NT, Windows 7, Windows 8, Mac OSX
Browser: IE8, IE7, Firefox, Chrome, Safari
Cookies: cookied enabled, cookies disabled
Channel inputs could include:
Channel: typical web transaction, typical web transaction, phone call center (<-- Note: double-weighting typical web transaction would make it show up twice as often as phone call center.)
Additional Environmental Variations involve file types and storage locations
Flight schedule information for flights is stored: Database Huey, Database Dooey, Database Louie
Frequent Flier Milage data stored in: United database, (Frontier Airlines database; not yet transferred into post-merger database), N/A
User variations relate to how different people might navigate through the System Under Test based on their particular habits as computer users. They can also include, as in the case of an "Admin User" with special rights, the particular features of the system that they have available to them based on the kind of user that they are.
For example, User Variations that you might consider including in end-to-end tests of hipmunk.com, might include:
User type: normal user (without a prior account), normal user (with an established account), call center agent
User persona: power user, clueless newbie user, typical user
Account characteristics: Gold Frequent Flier, Silver Frequent Flier, Bronze Frequent Flier, not a Frequent Flier
Other related ideas:
User navigates primarily by using: the keyboard (with hot-keys wherever possible), the keyboard (but without hot-keys), mouse
User rapidly clicks multiple times on submit buttons and next buttons: no, yes, no <-- (Note: double-weighting "no" will make it show up twice as often as "yes")
User enters information from top of screen to bottom of screen: always, usually, never
User enters all required information the first time they submit information: always, usually, never
Usage Variations relate to what different features of a system people might use, as well as how they might use those features, and the different types of data that might be used in different scenarios.
Actions that could vary from test to test in our hipmunk.com example could include:
One Way or Round Trip: One Way, Round Trip
Destination: South America, Europe, Asia, Africa, Australia, North America
Class of Travel: Economy, Business, First
Special Meal Requested: None, None, Vegetarian, Kosher
Saturday Night Stayover: Yes, No
Data that might vary will often be inextricably interrelated to usage examples (directly above). Each of the user selections above, for example, would create data that should be stored by the system. That's one of the reasons we categorize "Actions" and "Data" together as "Usage Variations." Even so, "data" can be useful as a trigger for additional testing ideas such as quantities, data ranges, and specific data formats. Here are some examples of test inputs that could be included from this example:
Date format: mm/dd/yy, dd/mm/yy, mm/dd/yyy, dd/mm/yyyy
Number of passengers: 1, 2-4, more than 4
Special characters included in customer name?: no, yes, no
"Hendrickson Variables" offer a great checklist to tick through as you think about possible variables to add.
Elisabeth Hendrickson published "Explore It!" in 2013. In our opinion, it is one of the best software testing books ever written. We highly recommend it to new and experienced software testers alike without reservations. It's available for purchase through PragProg and Amazon. Chapter 4, called "Finding Interesting Things to Vary," is particularly outstanding. The chapter does a superb job of articulating not only what kinds of variables you can change from test to test but also why it is such a good idea to vary them. These "Hendrickson Variables" come from Chapter 4 of "Explore It!"
Hendrickson lays out her list of Interesting Things to Vary as follows:
Countable Things -
Relative Positioning -
Files and Storage-
Geographic Locations -
Timing, Frequency, and Duration -
Input and Navigation -
A "rule of thumb" to guide you as you decide what things should be included and excluded in the Define Inputs screen:
Deciding whether to include a specific test idea often comes down to a judgement call by the test designer using Hexawise.
INCLUDE - If you have reason to believe that including a test input is likely to matter, and it would not take much extra time to vary that idea in each of the tests in your test set, include it. An example: if you're testing a transactional web application and prior releases have had quite a few defects associated with the IE8 browser, include "Browser Type" as a Parameter and "IE8" as one of your Values.
EXCLUDE - Conversely, if you suspect that a particular test idea is not going to make any difference, and that variation would take considerable time to include in each of your tests, do not enter those into the "Define Inputs" screen of Hexawise. Example: as pointed out by James Bach, defects have, on extremely rare occassions, been triggered by blocked cooling vents above servers which casused a server to overheat. Even so, it would rarely be sensible to invest the time necessary to test out multiple test scenarios with overheated servers on a given set of tests.
INCLUDING TEST IDEAS USING THE VALUE EXPANSION FEATURE TO ALLOW QUICK TESTING WITHOUT WASTING TIME "OVERTESTING" - An example: Imagine that you're testing a new feature of a financial transaction application. It will be used for foreign or domestic transactions. Setting up the testing environment for each foreign transaction requires dramatically more time than for domestic transactions. If you were to include Type of Transaction as a Parameter and use "Domestic" and "International" as the 2 Values for that Parameter, approximately 50% of your tests would be international transactions which would each require a great deal of extra setup time. In such situations, you might instead consider using Type of Transcation as a Value and then adding "Domestic, International, Domestic, Domestic, Domestic, Domestic, Domestic" as sub-values when you create your Value Expansion. Using this test design technique would result in at least one International transaction (for coverage of this potentially important scenario) but you would avoid spending too much time "over-testing" the idea.
Your ability to make judgement calls will increase as you gain experience. When you're first getting started, feel free to experiment. When you're experiementing, don't be afraid to add in a few more test ideas into the scope of your test sets than you're used to. You will notice that when you design tests with Hexawise you can include more test ideas in the scope of your tests than you're used to including. That's because:
- You will often find surprising defects when you add variation to your tests, yet adding the variation to your tests often hardly takes any time at all.
- Adding a Parameter with a few Values takes only a few seconds on the "Define Inputs" screen of Hexawise.
- Finally, unlike when you select and document tests by hand, generating the tests with the newly-added test ideas won't add noticeably more test documentation time; the tests generated by Hexawise will all automatically include that test idea.
- Provided you keep the number of Values per Parameter small, you usually won't increase the number of tests that Hexawise will generate.