Totally Data-Driven Automated Testing
A White Paper
By
Keith Zambelich
Sr. Software Quality Assurance Analyst
Automated Testing Evangelist
I have been involved in Software Testing and Software Quality Assurance for the past 15 years, and have tested a number of software applications on a variety of platforms. I have also been involved in some form of automated testing or another during this period.
At First Interstate Services Corporation, I was involved in testing Electronic Funds Transfer (EFT) applications written in ACP/TPF. Here I developed the "Transaction Simulation Test Facility", which allowed testers to simulate transactions to and from all of First Interstate’s connections (VISA, Cirrus, Great Western, etc.). This consisted of over 400 programs written in ACP/TPF, and enabled testers to verify all application modifications, including the front-end switching (Tandem) software.
At the Pacific Stock Exchange, I was in charge of testing all Exchange software, including their new PCOAST trading system. I developed and implemented a method of simulating Broker transactions, eliminating the need for live testing with Brokers. This resulted in a greatly improved implementation success rate.
During my employment at Omnikron Systems, Inc. (a software consulting company based in Calabasas, CA) I successfully implemented Automated Testing solutions using Mercury Interactive’s WinRunner® test tool for a variety of companies, including Transamerica Financial Services, J. D. Edwards Co., IBM Canada, PacifiCare Health Systems, and Automated Data Processing (ADP). While at Omnikron Systems, I developed a totally data-driven method of automated testing that can be applied to any automated testing tool that allows scripting.
I have been certified as a WinRunner® Product Specialist (CPS) by Mercury Interactive, Inc. Currently, I am the President and CEO of Automated Testing Specialists, Inc., a consulting firm specializing in the implementation of automated software testing solutions.
The case for automating the Software Testing Process has been made repeatedly and convincingly by numerous testing professionals. Most people involved in the testing of software will agree that the automation of the testing process is not only desirable, but in fact is a necessity given the demands of the current market.
A number of Automated Test Tools have been developed for GUI-based applications as well as Mainframe applications, and several of these are quite good inasmuch as they provide the user with the basic tools required to automate their testing process. Increasingly, however, we have seen companies purchase these tools, only to realize that implementing a cost-effective automated testing solution is far more difficult than it appears. We often hear something like "It looked so easy when the tool vendor (salesperson) did it, but my people couldn’t get it to work.", or, "We spent 6 months trying to implement this tool effectively, but we still have to do most of our testing manually.", or, "It takes too long to get everything working properly. It takes less time just to manually test.". The end result, all too often, is that the tool ends up on the shelf as just another "purchasing mistake".
The purpose of this document is to provide the reader with a clear understanding of what is actually required to successfully implement cost-effective automated testing. Rather than engage in a theoretical dissertation on this subject, I have endeavored to be as straightforward and brutally honest as possible in discussing the issues, problems, necessities, and requirements involved in this enterprise.
Simply put, what is meant by "Automated Testing" is automating the manual testing process currently in use. This requires that a formalized "manual testing process" currently exists in your company or organization. Minimally, such a process includes:
If your current testing process does not include the above points, you are never going to be able to make any effective use of an automated test tool.
So if your "testing methodology" just involves turning the software release over to a "testing group" comprised of "users" or "subject matter experts" who bang on their keyboards in some ad hoc fashion or another, then you should not concern yourself with testing automation. There is no real point in trying to automate something that does not exist. You must first establish an effective testing process.
The real use and purpose of automated test tools is to automate regression testing. This means that you must have or must develop a database of detailed test cases that are repeatable, and this suite of tests is run every time there is a change to the application to ensure that the change does not produce unintended consequences.
An "automated test script" is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development. Making effective use of any automated test tool requires at least one trained, technical person – in other words, a programmer.
Automated testing is expensive (contrary to what test tool vendors would have you believe). It does not replace the need for manual testing or enable you to "down-size" your testing department. Automated testing is an addition to your testing process. According to Cem Kaner, in his paper entitled "Improving the Maintainability of Automated Test Suites" (www.kaner.com), it can take between 3 to 10 times as long (or longer) to develop, verify, and document an automated test case than to create and execute a manual test case. This is especially true if you elect to use the "record/playback" feature (contained in most test tools) as your primary automated testing methodology. Record/Playback is the least cost-effective method of automating test cases.
Automated testing can be made to be cost-effective, however, if some common sense is applied to the process:
Every automated test tool vendor will tell you that their tool is "easy to use" and that your non-technical user-type testers can easily automate all of their tests by simply recording their actions, and then playing back the recorded scripts. This one statement alone is probably the most responsible for the majority of automated test tool software that is gathering dust on shelves in companies around the world. I would just love to see one of these salespeople try it themselves in a real-world scenario. Here’s why it doesn’t work:
After about 2 to 3 months of this nonsense, the tool gets put on the shelf or buried in a desk drawer, and the testers get back to manual testing. The tool vendor couldn’t care less – they are in the business of selling test tools, not testing software.
Now that we’ve eliminated Record/Playback as a reasonable long-term automated testing strategy, let’s discuss some methodologies that I (as well as others) have found to be effective for automating functional or system testing for most business applications
The main concept behind the "Functional
Decomposition" script development methodology is to reduce all test cases to their
most fundamental tasks, and write User-Defined Functions, Business Function Scripts,
and "Sub-routine" or "Utility" Scripts which perform
these tasks independently of one another. In general, these fundamental areas
include:
In order to accomplish this, it is necessary to separate Data from Function.
This allows an automated test script to be written for a Business Function, using data-files
to provide the both the input and the expected-results verification. A hierarchical
architecture is employed, using a structured or modular design. The highest level is the Driver script, which is the engine of the test. The Driver
Script contains a series of calls to one or more "Test Case" scripts. The
"Test Case" scripts contain the test case logic, calling the Business Function
scripts necessary to perform the application testing. Utility scripts and functions are
called as needed by Drivers, Main, and Business Function scripts.
Perform initialization (if required), then call the Test Case Scripts in the
desired order. Perform the application test case logic using Business Function Scripts Perform specific Business Functions within the application; Perform application specific tasks required by two or more Business scripts; General, Application-Specific, and Screen-Access Functions; Note that Functions can be called from any of the above script types. Example: The following steps could constitute a "Post a Payment" Test Case: A "Business Function" script and a "Subroutine" script could be
written as follows: Payment: Ver-Acct (Verify Account Summary & Transaction History): The "Business Function" and "Subroutine" scripts invoke "User
Defined Functions" to perform navigation. The "Test Case" script would call
these two scripts, and the Driver Script would call this "Test Case" script some
number of times required to perform all the required Test Cases of this kind. In each
case, the only thing that changes are the data contained in the files that are read
and processed by the "Business Function" and "Subroutine" scripts. Using this method, if we needed to process 50 different kinds of payments in order to
verify all of the possible conditions, then we would need only 4 scripts which are re-usable
for all 50 cases: If we were using Record/Playback, we would now have 50 scripts, each containing
hard-coded data, that would have to be maintained. This method, however, requires only that we add the data-files required for each test,
and these can easily be updated/maintained using Notepad or some such text-editor. Note
that updating these files does not require any knowledge of the automated tool, scripting,
programming, etc. meaning that the non-technical testers can perform this function, while
one "technical" tester can create and maintain the automated scripts. It should be noticed that the "Subroutine" script, which verifies the Account
Summary and Transaction History, can also be used by other test cases and business
functions (which is why it is classified as a "Subroutine" script rather than a
"Business Function" script) – Payment reversals, for example. This means
that if we also need to perform 50 "payment reversals", we only need to develop three
additional scripts. Since we already had the original 4 scripts, we can quickly clone these three new
scripts from the originals (which takes hardly any time at all). We can use the
"Subroutine" script as-is without any modifications at all.
If different accounts need to be used, then all we have to do is update the Data-Files, and not the actual scripts. It ought to be obvious that this is a much more cost-effective method than the Record/Playback method.
Advantages:
Disadvantages:
This method uses the actual Test Case document developed by the tester using a spreadsheet containing special "Key-Words". This method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire process is data-driven, including functionality. The Key Words control the processing.
Consider the following example of our previous "Post a Payment" Test Case:
| COLUMN 1 Key_Word |
COLUMN 2 Field/Screen Name |
COLUMN 3 Input/Verification Data |
COLUMN 4 Comment |
COLUMN 5 Pass/Fail |
| Start_Test: | Screen | Main Menu | Verify Starting Point | |
| Enter: | Selection | 3 | Select Payment Option | |
| Action: | Press_Key | F4 | Access Payment Screen | |
| Verify: | Screen | Payment Posting | Verify Screen accessed | |
| Enter: | Payment Amount | 125.87 | Enter Payment data | |
| Payment Method | Check | |||
| Action: | Press_Key | F9 | Process Payment | |
| Verify: | Screen | Payment Screen | Verify screen remains | |
| Verify_Data: | Payment Amount | $ 125.87 | Verify updated data | |
| Current Balance | $1,309.77 | |||
| Status Message | Payment Posted | |||
| Action: | Press_Key | F12 | Return to Main Menu | |
| Verify: | Screen | Main Menu | Verify return to Menu |
Each of the "Key Words" in Column 1 causes a "Utility Script" to be called which processes the remaining columns as input parameters in order to perform specific functions. Note that this could also be run as a manual test. The test engineer must develop and document the test case anyway – why not create the automated test case at the same time?
The data in red indicates what would need to be changed if one were to copy this test case to create additional tests.
How in the World Does This Work?
Architecture
The architecture of the "Test Plan Driven" method appears similar to that of the "Functional Decomposition" method, but in fact, they are substantially different:
Advantages:
This method has all of the advantages of the "Functional Decomposition" method, as well as the following:
Disadvantages:
Cost-Effectiveness:
In the example we gave using the "Functional Decomposition" method, it was shown that we could use previously created "Test Case" and "Business Function" scripts to create scripts for additional Test Cases and Business Functions. If we have 100 Business Functions to test, this means that we must create a minimum of 200 scripts (100 Test Case scripts, and 100 Business Function scripts).
Using the "Test Plan Driven" method, I currently am using 20 Utility scripts, that I have been able to use in every single automated testing engagement that I have been sent on. Let us examine what it takes on an average for me to walk into a company, and implement Automated Testing:
Depending on the complexity of the application, and how well the test tool works with the application, this process normally takes me no more than 3 days – 5 days is worst-case. With this "proof-of-concept" completed, if we are then contracted to do the job, it usually takes 2 or 3 weeks to develop the majority of the application-specific functions, and application-specific versions of the "utility" scripts. Naturally, if the application is quite complex, this process will take longer. At this point, testers can be trained to create the spreadsheet data (usually takes about a week) and then they are in business. It also takes about a week to train the "test tool technician" to use this methodology, provided that this person is a relatively competent programmer and has already been sufficiently trained by the tool vendor in the use of the tool.
What this demonstrates is that an organization can implement cost-effective automated testing if they go about it the right way.
Preparation is the Key:
The situation described above is pretty much an "ideal scene". It assumes that adequate preparations have been made by the organization before beginning the testing automation process. This is rarely the case, however. If adequate preparations have not been made, then the "ramp-up" time required is increased dramatically. What then, does an organization do to prepare themselves for this effort?
One of the main reasons organizations fail at implementing automated testing (apart from getting mired down in the "record/playback" quagmire) is that most testers do not welcome what they perceive as a fundamental change to the way they are going to have to approach their jobs. Typically, decisions as to what tool to use and how to implement it are made by management, often without consulting the people who are actually doing the testing, and who are now going to have to cope with all of this. This usually meets with a great deal of resistance from the testers, especially when management does not have a clearly defined idea of how to implement these changes effectively. Let us examine some concerns that might be expressed by testers, and some answers to these:
This is not even remotely true. The automated testing tool is just another tool that will allow testers to do their jobs better by:
The testers are still going to have to perform tests manually for specific application changes. Some of these tests may be automated afterward for regression testing.
If the "test-plan-driven" method (described above) is used, the testers will not have to learn how to use the tool at all if they don’t want to. All that they have to learn is a different method of documenting the detailed test cases, using the key-word/spreadsheet format. It is not that different from what they are doing currently, and takes only a few hours to learn.
Perhaps, but as we have already discussed, they will not have to use it. What will be required is that a "Test Tool Specialist" will need to be hired and trained to use the tool. This can either be a person who is already an expert with the particular tool, or can be a senior-level programmer who can easily be trained to use it. Most test-tool vendors offer training courses.
The "test-plan-driven" testing method will eliminate most of the testers’ concerns regarding automated testing. They will perform their jobs exactly as they do now. They will only need to learn a different method of documenting their test cases.
One area that organizations desiring to automate testing seem to consistently miss is the staffing issue. Automated test tools use "scripts" which automatically execute test cases. As I mentioned earlier in this paper, these "test scripts" are programs. They are written in whatever scripting language the tool uses. This might be C++ or Visual Basic, or some language unique to the test tool. Since these are programs, they must be managed in the same way that application code is managed.
To accomplish this, a "Test Tool Specialist" or "Automated Testing Engineer" or some such position must be created and staffed with at least one senior-level programmer. It does not really matter what languages the programmer is proficient in. What does matter, is that this person must be capable of designing , developing, testing, debugging, and documenting code. More importantly, this person must want to do this job – most programmers want nothing to do with the Testing Department. This is not going to be easy, but it is nonetheless absolutely critical. In addition to developing automated scripts and functions, this person must be responsible for:
It is often useful to hire a contractor (like me) who knows how to set this all up, help train the "Automation Engineer", and the testing staff, and basically get things rolling. In my experience, this can take from two to three weeks, or as long as two to three months, depending on the situation. In any case, it should be a short-term assignment, and if you find someone who really knows what they’re doing, it will be well worth it.
It is worth noting that no special "status" should granted to the automation tester(s). The non-technical testers are just as important to the process, and favoritism toward one or the other is counter-productive and should be avoided. Software Testing is a profession, and as such, test engineers should be treated as professionals. It takes just as much creativity, brain power, and expertise to develop effective, detailed test cases from business and design specifications as it does to write code. I have done both, and can speak from experience.
Copyright © 1998 by Keith Zambelich, all rights reserved