Example:

Make a loan payment, and verify: 1) Payment screen was updated, 2) Account Summary was updated, 3) Transaction History was updated.

	Driver:    calls	"PAYMENT"	("Main" Script)
		PAYMENT	calls	Pymt_Rtn:
				Access Payment Screen from Main Menu (function)
				Update Payment Screen using input file (function)
				Verify Payment Screen updated using verification file (function)
				Return to Main Menu
				Return (TRUE / FALSE / ERROR) to calling script
		PAYMENT:	If Pymt_Rtn	returns FALSE, return FALSE to Driver,
			else call	Acct_Sum:
				Access Account Summary Screen from Main Menu (function)
				Verify Account Summary Screen using verification file (function)
				Return to Main Menu
				Return (TRUE / FALSE / ERROR) to calling script
		PAYMENT:	If Acct_Sum	returns FALSE, return FALSE to Driver,
			else call	Trn_Hist:
				Access Transaction History Screen from Main Menu (function)
				Verify Transaction History Screen using verification file (function)
				Return to Main Menu
				Return (TRUE / FALSE / ERROR) to calling script
		PAYMENT:	Return	(TRUE / FALSE / ERROR) to Driver
	Driver: calls next	"Main" Script

In the above example, we would need the following data-files for each Test Case:

1. Input data ("field-names" + data to input) for Payment Screen
2. Verification data ("field-names" + data to verify) for Payment Screen
3. Verification data ("field-names" + data to verify) for Account Summary Screen
4. Verification data ("field-names" + data to verify) for Transaction History Screen

Processing Note:

If an error occurs in the processing, such that we cannot continue with the Test Case, a "FALSE" condition is returned to the calling script. This script, in turn, returns the "FALSE" condition to its calling script, etc., until control is returned back to the Driver script. If the Test Case dependencies have been properly controlled, the Driver can then continue with the next Test Case, otherwise the Driver would have to exit.

Advantages:

		Utilizing a modular design, and using files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts.
		Scripts may be developed while application development is still in progress. If functionality changes, only the specific "Business Function" script needs to be updated.
		Since scripts are written to perform and test individual Business Functions, they can easily be combined in a "higher level" test script in order to accommodate complex test scenarios.
		Data input/output and expected results is stored as easily maintainable text records. The user's expected results are used for verification, which is a requirement for System Testing.
		Functions return "TRUE" or "FALSE" values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed "recovery" routine, enables "unattended" execution of test scripts.

Disadvantages:

		Requires proficiency in the tool's Scripting Language;
		Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This requires data-files to be kept in separate directories by Test Case.
		Tester must not only maintain the Detail Test Plan with specific data, but must also re-enter this data in the various data-files.
		If a simple "text editor" such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or processing-errors will occur.

 

"Test Plan Driven" Method:

The "Test Plan Driven" method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire testing process is data-driven, including functionality. The Detail Test Plan is written in a specific format, then saved in a particular record-format which the pre-written "Utility" scripts use to control the entire processing of the Automated Test.

Example:

This example shows a Test Case document developed by the tester using a spreadsheet containing "Key-Words" in Column 1.   In this method, the entire process is data-driven, including functionality. The Key Words control the processing.  Note that this test case could also be executed manually, if necessary.

COLUMN 1 Key_Word	COLUMN 2 Field/Window Name	COLUMN 3 Input/Verification Data	COLUMN 4 Comment	COLUMN 5 Pass/Fail
Start_Test:	Window	Main Menu	Verify Starting Point
Enter:	Selection	3	Select Payment Option
Action:	Press_Key	Enter	Access Payment Screen
Verify:	Window	Payment Posting	Verify Screen accessed
Enter:	Payment Amount	125.87	Enter Payment data
	Payment Method	Check
Action:	Press_Button	Post	Process Payment
Verify:	Window	Payment Posting	Verify screen remains
Verify_Data:	Payment Amount	$ 125.87	Verify updated data
	Current Balance	$1,309.77
	Status Message	Payment Posted
Action:	Press_Button	Exit	Return to Main Menu
End_Test:	Window	Main Menu	Verify return to Menu

When the spreadsheet is completed, it is saved as a tab-delimited file, as well as in its original ".xls" format.  The Test Case could also be developed using a database such as Access, as long as it could be saved as a tab-delimited file.

Architecture:

The architecture of the "Test Plan Driven" method looks similar to that of the "Functional Decomposition" method, but in fact, they are substantially different:

		Driver Script		Performs initialization (if required);
				Calls the Application-Specific "Controller" Script, passing to it the file-name of the Test Case file (which has been saved in a "tab-delimited" format from the tester-generated Detail Test Plan spreadsheet);
		Controller Script		Reads and processes the "tab-delimited" file, using a "switch-case" statement to break on each Key Word;
				Builds a list of "processing actions" from the data within the rows / columns within the Key-Word group;
				Calls a specific "Utility" script associated with the Key Word, passing to it the created "processing list";
		Utility Scripts		Process input-list received from the "Controller" script;
				Perform specific tasks (e.g. press a key or button, enter data, verify data, etc.), calling "User Defined Functions" if required;
				Report any errors to a customized Test Report for the Test Case;
				Return to "Controller" script;
		Functions		General and Application-Specific functions may be called by any of the above script-types in order to perform specific tasks;

Advantages:

This method has most of the advantages of the "Functional Decomposition" method, as well as the following:

		Detail Test Plan can be written in Spreadsheet format containing all input and verification data. Therefore the tester only needs to write this once, rather than, for example, writing it in Word, and then creating input and verification files as is required by the "Functional Decomposition" method.
		Test Plan does not necessarily have to be written using a "spreadsheet" program. Any format can be used from which either "tab-delimited" or "comma-delimited" files can be saved (e.g. Access Database).
		If "utility" scripts can be created by someone proficient in the tool's Scripting Language, prior to the Detail Test Plan being written, then the tester can use the tool immediately and not be required to learn the TSL Scripting language. The tester need only learn the "Key Words" required, and the specific format to use within the Test Plan.
		If the Detail Test Plan already exists in some other format, it is not difficult to translate this into the "spreadsheet" format.
		Since a number of "generic" Utility scripts have already been created, we can usually get the user up and running (for most applications) with this method within a few days, rather than weeks.

Disadvantages:

		Development of "customized" (Application-Specific) Functions and Utilities still requires proficiency in the tool's Scripting Language.  Note that this is also true of the "Functional Decomposition" method, and, frankly of any method including "Record/Playback".
		If application requires more than a few "customized" Utilities, this will require the tester to learn a number of "Key Words" and special formats. This is preferable, however, to the tester having to learn an entire programming language in order to utilize a Test Tool.
		Spreadsheet Test Cases written in this manner are not re-usable.  If, for example the "Post a Payment" Business Function changes, then all test scripts that require the "Post a Payment" sequence will have to be updated.

 

Combined Methodologies:

In order to gain the advantage of "script re-usability" that we get from the "Functional Decomposition" method, and maintain the advantages obtained by using the   "Test Plan Driven" method, it can be advantageous to combine these two methods.

This can be accomplished by adding "Business Function Utility Scripts" and Key-Words to our "Test Plan Driven" method.

Example:

Using our "Post a Payment" example above, we can replace the "Enter:" and "Action:" Key-Words that we used to accomplish posting the payment, with a "Post_Payment" Business Function Key Word, that uses the same input information:

COLUMN 1 Key_Word	COLUMN 2 Field/Window Name	COLUMN 3 Input/Verification Data	COLUMN 4 Comment	COLUMN 5 Pass/Fail
Start_Test:	Window	Main Menu	Verify Starting Point
Enter:	Selection	3	Select Payment Option
Action:	Press_Key	Enter	Access Payment Screen
Verify:	Window	Payment Posting	Verify Screen accessed
Post_Payment:	Payment_Amount	125.87	Enter Payment data
	Payment_Method	Check	Select Payment Method
	Post_Method	Press_Button	(Option: Use_Menu)
Verify:	Window	Payment Posting	Verify screen remains
Verify_Data:	Payment Amount	$ 125.87	Verify updated data
	Current Balance	$1,309.77
	Status Message	Payment Posted
Action:	Press_Button	Exit	Return to Main Menu
End_Test:	Window	Main Menu	Verify return to Menu

We replace the field-names "Payment Amount" and "Payment Method" with parameters: Payment_Amount and Payment_Method, and also add a "Post_Method" allowing us an option between pressing the "Post" button, and using a menu_selection to perform the "action".  Now if the Business Function changes slightly, we do not have to update all of the "Post a Payment" Test Cases - we only have to update the "Post_Payment" Business Function Utility Script.  The change can be made in one place rather than in many.

The main disadvantage to this, is that rather than having a limited number of Key-Words, you may now have a rather large number of Key-Words to use, depending on the number of Business Function scripts you want to add.  This makes things more difficult for the tester, who now has a lot more Key-Words to deal with.  This difficulty can be eased somewhat by creating a Template spreadsheet that the tester can use to cut-and-paste from.

Parameterized Input:

Additional "re-usability" can be obtained by using parameters (representing variables) as input rather than actual input data.

Let us say, for example, we are testing a Web application, and must use both MSIE and Netscape browsers and yet run the exact same Test Cases using each of these.  In addition, we must run these tests on mirrored sites, where the main URL for each site is different.  Using our original "Test Plan Driven" example, we would need 4 separate sets of test cases!

This is easily solved using parameters as input, instead of data:

Start_Test:	Open_Browser	VAR<Browser>	Open Required Browser
Enter:	URL Field	VAR<Main_Url>	Set Main URL for Site
Action:	Image_Click	Register Now	Access Registration
Verify:	URL Field	VAR<Main_Url>/reg.html	Verify Registration URL

In this example, the Controller Script would contain a function that parses the input for the variable_string "VAR<" and resolves all such variables to their correct values.

To accomplish this, the Controller Script also must contain a function that reads a "Parameter File" and creates an associative array for all input values.

The "Parameter File" might look something like this:

#  VAR<Browser> = Netscape #  VAR<Main_Url> = http://www.buythisstuff.com # Note: Pound-Sign = comment VAR<Browser> = IE VAR<Main_Url> = http://www.buymorestuff.com

Each time the test suite is run, various combinations can be commented out, allowing the same script to be run for the 4 different scenarios: 2 mirrored sites with 2 browsers.  Another variation would be to have the Driver Script run the tests in a loop while incrementing a counter in order to reference 4 different parameter files (e.g. param1.txt, param2.txt, etc.).

Click Here for an spreadsheet example of a test case that was actually run.

 

Summary:

		Both the Functional Decomposition and Data-Driven methods of testing are platform and Test-Tool independent.
		Both methods can be used with any Test Tool which provides a "scripting language".
		The  Data-Driven method allows the tester to create Automated Test Scripts without having to learn the scripting language.
		The  Data-Driven method can be enhanced by adding Business Function Utilities which provides the advantages attainable by both methods.
		Additional Test Script re-usability can be obtained by using parameterized input rather than data at key points in the Test Case.
	Automated Testing Specialists can help you determine and quickly implement the most effective testing methodology for your automated testing requirements.

A Total Testing Solution:

The ATS ToolKit for WinRunner and our Test-Plan Driven Testing Methodology provides a comprehensive package that will enable you to develop cost-effective Automated Testing Solutions using Mercury Interactive's WinRunnerŪ test tool.   Click Here for further details!

Automated Testing Methodology Tools:

There are a few "packaged" methodologies available for Automated Testing, which you may find useful:

	TMS: Test Management System		Testmasters, Inc.

Visit the "Tools for Process Automation" section of our Automated Test Tools page to see additional tools.

 

Independent Consultant's Network		Home | Site Map | Resume | Articles & White Papers | Automated Testing Methodology   WinRunner ToolKit | Automated Test Tools | Consulting Services | Quality Assurance Links Downloads    |   Web Testing   |  Localization Testing   |   Contact Us   |   What's New		Automated Testing Resources