WHITE PAPER
Wipro Technologies
Innovative Solutions, Quality Leadership
Author: Selvam Ponnusamy
Orthogonal Array: Applicability in
Software Product Testing
Testing software product effectiveness can be improved by testing all possible combinations.
However, it is very difficult to test all possible combinations; many a times it is impossible. So the
other way to improve testing effectiveness is through automation. Both the processes require
high testing cost and time. There are solutions to this pain area using the famous Six Sigma
methodologies which provide tools like Orthogonal Arrays (OA).
Orthogonal Array and ideas from the Design of Experiments (DOE) help in improving the Software
product testing. This white paper introduces Orthogonal Arrays, explains the process of using
OA concepts to improve the testing process.
WHITE PAPER
TaPbaleg eo f: Contents
Table of Contents
Orthogonal Array: Applicability in Software Product Testing
INTRODUCTION ...................................................................................... 3
WHAT IS SIX SIGMA ................................................................................. 3
WHAT IS ORTHOGONAL ARRAY............................................................ 3
THE TAGUCHI APPROACH .........................................................................3
VERIFICATION SUITE DEVELOPMENT USING OA.............................. 7
AN EXPERIENCE USING OA .................................................................. 8
OA TEST CASE SUSTENANCE STRATEGY ....................................... 10
CONCLUSION ........................................................................................ 11
ABOUT THE AUTHOR ........................................................................... 11
ABOUT WIPRO TECHNOLOGIES ........................................................ 11
SIX SIGMA IN WIPRO............................................................................. 12
WIPRO IN SOFTWARE PRODUCT OUTSOURCING .......................... 12
WHITE PAPER
© Wipro Technologies Page : 03 of 12
Orthogonal Array: Applicability in Software Product Testing
Introduction
Testing today’s Software Products is complex as it is difficult to model the product in a way
that it is easy to understand the product’s behavior under various conditions. There are
multiple factors that affect the software product and these factors affect the behavior
simultaneously. This leads to spaghetti of test suites. The more the product evolves the
more complicated the test suite becomes. This makes the test result analysis, DR reporting
and DR analysis inefficient. The result is pouring more and more costly resources in to the
test activities.
These things have led to SW companies to spend as much on testing as they do on product
development. Mature industries such as the automotive industry, spend as little as 5% of
total costs on quality assurance, and have far better results.
What is Six Sigma
The term Six Sigma Quality comes from Motorola. Today, it is synonymous with world class.
Sigma is a statistical expression that indicates how defect free a process or a product is. Let
us say, to make a product we go through 10 steps. Each step is an opportunity to make a
defect. Thus, if we were to make 100,000 pieces of the given product, there will be a million
opportunities to make defects.
If our defect rate is at Three Sigma level, we make 66,807 mistakes. At Six Sigma level, the
defect rate drops to 3.4 mistakes in a million. Average processes and products operate at
Three Sigma level. The best in class processes and products are at Six Sigma level. The
secret of moving from Three Sigma to world class is use of quality at all levels and in all
functions.
What is Orthogonal Array
Orthogonal Array is a method of choosing a set of tests from a universe of tests, to make the
testing efficient and effective. It is based on creating Parameters (control Variables) and
Levels (values) for the parameters, which are the inputs to testable functions.
There are different algorithms to choose the parameters and levels from the universe of
available values. One such method was proposed by Dr Taguchi. This white paper also
deals with the OA method what Taguchi has suggested.
The quality engineering methods of Dr. Taguchi, employing design of experiments (DOE), is
one of the most important statistical tools of TQM for designing high quality systems at
reduced cost. Taguchi methods provide an efficient and systematic way to optimize designs
for performance, quality, and cost. Taguchi methods have been used successfully in Japan
and the United States in designing reliable, high quality products at low cost in such areas
as automobiles and consumer electronics.
WHITE PAPER
© Wipro Technologies Page : 04 of 12
Orthogonal Array: Applicability in Software Product Testing
The Taguchi Approach
Taguchi suggests five major steps in the designing process. The steps take the test team
from formulating the test problem to creating a good test design and refining the test design.
The steps are as follows:
• Formulate the problem
• Plan the experiment
• Analyze the results
• Confirm the experiment
• Adopt the new design
• Taguchi’s approach focuses on the first two steps since they are very important. The
stages are explained in the form of the following figure.
Formulate the Problem
Plan the Experiment
Analyze the results
Confirm the Improvement
Objective not met
steps in Taguchi approach
Objective met
Adopt the new design
Figure 1
Formulating the problem:
It is very much required for the design engineer to think the entire system in terms of parameters
that decide the outcome. Ideally, the problem definition should include two important things:
• The control variables
• The values each variable can take.
In this, the former is called as “Parameter” and the later is called as “Levels”. One more
important factor in this approach is that all the methodology is applicable only if the parameters
identified are independent.
Planning the experiment:
While planning the experiment, which is essentially the design of test case, Taguchi Analysis
uses a concept called Matrix Experiment using Orthogonal Arrays. It is an efficient way to
study the effect of several factors simultaneously. Orthogonal arrays offer many benefits.
First, the conclusions arrived at from such experiments are valid over the entire experimental
region spanned by the control factors and their settings. Second, there is a large saving in
the experimental effort. Third, the data analysis is very easy.
WHITE PAPER
© Wipro Technologies Page : 05 of 12
Orthogonal Array: Applicability in Software Product Testing
Matrix Experiment:
A Matrix Experiment consists of a set of experiments where we change the settings of the
various product or process parameters we want to study from one experiment to another.
After conducting a matrix experiment, the data from all experiments in the set taken together
are analyzed to determine the effects of the various parameters.
Constructing Matrix Experiments using special matrices, called “Orthogonal Arrays” allows
the effects of several parameters to be determined efficiently and is an important technique
in robust design. To actually construct an Orthogonal Array, control parameters or design
variables must be assigned to the columns of an array, and the integers in the array columns
are translated into the actual settings of the assigned parameters. The unassigned columns
are deleted from the array.
For instance, in a hypothetical design study, to evaluate effects of three design variables
(Parameters) on a product or process response, a designer selects two levels or settings
(Levels) for each design parameter:
Then as per Taguchi Approach
Parameter: 3
Level: 2
A suitable Matrix Experiment in this case uses the L4 Orthogonal Array as follows:
Experiment Column
Number
1
2
3
4
1 1 1
1 2 2
2 1 2
2 2 1
1 2 3
As depicted above, the L4 experiment consists of four rows and three columns where each
row corresponds to a particular experiment and each column identifies settings of a design
parameter. In the first run, for example, the three design variables are set at their low level
(level = 1). In the second run, the first parameter is set at level 1 and the remaining two
variables are set to high (level 2), and so on.
The following list contains some of the widely used Orthogonal Arrays, which use two and
three level design parameters:
L4: Accommodates three two-level design variables
Experiment Column
Number
1
2
3
4
1 1 1
1 2 2
2 1 2
2 2 1
1 2 3
WHITE PAPER
© Wipro Technologies Page : 06 of 12
Orthogonal Array: Applicability in Software Product Testing
i.e. 4 experiments have to be conducted for three factors two levels.
Without OA we need to conduct 23 = 8 experiments.
L8: Accommodates seven two-level design variables
Experiment
Number
1 2 3 4 5 6 7
Column
1
2
3
4
5
6
7
8
1 1 1 1 1 1 1
1 1 1 2 2 2 2
1 2 2 1 1 2 2
1 2 2 2 2 1 1
2 1 2 1 2 1 2
2 1 2 2 1 2 1
2 2 1 1 2 2 1
2 2 1 2 1 1 2
i.e. 8 experiments have to be conducted for 7 factors with 2 levels. Without OA we would have
to conduct 27 =128 experiments.
L9: Accommodates four three-level design variables
Experiment
Number
1 2 3 4
Column
1
2
3
4
5
6
7
8
9
1 1 1 1
1 2 2 2
1 3 3 3
2 1 2 3
2 2 3 1
2 3 1 2
3 1 3 2
3 2 1 3
3 3 2 1
i.e. 9 experiments have to be conducted for 4 factors with 3 levels. Without OA we would have
to conduct 34 = 81 experiments.
WHITE PAPER
© Wipro Technologies Page : 07 of 12
Orthogonal Array: Applicability in Software Product Testing
Verification Suite Development Using OA
A good analysis is required before deciding on the usage of OA for test suite development
with respect to identifying the parameters and levels. If the feature or the product, for which
the test case being developed, do not have a possibilities of defining the Parameters and
the respective Levels in a better way, it is suggested to look for alternatives than using OA.
It will be very effective and time saving if the test development team is closely interacts with
the feature or product development team from the RS phase with respect to defining the
Parameters and Levels. In case of test case re-writing, the Parameters and the Levels could
be arrived at by understating the feature/product itself. Help of the feature/product developer
may be obtained as applicable. Defining the Parameters and the respective Levels is the
crucial activity in the entire process.
Once the Parameters and Levels are defined, the next step is to come out with the optimized
combinations of test cases. Help of the tool called Minitab can be obtained for this purpose.
Minitab is a statistics tool from the company Minitab Inc. For Minitab the input is the Parameters
and levels and the output is the optimized test case combination.
Generally the code coverage is >90% by this optimized test combination, if the parameters
and levels selection is effective. At times it may not be practically possible to define parameters
and levels for all the functions defined in the product or feature and also there would be
requirements to have higher code coverage for products like mission critical applications. In
such cases manual test cases could be developed to support the optimized test combination.
This is advised since the effort what is required to identify the parameters in these cases
would be very high when compare to developing a test case in the conventional way.
Thus this method can give a very high code coverage which would ensure that the testing of
all functionality in a minimum time and cost.
In general the phase wise effort breakup in an OA test case development cycle could be as
given in the table below. As discussed earlier it is assumed that the test developer will
closely work with the feature/product developer from the RS defining stage to have the right
set of Parameters and the Levels which will make the test case more effective. In case of test
re-writing, there are chances that the study phase would be slightly higher since the test
developers would have to understand the feature/product on their own.
Phase Effort
Studying the feature and identifying the parameters • 15% of the test
development cycle time
Getting the optimum test combinations 5% of the test
Developing the test cases 60% of the test
development cycle time
Merging of the set-up scripts 5% of the test
development cycle time
Executing the tests and collecting data 5% of the test
development cycle time
Code coverage analysis, test enhancement and
test re-run 10% of the test
development cycle time
WHITE PAPER
© Wipro Technologies Page : 08 of 12
Orthogonal Array: Applicability in Software Product Testing
The table below provides the comparison between the conventional method and OA method
with respect to lines of test code, test execution time and code coverage. It is assumed that
the feature/product will be of medium complexity and code size.
Description Conventional OA Test Case
Test Case
Lines of Code 100% 30-40% of the
conventional test case
Test execution time 100% ~ 30% of the
conventional test case
Code coverage industry norm is Can be as high as 100%
60-70% (with the support of
manual test cases)
An experience Using OA
Re-Writing Join Index (JNIX) feature test suite using Orthogonal Array Concept (Taguchi
Method). This is an attempt to re-write the test cases for Join Index (JNIX) feature test suite
of a RDBMS. Orthogonal Array concept from Wire’s customized Six Sigma methodology was
used to get the best test combinations. The focus here is to cover the feature completely.
Hence all the possible parameters and levels were considered while developing the test
cases. 13 different parameters and their levels were identified. 36 valid test combinations
were obtained using Taguchi Method of Orthogonal Array. 2 additional test cases were
written to test the join index properties which are not part of the 36 original test combinations.
In total, 38 test cases (with 1367 LOC) were developed. These 38 test cases were executed
and the code coverage data is obtained.
If we had used Full Factorial method of Orthogonal Arrays instead of Taguchi method, we
would have ended up writing 139968 test cases to test all possible combinations of
parameters. This means that what the 139968 test cases of Full Factorial method covers,
Taguchi method of OA covers the same with just 38 test cases, without sacrificing on
confidence level. This would have been the same case if we had used the conventional
method of developing test cases.
“Minitab” tool was used to get the test combinations for OA Taguchi method. A code coverage
tool, which is part of an OS, has been used to measure the code coverage.
Following are the example of parameters and levels which were considered for testing join
index on tables. Each row indicates one test case which gives various combinations of
levels for every parameter.
WHITE PAPER
© Wipro Technologies Page : 09 of 12
Orthogonal Array: Applicability in Software Product Testing
Parameters Levels
1 Base Table type Set Multiset
2 Fault tolerance for base tables Fallback
No fallback
3 Database (s) in which the base tables reside 1
2
4 Fault Tolerance for Join Index Fallback
No fallback
The table below has the example of the optimized test combination which is the output of the
tool Minitab. Each row can be treated as input/combination for a test case/suite.
The table below gives the actual result/benefit of OA test case when compared with the
existing test case which was generated using conventional method. The data shows
encouraging results of the optimized test case.
Existing Test Case New Test Case
Number of test cases 117 38
LOC for test cases 28122 1367
Execution time 8 hours 25 minutes
Source lines covered 70% of the feature 95% of existing test case
WHITE PAPER
© Wipro Technologies Page : 10 of 12
Orthogonal Array: Applicability in Software Product Testing
OA Test Case Sustenance Strategy
After the product/feature reaches the market, generally, the field errors, customer feed back,
product/feature enhancement etc. are carried out either through DR (defect report) process
or RFC (request for change) process. This results in addition of test cases to test the new
code.
Very often the newly generated test cases to verify the code changes in the feature/product
are added to existing optimized test suites directly without any optimization. This practice
may defeat the purpose of optimized test concept using OA at times. It is time consuming to
identify Parameters and Levels for small fixes, use tools like Minitab to define the optimization
etc. In such cases it is advised to revisit the test case in a fixed interval, based on the field
error rate or the feature enhancement, to address the newly added test for optimization.
Though it looks like taking additional effort, in practical it is very small when compare to the
effort and time what is saved because of the optimization.
Benefits of OA
• Productivity improvement:
• Implementation time is less: The implementation time is less, as the test
cases are less. Even though the test suit count is less, the combination
will be optimum and covers the basic feature completely. This is evident
from the fact that only 38 test cases of 1,367 LOC were written compared
to the existing 117 test cases of 28,122 LOC to test the join index feature.
• Execution time is less: Since the test cases are less in new test suite
compared to the conventional methods, test execution time is less. The
existing tests used to take 8 hours to complete where as the new test
cases runs in just 25 minutes to test the join index feature.
• Result analysis takes less time: Since the number of test cases and the
LOC for the test cases is less, the result file size is going to be less
compared to the old tests. Hence time taken to compare the result files
with the standard files is going to be less. This will further reduce the
overall test execution time.
• Increase in overall productivity: All the above points provide evidence to the
fact that the OA methodology increases the overall productivity since lesser
number of test cases are written. Implementation time, test execution time
and the result analysis time is less compared to the other methods. Hence,
OA concept increases the overall productivity by 40%-50%.
• Quality:
• High code coverage: As evidenced in the join index experience, OA
methodology covers close to 95% of the feature code compared to the
usual methods. This is achieved with less number of test cases and with
less execution time. For the remaining 5% of the code, tests have to be
written manually. In this way, close to 100% of the feature code is covered.
• Right the first Time: In a conventional method of Test case development,
the Test Design mostly depends on the capabilities of the test developer.
This might result in redesign, large review defects and thus impact the
over all quality of the Test cases. This is avoided to a large extent if OA is
used. Since the parameters and levels are defined after good analysis
and input from the feature/product developers themselves, this can result
in Right the First Time delivery and thus reduce the probabilities of
redesign, and review rework.
WHITE PAPER
Page :
Conclusion
It is beneficial to the Software product testing to make use of the Orthogonal Arrays. The only
constraint would be the applicability of the concept to certain types of products. There are 3
areas in a product life cycle management we can get benefited by adopting OA.
• Test case development for new feature/product
• Re designing the existing test cases
• Sustenance of the test cases thus developed.
In all the three areas, Software product companies can enjoy the benefits of OA, which were
detailed through out this document.
The most critical portion of the adoption process is the product understanding and modeling
the product/ feature as Parameters and levels. The care and efforts put in to develop this
model will drive the success of the test suite design. If the Software product company does
not have enough experience with OA and Six Sigma concepts, it will be worthwhile investing
in hiring professionals trained in these concepts and experienced in software testing.
About the Author
Selvam Ponnusamy: Has more than 15 years of IT experience. He has spent more than 10
years on hardware and software testing. Selvam is a certified Black Belt in the applications
of Six Sigma concepts and one of the pioneers of Wipro in adapting OA for testing software
products. Selvam has been a practitioner of quality systems like ISO 9001, CMMi, Six Sigma
concepts and software testing. He is a thought leader in Wipro in software product testing.
About Wipro Technologies
Wipro is the first PCMM Level 5 and SEI CMMi Level 5 certified IT Services Company globally.
Wipro provides comprehensive IT solutions and services (including Systems Integration, IS
Outsourcing, Package Implementation, Software Application Development and Maintenance)
and Research & Development services (hardware and software design, development and
implementation) to corporations globally.
Wipro’s unique value proposition is further delivered through the pioneering Offshore
Outsourcing Model and stringent quality processes of SEI and Six Sigma.
© Wipro Technologies 11 of 12
Orthogonal Array: Applicability in Software Product Testing
WHITE PAPER
Page : 12 of 12
For further information visit us at: http://www.wipro.com/productoutsourcing
For more whitepapers logon to: http://www.wipro.com/insights
© Copyright 2004. Wipro Technologies. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in
any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without express written permission from Wipro Technologies.
Specifications subject to change without notice. All other trademarks mentioned herein are the property of their respective owners. Specifications
subject to change without notice.
Wipro Technologies
Innovative Solutions, Quality Leadership
www.wipro.com
eMail: info@wipro.com
Six Sigma in Wipro
Wipro studied many existing quality systems – the Japanese and the American models.
After carefully studying these, Six Sigma was chosen. The reasons were:
• It is very measurable unlike the other quality systems which are more open ended.
• It is a tool, a methodology to remove defects in products, processes and services.
• It encompassed manufacturing and transactional quality.
• There was a world-class consulting agency to guide us for the first 24 months.
• It can give us a competitive edge which could sustain us for a decade or more.
• It is global and world leaders like GE and Citi Bank have it.
• It focused on the customer who has become so important in today’s business.
• It was project based and hence project successes gave us intermediate milestones
to measure ourselves by.
• Six Sigma is based on data, on facts.
Six Sigma can be integrated with the other quality initiatives that various units have embarked
upon – like SEI and ISO. If SEI and ISO form the base, Six Sigma gave the tools and
methodology to remove the defects.
Wipro in Software Product Outsourcing
Wipro provides development, maintenance, QA, and technical support services to a number
of software product companies. Wipro has a unique track record of working with 14 of the 23
leading Fortune 500 technology firms that work with Indian partners. In 2003, Wipro become
the first organization outside the US to be awarded the prestigious IEEE Software Process
Achievement award.
Wipro Technologies
Innovative Solutions, Quality Leadership
Author: Selvam Ponnusamy
Orthogonal Array: Applicability in
Software Product Testing
Testing software product effectiveness can be improved by testing all possible combinations.
However, it is very difficult to test all possible combinations; many a times it is impossible. So the
other way to improve testing effectiveness is through automation. Both the processes require
high testing cost and time. There are solutions to this pain area using the famous Six Sigma
methodologies which provide tools like Orthogonal Arrays (OA).
Orthogonal Array and ideas from the Design of Experiments (DOE) help in improving the Software
product testing. This white paper introduces Orthogonal Arrays, explains the process of using
OA concepts to improve the testing process.
WHITE PAPER
TaPbaleg eo f: Contents
Table of Contents
Orthogonal Array: Applicability in Software Product Testing
INTRODUCTION ...................................................................................... 3
WHAT IS SIX SIGMA ................................................................................. 3
WHAT IS ORTHOGONAL ARRAY............................................................ 3
THE TAGUCHI APPROACH .........................................................................3
VERIFICATION SUITE DEVELOPMENT USING OA.............................. 7
AN EXPERIENCE USING OA .................................................................. 8
OA TEST CASE SUSTENANCE STRATEGY ....................................... 10
CONCLUSION ........................................................................................ 11
ABOUT THE AUTHOR ........................................................................... 11
ABOUT WIPRO TECHNOLOGIES ........................................................ 11
SIX SIGMA IN WIPRO............................................................................. 12
WIPRO IN SOFTWARE PRODUCT OUTSOURCING .......................... 12
WHITE PAPER
© Wipro Technologies Page : 03 of 12
Orthogonal Array: Applicability in Software Product Testing
Introduction
Testing today’s Software Products is complex as it is difficult to model the product in a way
that it is easy to understand the product’s behavior under various conditions. There are
multiple factors that affect the software product and these factors affect the behavior
simultaneously. This leads to spaghetti of test suites. The more the product evolves the
more complicated the test suite becomes. This makes the test result analysis, DR reporting
and DR analysis inefficient. The result is pouring more and more costly resources in to the
test activities.
These things have led to SW companies to spend as much on testing as they do on product
development. Mature industries such as the automotive industry, spend as little as 5% of
total costs on quality assurance, and have far better results.
What is Six Sigma
The term Six Sigma Quality comes from Motorola. Today, it is synonymous with world class.
Sigma is a statistical expression that indicates how defect free a process or a product is. Let
us say, to make a product we go through 10 steps. Each step is an opportunity to make a
defect. Thus, if we were to make 100,000 pieces of the given product, there will be a million
opportunities to make defects.
If our defect rate is at Three Sigma level, we make 66,807 mistakes. At Six Sigma level, the
defect rate drops to 3.4 mistakes in a million. Average processes and products operate at
Three Sigma level. The best in class processes and products are at Six Sigma level. The
secret of moving from Three Sigma to world class is use of quality at all levels and in all
functions.
What is Orthogonal Array
Orthogonal Array is a method of choosing a set of tests from a universe of tests, to make the
testing efficient and effective. It is based on creating Parameters (control Variables) and
Levels (values) for the parameters, which are the inputs to testable functions.
There are different algorithms to choose the parameters and levels from the universe of
available values. One such method was proposed by Dr Taguchi. This white paper also
deals with the OA method what Taguchi has suggested.
The quality engineering methods of Dr. Taguchi, employing design of experiments (DOE), is
one of the most important statistical tools of TQM for designing high quality systems at
reduced cost. Taguchi methods provide an efficient and systematic way to optimize designs
for performance, quality, and cost. Taguchi methods have been used successfully in Japan
and the United States in designing reliable, high quality products at low cost in such areas
as automobiles and consumer electronics.
WHITE PAPER
© Wipro Technologies Page : 04 of 12
Orthogonal Array: Applicability in Software Product Testing
The Taguchi Approach
Taguchi suggests five major steps in the designing process. The steps take the test team
from formulating the test problem to creating a good test design and refining the test design.
The steps are as follows:
• Formulate the problem
• Plan the experiment
• Analyze the results
• Confirm the experiment
• Adopt the new design
• Taguchi’s approach focuses on the first two steps since they are very important. The
stages are explained in the form of the following figure.
Formulate the Problem
Plan the Experiment
Analyze the results
Confirm the Improvement
Objective not met
steps in Taguchi approach
Objective met
Adopt the new design
Figure 1
Formulating the problem:
It is very much required for the design engineer to think the entire system in terms of parameters
that decide the outcome. Ideally, the problem definition should include two important things:
• The control variables
• The values each variable can take.
In this, the former is called as “Parameter” and the later is called as “Levels”. One more
important factor in this approach is that all the methodology is applicable only if the parameters
identified are independent.
Planning the experiment:
While planning the experiment, which is essentially the design of test case, Taguchi Analysis
uses a concept called Matrix Experiment using Orthogonal Arrays. It is an efficient way to
study the effect of several factors simultaneously. Orthogonal arrays offer many benefits.
First, the conclusions arrived at from such experiments are valid over the entire experimental
region spanned by the control factors and their settings. Second, there is a large saving in
the experimental effort. Third, the data analysis is very easy.
WHITE PAPER
© Wipro Technologies Page : 05 of 12
Orthogonal Array: Applicability in Software Product Testing
Matrix Experiment:
A Matrix Experiment consists of a set of experiments where we change the settings of the
various product or process parameters we want to study from one experiment to another.
After conducting a matrix experiment, the data from all experiments in the set taken together
are analyzed to determine the effects of the various parameters.
Constructing Matrix Experiments using special matrices, called “Orthogonal Arrays” allows
the effects of several parameters to be determined efficiently and is an important technique
in robust design. To actually construct an Orthogonal Array, control parameters or design
variables must be assigned to the columns of an array, and the integers in the array columns
are translated into the actual settings of the assigned parameters. The unassigned columns
are deleted from the array.
For instance, in a hypothetical design study, to evaluate effects of three design variables
(Parameters) on a product or process response, a designer selects two levels or settings
(Levels) for each design parameter:
Then as per Taguchi Approach
Parameter: 3
Level: 2
A suitable Matrix Experiment in this case uses the L4 Orthogonal Array as follows:
Experiment Column
Number
1
2
3
4
1 1 1
1 2 2
2 1 2
2 2 1
1 2 3
As depicted above, the L4 experiment consists of four rows and three columns where each
row corresponds to a particular experiment and each column identifies settings of a design
parameter. In the first run, for example, the three design variables are set at their low level
(level = 1). In the second run, the first parameter is set at level 1 and the remaining two
variables are set to high (level 2), and so on.
The following list contains some of the widely used Orthogonal Arrays, which use two and
three level design parameters:
L4: Accommodates three two-level design variables
Experiment Column
Number
1
2
3
4
1 1 1
1 2 2
2 1 2
2 2 1
1 2 3
WHITE PAPER
© Wipro Technologies Page : 06 of 12
Orthogonal Array: Applicability in Software Product Testing
i.e. 4 experiments have to be conducted for three factors two levels.
Without OA we need to conduct 23 = 8 experiments.
L8: Accommodates seven two-level design variables
Experiment
Number
1 2 3 4 5 6 7
Column
1
2
3
4
5
6
7
8
1 1 1 1 1 1 1
1 1 1 2 2 2 2
1 2 2 1 1 2 2
1 2 2 2 2 1 1
2 1 2 1 2 1 2
2 1 2 2 1 2 1
2 2 1 1 2 2 1
2 2 1 2 1 1 2
i.e. 8 experiments have to be conducted for 7 factors with 2 levels. Without OA we would have
to conduct 27 =128 experiments.
L9: Accommodates four three-level design variables
Experiment
Number
1 2 3 4
Column
1
2
3
4
5
6
7
8
9
1 1 1 1
1 2 2 2
1 3 3 3
2 1 2 3
2 2 3 1
2 3 1 2
3 1 3 2
3 2 1 3
3 3 2 1
i.e. 9 experiments have to be conducted for 4 factors with 3 levels. Without OA we would have
to conduct 34 = 81 experiments.
WHITE PAPER
© Wipro Technologies Page : 07 of 12
Orthogonal Array: Applicability in Software Product Testing
Verification Suite Development Using OA
A good analysis is required before deciding on the usage of OA for test suite development
with respect to identifying the parameters and levels. If the feature or the product, for which
the test case being developed, do not have a possibilities of defining the Parameters and
the respective Levels in a better way, it is suggested to look for alternatives than using OA.
It will be very effective and time saving if the test development team is closely interacts with
the feature or product development team from the RS phase with respect to defining the
Parameters and Levels. In case of test case re-writing, the Parameters and the Levels could
be arrived at by understating the feature/product itself. Help of the feature/product developer
may be obtained as applicable. Defining the Parameters and the respective Levels is the
crucial activity in the entire process.
Once the Parameters and Levels are defined, the next step is to come out with the optimized
combinations of test cases. Help of the tool called Minitab can be obtained for this purpose.
Minitab is a statistics tool from the company Minitab Inc. For Minitab the input is the Parameters
and levels and the output is the optimized test case combination.
Generally the code coverage is >90% by this optimized test combination, if the parameters
and levels selection is effective. At times it may not be practically possible to define parameters
and levels for all the functions defined in the product or feature and also there would be
requirements to have higher code coverage for products like mission critical applications. In
such cases manual test cases could be developed to support the optimized test combination.
This is advised since the effort what is required to identify the parameters in these cases
would be very high when compare to developing a test case in the conventional way.
Thus this method can give a very high code coverage which would ensure that the testing of
all functionality in a minimum time and cost.
In general the phase wise effort breakup in an OA test case development cycle could be as
given in the table below. As discussed earlier it is assumed that the test developer will
closely work with the feature/product developer from the RS defining stage to have the right
set of Parameters and the Levels which will make the test case more effective. In case of test
re-writing, there are chances that the study phase would be slightly higher since the test
developers would have to understand the feature/product on their own.
Phase Effort
Studying the feature and identifying the parameters • 15% of the test
development cycle time
Getting the optimum test combinations 5% of the test
Developing the test cases 60% of the test
development cycle time
Merging of the set-up scripts 5% of the test
development cycle time
Executing the tests and collecting data 5% of the test
development cycle time
Code coverage analysis, test enhancement and
test re-run 10% of the test
development cycle time
WHITE PAPER
© Wipro Technologies Page : 08 of 12
Orthogonal Array: Applicability in Software Product Testing
The table below provides the comparison between the conventional method and OA method
with respect to lines of test code, test execution time and code coverage. It is assumed that
the feature/product will be of medium complexity and code size.
Description Conventional OA Test Case
Test Case
Lines of Code 100% 30-40% of the
conventional test case
Test execution time 100% ~ 30% of the
conventional test case
Code coverage industry norm is Can be as high as 100%
60-70% (with the support of
manual test cases)
An experience Using OA
Re-Writing Join Index (JNIX) feature test suite using Orthogonal Array Concept (Taguchi
Method). This is an attempt to re-write the test cases for Join Index (JNIX) feature test suite
of a RDBMS. Orthogonal Array concept from Wire’s customized Six Sigma methodology was
used to get the best test combinations. The focus here is to cover the feature completely.
Hence all the possible parameters and levels were considered while developing the test
cases. 13 different parameters and their levels were identified. 36 valid test combinations
were obtained using Taguchi Method of Orthogonal Array. 2 additional test cases were
written to test the join index properties which are not part of the 36 original test combinations.
In total, 38 test cases (with 1367 LOC) were developed. These 38 test cases were executed
and the code coverage data is obtained.
If we had used Full Factorial method of Orthogonal Arrays instead of Taguchi method, we
would have ended up writing 139968 test cases to test all possible combinations of
parameters. This means that what the 139968 test cases of Full Factorial method covers,
Taguchi method of OA covers the same with just 38 test cases, without sacrificing on
confidence level. This would have been the same case if we had used the conventional
method of developing test cases.
“Minitab” tool was used to get the test combinations for OA Taguchi method. A code coverage
tool, which is part of an OS, has been used to measure the code coverage.
Following are the example of parameters and levels which were considered for testing join
index on tables. Each row indicates one test case which gives various combinations of
levels for every parameter.
WHITE PAPER
© Wipro Technologies Page : 09 of 12
Orthogonal Array: Applicability in Software Product Testing
Parameters Levels
1 Base Table type Set Multiset
2 Fault tolerance for base tables Fallback
No fallback
3 Database (s) in which the base tables reside 1
2
4 Fault Tolerance for Join Index Fallback
No fallback
The table below has the example of the optimized test combination which is the output of the
tool Minitab. Each row can be treated as input/combination for a test case/suite.
The table below gives the actual result/benefit of OA test case when compared with the
existing test case which was generated using conventional method. The data shows
encouraging results of the optimized test case.
Existing Test Case New Test Case
Number of test cases 117 38
LOC for test cases 28122 1367
Execution time 8 hours 25 minutes
Source lines covered 70% of the feature 95% of existing test case
WHITE PAPER
© Wipro Technologies Page : 10 of 12
Orthogonal Array: Applicability in Software Product Testing
OA Test Case Sustenance Strategy
After the product/feature reaches the market, generally, the field errors, customer feed back,
product/feature enhancement etc. are carried out either through DR (defect report) process
or RFC (request for change) process. This results in addition of test cases to test the new
code.
Very often the newly generated test cases to verify the code changes in the feature/product
are added to existing optimized test suites directly without any optimization. This practice
may defeat the purpose of optimized test concept using OA at times. It is time consuming to
identify Parameters and Levels for small fixes, use tools like Minitab to define the optimization
etc. In such cases it is advised to revisit the test case in a fixed interval, based on the field
error rate or the feature enhancement, to address the newly added test for optimization.
Though it looks like taking additional effort, in practical it is very small when compare to the
effort and time what is saved because of the optimization.
Benefits of OA
• Productivity improvement:
• Implementation time is less: The implementation time is less, as the test
cases are less. Even though the test suit count is less, the combination
will be optimum and covers the basic feature completely. This is evident
from the fact that only 38 test cases of 1,367 LOC were written compared
to the existing 117 test cases of 28,122 LOC to test the join index feature.
• Execution time is less: Since the test cases are less in new test suite
compared to the conventional methods, test execution time is less. The
existing tests used to take 8 hours to complete where as the new test
cases runs in just 25 minutes to test the join index feature.
• Result analysis takes less time: Since the number of test cases and the
LOC for the test cases is less, the result file size is going to be less
compared to the old tests. Hence time taken to compare the result files
with the standard files is going to be less. This will further reduce the
overall test execution time.
• Increase in overall productivity: All the above points provide evidence to the
fact that the OA methodology increases the overall productivity since lesser
number of test cases are written. Implementation time, test execution time
and the result analysis time is less compared to the other methods. Hence,
OA concept increases the overall productivity by 40%-50%.
• Quality:
• High code coverage: As evidenced in the join index experience, OA
methodology covers close to 95% of the feature code compared to the
usual methods. This is achieved with less number of test cases and with
less execution time. For the remaining 5% of the code, tests have to be
written manually. In this way, close to 100% of the feature code is covered.
• Right the first Time: In a conventional method of Test case development,
the Test Design mostly depends on the capabilities of the test developer.
This might result in redesign, large review defects and thus impact the
over all quality of the Test cases. This is avoided to a large extent if OA is
used. Since the parameters and levels are defined after good analysis
and input from the feature/product developers themselves, this can result
in Right the First Time delivery and thus reduce the probabilities of
redesign, and review rework.
WHITE PAPER
Page :
Conclusion
It is beneficial to the Software product testing to make use of the Orthogonal Arrays. The only
constraint would be the applicability of the concept to certain types of products. There are 3
areas in a product life cycle management we can get benefited by adopting OA.
• Test case development for new feature/product
• Re designing the existing test cases
• Sustenance of the test cases thus developed.
In all the three areas, Software product companies can enjoy the benefits of OA, which were
detailed through out this document.
The most critical portion of the adoption process is the product understanding and modeling
the product/ feature as Parameters and levels. The care and efforts put in to develop this
model will drive the success of the test suite design. If the Software product company does
not have enough experience with OA and Six Sigma concepts, it will be worthwhile investing
in hiring professionals trained in these concepts and experienced in software testing.
About the Author
Selvam Ponnusamy: Has more than 15 years of IT experience. He has spent more than 10
years on hardware and software testing. Selvam is a certified Black Belt in the applications
of Six Sigma concepts and one of the pioneers of Wipro in adapting OA for testing software
products. Selvam has been a practitioner of quality systems like ISO 9001, CMMi, Six Sigma
concepts and software testing. He is a thought leader in Wipro in software product testing.
About Wipro Technologies
Wipro is the first PCMM Level 5 and SEI CMMi Level 5 certified IT Services Company globally.
Wipro provides comprehensive IT solutions and services (including Systems Integration, IS
Outsourcing, Package Implementation, Software Application Development and Maintenance)
and Research & Development services (hardware and software design, development and
implementation) to corporations globally.
Wipro’s unique value proposition is further delivered through the pioneering Offshore
Outsourcing Model and stringent quality processes of SEI and Six Sigma.
© Wipro Technologies 11 of 12
Orthogonal Array: Applicability in Software Product Testing
WHITE PAPER
Page : 12 of 12
For further information visit us at: http://www.wipro.com/productoutsourcing
For more whitepapers logon to: http://www.wipro.com/insights
© Copyright 2004. Wipro Technologies. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in
any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without express written permission from Wipro Technologies.
Specifications subject to change without notice. All other trademarks mentioned herein are the property of their respective owners. Specifications
subject to change without notice.
Wipro Technologies
Innovative Solutions, Quality Leadership
www.wipro.com
eMail: info@wipro.com
Six Sigma in Wipro
Wipro studied many existing quality systems – the Japanese and the American models.
After carefully studying these, Six Sigma was chosen. The reasons were:
• It is very measurable unlike the other quality systems which are more open ended.
• It is a tool, a methodology to remove defects in products, processes and services.
• It encompassed manufacturing and transactional quality.
• There was a world-class consulting agency to guide us for the first 24 months.
• It can give us a competitive edge which could sustain us for a decade or more.
• It is global and world leaders like GE and Citi Bank have it.
• It focused on the customer who has become so important in today’s business.
• It was project based and hence project successes gave us intermediate milestones
to measure ourselves by.
• Six Sigma is based on data, on facts.
Six Sigma can be integrated with the other quality initiatives that various units have embarked
upon – like SEI and ISO. If SEI and ISO form the base, Six Sigma gave the tools and
methodology to remove the defects.
Wipro in Software Product Outsourcing
Wipro provides development, maintenance, QA, and technical support services to a number
of software product companies. Wipro has a unique track record of working with 14 of the 23
leading Fortune 500 technology firms that work with Indian partners. In 2003, Wipro become
the first organization outside the US to be awarded the prestigious IEEE Software Process
Achievement award.
No comments:
Post a Comment