Executable Test Matrix
And
Field Validation Matrix
International Software Testing Conference - 2009
Author:
G. Shanmugapriyan
Shanmugapriyan.ganapathy@honeywell.com
Honeywell Technology Solutions, Bangalore
2
1.0 Abstract:
1.1 Introduction:
In scenarios where the application has complex UI’s and have more data input
fields it will be time consuming for designing the test cases for entire UI and for each
individual field level validations. Also assuring 100% testing coverage would be herculean
task. This document is all about describing the new test case design technique called
Executable Test Matrix (ETM) and Field Validation Matrix (FVM) which identified to
overcome the current complex situations. ETM and FVM techniques yields better test
coverage and productivity savings and also help us to have the consolidated defect
metrics for each individual build automatically with better metrics management.
1.2 Audience:
This technique will help the testing team at various levels by helping the tester for
quicker test design and execution with improved accuracy, helps in better test coverage
and certification for leads and metrics management for managers. Also this will help the
team to design the test cases for each individual field level with Data type, Field type,
Boundary value analysis, Equivalence class partitioning etc., with positive and negative
validations.
1.3 Benefits:
To design the test cases in very short duration for huge and complex UI base
applications
To design the test cases for individual field level validations with positive and
negative scenarios
To design the test cases when different set of modules having the same kind of
verification conditions
Productivity savings through framework reuse in multiple projects
Less rework and COPQ
100% test coverage
Quick turn around time for test design phase
3
2.0 Table of Contents:
1.0Abstract:............................................................................................................... 2
2.0Table of Contents:................................................................................................ 3
3.0Introduction:......................................................................................................... 4
4.0Problem Description:............................................................................................ 4
4.1Description 1:................................................................................................... 4
4.2Description 2:................................................................................................... 4
5.0Problem Solution:................................................................................................. 5
5.1Solution 1:........................................................................................................ 5
5.2Solution 2:........................................................................................................ 5
6.0Test design Models:..............................................................................................6
6.1Executable Test Matrix:................................................................................... 6
6.2Field Validation Matrix:................................................................................... 9
7.0Conclusions:....................................................................................................... 11
8.0Acronyms:.......................................................................................................... 12
9.0 Bibliography about the author:.......................................................................... 12
4
3.0 Introduction:
In most of the cases even though the functionality is heart of the
application, the UI become the face of it. It inspires the users in such a way
that the user will revisit the application again because of the UI. To have
such feelings, when the application is developed it should have the entire
coverage and focus on both functionality and UI as well.
In this process, system testing will do the major role to ensure the
quality of the application, look & feel and usability stuffs. In this system
testing, designing the test cases will play the major role. If the test case
ensures the test coverage, then the execution part becomes easier. So
designing the test cases become the important aspect for the testers.
4.0 Problem Description:
4.1 Description 1:
If the size of the application is very huge and complicated, it will be
more time consuming for designing the test cases for UI and also it will eat
up the time of test design for the functionality as well. Also it is very risky to
ensure the 100% test coverage and no schedule slippage due to the
enlarged test case design for huge UI.
Challenges faced:
Time consuming for designing the test cases for entire
application UI (Including alignment, spell check, navigation
etc.,)
To ensure the 100% test coverage
Schedule slippage due to time consuming for designing the
test cases in normal way
4.2 Description 2:
Also, if the application business core is to get large amount of data
from user and save it to the database means then the application will
contain more input fields like text boxes, text area, radio button and combo
buttons etc., Designing the test cases for all those input boxes with all
5
positive and negative validations will be more time consuming and also
there will be a chance of missing the fields since need to design the test
cases for all input fields in normal way of design.
Challenges faced:
Time consuming for designing the test cases for all the
input fields including positive and negative validations
To ensure the 100% test coverage
Schedule slippage due to time consuming for designing the
test cases in normal way
5.0 Problem Solution:
5.1 Solution 1:
Identified the new test design technique, Executable Test Matrix
(ETM) which will help you to design the test cases in very short duration and
ensure the 100% test coverage. We will have the coverage for functionality
and UI in easily understandable way. Also, we could able to complete the
test design in very short duration which will result you in saving the
productivity.
Challenges overcome:
Able to design the test cases in very short duration
100% test coverage ensured
Able to finish the test design activity before stipulated time
period
Productivity savings
5.2 Solution 2:
Identified the new test design technique, Field Validation Matrix
(FVM) which will help you to design the test cases in very short duration and
ensure the 100% test coverage. We will have the focus on each individual
field level validation for all the input fields. Also, all the positive and negative
6
validations will be taken care and could able to complete the test design in
very short duration which will result in the productivity savings.
Challenges overcome:
Able to design the test cases in very short duration
Focus on each individual field level validation
100% test coverage ensured
Able to finish the test design activity before stipulated time
period
Productivity savings
6.0 Test design Models:
6.1 Executable Test Matrix:
Objective of this Executable Test Matrix is to have the 100% test
coverage of the UI and able to finish the test design in very short duration.
The executable test matrix has been designed using Microsoft excel. The
modules which are going to test are aligned in horizontal manner and the
test actions we are going to perform are aligned in vertical manner. Please
find the below sample format of ETM
7
In the above format, 0 indicates the pass and 1 indicates the fail.
Also, this technique includes the calculation of metrics with number
of checks passed and number of checks failed. With this metric we could
able to calculate the total checks passed and failed report for the respective
builds executed. Please refer the below picture for the consolidated pass fail
report in the build 1
When we start the test execution for the build 2 we will identify the
difference with the set of color code to the cell. For example see the below
picture in which the failed test cases are verified in the build 2 which are
identified by different color code.
In the above picture the different color code shows about the
defects which are verified in the build 2. Also, if there are any reopened
defects then the respective cell will be differentiated with different color code
and cell values. Please note the above picture which is mentioned the value
as “2” for reopened defects.
8
We also having the forward and backward traceability implemented
with the above technique. We have captured the requirement tag for each
module and from we derived the test case id for each and every check. If
any of the checks get fails then we will differentiate with color and mention it
as “1”. Also, defect id which logged for that particular defect will be captured
in the comment for that particular cell. In this way we will have the forward
traceability from requirements and backward traceability from defects
logged.
When all the defects get fixed and verified in the final build, the
consolidated defect metrics will get calculated automatically based on the
test result inputs. If there are any opened defects then we could able identify
with the consolidated metrics displayed. If all the checks are passed then
the metrics will be displayed as follows;
The below results indicates the outcome of executable test matrix
implementation in real time.
9
40 36
35
30
25
Normal
20 16
13.5 ETM
15
10 6
5
0
Hours taken TC's/hour
Advantages:
Easy to use and able to design it in very short duration
Ensured amount of test coverage
Productivity savings due to very short duration of test design
Pass Fail report for each individual module
Automatically calculated consolidated test result based on the test
results input
Limitations:
ETM cannot be used with the same design across multiple builds;
you have to copy the design and paste it and have the track of
defects for the subsequent builds.
6.2 Field Validation Matrix:
Objective of this Field Validation Matrix is to have the coverage and
focus on each individual field level. The Field Validation Matrix has been
designed using Microsoft excel. The fields which are going to test are
aligned in the horizontal manner and the test actions we are going to
perform are aligned in the vertical manner. The FVM contains the positive
10
and negative validations for each individual field level. Please find the below
sample format of FVM.
In the above format, 0 indicates pass and 1 indicates fail.
Limitations:
FVM cannot be used with the same design across multiple
builds; you have to copy the design and paste it and have the track of
defects for the second build. For example see the below picture in which the
failed test cases are verified in the build 2 which are identified by different
color code.
11
The below results indicates the outcome of field validation matrix
implementation.
30
24.16
25
20 18.12
Normal
15
FVM
10 8
6
5
0
Hours taken TC's/hour
7.0 Conclusions:
The above best practices would certainly help the system testers to
design the test cases in efficient way. They could have the 100% test
coverage and able to complete it within very short duration and achieve the
productivity savings. This could help the users to design the test cases for
individual field level with positive and negative validations within the
stipulated time.
12
8.0 Acronyms:
Acronyms Description
ETM Executable Test Matrix
FVM Field Validation Matrix
TC’s Test cases
UI User Interface
9.0 Bibliography about the author:
Shanmugapriyan Ganapathy is a Software Engineer with
Honeywell Technology Solution Pvt. Ltd. He has 4 years of experience in
software testing. Priyan being a first class post graduate in Master of
Computer Application from Anna University, joined Master mind info tech,
Bangalore in 2005 April as a Test engineer. After a year he joined
Honeywell Technology solutions as a Software Test Engineer and was
inducted into testing team. Priyan who is an ISTQB certified tester worked
on various web service and desktop applications. He is also a Six Sigma
green belt certified test engineer with a good capability of learning quickly
the projects business as well as functional requirements. Priyan has
received many awards and acknowledgements for his meritorious
contribution to the projects & quick learning.
