Release Based Testing

Author : Rupa Joshi .

rupaj@techmahindra.com

STC Conference 2009 Tech Mahindra Limited

Sharda Centre, Off Karve Road,
Pune – 411004
Maharashtra, India.

© Tech Mahindra Limited 2008

 -2-

Abstract
The purpose of this white paper is to introduce the concept of Release Based Testing and to
help the testing teams to build the culture where testing teams work in unison for successful
and timely release. The release based testing is not a new type of testing; it is a process
and a culture that has to be built across all teams involved in the release.

The white paper covers the Organisation level impact of changing the trend of product based
testing to release based testing. The paper explains how to model your release based
testing process and phase wise approach to release testing.

The paper is concluded with the Practical Experience in transforming the product based
testing practice to release based testing for a major Telecom Service Provider.

Table of Contents
RELEASE BASED TESTING..................................................................................................................................... 1
INTRODUCTION......................................................................................................................................................... 3
MOVING FROM PRODUCT BASED TESTING TO RELEASE BASED TESTING......................................... 4
Organisation Level Impact.......................................................................................................................................... 4
Involvement of Various Teams .................................................................................................................................. 4
Impact on Testing Teams........................................................................................................................................ 6
Impact on Release Teams........................................................................................................................................6
Why Test Software Releases?..................................................................................................................................... 7
INTRODUCING RELEASE BASED TESTING PROCESS....................................................................................8
Model Your Release Testing Process........................................................................................................................ 10
RELEASE BASED TESTING – PRACTICAL EXPERIENCE: .......................................................................... 11
CONCLUSION:...........................................................................................................................................................12
REFERENCES............................................................................................................................................................ 13
AUTHORS' PROFILE................................................................................................................................................13

Release Based Testing White Paper STC Conference 2009

INTRODUCTION

All managers were seated in that cozy, big, well lit Meeting Room. There was hushed
discussion going on amongst all managers attending the meeting called by their CEO. The
invitees were the managers from many teams - Change Management team, Release
Management team, Product Development Teams, Configuration Management team,
Environment Management Team and of course the Centralized Testing Team.
The agenda of the meeting was conveyed to all the managers. It simply said, “Moving from
Product Based Testing to Release Based Testing”. Many managers except the Test Manager
were a bit baffled because they never thought that the subject could have any relevance to
their work.

At the stroke of 10:00 AM, the CEO entered the room, a respected personality in the
organisation, known for his vision and lookout of future trends. He glanced at the audience
and gave a nod of satisfaction.
“My dear Colleagues”, he started with the meeting, “as you all know, today’s one and only
agenda is to discuss the implications of the management decision of moving from the
culture of ‘Product Based testing to Release Based Testing’. You might be wondering as to
how the other streams are related to this topic and why I have not briefed it personally to
only the Test Manager”.

“Well, I had a series of discussions with the Test Manager before this meeting, and I am
sure she has come prepared today to abreast you all about what is release based testing,
what will be your involvement in this transformation process and what are the implications
of moving to release based testing.
Over here I would like to emphasize only one point before handing over the control to the
Test Manager. Remember, that Release Based Testing is not, I repeat not, a new type of
testing. It is a cultural change brought in the organisation and we must be party to this
transformation. Our Test Manager will be shouldering the responsibility of implementing the
Release Based Testing practice across organization. Now over to our Test Manager.”

The Test Manager rose from her seat; she was ready to explain the concept with the help of
comprehensive presentation, diagrams and examples of such exercise done in other
organizations.

“Thank You Sir, and a warm welcome to all attendees. Our CEO has already given you brief
idea about today’s discussion, so I will directly move to explaining the Release Based
Testing concept. You are free to raise your queries during this discussion.

She started –

“We all know that our organisation has decided to go into the market with the roadmap of
services to be offered to our end customers. The services will be offered through a series of
releases timed at fixed intervals. Though you are all aware of this information, I repeat it
here to set the tone of today’s meeting.

The software release life cycle is composed of different stages that describe the stability of a
piece of software and the amount of development it requires before final release. Each
major version of the release is composed of the services offered by various products. The
release cycle usually goes through a stage when new features are added; a stage when it is
being actively debugged; and finally a stage when all important bugs have been removed.

Release management is often that part of the software development lifecycle which has the
entire team chewing its nails. If your release process goes wrong, your software isn't
making you money, and your teams aren't working on new features.”

The Release Manager nodded his head in agreement. The Test Manager Continued -
“Before we discuss about Release Based Testing, let us understand the meaning of the term.
Release Based Testing should not be confused with the term Release Testing. The term
Release Testing is more commonly used for testing the integrated system which forms the
release and is done towards end of release cycle, in most cases just before the release date.
Whereas release based testing is as mentioned earlier by our CEO, is a cultural change to be
brought across organisation.”

The Release Manager raised his hand to ask a query, “We are providing our services through
series of releases for a long time now. So, what is the big difference in the earlier way of
working and when the Release Based Testing is implemented?”

The Test Manager smiled, “Precisely this is what I am going to explain next, it is about the
changes to be brought as part of moving to release based testing.

We all know that till now our release scope was purely based on what our various product
development teams had to offer in their own releases. Many a times, we could not deliver a
particular service in demand, simply because our product teams had not implemented the
changes required to employ those services.

We all are aware that with the advent of Service Oriented Architecture (SOA), we are able to
combine the core services and offer new services with much less efforts. Rather than going
into technicalities of the SOA, I would like to emphasize on the point that this change has
already brought in changes in the processes and responsibilities of each team present here.”

“So what are the implications of moving from Product Based Testing to Release Based
Testing? Let’s discuss this in detail. We will check the organisation level impact and the role
of each team in Release Based Testing”.

MOVING FROM PRODUCT BASED TESTING TO RELEASE BASED TESTING

“A Software Release is a fully constructed version of a software product which is ready for
use by specific audience. The Software Releases are typically directed at end users, and we
all agree that every release has to be tested thoroughly to ensure that we provide quality
products to our customers and end users.”

Organisation Level Impact

“It is imperative here to assess the impact of transformation from Product based testing to
Release Based Testing. Since many internal and external teams are directly or indirectly
involved into release based testing process; it would be appropriate here if we acknowledge
the impact of change on various factors.

I will briefly talk about the role of each team in the Release Based Testing Process, while the
detailed explanation will follow when I will talk about the Release Based Testing”.

The Test Manager showed a slide where a table containing name of teams and their role in
release based testing was explained.

Involvement of Various Teams

Team Role in Release Based Testing

Change This team is coming up with the demands that are converted into
Management requirements for the release. The feasibility analysis is carried out before
Team each demand is converted into the requirement, which will be part of
Team Role in Release Based Testing
release scope.
The Change Management team will involve testing team during feasibility
analysis to verify the testability of each demand. The Change
Management Manager will discuss with the Test Analysts about types and
levels of testing required to test the demand and organizational capability
of carrying out that type/s of testing.

Release The Release Management team is arranging the release planning

Management meetings. The Release Manager will discuss with Test Manager about
Team synchronization of test schedule with release schedule. Various product
development teams will provide their product functionality delivery dates,
so that the Test Manager can plan for System Integration and End to End
testing.
The Test Analysts will be involved in preparing the Impact Analysis
document and will get insight about the test scenarios to be newly
developed or existing ones to be changed to test the new functionality.
This will be used as an input for estimating the testing efforts for that
release.

Configuration The Configuration Manager will discuss with the Test Manager about the
Management Test Asset management for the particular release and how the test
Team scenario stack and Regression test pack to be version controlled for each
build provided by development teams during release cycle.
The access control and level of access will be defined for each release
cycle.
The Test Automation Lead will discuss how the Configuration Management
Database (CMDB) maintained by our Test Management tool through its
configuration management utility will be synchronized with main CMDB.

Product The Product development teams will discuss with the Test Manager about
Development their functionality release schedules and number of builds through which
Teams the functionality will be released to testing team.
The Test Analysts will discuss with the Solution Designers and
development leads about the stubs and simulators to be developed till the
functionality is developed. The Testing team will provide stub
requirements and eventually will test the stubs developed for early
integration of system.

Environment The Environment management team is managing the test environments.

Management The Environment Manager will discuss with the Test Manager about the
Team schedule of environment builds, the sequence of builds and the escalation
process in case of environment downtime.

Each Manager agreed to the role they are to play in the Release Based Testing. It was
Change Management Manager’s hand which rose this time to ask the query, “We understood
the role each team has to play in Release Based Testing. However, you were also talking
about cultural change brought in at organisation level. Could you please explain what
changes will be brought in the testing processes?”

“Sure”, was the reply of the Test Manager. She showed another slide with the diagram
showing Organisational Impact.

“Well, here is a diagram showing impact of adopting Release Based Testing. As you see it is
affecting three major areas of management, namely, Responsibility, Hierarchy and
Standardization. The Testing Process will undergo major changes in terms of –
Figure 1: Impact on Testing Processes

The Change Management Manager nodded his head to indicate that he understood the
concept, “alright, now I have another query. We have seen the impact on Testing Processes,
what will be impact on testing teams or any other teams?”

“Good Question. Let me explain you the impact on testing team and release management
team” Said the Test Manager.

Impact on Testing Teams

“Gone are the days when Component testing teams used to develop the features which then
were accommodated in the release scope. Since the release scope is the driving force for
defining the component level scope, each team will have to change its mindset of working in
Silos.

Increased coordination across testing teams as well as other support teams will reflect the
positive effect of united team working for a common cause. While working on the project,
we will observe the positive change in communication across various teams. The responses
will be faster than earlier and there will be fewer escalations!!”

Impact on Release Teams

“The Release Management team is usually the most stressed out team under the typical
scenario where the ownership of success of release is put on their shoulders. With the
changes brought through the Release Based Testing process, the Release Management
Team can take a sigh of relief since they have many helping hands now to share their
responsibility.”
The Test Manager paused for a moment to check if anyone has any queries over the concept
explained so far. All were attentive to what was being explained and seemed to have
understood the concept well. So she moved on.

“Having explained the roles of teams and impact on various teams, let’s move to the
fundamental question now and that question is Why to Test Software Releases; especially
when the individual product development teams have tested their functionality thoroughly.
Well, here is the answer to that question.”

Why Test Software Releases?

“Given today’s integrated environment, releasing production software is an accomplishment

in itself. With a production release, the design is complete, testing is complete,
requirements are verified, outstanding problems are mitigated, contractual obligations have
been met, the schedule no longer is a plan, it is the actuals for the program, and significant
management oversight – sometimes known as help – is provided, making the path for a
production release familiar and the process well known. Accompanying a production release
is a sense of accomplishment for a job well done and possibly the end of the program.

When the Release Management Team deploys new software without adequate (or any)
testing, it tends to get what it deserves—lots of headaches, late nights and extra work.
What the business and users receive from this oversight can be disastrous. Release
Management Team need to understand that any new software or upgrade to a critical
system must be tested in near real-world environments.

Release Management Team has a lot on their plates. One critical responsibility is the
deployment of new mission-critical applications or updates to those systems. Bypassing or
limiting testing of new releases, whether that code is from reputable software vendors or
developed in-house, is not a good idea. These problems are not widely known or reported
outside of the groups directly affected, but they show that thorough testing should be done
for any new software deployment.

To avoid such situations, the Organization must adopt the Release Based Testing Process.

There are two types of organisation level goals that the testing team is expected to fulfill –
Process Goals and Business Goals; the Release Based Testing is no exception to this.

Following are the Process Goals and Business Goals that should be fulfilled by the Release
Based Testing Team.”

Process Goals Business Goals

Continuous approach to achieve quality level Confirm utility / correctness of released
initiated at early stages of Release Life Cycle functionality

Repeatable, systematic and documented Ensure robustness of the release

testing approach

Achieve as much as automation to Testing Assured performance of released

Cost and Test Cycle Time functionality

Standardization of test process across Reduced cost of quality

organisation

Coordinated testing efforts across all Quantitative representation of release quality

component teams involved in a release through test metrics oriented reports

Table 1: Organisation Level Goals

There was a slight discomfort in some members, and that is when the CEO realized that it is
almost a time to have a Lunch break. He interrupted the Test Manager’s flow of speech, “I
can see that you all are quite engrossed in the discussion, but May I suggest a lunch break
now? Let us all enjoy our lunch and we will assemble for further discussion after the lunch
break.”

All managers were still discussing over the new concept while heading towards the cafeteria.

The Manager’s assembled in the meeting room after the lunch. A diagram of Release Based
Testing Process was shown on the screen. The Test Manager resumed her explanation.

“Having understood the impact on various teams and brief understanding of the organisation
level goals, we are now ready to understand the Release Based Testing Process. Since
various teams will be involved in the Release Based Testing, it will be interesting to see the
flow of events or operations performed as part of Release Based Testing.”

INTRODUCING RELEASE BASED TESTING PROCESS

“Release Testing is performed in many phases. First the software to be released is verified
by the component level testing teams. Corrections & refinements are done. Once the
software from all product teams is integrated, the release is built with the tested software. If
necessary, the corrections are applied later in the form of software patches. The integrated
software is then tested by the System Integration Testing Team. Once confident that the
integrated system is performing its tasks as per requirements, the baton is then passed on
to End to End testing team. We all know that the E2E testing team focuses more on the
business scenarios and tests the systems from end user’s perspective.

Test managers often are under pressure to speed the release of software. To ensure
adequate testing, they must clearly communicate with the other team managers, the testing
scope, testing plans, and risks involved.

“Here is the diagrammatical representation of the typical Release Based Testing Process”,
the Test Manager pointed towards the diagram on the screen and started explaining the
diagram.

“A new role of the Release Test Manager in the Release Coordination Team is recommended.
The Release Test Manager is responsible for the overall Quality of the release from
conceptualization stage through to operational stage. The Test Manager will act as the
pivotal point for managing and ensuring the quality of reviews and testing within the
release. Fundamental to the role is to liaison and co-ordination between the Change
Management, Release Management, Test Environment Management and Configuration
Management and Various Product level testing teams.”
 Market Businesses End-Users Product Security
Forces Defects Groups BCP
IT Infrastructure
Support

Requests for Change (RFCs)

Change
Management Team

Release Scope Finalization

Inputs From
Component , SIT, E2E Test Scope Finalization Release
Testing Teams & Management Team
Business Users

Release Assurance
Initiate Release Based Testing

Environment
Component Testing Component Testing Management Team
Teams
Test
Integration Testing Automation
Internal SIT and E2E
Testing Teams End To End Testing

Domain Experts TEST MANAGEMENT Configuration

End Users Management Team

Storage of Test Assets

Figure 2: Release Based Testing Process

“A Release Test Manager will coordinate with the Release Manager. Typical responsibilities of
the Release Test Manager includes –

• Delivering the release to agreed Quality level.

• Implementing the review and test strategy throughout the release.

• Liaising with the Requirements, Design, Development, Test, Deployment and Operations
teams to understand their position and communicate release testing expectations and
principles.

• Communicating regularly with the release manager to provide progress and

communicate risks.

• Constantly reviewing release lifecycle to review and improve test processes.

• Enabling successful planning and execution of all review and testing phases for a release.

• Ensuring effective coverage of functional and non-functional testing, in line with the
review and test strategy.

• Ensuring the use of the agreed test management tool for test planning, progress
monitoring and fault management.
• Ensuring that testing teams are included in reviews of requirements, design and other
early lifecycle deliverables.

• Completing the release test summary report.

• Liaising with component development teams, to aid effective communication and

collaboration.

• Clarifying uncertainty and misunderstandings in requirements.

• Reviewing requirements, design development and test documentation. This may include
participation in requirements testability reviews, design walkthroughs and reviews of
development testing coverage and results.

• Conducting traceability analysis of requirements, design and test documentation. These

may take the form of compliance statements/tables or validation matrices.

• Actively working with Test Environment Team to ensure the availability of the
appropriate test environment, configuration of the environment and ensuring that the
releases are loaded on the respected environment when needed.

• Produce early test strategy focusing Test environment, testing approach, identified risks
and cost.

• Coordinating defect resolution.

• Monitoring the Test Execution progress against agreed plans.”

One of the Product Development Managers interrupted the Test Manager to ask his query,
“This is quite an impressive list of responsibilities of Release Test Manager and it gives us a
fair idea as to how you will interact with us in the capacity of Release Test Manager
henceforth. However, do you have a model of the Release Based Testing Process?”

“Certainly, our organisation encourages developing a process model before working on any
new concept or activity. I have developed a process model and I would like to verify its
correctness and completeness with you today”.

Model Your Release Testing Process

“The model has been developed by mapping the release testing process with the release life
cycle stages. In the current business scenario where every organisation is trying to achieve
faster ‘Go-To-Market’ strategies, the Component Development teams and other support
teams aim to spend as few resources as possible on a process which does not seem to be
part of their own stack of work. Compromises are made to realize a working system in time
and according to objective.

The most prominent factors to assure quality and success of the software release are as
follows:

• Give ample time and importance to building up a ‘Release Based Culture’. Common
understanding of the release functionality and the environment where it will be
deployed, a common working language and the use of external and in-house standards
are the basis of fruitful release based testing process.
• Build change management in your testing process to give flexibility when requirements
are modified while the software environment is evolving.

• Define the scope of release functionality for testing at an early stage for maximum pay-
off in the testing process. Misunderstanding and mistakes made at this stage when
converting the user requirements into test requirements or test specifications calls for
time consuming re-work at later stages.

• Review the requirements thoroughly. I would recommend formal review for conceptually
prominent components. Light inspection can be applied once the release functionality is
understood and accepted by the concerned teams.

• Mark each milestone in release lifecycle with the successful completion of testing at that
stage. Plan resources and activities in advance to carry out phase wise testing.

• Adopt from the start the concept of planned releases with ‘Check Targets’.

• Designated persons / team looking into the test results generated by testing teams
o Release Coordination team is deeply involved in the validation activities
o Active participation of developers for quick resolution of defects
o Coordination and test planning across various business verticals

• Weekly status reports and bi-weekly meetings to share the knowledge about
o The status of the release
o Defect identification and propagation to concerned teams
o Sharing of testing documents

I would recommend the Increment-Build model for release based testing. I.e. the release
testing team will start with the simple Testing Framework. The idea is to always have the
framework running without any problems. New components and new levels of testing get
tied into this framework as the release cycle progresses. Initial versions of this framework
might be boring with incomplete features, but important part is that it will still be in running
condition. So rather than doing lot of planning up-front, evolve the framework when needed
and continually add & repair the test artifacts when necessary.”

There was short Tea Break after which Test Manager assured everyone that she is almost
through with her explanation and that she will now share the experience of one of the
service provider who has implemented the process of Release Based Testing in their
organisation.

RELEASE BASED TESTING – PRACTICAL EXPERIENCE:

“Recently, one of the Telecom Service Provider opted for moving from Product Based Testing
to Release Based Testing. The consultants were appointed to assess and measure the
organisation level impact of this transformation.

The Approach adopted by the Service Provider :

Testing team will devise the scope of testing once the release management team has
authorized the scope for a release. Any changes to the scope then will undergo the change
management process before being accommodated in the testing scope. There will be defined
timeline for freezing the scope of the current release; the scope changes after that timeline
would be just to accommodate fixes to faults found during testing.
Figure 3: Release Based Testing - An Example

The Release based testing approach will cover following activities:

• Defining the scope of testing based on the scope of current planned release
• Verification of the system in release to check the conformance with defined business
processes
• Implementing the review techniques throughout the release

The release based approach for testing will ensure adherence to release schedule and
release date. Thus the project planning phases will be mapped to release timelines. The
Release Test Manager will liaise with the Release Manager to report progress and
communicate risks. The release manager will be involved in the periodical weekly meetings
to determine the readiness of the system for launch.”

The Test Manager took a brief pause before concluding her today’s session.
“So we are at the end of today’s session on Release Based Testing. If you do not have any
queries, I will conclude this session by summarizing our discussion.”

CONCLUSION:

“The main objective of the Release Based Testing process is to build the culture of ‘One
Release – One Team’ to ensure successful and on time release. The other important
objectives to be considered are -

• To identify the problems earlier in the release cycle and drastically reduce total cycle
time.

• Clear and defined hierarchy with defined roles and responsibilities

• Ensuring involvement of all concerned parties in the release process

In nutshell, focus on Release Based Testing will bring in following organisation level
changes:
 CONTEXT FROM TO

Decision making Operational Strategic

Testing Approach Application Complete Release

Components

Evaluation Perspective Development Quality Business Acceptance

Quality

Delivery Solution Project Test Comprehensive Release

Execution Testing

Business Model Project Based Release Based

Expected Benefits Cost Reduction Higher ROI, Lower COQ,

Faster Time to Market

Reduced Cycle Time Optimized test life cycle

Figure 4: Moving towards Release Based Testing

There was huge applaud from the audience. Every member was happy with the outcome of
the meeting and the CEO was personally congratulating the Test Manager for excellent
presentation of the concept.

REFERENCES
• Why Test New Software Releases? -> McLean Report: Research Note
• Future of Testing -> James Whittaker
• Software Release Management is a Thankless Task –> Christopher Jones
• Release Management – Service Management Function -> Microsoft Operations
Framework
• Software Release Validation -> Manual Gallas, CERN
• Software Release Management -> Lorette Clement - Smith

AUTHORS' PROFILE

I have 19+ years of experience in Information Technology and 12+ years experience in
Software Testing with opportunities to work for customers in different countries like USA,
UK, Singapore, Jamaica, New Zealand and Germany.

Currently working as Senior Quality Consultant with Tech Mahindra, and working as a senior
member of Testing Practice Group.

I possess Masters Degree in Information Technology (MBA – IT). I am Certified Software

Test Engineer (CSTE), a certification awarded by QAI, USA. I acquired my CSTE certification
in 2003 and since then I have been acting as a visiting faculty at various institutions to
teach aspiring CSTE exam entrants.

I have worked on large variety of application systems, and have managed delivery of large
projects in the areas of Functional testing, Test Automation and performance testing. I have
setup the Test Automation Center of Excellence (Automation CoE) for Tech Mahindra and
acted as Test Automation CoE head for 2 years.

Currently, I am providing Test Process Consultancy to a number of projects for domestic and
international clients.

I am on the Steering Committee of SPIN, Pune Chapter. SPIN – Software Process

Improvement Network is the international forum formed to bring awareness about quality
related best practices for IT professionals.