PNNL-17772
Prepared for the U.S. Department of Energy
Under Contract DE-AC05-76RL01830
Software Verification and Validation
Procedure
March 2008
DISCLAIMER
This report was prepared as an account of work sponsored by an agency of the
United States Government. Neither the United States Government nor any
agency thereof, nor Battelle Memorial Institute, nor any of their employees,
makes any warranty, express or implied, or assumes any legal liability or
responsibility for the accuracy, completeness, or usefulness of any
information, apparatus, product, or process disclosed, or represents that its
use would not infringe privately owned rights. Reference herein to any
specific commercial product, process, or service by trade name, trademark,
manufacturer, or otherwise does not necessarily constitute or imply its
endorsement, recommendation, or favoring by the United States Government or
any agency thereof, or Battelle Memorial Institute. The views and opinions of
authors expressed herein do not necessarily state or reflect those of the United
States Government or any agency thereof.
PACIFIC NORTHWEST NATIONAL LABORATORY
operated by
BATTELLE
for the
UNITED STATES DEPARTMENT OF ENERGY
under Contract DE-AC05-76RL01830
Printed in the United States of America
Available to DOE and DOE contractors from the
Office of Scientific and Technical Information,
P.O. Box 62, Oak Ridge, TN 37831-0062;
ph: (865) 576-8401
fax: (865) 576-5728
email: reports@adonis.osti.gov
Available to the public from the National Technical Information Service,
U.S. Department of Commerce, 5285 Port Royal Rd., Springfield, VA 22161
ph: (800) 553-6847
fax: (703) 605-6900
email: orders@ntis.fedworld.gov
online ordering: http://www.ntis.gov/ordering.htm
This document was printed on recycled paper.
(9/2003)
PNNL-17772
Software Verification and Validation
Procedure
March 2008
Prepared for
CH2M HILL Hanford Group, Inc.
under Contract DE-AC06-76RLO 1830
Information Sciences & Engineering
Pacific Northwest National Laboratory
Richland, Washington 99352
PNNL-17772
Acronyms and Abbreviations
ATPR Acceptance Test Plan and Results
CRTT Change Request Tracking Tool
DOE U.S. Department of Energy
IEEE Institute of Electrical and Electronics Engineers
PNNL Pacific Northwest National Laboratory
SSEP Software Systems Engineering Process
TWINS Tank Waste Information Network System
v
PNNL-17772
Contents
Acronyms and Abbreviations ............................................................................................................... v
1.0 Purpose and Scope........................................................................................................................ 1
2.0 Implementation............................................................................................................................. 1
3.0 Responsibilities............................................................................................................................. 1
4.0 Procedures .................................................................................................................................... 2
4.1 Procedure Description .......................................................................................................... 2
4.2 Procedure Flow Diagram ..................................................................................................... 3
5.0 Definitions .................................................................................................................................... 4
6.0 Records ......................................................................................................................................... 5
7.0 References .................................................................................................................................... 5
8.0 Distribution................................................................................................................................... 5
Appendix A........................................................................................................................................... A.1
Appendix B ........................................................................................................................................... B.1
Appendix C ........................................................................................................................................... C.1
vii
PNNL-17772
1.0 Purpose and Scope
This Software Verification and Validation procedure provides the action steps for the Tank Waste
Information Network System (TWINS) testing process. The primary objective of the testing process is to
provide assurance that the software functions as intended, and meets the requirements specified by the
client. Verification and validation establish the primary basis for TWINS software product acceptance.
The TWINS project conforms to the requirements of the Pacific Northwest National Laboratory
(PNNL) Information Sciences and Engineering Software Systems Engineering Process (SSEP). The
SSEP has the testing process (verification and validation) integrated into its defined software lifecycle.
The SSEP review and test process is defined at http://ssep.pnl.gov/VVT. The methods defined in this
procedure are derived by applying a graded approach, adapting the SSEP Reviews and testing processes
to the specific risks associated with the TWINS project.
This Software Verification and Validation procedure covers all software changes relating to the
TWINS system. This includes web pages, scripts (server-side and client-side), code, and MS Access files
(tables, reports, queries, modules).
2.0 Implementation
This procedure is effective on the Effective Date shown in the header above.
3.0 Responsibilities
Project Manager: The project manager of the TWINS project is responsible for ensuring that all
aspects of this Plan are implemented. Other responsibilities are contained within Section 4.0.
1
PNNL-17772
4.0 Procedures
4.1 Procedure Description
Project Manager 1 Enter the statement of work, change order, or glitch information into the
Change Request Tracking Tool (CRTT).
Project Manager 2 Determine if the software change (prompted by a statement of work,
change order or problem report) requires a formal Acceptance Test Plan
and Results (ATPR) and enter the determination in the CRTT. If a formal
test plan is required, skip to Step 8. If a formal test plan is not required,
complete steps 3 through 7 inclusive.
Note: Criteria for making a formal test plan determination are found in
Attachment A.
Project Manager 3 Assign programming of the software changes to a programmer.
Programmer 4 Complete programming and perform unit testing.
Programmer 5 Notify the Independent Reviewer that programming and unit testing are
complete.
Independent 6 Review each task performed by the Programmer, updating the CRTT as
Reviewer appropriate. Complete acceptance testing, documenting the results as
indicated in Attachment B.
Independent 7 Enter the date of verification, verifier and the verification information of
Reviewer Step 6 into the appropriate place in the CRTT log. Procedure ends with
this step.
Project Manager 8 Assign programmer and Independent Reviewer to complete and unit test
the changes.
Independent 9 Complete ATPR per template Attachment C and submit to Project
Reviewer Manager for approval. In the case where CH2MHill subject matter
expert(s) will perform independent testing to verify results, the
programmer will act as the Independent Reviewer in that he/she will
document the test results in the ATPR.
Project Manager 10 Resolve comments and concerns with Independent Reviewer and
Programmer as needed and Approve ATPR.
Programmer 11 Complete programming and unit testing of changes.
Independent 12 Submit ATPR and code to Tester per protocols in the Software
Reviewer Configuration Management Plan for acceptance testing.
Tester 13 Complete acceptance testing and document on the ATPR form prepared
in Step 9. If any tests fail, have the programmer make appropriate
programming corrections, or correct test procedures, and rerun the tests.
Indicate on the test forms or tables in ink the initials of the tester, the date
of successful retest, and the notation that the test passed.
2
PNNL-17772
Tester 14 Sign the ATPR and forward to Project Manager.
Quality Engineer 15 Review and approve the ATPR, resolving comments with the Tester as
needed.
Project Manager 16 Maintain a file copy of the completed and approved ATPR.
4.2 Procedure Flow Diagram
3
PNNL-17772
5.0 Definitions
Acceptance Testing Testing to ensure that the given requirements for a software change are met
and that the overall system is not adversely affected by the change.
Acceptance testing is performed by a person other than the developer. This
type of testing is conducted at a higher conceptual level than Unit Testing and
is focused on inputs and generated results at the user level. The scope of this
type of testing includes:
1. Test cases to cover the range of possible input data, including valid
and invalid values.
2. Ensuring that the system handles invalid input data appropriately to
minimize the risk of generating inaccurate results.
3. Verifying the accuracy of results generated from valid input data
against another trusted/verified source.
It is the judgment of the Independent Reviewer and ultimately the Project
Manager that determine the scope and depth of coverage for Acceptance
Testing.
If the independent testing is performed by CH2MHill subject matter expert(s),
then the developer assigned to the change request or bug is responsible for
documenting the test cases and results. This data will most likely be in the
form of email correspondence and data dumps collected in Excel.
Software Verification The process of determining whether the requirements for a system or
and Validation component are complete and correct, the products of each development phase
fulfill the requirements or conditions imposed by the previous phase, and the
(from IEEE Std 610.12- final system or component complies with specified requirements.
1990, IEEE Standard
Glossary of Software
Engineering
Terminology)
Unit Testing Testing of individual software components (stored procedures, triggers, batch
executables, queries/reports, functions, subroutines, modules, and other
individual classes). This testing is carried out by the developer using their
knowledge of the code details. This type of testing focuses on verifying the
operation of each component separately. The items generally included in this
type of testing include:
1. For each variable in the component:
a. A condition inside the variable’s acceptable boundaries.
4
PNNL-17772
b. A condition outside the variable’s acceptable boundaries.
2. Each significant logic branch or decision in the component.
3. Each significant operation or calculation that affects the output of the
component.
It is ultimately the judgment of the developer that determines the degree of
coverage for Unit Testing.
6.0 Records
Records generated by this procedure include entries into the CRTT log and the ATPR forms. Records
kept prior to January 2, 2008 have been recorded in RIDS. Records after this date shall be kept in TRIM
which is maintained by the Project Administrator.
7.0 References
1. TWINS, Software Configuration Management Plan, Revision 6, March 2008.
2. TRIM and Electronic Records Management http://cer.pnl.gov/records/trim/index.stm
8.0 Distribution
1. TWINS Project Manager
2. TWINS Software Developers
3. TWINS Project Quality Engineer
5
Appendix A
PNNL-17772
Appendix A
Criteria for Determination of Need for a
Formal Acceptance Test Plan
The above decision matrix defines the general criteria and process for determining whether or not a
formal test plan is required for a given change. This matrix is utilized by the Project Manager as an aid in
making that determination.
The “Probability of Failure” axis addresses how likely it is that the given change will be problematic.
Issues typically considered in this area are listed below:
Difficulty or complexity of change
Size/scope of change
Experience level of developers doing the change
The “Impact of Failure” axis addresses the implications if the given change does have problems.
Issues typically considered in this area are listed below:
Safety of workers or public compromised
Regulatory compliance jeopardized
Financial loss for PNNL and/or customer
Lowered public image for PNNL, DOE, and customer
Reduction of operational capability for PNNL and/or customer
Compromised security of PNNL, DOE, or customer assets
A.1
Appendix B
PNNL-17772
Appendix B
Essential Information for Acceptance Test Plans and Results
The following information is essential for documenting acceptance testing when a formal ATPR are
not required:
CRTT identification number of the change being tested
Name of the software and revision being tested
Tester’s name, date of test
Any special tools, conditions, or configurations needed to perform the test
Test case descriptions, including expected outputs
Actual results for each test case, including a “pass/fail” notation
This information is a subset of the information contained in the ATPR template provided in
Attachment C.
B.1
Appendix C
PNNL-17772
Appendix C
Template for Acceptance Test Plans and Results
Bug/Change Bug/Change
Request ID: Request ID:
Item(s) Rev: Date:
Tested:
Test
Responsibilities:
Test
Environment:
Test Data Set(s), and
Execution Log(s):
Pass/Fail Pass Each test case shall have criteria to define a
Criteria Pass outcome for this test.
Conditional When all items identified as Corrective Actions
Pass have been completed and retested, the Conditional
Pass will be changed to a Pass.
Fail When a Pass or Conditional Pass is not received.
Pass after After a Fail outcome and retest with a Pass
Fail outcome, the Fail outcome is replaced with this
outcome.
Test Setup Specific instructions regarding how to set the environment up
for the testing.
Test Procedure Specific instructions for how to perform the testing.
Test Restart Specific instructions regarding how to restart testing after a
“stop”, usually just “Redo Test Setup”.
Test Stop Specific instructions regarding when to abort the testing, in
the event of a problem or failure, prior to completion.
Test Wrap Up Specific instructions regarding when the testing is complete.
Test Case Log
Test
Case Test Description Input Expected Actual Pass/
ID Date Result Result Fail
T1
T2
T3
T4
C.1
PNNL-17772
Corrective Bug/ Retest
Actions and Problem Test Bug/Problem Outcome
Retest ID Case Description Description (Pass/Fail
Criteria
Test Outcome Pass Conditional Pass Fail Pass After Fail
Approvals Approver Name Signature Date
Tester
Quality
Carrie Carlson
Engineer
Project Manager Tom Olund
C.2
