104-EMD-001

ECS Maintenance and Development Project

Software Quality Assurance Plan

for the EMD Project

Revision -

October 2003

Raytheon Company
Upper Marlboro, Maryland
 Software Quality Assurance Plan
for the
EMD Project

Revision -

October 2003

Prepared Under Contract NAS5-03098

CDRL Item # 004

RESPONSIBLE AUTHOR

Carla Baglione /s/ 10/30/03

Carla Baglione, Quality Engineer Date
ECS Maintenance and Development Project

RESPONSIBLE OFFICE

Kenneth L. Wise /s/ 10/30/03

Kenneth L. Wise, Quality Assurance Manager Date
ECS Maintenance and Development Project

Raytheon Company
Upper Marlboro, Maryland

104-EMD-001
This page intentionally left blank.

104-EMD-001
 Preface

This document is a formal contract deliverable. It requires Government review and approval
within 20 business days. Changes to this document will be made by document change notice
(DCN) or by complete revision.
Any questions should be addressed to:
Data Management Office
The EMD Project Office
Raytheon Company
1616 McCormick Drive
Upper Marlboro, Maryland 20774-5301

Revision History

Document Number Status/Issue Publication Date CCR Number

104-EMD-001 Initial Release October 2003 03-0731

iii 104-EMD-001
This page intentionally left blank.

iv 104-EMD-001
 Abstract

The Software Quality Assurance Plan (SQAP) establishes the Quality Assurance program for the
EMD contract. This plan describes the Quality Assurance (QA) organization and audit,
evaluation and monitoring activities applicable for the EMD Program. It describes EMD Quality
Assurance activities performed by QA staff, directed by documented procedures. This plan also
references activities performed by other organizations or functions that are monitored or audited
by QA.
Keywords: SQAP, quality, ISO, AS9100, QMS, audit, evaluation, nonconformance,
noncompliance, deficiency, process, procedure

v 104-EMD-001
This page intentionally left blank.

vi 104-EMD-001
 Contents

1. Purpose
1.1 Purpose and Scope ......................................................................................................... 1-1
1.2 Software Items Covered................................................................................................. 1-1
1.3 Software Lifecycle ......................................................................................................... 1-1

2. Reference Documents
2.1 Documents Referenced Herein ...................................................................................... 2-1
2.2 Documents Used to Develop this SQAP ....................................................................... 2-1

3. Management
3.1 EMD Project Organization ............................................................................................ 3-1
3.1.1 Program Organizational Structure ..................................................................... 3-1
3.1.2 Quality Assurance Organization........................................................................ 3-1
3.1.3 AS9100/ISO 9000 Quality Management System .............................................. 3-1
3.2 Tasks .............................................................................................................................. 3-2
3.2.1 Maintenance and Development Lifecycles........................................................ 3-2
3.2.2 Maintenance and Development Tasks ............................................................... 3-2
3.2.3 Quality Assurance Processes ............................................................................. 3-4
3.3 Roles and Responsibilities ............................................................................................. 3-4
3.3.1 Quality Assurance Manager............................................................................... 3-4
3.3.2 Quality Assurance Engineers............................................................................. 3-4
3.4 Quality Assurance Estimated Resources ....................................................................... 3-5

4. Documentation
4.1 Purpose........................................................................................................................... 4-1
4.2 Minimum Documentation Requirements....................................................................... 4-1

vii 104-EMD-001
 4.2.1 Software Requirements Description .................................................................. 4-1
4.2.2 Software Design Description ............................................................................. 4-1
4.2.3 Verification and Validation Plans...................................................................... 4-1
4.2.4 Formal Verification Results Report and Validation Results Report.................. 4-2
4.2.5 User Documentation .......................................................................................... 4-2
4.2.6 Configuration Management Plan ....................................................................... 4-2
4.3 Other Documentation..................................................................................................... 4-2
4.3.1 Transition Plan ................................................................................................... 4-2
4.3.2 Program Management Plan................................................................................ 4-3
4.3.3 Development and Program Plans....................................................................... 4-3
4.3.4 Review Documentation...................................................................................... 4-3

5. Standards, Practices, Conventions, and Metrics

5.1 Purpose........................................................................................................................... 5-1
5.2 Content........................................................................................................................... 5-1
5.2.1 Process Requirements ........................................................................................ 5-1
5.2.2 Metrics ............................................................................................................... 5-1

6. Software Reviews
6.1 Purpose........................................................................................................................... 6-1
6.1.1 Technical and Peer Review................................................................................ 6-1
6.2 Minimum Requirements ................................................................................................ 6-1
6.2.1 Software Specifications Review (SSR) ............................................................. 6-1
6.2.2 Architecture Design Review (ADR).................................................................. 6-1
6.2.3 Detailed Design Review (DDR) ........................................................................ 6-2
6.2.4 Verification and Validation Plan Review .......................................................... 6-2
6.2.5 Functional Audit ................................................................................................ 6-2
6.2.6 Physical Audit.................................................................................................... 6-2
6.2.7 In-Process Audits ............................................................................................... 6-2
6.2.8 Managerial Reviews .......................................................................................... 6-3
6.2.9 Software Configuration Management Plan Review .......................................... 6-3
6.2.10 Post-Implementation Review............................................................................. 6-3
6.3 Other Reviews and Audits ............................................................................................. 6-3

viii 104-EMD-001
 7. Test

8. Problem Reporting and Corrective Action

8.1 Hardware and Software Items........................................................................................ 8-1
8.2 EMD Process and Work Products.................................................................................. 8-1
8.3 Quality Management System......................................................................................... 8-1

9. Tools, Techniques, and Methodologies

9.1 Tools and Equipment ..................................................................................................... 9-1
9.2 Methodologies and Process ........................................................................................... 9-1

10. Media Control

11. Supplier Control

11.1 Supply Chain Management.......................................................................................... 11-1
11.2 Quality Assurance Role ............................................................................................... 11-1

12. Records Collection, Maintenance, and Retention

12.1 Quality Assurance Tracking Database......................................................................... 12-1
12.2 Quality Assurance Repository ..................................................................................... 12-1

13. Training
13.1 Training Requirements ................................................................................................ 13-1
13.2 Raytheon Provided Training ........................................................................................ 13-1

14. Risk Management

14.1 Risk Management Approach........................................................................................ 14-1
14.2 Quality Assurance Role ............................................................................................... 14-1

ix 104-EMD-001
 15. Glossary

16. SQAP Change Procedure and History

16.1 Change Control ............................................................................................................ 16-1
16.2 Modification History.................................................................................................... 16-1

x 104-EMD-001
 1. Purpose

1.1 Purpose and Scope

This Software Quality Assurance Plan (SQAP) establishes the Quality Assurance program for
the ECS Maintenance and Development (EMD) contract (NAS5-03098). This document is
CDRL Item # 004 and is developed in accordance with the requirements of Data Item
Description (DID) EMD-SQAP-4.
The SQAP addresses the organization, responsibilities, procedures, methods and tools employed
by the Raytheon Team in maintaining, sustaining, and enhancing the EOSDIS Core System
(ECS) under the EMD contract. This plan describes the Quality Assurance (QA) organization
and the specific activities applicable for the EMD Program. In accordance with IEEE Std-730-
2002, this plan will cover EMD Quality Assurance activities performed by members of the
Quality Assurance organization and directed by applicable QA documented procedures. In
addition, it will reference activities monitored by QA that are performed by other organizations
or functions.

1.2 Software Items Covered

This SQAP covers the maintenance of all existing components of the ECS Science Data
Processing System (SDPS). The SDPS supports science data management activities for 19 Earth
Science instruments on 10 spacecraft. It interfaces with 34 external systems and supports a wide
range of stakeholders. The existing ECS consists of more than a million lines of C++ custom
code and 155 thousand lines of Java custom code, contained in more than 15 thousand modules.
During the EMD Program, the SDPS will require further development to address significant
changes in system throughputs and capacities, in addition to supporting new missions and
satisfying stakeholder requirements.

1.3 Software Lifecycle

Since this is both a maintenance and development contract, the entire software lifecycle will be
covered by the SQAP.
Requirements for new development may be received by the contractor or proposed by the EMD
contractor as system enhancement proposals (SEP). New capability development is documented
in Tickets, which provide operations concept, Level 4 requirements and verification criteria.
Tickets are subject to internal peer review, which may include participation from Earth Science
Data and Information System (ESDIS) and Distributed Active Archive Center (DAAC)
representatives. Once requirements are adequately defined, software design and development
may begin and will proceed through preliminary design and detailed design, with peer review at
each design stage. When the design phase has concluded, software code will be developed and
inspected, and unit tests will also be developed, reviewed, and executed. Integration and

1-1 104-EMD-001
subsequent system level verification and validation will take place prior to deployment. Early
defect detection is emphasized in all phases through mature peer review and inspection activities.
Custom code is enhanced via Modification Requests and maintained based on trouble tickets and
non-conformance reports (NCRs) from system users. The development process for such ‘fixes’
depends on established thresholds for changes to software lines of code and may include design
and/or code peer reviews at specified stages.

1-2 104-EMD-001
 2. Reference Documents

2.1 Documents Referenced Herein

AS9100, Quality Systems - Aerospace - Model for Quality Assurance in Design, Development,
Production, Installation and Servicing, Revision A, August 2001
IEEE Std-730-2002, IEEE Standard for Software Quality Assurance Plans,
23 September 2002
ISO 9001:2000, Quality Management Systems — Requirements, 15 December 2000
Landover Facility Procedures — Quality Assurance series
ECS/EMD Project and Work Instructions
ECS DID 305 Segment Design Specification
ECS DID 311 Database Design and Database
ECS DID 313 Internal Interface Control Document (ICD)
DID EMD-PP-5 Property Management Plan
DID EMD-RMP-6 Risk Management Plan
DID EMD-TP-1 Transition Plan
DID EMD-MCMR-12 Monthly Contractor Manpower Reporting
DID EMD-533-11 Monthly Contractor Cost Reporting — 533 Requirements
108-EMD-001 Program Management Plan for the EMD Project
110-EMD-001 Configuration Management Plan
117-EMD-001 Procurement Management Plan
207-EMD-001 EMD Security Management Plan
302-EMD-001 Software Maintenance and Development Plan
303-EMD-001 Hardware Maintenance and Development Plan

2.2 Documents Used to Develop this SQAP

IEEE Std-730-2002, IEEE Standard for Software Quality Assurance Plans,
23 September 2002
EMD Task 101 Statement of Work for ECS SDPS Maintenance, August 2003

2-1 104-EMD-001
61-0010, Raytheon Quality Management System Manual
Raytheon Company Policy 000000079-RP, Product Assurance Requirements
Landover Facility Procedure 19-0-4, Software Quality Assurance Program Planning

2-2 104-EMD-001
 3. Management

3.1 EMD Project Organization

3.1.1 Program Organizational Structure

The EMD is organized and managed using integrated product teams (IPTs) and cross product
teams (CPTs). The IPT is an integrated multidisciplinary team designed to meet common
objectives and organized around a specific task, product, or service. CPTs are made up of the
functions that will apply to multiple tasks on the EMD Contract and will typically provide
similar services across many IPTs.
Program Management provides management oversight during all task life cycle phases and
ensures that adequate support services are available for all tasks to include cost and schedule
tracking, configuration management, procurement, quality assurance, computer infrastructure,
contracts, and safety. More detailed information is provided in the Program Management Plan
for the EMD Project (108-EMD-001).

3.1.2 Quality Assurance Organization

The Raytheon Information Technology Systems (ITS) Quality Assurance (QA) organization has
the responsibility for implementing the QA program for the EMD Project. The ITS QA
organization has dual reporting roles: to the Intelligence and Information Systems (IIS) division
Product Assurance/Quality Director, and to the ITS General Manager, thereby ensuring program
organizational independence.
The ITS Quality Manager has overall responsibility for quality assurance programs and will
identify and direct the EMD Quality Assurance team members. The EMD QA team will provide
independent and objective evaluation of process execution and work product quality. This
separation of QA responsibilities from EMD product team organizations ensures the
independence and organizational freedom to objectively evaluate product and process, monitor
EMD development and maintenance activities, and to determine corrective action and verify
compliance.
The QA Manager will be the primary interface with the ESDIS System Assurance Manager and
System Assurance representatives. The EMD QA organization advocates open communication
and interaction with ESDIS to ensure understanding and the achievement of our common
objectives.

3.1.3 AS9100/ISO 9000 Quality Management System

Raytheon’s fundamental standards for quality management are AS9100, ISO 9000 and best
industry practices included in the Raytheon Integrated Product Development System (IPDS).
Quality requirements are further flowed-down through Raytheon IIS division-level directives

3-1 104-EMD-001
and tailored specifically for EMD processes. Our Quality Management System (QMS) has been
registered ISO 9000 compliant by National Quality Assurance (NQA), a Registrar Accreditation
Board (RAB) accredited third-party auditing and assessment company. In August 2003, the
Landover facility successfully completed an audit to requirements of AS9100 and was deemed
compliant.

3.2 Tasks

3.2.1 Maintenance and Development Lifecycles

The EMD comprises both maintenance and development activities. Modification requests lead to
system enhancements and may involve the complete software development lifecycle:
requirements, design, development, code, unit test, integration, system-level test, and
deployment and includes system, installation, and operational documentation. Maintenance of
the system continues with evaluation of operational trouble tickets and NCRs, code changes,
baseline control, and patch deployment.

3.2.2 Maintenance and Development Tasks

Throughout the EMD Program, QA will conduct planned audits, evaluations, and monitoring
activities aligned with those program activities identified in the software and hardware
maintenance and development plans (SMDP/HMDP) and the EMD project schedule. In a
maintenance environment, QA will establish ongoing programmatic tasks to ensure continuous
monitoring, corrective action, improvement, and quality results. In the case of planned custom
code development task orders, QA will define tasks directly tied to scheduled design,
development, test, and deployment events within the task order scope of work. QA will identify
these tasks and activities and track them in an EMD QA project schedule.

3.2.2.1 Program Management Tasks

Program Management (PM) tasks are defined as those that are ongoing activities, performed
primarily by members of the PM Team and not tied to a scheduled event or milestone. As such,
these tasks may have general or implied entry/exit criteria, which may be defined in EMD
Project and Work Instructions. QA functions in both a monitoring capacity and conducts
periodic process audits and product evaluations, as appropriate. PM tasks include, but are not
limited to: Program Planning, Program Controls, Risk Management, Customer Interface,
Program Review, Documentation, SEP and Task Order Planning.

3.2.2.2 Maintenance and Engineering Support Tasks

Maintenance and engineering support tasks are those associated with both development and
maintenance activities. They may be ongoing, as for code and hardware maintenance, or tied to
events identified by Task leads in the Primavera (P3) scheduling tool. Since these are not
discreet tasks, entry criteria may be general, time-phased, or implied for ongoing activities, such
as those based on the completion of a related activity. Engineering support tasks include, but are
not limited to: COTS Maintenance, Failure Review, Problem Management (trouble ticket/NCR),

3-2 104-EMD-001
Help Desk Operations, Configuration Management (baseline management, configuration change
request and control). Maintenance and support tasks will be performed by EMD IPTs and CPTs,
as described in the Program Management Plan. QA will function in a monitoring capacity and
also conduct periodic process audits, configuration audits and product evaluations, as
appropriate.

3.2.2.3 Development Tasks

Development tasks include those activities associated with a typical software development
lifecycle. Development tasks may have a specified delivery schedule and may therefore be
identified in P3. During the ECS contract, effective processes were established and documented
in Project Instructions (PI) that direct the activities as listed below. More specific entry/exit
criteria may be detailed for each in its applicable PI. Further information will also be provided in
the Software Maintenance and Development Plan (SMDP), DID EMD-SMDP-2. The Quality
Assurance organization participates in development related planning and review activities and
audits them for compliance to established processes and entry/exit criteria.

Activity Entry Criteria Exit Criteria

Development planning Receipt of new requirements Plan Review and Approval.
Ticket (requirements) Planning document (draft ticket) Ticket Peer Review, Action Item
development Work-off, Authorization to proceed
Design Ticket/Ops Concept Design Peer Review, Action Item
Previous stage authorization to Work-off, Authorization to proceed
proceed
Code Design Completion Code Peer Review, Action Item
Previous stage authorization to Work-off, Authorization to proceed
proceed
Test plan/procedures Developed in conjunction with Test Plan/Procedures completed,
development Design & Code phase reviewed, Authorization to
proceed as artifact of Design and
Code phase
Test execution Code merged, test environment Test completion, as defined for
(Unit and Integration) established, test procedures specified test (e.g., no severity 1
complete. NCR)

3.2.2.4 Deployment Tasks

Deployment tasks include those associated with the release of custom code (test executables,
patches and system release) and COTS (new and upgrade). Deployments are planned and
coordinated, and activities may be identified in Primavera and the Patch Plan. Release
management includes various phase-related entry/exit criteria leading to a Pre-Ship Review
(PSR) and includes completion of testing, NCR status and release-related documentation
readiness. The Operations Deployment CPT is primarily responsible for performing deployment
tasks. QA is responsible for monitoring the activities and conducting audits for compliance to
established process.

3-3 104-EMD-001
3.2.3 Quality Assurance Processes
The QA organization has well established, documented processes, referred to as Landover
Facility Procedures (LFP). The QA LFPs outline the processes for conducting objective audits
and product evaluations, including developing appropriate audit/evaluation criteria and audit
checklists, conducting the audit, documenting and reporting results, to include any
nonconformances, which are documented based on defined criteria, as Deficiency Reports (DR)
or Corrective and Preventive Action Report (C/PARs), conducting follow-up and verification
audits to ensure closed-loop corrective action, collecting metrics, and maintaining audit records.
See section 9.2 for a complete list of current Quality Assurance LFPs.

3.3 Roles and Responsibilities

3.3.1 Quality Assurance Manager

The Quality Assurance organization will be led by a Quality Assurance Manager with the
responsibility to:
• Ensure the EMD QA team is staffed appropriately.
• Manage the EMD Quality Assurance budget and schedule.
• Evaluate QA staff performance; provide direction and guidance, as appropriate.
• Provide appropriate resources: office space, materials, tools, and training to ensure
that Quality Assurance activities can be carried out effectively.
• Support AS9100/ISO 9000 related activities and ensure continued conformance to the
requirements of the standards.

3.3.2 Quality Assurance Engineers

Quality Assurance Engineers (QAE) are assigned by the QA Manager to support the EMD
Program. Each QAE is responsible for quality assurance activities, such as:
• Monitor EMD activities and ensure the appropriate standards, processes, and
procedures are identified and implemented by EMD Program staff.
• Conduct audits and perform objective evaluations of EMD processes and work
products, based on defined criteria, to ensure they are in accordance with and
conform to specified standards, processes, and procedures.
• Record and communicate results of audits and evaluations, which include identifying
nonconformances and documenting them as DRs or C/PARs.
• Collect and analyze QA activity metrics and report results to QA Manager and
Program management, as appropriate.
• Assign, track and verify corrective actions resulting from audits, evaluations,
monitoring or Quality Action Requests (QAR).

3-4 104-EMD-001
 • Ensure continuous process improvement by participating in process teams, lessons
learned, management reviews, and Raytheon Six Sigma projects.
• Manage QA-related activities, as assigned, in accordance with EMD Program
schedule.
• Support AS9100/ISO 9000 activities and ensure continued conformance to the
requirements of the standards.

3.4 Quality Assurance Estimated Resources

The QA Manager has the responsibility for ensuring the EMD Program is staffed appropriately
to support the quality assurance requirements of the applicable task orders. EMD staffing status
will be reported monthly in the Contractor Manpower Report, DID EMD-MCMR-12.
As the Control Account Manager (CAM), the QA Manager is also responsible for maintaining
and monitoring the allocated budget for the Quality Assurance Work Breakdown Structure
(WBS). Financial management status will be reported monthly in the Contractor Cost Reporting
— 533 Requirements report, DID EMD-533-11.

3-5 104-EMD-001
This page intentionally left blank.

3-6 104-EMD-001
 4. Documentation

4.1 Purpose
This section will identify the documentation governing the development, verification, validation,
use and maintenance of the ECS Science Data Processing System.
Quality Assurance engineers will conduct evaluations of specified documents to determine
conformance to applicable standards and Data Item Descriptions (DID), as well as to ensure
format, editorial, and technical adequacy. In accordance with LFP 19-0-22, Quality Assurance
Product Evaluation Procedure, QA engineers will use a prescribed checklist to conduct
document product evaluations. Document evaluation results will be provided to the authoring or
responsible engineer as redlines for editorial comments or corrective actions for nonconformance
to requirements.

4.2 Minimum Documentation Requirements

4.2.1 Software Requirements Description

Level 3 requirements are specified by ESDIS in the ECS Functional & Performance
Specification. Level 4 requirements, derived from the Level 3s are documented in EMD Tickets,
which are controlled documents that include interfaces, operational concept, design
requirements, design directives, test criteria (functional, error and performance) and mappings to
test procedures.
QAEs review Requirements Tickets concurrent with their participation in Ticket Peer Reviews.
QAEs have a dual role in the Peer Review process: in addition to review of the requirements
document (Ticket), the process itself is subject to audit (see Section 3.2.3. Quality Assurance
Processes).

4.2.2 Software Design Description

The software design description documents developed under the ECS SDPS contract include
ECS DID 305, Segment Design Specification, DID 311, Database Design and Database, and
DID 313, Internal Interface Control Document (ICD). These documents will be maintained
throughout the EMD contract period and will be subject to the QA review/evaluation described
in Section 4.1 above.

4.2.3 Verification and Validation Plans

Verification and validation (V&V) activities occur throughout the design and development
lifecycle and include peer review of requirements, design artifacts, and code, as well as unit,
integration and formal (witnessed) tests. For each design element, test plans for unit-level and

4-1 104-EMD-001
integration tests are developed and peer reviewed. Development unit and integration tests are
scheduled and test plans and procedures are documented in accordance with specified formats.
Verification and validation activities are identified during the planning process and will be
documented in the Planning Input Document (PID) and associated Ticket. Depending on several
factors, e.g., complexity of the design, operational needs, and system impact, formal witnessed
verification & validation tests may be planned. Formal tests will be documented in specified
formats and test plans and procedures will be peer reviewed. Formal test plans may be submitted
to ESDIS for review and approval prior to test execution, or reviewed concurrent with the
development design peer review.

4.2.4 Formal Verification Results Report and Validation Results Report

The Verification Database (VDB) is the repository for formal verification and validation results
and provides various reports on demand. At the conclusion of a witnessed, formal integration or
acceptance test, results are initially documented in Criteria Verification Logs. The EMD test
team reviews the log results with ESDIS Independent Verification and Validation (IV&V)
representatives. Verification status requires IV&V concurrence prior to being formally
documented as ‘verified’ in the VDB.
EMD test artifacts (e.g., execution forms, output logs and GUI screen captures) are maintained in
formal Integration or formal Acceptance Test Folders and are subject to post-test and periodic
QA audit and evaluation.

4.2.5 User Documentation

User documentation developed for the DAACs under the ECS SDPS contract include ECS DID
609, Operation Tool Manual, DID 625 Training Material, and DID 611, Mission Operations
Procedures. These documents will be maintained throughout the EMD contract period and will
be subject to QA review and evaluation as described in Section 4.1 above. In addition, software
release documents such as release notes and installation documents are reviewed by QA during
the deployment internal review process and pre-ship evaluations.

4.2.6 Configuration Management Plan

The EMD contractor will develop and deliver a Configuration Management Plan (DID EMD-
CMP-19), three months after task authorization, as specified by the CDRL for EMD Task 101.
QA will review and evaluate the CMP in accordance with the QA process as described in
Section 3.2.3.

4.3 Other Documentation

4.3.1 Transition Plan

The Transition Plan (DID EMD-TP-1), EMD Contractor document number 101-EMD-001, was
developed and delivered in August 2003 – two weeks after task authorization, as specified by the

4-2 104-EMD-001
CDRL for EMD Task 101. The document was reviewed by QA and results recorded in Product
Evaluation #0388, in accordance with QA process as described in Section 3.2.3.

4.3.2 Program Management Plan

The Program Management Plan (DID EMD-PMP-8), EMD Contractor document number 108-
EMD-001, was developed and delivered in September 2003 — one month after task
authorization, as specified by the CDRL for EMD Task 101. The document was reviewed by QA
and results were recorded in Product Evaluation #0389.

4.3.3 Development and Program Plans

The EMD contractor will develop and deliver the following documents according to the
specified schedule of the CDRL for EMD Task 101. QA will review and evaluate these
documents in accordance with QA process as described in Section 3.2.3.
• DID EMD-SMDP-2, Software Maintenance and Development Plan
• DID EMD-HMDP-3, Hardware Maintenance and Development Plan
• DID EMD-PP-5, Property Management Plan
• DID EMD-RMP-6, Risk Management Plan
• DID EMD-SMP-7, EMD Security Management Plan
• DID EMD-PM-17, Procurement Management Plan

4.3.4 Review Documentation

Quality Assurance will participate in program and milestone reviews, as appropriate. As above,
QA will review and evaluate presentation and report materials in review packages, as indicated
for CDRL items, to ensure compliance to review agenda, DID requirements, and the statement of
work.

4-3 104-EMD-001
This page intentionally left blank.

4-4 104-EMD-001
 5. Standards, Practices, Conventions, and Metrics

5.1 Purpose
The purpose of this section of the SQAP is to identify the processes, product standards, and
metrics used by the Sustaining Engineering IPT, and to explain how QA monitors and assures
conformance to these requirements.

5.2 Content

5.2.1 Process Requirements

Documentation, design and coding standards, development and maintenance processes, and test
standards and practices are all described in EMD Project Instructions (PI) and Work Instructions
(WI) posted to an internal web page, accessible to EMD Program personnel. EMD product and
process metrics are described in the Program Management Plan, also available via the web.
Project PIs, WIs and plans are reviewed regularly and updated as appropriate.
QA will monitor conformance to these requirements through regularly scheduled, formally
documented audits. Audit findings will be recorded in the Quality Assurance Tracking Database,
and associated physical artifacts will be filed in the QA Repository. Findings that require
corrective action will continue to be monitored and tracked to acceptable resolution.

5.2.2 Metrics
QA participates in the development, implementation, and data production for the metrics defined
in the EMD PMP. These metrics, developed in coordination with the customer, are intended to
enable management visibility into the health of the development/maintenance process and the
operational ECS system. QA provides ESDIS management an independent analysis of these
metrics on a monthly basis.
In addition to the metrics described in the EMD PMP, QA maintains internal metrics to assess
the workload and effectiveness of the QA staff. These metrics are reviewed at staff meetings on a
monthly basis, and are used to adjust workload assignments and to focus QA attention to
potential problem areas.
The EMD QA Manager uses analysis of the EMD metrics, along with results from internal QA
metrics to apprise Raytheon program and executive management at quarterly Quality System
Management Reviews (QSMR). Analysis of the EMD metrics provides input into the Quality
Program Indicator (QPI), an internal tool used for evaluating and reporting project health status
to upper Raytheon management. QA internal metrics are used to report problems that require
upper management’s attention, as well as to report quality improvements.

5-1 104-EMD-001
This page intentionally left blank.

5-2 104-EMD-001
 6. Software Reviews

6.1 Purpose

6.1.1 Technical and Peer Review

Software reviews have a dual purpose on the EMD Program. First, software reviews present
opportunities for defect containment. Secondly, they may serve as gates to assure appropriate
progression to the next phase of the software lifecycle. EMD software reviews will be consistent
with the applicable technical review descriptions in the EMD SOW and the criteria for peer
reviews of software work products documented in the EMD Peer Review Process PI. Minimum
requirements established by IEEE Std-730-2002 are described for EMD in Section 6.2. The
schedule for software reviews will be determined for each project task order issued by ESDIS
and for each individual sustaining engineering NCR that meets the thresholds for software peer
reviews.
The reviews shall be accomplished according to governing EMD procedures and applicable
technical review descriptions in the EMD SOW. Quality Assurance will attend significant
reviews and audit the content, conduct, and follow-through on the review. As appropriate, QA
will be a participant and may be included on the agenda to report on activities, findings, and
software status.

6.2 Minimum Requirements

6.2.1 Software Specifications Review (SSR)

The EMD implementation of the SSR is the Requirements Management process that includes
Requirements Ticket peer reviews. The Ticket peer review is conducted to ensure adequacy of
requirements prior to review and/or approval of the Ticket by ESDIS and baseline in the VDB.
QA will participate in Ticket peer reviews and audit the process using criteria based on the
applicable governing procedures.

6.2.2 Architecture Design Review (ADR)

The EMD implementation of the ADR is the Preliminary Design Review (PDR) stage of the Peer
Review process. The PDR peer review is conducted to ensure adequacy of the preliminary
design, to uncover errors and improve design quality, and to determine readiness to proceed to
the next design phase. QA will participate in PDR peer reviews and will audit the process using
criteria based on the applicable governing procedures.

6-1 104-EMD-001
6.2.3 Detailed Design Review (DDR)
The EMD implementation of the DDR is the Detailed Design stage of the Peer Review process.
The DDR peer review is conducted to determine sufficient design detail, to uncover errors and
improve design quality, and to determine readiness to proceed to the next design phase. QA will
participate in DDR peer reviews and audit the process using criteria based on the applicable
governing procedures.

6.2.4 Verification and Validation Plan Review

The EMD implementation of V&V plan reviews occurs during Design and Code and Unit Test
Peer Reviews. At each successive peer review, the Integration Plan and Test Procedures will be
evaluated to ensure the V&V methods are complete and appropriate based on the applicable
requirements As above, QA will participate in these V&V peer review activities and audit the
process using criteria based on governing procedures.

6.2.5 Functional Audit

EMD QA will conduct a Functional Configuration Audit (FCA) when a release Consent to Ship
Review (CSR) is scheduled. This is also implemented on EMD through the continuous
monitoring of verification activity for new functionality delivered as a patch rather than a full
release.

6.2.6 Physical Audit

EMD conducts a Physical Configuration Audit (PCA) when a release CSR or Release Status
Review (RSR) is scheduled. The Configuration Management CPT will perform PCAs with
appropriate support from QA. The PCA process will be periodically audited by QA to ensure its
effectiveness and conformance to documented procedures.

6.2.7 In-Process Audits

As discussed previously, in-process audits of the design will be accomplished for the EMD
Program through design and development peer reviews monitored and audited by QA.
Interfaces are considered and addressed early in the requirements Ticket peer review and
continue to be evaluated through design and development peer reviews (PDR/DDR/CUT).
Code is evaluated with design documentation and artifacts during the Code and Unit Test (CUT)
inspection peer review.
Design implementation is evaluated to ensure functional requirements are appropriately met.
This is achieved through peer review, verification testing and FCAs, monitored and audited by
QA.
Test plans and procedures are reviewed for adequacy and suitability to demonstrate
implementation of functional requirements. This is achieved through the design/development and

6-2 104-EMD-001
CUT peer reviews, with emphasis on Integration Plans and Test Procedures, Ticket
requirements, and acceptance criteria.

6.2.8 Managerial Reviews

EMD Program Management will conduct regular managerial reviews, such as program monthly
reviews (PMR) to present general program and technical system status; and weekly status
reviews that addresses an agreed-upon agenda and provides the opportunity to ensure common
understanding of technical issues and priorities, discuss specific actions and mitigate overall
program risks.
The Quality Management System (QMS) and Quality Assurance activities are reviewed and
reported to program and executive management. QA will prepare a report for the EMD Program
Manager that lists previous month QA-related activities, including monitoring, audits and
product evaluations. The report will indicate the number of audit/evaluations performed, the
number of deficiencies, if any, and issues, concerns or observations resulting from trends;
including positive feedback. QA also presents Quality System Management Reviews (QSMR) to
executive management quarterly. This review provides program and process insight to enable
management to assess the suitability and effectiveness of the Landover facility QMS.

6.2.9 Software Configuration Management Plan Review

The EMD implementation of this will occur during the document review cycles the EMD CM
Plan will go through from draft to final contract deliverable. There are many functional
reviewers and QA will focus on DID compliance and the adequacy and completeness of the CM
methods defined for EMD.

6.2.10 Post-Implementation Review

The EMD implementation of this will be the Lessons Learned Review scheduled according to
the EMD SOW.

6.3 Other Reviews and Audits

EMD process audits are scheduled by QA for key processes implementing EMD activities and
tasks described in Section 3.2.2 at a frequency appropriate to their maturity and impact on EMD
and the ECS system. EMD product evaluations are conducted on all contract deliverables during
the document preparation review cycles, and key products from the tasks in Section 3.2.2. The
QA schedule will reflect the status, accomplishment and frequency of these process audits and
product evaluations.

6-3 104-EMD-001
This page intentionally left blank.

6-4 104-EMD-001
 7. Test

The EMD SOW requires testing in addition to the software verification and validation testing
described in Section 4.2.3. End-to-end performance tests of extended duration will be monitored
by QA when scheduled. Regression and Installation testing will be executed for each patch or
major release and the resulting test reports audited by QA as part of the deployment audit
process.

7-1 104-EMD-001
This page intentionally left blank.

7-2 104-EMD-001
 8. Problem Reporting and Corrective Action

This section of the SQAP describes the process for reporting, tracking, and resolving
problems/issues from software items and software development and maintenance processes and
work products.

8.1 Hardware and Software Items

Problems concerning SDPS custom code, COTS, and associated software/hardware items may
be reported by the Sustaining Engineering IPT and/or the operational DAACs as
nonconformance reports (NCRs) or Trouble Tickets. The responsible organizations and the
processes and tools used for reporting, tracking and resolving these problem reports are
described in EMD Project and Work Instructions.

8.2 EMD Process and Work Products

Problems resulting from EMD software development and maintenance processes and work
products may be reported by project personnel or identified by QA as a result of formal audits
and evaluations or informal surveillance. Project personnel may also identify and report
problems to QA via a Quality Action Request (QAR), which will initiate an investigation by QA.
Results from formal audits/evaluations or informal surveillance that require corrective actions, as
well as QARs, are documented and tracked in the QA Tracking Database. Associated physical
artifacts are filed and maintained in the QA Repository. In accordance with established LFPs
(see Section 9.2), all corrective actions will be monitored and tracked to acceptable resolution.

8.3 Quality Management System

Problems uncovered during ISO audits or project process audits that indicate a failure of the
Quality Management System, are documented as C/PARs. QA is responsible for the tracking and
status reporting of C/PARs. The QA manager, who is also the Landover Site ISO Coordinator,
reports on ISO activities (ISO training, scheduled audits, C/PAR status) to EMD Program and
Executive Management.

8-1 104-EMD-001
This page intentionally left blank.

8-2 104-EMD-001
 9. Tools, Techniques, and Methodologies

9.1 Tools and Equipment

QA will use standard tools and procedures already established and proven effective during the
ECS contract, to continue support on the EMD Program. QAEs will use standard Microsoft
Windows-based computers and software applications to ensure collaborative ease of use and
consistency across the EMD Program:
• QA Tracking Database – is a Microsoft Access database, custom designed by and
accessible only to QA. It was developed to document audit and evaluation results,
deficiency reports, and status. Records in the database are updated, maintained and
statused as corrective action and verification activities are carried out to closure. The
database will be used to generate data and metrics necessary for program and executive
management reports and reviews, as well as to demonstrate compliance to the
requirements of ISO 9000 and AS9100.
• QA will use Microsoft Project to schedule and track QA activities planned as ongoing or
aligned with the EMD Program schedule (in Primavera).
• Microsoft Office products, such as Word, Excel and PowerPoint are standard and will be
used to develop documentation, QA metrics reports and graphs, and review or
presentation material, as appropriate. Visio Professional may also be used to develop
process flows and other diagrams.

9.2 Methodologies and Process

The EMD QA organization relies on a well-established hierarchy of documented processes to
direct QA activities, stemming from the Raytheon Quality Management System Manual, IPDS,
IIS Product Assurance/Quality policies and procedures. The latter of which have been tailored
and documented at the local level as Landover Facility Procedures and listed below:
19-0-0 ISO Implementation Map
19-0-2 Quality System Management Review
19-0-4 Software Quality Assurance Program Planning
19-0-14 Quality Assurance Deficiency Reporting
19-0-16 Quality Management System Records
19-0-17 Internal Quality System Audits
19-0-19 Quality Assurance Audit Procedure
19-0-20 Quality Assurance Evaluation Criteria Procedure
19-0-22 Quality Assurance Product Evaluation Procedure
19-0-23 Quality Assurance Status and Metrics Reporting
19-0-24 Corrective and Preventive Action

9-1 104-EMD-001
This page intentionally left blank.

9-2 104-EMD-001
 10. Media Control

During the EMD Program ECS SDPS and COTS software and media will be controlled by the
Configuration Management organization through established process and use of appropriate
tools (e.g., Rational® ClearCase®). For more information and detail, refer to the EMD
Configuration Management Plan (DID EMD-CMP-19).

10-1 104-EMD-001
This page intentionally left blank.

10-2 104-EMD-001
 11. Supplier Control

11.1 Supply Chain Management

Raytheon’s Supply Chain Management (SCM) organization is responsible for subcontract and
procurement management. SCM will ensure that all required products and services are
competitively selected, to the fullest extent possible, in accordance with prime contract terms
and conditions, company policies and procedures, and all applicable statutes and regulations.
Further details and information will be available in a separate Procurement Management Plan to
be developed in accordance with DID EMD-PM-17 and evaluated by QA prior to delivery.

11.2 Quality Assurance Role

As part of the Raytheon Quality Management System, SCM procedures and procurement records
are subject to audit and inspection. Quality Assurance performs annual audits to the requirements
of AS9100 and ISO 9000, as well as internal Raytheon policies and procedures to ensure
continued compliance.
All subcontractor team members working onsite are directed to adhere to EMD processes and
procedures, and are subject to process audit and work product evaluations. In addition, QA
participates in and contributes to regular subcontractor performance evaluation.

11-1 104-EMD-001
This page intentionally left blank.

11-2 104-EMD-001
 12. Records Collection, Maintenance, and Retention

12.1 Quality Assurance Tracking Database

Quality Assurance engineers document audit and product evaluation results in an MS Access-
based database, i.e., the QA Tracking Database. The database is access-restricted to QAEs. Data
entry correction and database modifications is further restricted to the Quality Assurance
Manager or designee. The database resides on a network server that is included in the
Information Technology (IT) routine back-up schedule.

12.2 Quality Assurance Repository

The Quality Assurance organization maintains a physical records repository collocated within
the QA office space. Complete audit records include the formal audit or product evaluation
report (generated from the QA Tracking Database), and as appropriate, Deficiency Reports
(DRs), checklists, and other artifacts and evidence. Quality Assurance audit and product
evaluation records are retained three years minimum, in accordance with Raytheon Company
Policy 000000024-RP, unless otherwise stipulated by EMD Contract. Upon request, audit
records are available for EMD Program and ESDIS System Assurance review.

12-1 104-EMD-001
This page intentionally left blank.

12-2 104-EMD-001
 13. Training

In order to carry out the Quality Assurance activities described herein, each QAE is expected to
have fundamental knowledge of the following areas through prior experience or training.

13.1 Training Requirements

It is the responsibility of the Quality Assurance Manager to establish basic QAE skills, training
and experience requirements. The QA Manager will assess QAE capability, identify training
needs, and facilitate training delivery via applicable Raytheon and Landover procedures.
EMD QAEs either already possess the following knowledge through work experience or receive
targeted training prior to engaging in quality assurance tasks.
• Software Engineering Principles – General understanding of software lifecycle and
software work products.
• ECS SDPS – General understanding of the EOSDIS Core System Science Data
Processing System.

13.2 Raytheon Provided Training

The training listed below was developed by Raytheon for the specified purpose and is available
to all EMD Quality Engineers.
• Internal Auditor Training – Describes the fundamental principles in preparing,
conducting and documenting an internal audit.
• ISO 9000/AS9100 – Overview describing the requirements of the ISO 9000 Quality
Management Systems standard and additional Aerospace requirements.
• Raytheon Integrated Product Development System (IPDS) – Overview and
understanding of the structure within which Raytheon common processes are
integrated and deployed.
• Raytheon Six Sigma (R6σ) – Overview of the fundamentals of R6σ, which is broader
in scope than traditional six sigma. Specialist training provides specific guidance for
applying the principles and tools to achieve improvement objectives.

13-1 104-EMD-001
This page intentionally left blank.

13-2 104-EMD-001
 14. Risk Management

14.1 Risk Management Approach

Raytheon has a well-structured continuous risk management system (CRMS) in place that meets
the guidelines of NPG 7120.5A. Risk factors such as technical complexity, staff
experience/availability, external dependencies, and COTS integration are considered in costing
and scheduling and form an integral part of the EMD planning process for system enhancements.
As a result, potential risks are identified and addressed early in the process and tracked
throughout the development process until they can be closed. Further detail will be provided in a
separate Risk Management Plan to be developed and delivered in accordance with the CDRL
schedule and DID EMD-RMP-6.

14.2 Quality Assurance Role

The Quality Assurance program monitors the risk process via scheduled program audits and
annual AS9100/ISO 9000 audits. QA participates in risk mitigation strategies, such as
monitoring PVC testing, auditing design and development reviews, and participation in
management/customer reviews. QA identifies and reports program risk through its analysis of
noncompliance and corrective action trends resulting from program audits and evaluations.

14-1 104-EMD-001
This page intentionally left blank.

14-2 104-EMD-001
 15. Glossary

Terms and abbreviations used in this SQAP are listed below. Most are well understood within
the EMD Program and similar industry and programs. Abbreviations and acronyms referenced
herein have been defined within this plan.

ADR Architecture Design Review

CAM Control Account Manager
CCB Configuration Control Board
C/PAR Corrective and Preventive Action Report
CPT Cross Product Team
CRMS Continuous Risk Management System
CSR Consent to Ship Review
CUT Code and Unit Test
DAAC Distributed Active Archive Center
DDR Detailed Design Review
DID Data Item Description
DR Deficiency Reports
ECS EOSDIS Core System
EMD ECS Maintenance and Development
ESDIS Earth Science Data and Information System
FCA Functional Configuration Audit
HMDP Hardware Maintenance and Development Plan
ICD Interface Control Document
IIS Intelligence and Information Systems
IPDS Integrated Product Development System
IPT Integrated Product Team
IT Information Technology
ITS Information Technology Systems
IV&V Independent Verification and Validation
LFP Landover Facility Procedures
NCR Non-Conformance Report
NQA National Quality Assurance
PCA Physical Configuration Audit
PDR Preliminary Design Review
PI Project Instruction
PID Planning Input Document
PM Program Management
PSR Pre-Ship Review
QA Quality Assurance
QAE Quality Assurance Engineer
QAR Quality Action Request
QMS Quality Management System

15-1 104-EMD-001
QPI Quality Program Indicator
QSMR Quality System Management Review
R6σ Raytheon Six Sigma
RAB Registrar Accreditation Board
RSR Release Status Review
SCM Supply Chain Management
SDPS Science Data Processing System
SEP System Enhancement Proposal
SMDP Software Maintenance and Development Plan
SQAP Software Quality Assurance Plan
SSR Software Specifications Review
VDB Verification Database
WBS Work Breakdown Structure
WI Work Instruction

15-2 104-EMD-001
 16. SQAP Change Procedure and History

16.1 Change Control

This document, produced by the Quality Assurance organization, will undergo a review by
members of Program Management and the Configuration Control Board (CCB), and will be
released upon their approval. Thereafter, the document will be under Configuration Management
control. Subsequent changes or modifications to this document will be the responsibility of QA.
Proposed changes will be coordinated by QA and reviewed and approved by the CCB.

16.2 Modification History

This is the initial release of the Software Quality Assurance Plan for the EMD Project. Future
versions will be identified by a revision indicator (letter or number) and the revision history will
be documented in the front matter of this SQAP.

16-1 104-EMD-001
This page intentionally left blank.

16-2 104-EMD-001