USOAP Continuous Monitoring
Approach (CMA) Workshop
Module 4
How to Conduct PQ Self-Assessment and
Submit/Update Corrective Action Plans
(CAPs)
March 2017 CMA Workshop Module 4 1
Objective
The objective of this module is to explain:
• How to conduct PQ self-assessment; and
• How to submit and update CAPs.
March 2017 CMA Workshop Module 4 2
Outline
How to Conduct PQ Self-Assessment.
How to Submit CAPs.
How to Update CAPs.
March 2017 CMA Workshop Module 4 3
How to Conduct
PQ Self-Assessment
March 2017 CMA Workshop Module 4 4
PQ Self-Assessment
• PQ self-assessment essentially means that States answer the
Protocol Questions (PQs) on the CMA Online Framework (OLF)
as a way to assess the state of their civil aviation safety
oversight systems.
• Specifically, they:
– Assess the “Status of Implementation” of
each PQ (i.e. Satisfactory, Not Satisfactory
or Not Applicable);
– Provide “Remarks” to explain “Status”; and
– Attach supporting “Evidence”.
March 2017 CMA Workshop Module 4 5
PQ Self-Assessment
• Before conducting self-assessment on the OLF, States should
implement their CAPs and update status of CAP
implementation for each PQ finding.
• Through the “Self-Assessment” tool on the OLF, States can:
‒ Indicate/update the status of implementation of each PQ;
‒ Submit supporting evidence;
‒ Conduct internal self-audits;
– Prepare for ICAO USOAP CMA activities; and
– Monitor their own civil aviation safety oversight systems.
March 2017 CMA Workshop Module 4 6
PQ Self-Assessment
• As a priority, States’ PQ self-assessment should focus on:
a) “Not Satisfactory” PQs from previous USOAP activity;
b) New PQs introduced through PQ amendments (classified
as “undetermined PQs”);
c) Amended PQs which impact implementation status of
PQs; and
d) PQs whose implementation status may have changed due
to changes in States’ aviation system, regulations and/or
procedures.
March 2017 CMA Workshop Module 4 7
After Self-Assessment…
• ICAO uses the results of a State’s completed self-assessment to
prepare for CMA activities.
• Through the CMA activities, ICAO reviews and validates a
State’s self‒assessment to determine the latest level of
efficient implementation (EI) of State’s safety oversight
capacity.
• ICAO will periodically revise the PQs in order to reflect
amendments made to the referenced documents.
• It is important for States to always refer to the stated versions
of the referenced documents.
March 2017 CMA Workshop Module 4 8
How to Conduct PQ Self-Assessment
• Go to the USOAP CMA Online
Framework (OLF) website using
the following link:
www.icao.int/usoap.
• Click the “Self-Assessment” icon.
March 2017 CMA Workshop Module 4 9
How to Submit CAPs
March 2017 CMA Workshop Module 4 10
How to Submit CAPs
States are required to submit:
• Effective CAPs;
• Provide regular CAP progress as required by the
MOU.
March 2017 CMA Workshop Module 4 11
How to Submit CAPs
• States should ensure that CAPs submitted:
− Fully address PQ findings;
− Are relevant, clear, detailed and comprehensive;
− Have realistic target implementation dates; and
− Work systematically towards implementing corrective
actions.
March 2017 CMA Workshop Module 4 12
How to Submit CAPs
Why is it important for States to provide effective CAPs
and CAP updates?
• It is required by the MOU.
• It ensures that States fully address PQ findings.
• CAP updates allow States to report the level of
progress made.
• It enables ICAO to determine the next course of
action.
March 2017 CMA Workshop Module 4 13
How to Submit CAPs
• Go to the USOAP CMA Online
Framework (OLF) website
using the following link:
www.icao.int/usoap.
• Click “CAP” icon.
March 2017 CMA Workshop Module 4 14
Conducting PQ Self-Assessment
View the toolbar and select the correct options
to obtain the desired information, as shown here.
March 2017 CMA Workshop Module 4 15
PQ Self-Assessment — for ICVM
States preparing for an ICVM should conduct a self-assessment
on all “not satisfactory” PQs of selected audit area(s).
Select the option “Not satisfactory” to view the full list of
“not satisfactory” PQs in the selected audit area.
March 2017 CMA Workshop Module 4 16
PQ Self-Assessment — for Audit
States preparing for an audit should conduct a self-assessment on all PQs.
Select the option “All” to view all PQs.
March 2017 CMA Workshop Module 4 17
Conducting PQ Self-Assessment
Select the desired PQ.
March 2017 CMA Workshop Module 4 18
Conducting PQ Self-Assessment
Refer first to ICAO’s comments from
previous activity report in conducting self-assessment.
Click “Translate” button to obtain
translation of text into desired ICAO language.
March 2017 CMA Workshop Module 4 19
Translation of Text
Click to select the desired language.
The translated text will appear in the top box, while the
text in the original language appears in the lower box.
Use the scroll bar to navigate up/down the textboxes.
March 2017 CMA Workshop Module 4 20
Viewing Guidance and References
To view guidance and references of selected PQ,
double-click the PQ number (first column on left) and
“Review Evidence & References” window will appear.
March 2017 CMA Workshop Module 4 21
Example
PQ No. Question
4.223 Does the aircraft operations organization ensure that the air operator
OPS develops an ACAS training programme on ACAS-equipped aircraft for its
CE-6 pilots?
Review Evidence:
1) Verify implementation of operations inspectors’ procedures as part of the
certification process.
2) Sample air operator’s training manuals.
3) Sample ACAS training programme
and records to confirm implementation.
Reference
STD: A6, Part I App. 2, 2.1.31
A6, Part III Sec 3, C3, Att. A
GM: Doc 8168, Vol I
March 2017 CMA Workshop Module 4 22
Conducting PQ Self-Assessment
For “Not Satisfactory” PQs, review the “Description” and “Progress Documented”
under the “Status of Implementation by ICAO” toolbar.
Note.—“Satisfactory” PQs will not have these information.
March 2017 CMA Workshop Module 4 23
Viewing CAP
To view CAP of related PQ, click “Click To View CAP” link on the right. OR
Scroll to bottom of page to view CAP inside “Corrective Action Plan” box.
March 2017 CMA Workshop Module 4 24
Conducting PQ Self-Assessment
Select “Yes” or “No” to “Reply to Protocol Question” based on
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f), describes requirements for air operators to establish an ACAS
programme whether or not the State has responded to the relevant PQ.
• Inspector guidance is detailed in Operation inspector Manual, chapter 4, section 2
Indicate the State’s status of implementation for said PQ.
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1, detail the certification process
• “Satisfactory”
• CAA-AC-OPS050 means
Contents of an Operationsthat the
Manual State
Section has
9.3.2(f) complied
includes with all the
the CAA requirements. guidance
• Operation
requirements of the
inspector manual, chapter PQ. 2
4, section
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1
• “Not satisfactory” applies when State has not fully complied with any of
the guidance requirements of the PQ.
• “Not applicable” means that the PQ does not apply in the State (e.g. PQ
related to ROs and the State does not have any).
• “BLANK” indicates the PQ is undetermined, typically for new PQs that
have not been audited.
March 2017 CMA Workshop Module 4 25
Providing Remarks…
Use the guidance in the “Review Evidence & References” box
to prepare and compose “Remarks” to explain “Status of Implementation”.
In the “Remarks” box, provide one sentence to describe progress of implementation
for each guidance element outlined in “Review Evidence & References”.
March 2017 CMA Workshop Module 4 26
Providing Remarks…
Points to keep in mind when providing “Remarks”:
• The State should provide “Remarks” to explain the reason
for the “Status of Implementation” (Satisfactory/Not
satisfactory/Not applicable) of the pertinent PQ.
• The “Remarks” should respond, point-by-point and in a
single sentence, to each of the corresponding guidance
elements found in “Review Evidence & References”.
• If supporting evidence is attached, the State should
provide detailed references, such as chapter, page,
paragraph, etc.
March 2017 CMA Workshop Module 4 27
Providing Evidence…
If necessary, provide/attach evidence to support
corresponding “Remarks” and guidance elements.
Include document name and specific reference(s).
For large documents, include detailed references to
chapters, sections, pages, etc.
March 2017 CMA Workshop Module 4 28
Completing PQ Self-Assessment
Copies of references to be attached
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f), describes requirements for air operators to establish an
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f), describesfor eachforitem
requirements of evidence.
air operators to establish an ACAS
ACAS programme
programme.
• Inspector guidance is detailed in Operation inspector Manual, chapter 4, section 2
• Inspector guidance is detailed in Operation inspector Manual, chapter 4, section 2
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1, detail the certification process
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1, detail the certification process
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f) includes the CAA requirements.
•• CAA-AC-OPS050
Operation inspectorContents of an Operations
manual, chapter 4, sectionManual
2 Section 9.3.2(f) includes the CAA requirements.
•• Operation inspector manual, chapter 4, section 2
The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1
Click “Select” to open window and select file for upload.
Click “Upload Files” to upload.
Until self-assessment is completed,
save your work using the button “Save as Draft”.
Once self-assessment is fully completed, click “Submit to ICAO”.
March 2017 CMA Workshop Module 4 29
Example of Completed Self-Assessment
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f), describes requirements for air operators to establish an ACAS
programme
• Inspector guidance is detailed in Operation inspector Manual, chapter 4, section 2
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1, detail the certification process
• CAA-AC-OPS050 Contents of an Operations Manual Section 9.3.2(f) includes the CAA requirements.
• Operation inspector manual, chapter 4, section 2
• The Air Operator Certification Manual Chapter 10, and checklist in Appendix 1
Helpful Tips in Revising/Updating “Remarks”:
1) Copy the original “Remarks” text to a WORD document.
2) Revise/update the original text as required.
3) In the “Remarks” box, add the date of submission.
4) Below the date, add the subject matter in CAPITAL LETTERS.
March 2017
5) CopyCMA
and Paste the revised/updated text into the “Remarks”30
Workshop Module 4
box.
6) Click “Save as Draft” to save your work.
Exporting Self-Assessment Checklist
Under “Protocol Question (PQ)” tab, scroll down and
select “Export State Self-Assessment”.
March 2017 CMA Workshop Module 4 31
Exporting Self-Assessment Checklist —
“All Protocols”
To view all PQs, Click “All Protocols”.
March 2017 CMA Workshop Module 4 32
Exporting Self-Assessment Checklist —
“Only Not satisfactory PQs”
To view “Not Satisfactory” PQs only,
click “Only Not satisfactory Protocols”.
March 2017 CMA Workshop Module 4 33
Exporting Self-Assessment Checklist —
Selected PQs
To view PQs of a particular audit area (1 area at a time),
check box next to selected audit area.
March 2017 CMA Workshop Module 4 34
Exporting Self-Assessment Checklist
Once PQ selection is made, click “Submit” to download file.
!!! Wait a few seconds for the file to download, depending on the
Internet speed. DO NOT press the “Submit” button twice.
March 2017 CMA Workshop Module 4 35
Exporting Self-Assessment Checklist
Save the file in the desired location for use.
March 2017 CMA Workshop Module 4 36
Exporting Self-Assessment Checklist
March 2017 CMA Workshop Module 4 37
Importing Self-Assessment Checklist
Under “Protocol Question (PQ)” tab, scroll down and select
“Import Protocol Self-Assessment Checklist”.
March 2017 CMA Workshop Module 4 38
Importing Self-Assessment Checklist
Click “Select” to open window and select file to be imported.
March 2017 CMA Workshop Module 4 39
Importing Self-Assessment Checklist
Select file to be imported and close window.
March 2017 CMA Workshop Module 4 40
Importing Self-Assessment Checklist
Click “Open” to import the file.
March 2017 CMA Workshop Module 4 41
Importing Self-Assessment Checklist
March 2017 CMA Workshop Module 4 42
Importing Self-Assessment Checklist
Once filename appears in “Select” field, click “Submit”.
March 2017 CMA Workshop Module 4 43
How to Submit CAPs
Review “Description” and “Progress Documented” for
pertinent PQ finding from previous USOAP activity.
“Preview mode” vs “Live mode”
March 2017 CMA Workshop Module 4 44
How to Submit CAPs
Select “Corrective Action Plan (CAP)” tab.
Refer to guidance and references of selected PQ.
Double-click PQ number (first column on left) and
“Review Evidence & References” window will appear.
March 2017 CMA Workshop Module 4 45
How to Submit CAPs
Provide a realistic “Estimated Implementation Date (EID)”
for each step of the CAP.
Enter the completion date when the CAP is completed.
Enter CAP in a step-by-step format, describing the action and
responsible action office per step in clear, concise term.
March 2017 CMA Workshop Module 4 46
How to Submit CAPs
Enter comments on CAP submission/update in “State Comments”
box, if necessary. Completion of this field is optional.
March 2017 CMA Workshop Module 4 47
How to Submit CAPs
Scroll down to the bottom of the page and click
Provide any changeslink
“Self-Assessment” impacting
to open the PQ self-assessment
“Self-Assessment” box
and attach evidence as necessary.
(read only).
Submit to ICAO even if action is not 100% completed so
Save as draft for the continuation of the work later.
that ICAO can monitor the progress
March 2017 CMA Workshop Module 4 48
Example
Example
PQ 5.453 (CE-8)
Does the AID initiate a special evaluation or impose special
operational restrictions if information obtained from reliability
monitoring indicates degraded level of safety?
Guidance
Review examples of action taken.
Reference
Note. — If oversight of reliability
programmes is not conducted,GM thisDoc 9760 Vol. II Part B
PQ is automatically “Not 7.4
satisfactory”.
.
March 2017 CMA Workshop Module 4 49
Example
“Description” and “Progress Documented” by ICAO:
“CAA does not initiate a special evaluation or impose special operational restrictions if information
obtained from reliability monitoring indicates degraded level of safety as part of its surveillance
reliability programme.”
March 2017 CMA Workshop Module 4 50
Example of Completed CAP
Corrective Action Plan, Action Items: 31/12/2012
Rev. Date of
Action Est. Imp.
Step Proposed Action Evidence Reference Imp. Completi Progress
Office Date
Date on
1 Programme AIR participation in all future AID 25/6/2013 Completed
reliability meetings held by the air operators.
2 Collect and analyse existing reports from AID 13/7/2013 Completed
reliability monitoring.
3 Establish a process to ensure the initiation of AID 11/10/2013 25%
special evaluation or imposition of operational
restrictions in cases of degraded level of safety.
4 Define possible operational restrictions to be AID 11/10/2013 Not started
taken.
March 2017 CMA Workshop Module 4 51
How to Add New CAPs
1 - Select PQ for which a new CAP is to be added.
2 - Click “Add new record”.
March 2017 CMA Workshop Module 4 52
How to Add New CAPs
Fill in the following fields:
• Step number: A CAP may
have several steps based on
complexity of action. Each
action will require a separate
step to facilitate CAP
assessment and validation.
• Estimated Implementation
Date: This estimated date
should indicate when the step
is expected to be fully
implemented.
• Progress status
• Action office is the office in
charge of completing this
step.
March 2017 CMA Workshop Module 4 53
How to Add New CAPs
Fill in the following fields:
• Proposed Action: Make sure
to provide a proposed action
item for the corresponding
step.
• Evidence Reference: Each
reference should be clearly
indicated (i.e. chapter,
section, paragraph, etc.).
• Save as draft
• Close: To return to the
previous page and to add
next steps.
March 2017 CMA Workshop Module 4 54
How to Add New CAPs
Click this link to provide self-assessment
and attach evidence.
Provide additional comments, if necessary.
This field is optional.
Click this button when all the steps related to the CAP are
completed and ready to be submitted to ICAO.
March 2017 CMA Workshop Module 4 55
How to Update CAPs
March 2017 CMA Workshop Module 4 56
How to Update CAPs
Click “Edit” to revise/update CAPs.
Select the PQ whose CAP is to be
revised/updated.
March 2017 CMA Workshop Module 4 57
How to Update CAPs
Fill in the following fields:
• Step Number: A CAP may
have several steps based on
the complexity of the action.
Each action will require a
separate step to facilitate
CAP assessment and
validation.
• Revised Implementation
Date: Based on the CAP
update, if the EID is no longer
realistic, then a revised
implementation date should
be entered.
• Date of Completion: The
date when the action was
completed should be entered.
• Progress Status
• Action Office
March 2017 CMA Workshop Module 4 58
How to Update CAPs
Fill in the following fields:
• Updated Proposed Action:
Make sure to provide a
proposed action item for each
corresponding step.
• Evidence Reference: Each
reference should be precise
and detailed (with chapter,
section, paragraph, etc.).
• Save as Draft
• Close: To return to previous
page and to edit other steps.
• Submit to ICAO: When all
steps are fully edited and
ready for submission.
Note.— Evidence may be attached using the instructions mentioned previously.
March 2017 CMA Workshop Module 4 59
Export/Import Self-Assessment Checklist
Points to keep in mind when exporting/importing files:
• If exported files are to be used solely for information purposes, make
sure to manually Unlock the file after exporting.
• Do NOT make any modifications to the structure of the document,
entering information ONLY in the prescribed areas.
! Changing the document structure will prevent you from importing said
file back into the OLF.
• Do NOT change the document name.
• Once all work on the document is completed, make sure to uncheck the
“Import the Self-Assessment draft results to the OLF and maintain the
lock on said checklist” button.
March 2017 CMA Workshop Module 4 60
Responsibility of State
State is required to:
• Provide a CAP that fully addressed the PQ Finding
(Refer to the 6 criteria of a good CAP-module 2);
• Submit to ICAO the progress of each CAP;
• When fully implemented, attached the evidence.
March 2017 CMA Workshop Module 4 61
Six Criteria for a Good CAP (“RCDSRC”)
1) Relevant: CAP addresses the issues and requirements related
to the finding and corresponding PQ and CE.
2) Comprehensive: CAP is complete and includes all elements
or aspects associated with the finding.
3) Detailed: CAP outlines implementation process using
step-by-step approach.
4) Specific: CAP identifies who will do what, when and in
coordination with other entities, if applicable.
5) Realistic: In terms of contents and implementation timelines.
6) Consistent: In relation to other CAPs and with the
State’s self-assessment.
21 February 2017 CMA Workshop Module 2 62
Responsibility of ICAO
ICAO is required to:
• Provide status of submitted CAP if it does not,
partially or fully addressed the Finding;
• Provide the status of the CAP review (not started,
in progress, completed);
• Verify if a completed CAP submitted meets the 3
requirements for validation.
March 2017 CMA Workshop Module 4 63
Review
How to Conduct PQ Self-Assessment.
How to Submit CAPs.
How to Update CAPs.
The system is evolving.
Feedback and suggestions from users is
required for the system to mature!
March 2017 CMA Workshop Module 4 64
