Keywords

1 Introduction

In the fourth edition of the Safety Management Manual (SMM) of the International Civil Aviation Organization (ICAO), the concept of “safety intelligence” was put forward, which provides a new perspective for the future improvement of safety management in the world civil aviation. Safety intelligence concerned with leveraging safety data and safety information to develop actionable insights which can be used by an organization’s leadership to make data-driven decisions, including those related to the most effective and efficient use of resources.

Also, according to the SMM, each State should consider the acceptable level of safety performance (ALOSP). The ALOSP expresses the safety levels the State expects of its aviation system, including the targets that each sector needs to achieve and maintain in relation to safety, as well as measures to determine the effectiveness of their own activities and functions that impact safety. The responsibility for establishing the ALOSP rests with the State’s aviation authorities, and will be expressed through the set of Safety Performance Indicators (SPIs) for the State, sectors and service providers under their authority.

But compared with service providers, the evaluation of the State’s safety performance is more complicated, because a large amount of operational information data is distributed in different operating and management systems. What kinds of SPIs can be represented as a national safety level? How to obtain the raw data to support these SPIs? Are these data sources and channels stable and long-term? All these questions are needed to solve.

2 Available Representative Data and Their Implications

2.1 High Risk Categories (HRCs) of Occurrences from ICAO and China Civil Aviation

The selection of types of occurrences which are deemed HRCs are based on high fatality risk per accident or the number of accidents and incidents. Based on results from the analysis of safety data collected from proactive and reactive sources of information (e.g. accidents, incidents, events), as well as from ICAO and other non-governmental organizations, the following HRCs, in no particular order, have been identified for the 2017–2019 edition of the Global Aviation Safety Plan (GASP) [1]: Runway safety (RS) related events, controlled flight into terrain (CFIT); and loss of control in-flight (LOC-I).

ICAO uses these HRCs as a baseline in its safety analysis. Figure 1 shows that in 2018, the three categories represented 96% of all fatalities, 73% of fatal accidents, 54% of the total number of accidents and 80% of the accidents that destroyed or caused substantial damage to aircraft [2].

Fig. 1.
figure 1

High-Risk Category accident distribution

Full size image

Figure 2 shows a breakdown of the three HRCs in 2018 and the respective distribution of accidents, fatal accidents, fatalities and accidents in which aircraft were destroyed or substantially damaged.

Fig. 2.
figure 2

High-Risk Category accident overview

Full size image

Accidents related to runway safety (RS) accounted for nearly half of all accidents in 2018 (48%, compared with 53% in 2017), and included 4 fatal accidents with 54 fatalities. Loss of Control In-Flight (LOC-I) represented 36% of fatal accidents (up from 20% in 2017) with total 438 fatalities. There were no fatal accidents related to controlled flight into terrain (CFIT) in 2018.

According to ICAO, runway safety related events include the following ICAO accident occurrence categories: abnormal runway contacts, runway excursion, runway incursion, loss of control on ground, ground collision, collision with obstacles, undershoot/overshoot. But according to the event types of Chinese civil aviation, runway safety mainly involves heavy landing, touching the ground except for landing gear wheels, bird strike, runway excursion, runway incursion, landing on the ground outside the runway, and ground collision with obstacles. In order to further identify the major concerns of China Civil Aviation, this study need to combine the data analysis of the past decade in China.

According to the statistics of flight accidents from 2008 to 2017 from the Aviation Safety Information System (ASIS) of China Civil Aviation, as shown in Fig. 3, the statistical results show that CFIT, LOC-I and RS related events (runway excursion) are the types with high frequency of occurrence in China in the past decade.

Fig. 3.
figure 3

Statistical analysis chart of accidents from 2008 to 2017

Full size image

Therefore, combined with global safety priorities and China’s civil aviation safety information, it can be set to CFIT, LOC-I and runway excursion(RE) as major concerns of China civil aviation. These are also seen as outcome indicator of the operational risk SPIs in national safety performance evaluation indicator system, and it can be provided intuitive data form the aviation safety information network such as the numbers of accidents, incident, general event, or specific types of events such as CFIT, LOC-I and RE.

2.2 Flight Operation Quality Monitoring Information of Civil Aviation Administration of China

In 2013, Civil Aviation Administration of China (CAAC) approved the establishment of CAAC Flight Operations Quality Assurance (FOQA) Station. In 2017, the China Civil Aviation Flight Quality Monitoring Service Platform (FQMSP) was launched, and by the end of 2019, the FOQA station could real time monitor 3700 aircraft from 54 CAAC transport airlines. Since 2018, the analysis has been focused on the red events which are exceed established standards monitored by the base station, and includes CFIT, LOC-I, and RE risk and related monitoring parameters.

According to the statistical analysis report on FOQA of CAAC on 2017 [3], the FOQA monitoring items with the highest influence degree on CFIT risk in the A320 series are respectively: GPWS Warning, Glide slope deviation, High approach speed, Localizer deviation, Roll high in landing, and in the B737 series are respectively: High IVV, Glide slope deviation, High approach speed, Localizer deviation and Roll high in approach.

The FOQA monitoring items with the highest influence degree on LOC-I risk in the A320 series are respectively: Pitch platform, Double side lever Input, and in the B737 series are respectively: Pitch high at takeoff, Pitch low in landing, Roll high in approach and Stick shaker.

The FOQA monitoring items with the highest influence degree on RE risk in the A320 series are respectively: High landing speed, unstable landing glide direction, Glide slope deviation. But in the B737 series are respectively: unstable takeoff or landing glide direction, ILS glide slope deviation, High approach speed, ILS localizer deviation, High landing speed and late landing flaps setting.

FQMSP can directly display the fusion value of the three risks and the actual value of the corresponding monitoring items. Take the risk value of LOC-I of CAAC in May 2018 as an example and the corresponding “Roll high in approaching” of one of the monitoring items, as shown in Fig. 4 and Fig. 5.

Fig. 4.
figure 4

The daily risk of LOC-I in May 2018

Full size image
Fig. 5.
figure 5

The daily risk of roll high in approaching in May 2018

Full size image

Also, the CAAC FQMSP can further show the number and incidence of different over-limit events in the A320 series and B737 series.

Therefore, through the civil aviation flight quality monitoring information research, the CFIT, LOC-I and RE risks of China’s civil aviation, can be further decomposed into corresponding FOQA monitoring items as process indicators of the operational risk SPIs, and the flight quality monitoring platform is able to provide sustained, stable and real data for these indexes.

2.3 Flight Standards Oversight Information of CAAC

Flight standards oversight is one of the important methods for civil aviation regulatory departments to carry out safety management for airlines. Through years of efforts, CAAC has gradually set up a flight standards oversight program (FSOP). Based on the flight standards rules and regulations, the system is used for flight standards department of the civil aviation administration, regional administration and supervision bureau. In 2013, CAAC promoted the implementation of FSOP in the industry. At present, some data related to industry safety performance can be obtained continuously from FSOP system.

The source of data comes two parts. One is basic operating data that includes aircraft number, flights time, pilots (captains and copilots) number, fatigue index, effective flight attendants of each transport airlines, and these data can be collected and calculated monthly. The other is oversight data checked and found by the regulatory inspectors. The FSOP system can display the number of problems found by all inspectors in a certain period of time, as shown in Table 1.

Table 1. Number of problems identified in regulatory supervision from January to May, 2018
Full size table

Meanwhile, the system can display the distribution diagram of the number of findings classification, such as procedures, controls, tools, manuals, personnel.

As a result, the FSOP system can provide some reliable data for safety management and foundation indicators of the operational risk SPIs, such as problem rate found by regulatory supervision, the problem rate found by different classification, as well as fatigue index and pilot-aircraft ratio.

2.4 State Safety Profile Information from ICAO

The integrated Safety Trend Analysis and Reporting System (ISTARS) is a web-based system on the ICAO USOAP audit consequence. ISTARS provides a quick and convenient interface to a collection of safety and efficiency datasets and web applications to make safety, efficiency and risk analyses. “State Safety Briefing” is a module of the system, which extracts the aviation safety profile of each State and gives a brief description of the 13 indicators from both target and completion values through the Dashboard, as shown in Fig. 6.

Fig. 6.
figure 6

An overview of China’s aviation safety profile in 2019

Full size image

From the dashboard, ISTARS can provide some direct indictors as the process implementation SPIs, such as USOAP EI, significant safety concerns, SSP foundation, global Aviation training activities, and corrective action plan update.

2.5 Other Authorization Information

In addition to the stable data provided by the above systems, relevant information can also be collected according to the authorization to continuously improve the national safety performance indicators. For example, SMS maturity information of some enterprises can be collected through SMS auditing tool.

3 Design Principles of Indicator System

Based on the above-mentioned in-depth investigation of data and information related to civil aviation safety performance of CAAC, this study classifies the sources, functions and types of data and information, and proposes to establish a safety performance indicator system of civil aviation industry. In order to establish a more scientific and accurate national safety performance indicator system, the selection and design of indicators mainly follow several basic principles [5,6,7]:

  1. 1)

    Clarify safety objectives

    It is determined that the safety objectives is to control the three core risks of CFIT, LOC-I and RE, and the safety objectives of industry safety performance can be adjusted according to the change of international and domestic safety concerns.

  2. 2)

    Set index category

    Based on the safety principles such as accident cause theory and actual deviation theory as the design theoretical basis of indicators, referring to the ICAO guidance document (DOC9859) [4] and the concept of safety performance management of operators, a rich and multi-dimensional industry safety performance indicator system is established, and the operation risk category is set (which can be divided into safety result category, operation process category, safety management category and safety foundation category), as well as process implementation category, to realize comprehensive evaluation of industry safety status from process and result, supervision and operation, resources and guarantee.

    Operational risk SPIs mainly reflects the safety performance of service providers, which is used to reflect the overall operation level of the industry. Specifically divided into:

    • safety result category: to evaluate accidents, incidents and other events related to core risks of the industry.

    • operation process category: to assess the degree of operational deviation affecting the core risks of the industry.

    • safety management category: to evaluate the ability and effect of industry supervision and management on Enterprises.

    • safety foundation category: to assess the overall resource allocation of the industry to the enterprise.

    Process implementation SPIs mainly reflects the risks in the process of management and operation, which is used to reflect the management ability of the industry.

  3. 3)

    Ensure sustainable and effective access to monitoring data

    Under the premise of ensuring that the industry safety performance indicators are quantifiable and measurable, considering that the monitoring data corresponding to the indicators can be obtained continuously and effectively, the research considers that:

    • Aviation Safety Information System of CAAC (ASIS) can provide stable data sources for operational risk (safety result category) SPIs.

    • China Civil Aviation Flight Quality Monitoring Service Platform (FQMSP) can provide reliable data for operational risk (operation process category) SPIs.

    • Flight Standards Oversight Program of CAAC (FSOP) can provide reliable data for operational risk (operation process category and safety foundation category) SPIs.

    • ICAO Integrated Safety Trend Analysis and Reporting System (ISTARS) can provide reliable data for process implementation SPIs.

  4. 4)

    Ensure sustainable and effective access to monitoring data

    Pareto’s principle tells us that 80% of output comes from 20% of important input. For industry safety performance management, it is impossible to identify, measure and manage all indicators related to industry safety performance. And in order to save the cost of management and achieve the purpose of managing the safety status, it is necessary to develop 20% representative key SPIs. Through the management of these KPI, it will achieve the purpose of monitoring the overall safety status.

4 Framework and Examples of National Safety Performance Evaluation Indicator System

On the basis of following the design principle of index system, it designs the industry safety performance evaluation indicator system. The specific framework is as follows in Fig. 7, and the following Table 2 is an example of national safety performance evaluation indicator system.

Fig. 7.
figure 7

Framework of national safety performance evaluation indicator system

Full size image
Table 2. An example of national safety performance evaluation indicator system
Full size table

5 Prospect Application

The establishment of national safety performance evaluation indicator system breaks the data barrier of each operation management system to some extent and makes the data used for decision-making more objective and diversified. The study will continue to enrich indicators that can be used to evaluate the safety performance of the industry, and try to use mathematical models to evaluate and warn the comprehensive risks on the basis of these indicators, so that CAAC can conduct risk prevention and control of the industry safety trends in real time, and timely intervene to avoid serious consequences.