CN115587693A - A Method for Rating Flight Training Quality Based on Observable Behaviors - Google Patents

Abstract

The invention relates to the technical field of row training rating, and particularly discloses a flight training quality rating method based on observable behaviors, which is based on a traditional flight training performance assessment operation mode and a CBTA theory constructed by ICAO (independent component analysis) to design a typical subject observation item measurement table and construct a competence assessment matrix, so as to provide a solution for optimizing the competence rating of each stage of initial flight training.

Description

A Method for Rating Flight Training Quality Based on Observable Behaviors

technical field

The invention relates to the technical field of flight training rating, in particular to a method for rating flight training quality based on observable behavior.

Background technique

Flight training is the basis for ensuring flight safety and realizing the high-quality development of civil aviation. A scientific and standardized training quality evaluation system is a key link to control training quality and improve training efficiency. In order to mitigate the risks encountered by flight crew members in flight, especially in response to the increasingly complex civil aviation system, the International Civil Aviation Organization (hereinafter referred to as ICAO) proposes that flight training and assessment should focus on knowledge application (KNO), program application and compliance with regulations (APK), Automated Path Management (FPA), Manual Path Management (FPM), Communication (COM), Leadership and Teamwork (LTW), Situational Awareness and Information Management (SAW), Workload Management (WLM), Problem-solving and decision-making (PSD) and other nine core competency concepts of pilots, and in the "Global Aviation Safety Plan" (GASP) emphasized the use of competency-based training and assessment (Competency Based Training System, CBTA) as a continuous improvement of the global Aviation Safety Strategy.

In addition, pilot skills life cycle management (PLM) is the development direction of pilot skill management paradigm in the global civil aviation industry, and its premise is to establish a set of ability evaluation index system that runs through the life cycle of pilot skills. For this reason, ICAO and IATA established The nine competency index systems are defined, and the inspection dimensions of each competency are defined. Therefore, in order to promote the implementation of the PLM management paradigm in the whole industry, it is necessary to improve the pilot training quality evaluation method according to the core competency evaluation index system built by ICAO from the initial flight training stage.

The current temporary flight training evaluation method is implemented with subjects as the training carrier. The examiners mainly give the conclusions of "pass" and "fail" through the observation of the flight process of the students, which cannot realize the various competency levels of the students. and its distribution for scientific evaluation. Therefore, the existing training quality assessment methods urgently need to be reformed and optimized, from the old "fixed subject" training quality assessment to the core competency training assessment (CBTA) based on observable behavior. Based on the core competency evaluation index system, a flight training quality rating method based on observable behavior is established.

Contents of the invention

In order to solve the problems existing in the prior art, the present invention provides a method of flight training quality rating based on observable behavior, based on the traditional flight training performance appraisal operation mode and the CBTA theory constructed by ICAO, a typical subject observation item measurement table is designed And build the evaluation matrix, and construct the competency rating model based on OB, obtain the rating standard, provide a solution for solving the optimization of the competency rating in each stage of the initial flight training, and solve the problems mentioned in the above-mentioned background technology.

In order to achieve the above object, the present invention provides the following technical solutions: a method of flight training quality rating based on observable behavior, comprising the following steps:

S1. Construct a training evaluation worksheet, which includes typical subjects to be investigated, observation items for each subject, scoring standards and scoring;

S2. Construct observation item-OB correlation matrix;

S3. The examiner obtains the measurement vector A according to the training evaluation worksheet;

S4. Construct the competency evaluation matrix Y;

表征得到可观察行为 OB的展现频度f_ofn和展现数量f_mny的评估模型；S5. According to the competency evaluation matrix Y, introduce the relative norm

Characterize and obtain the evaluation model of the display frequency f _ofn and the display quantity f _mny of the observable behavior OB;

和γ₁、γ₂、γ₃，然后根据所划分的胜任力评级标准对学员进行评级。S6. According to the evaluation model, the classification threshold is obtained by solving the optimization problem

And γ ₁ , γ ₂ , γ ₃ , and then grade the trainees according to the divided competency rating standards.

Preferably, in step S2, a correlation is established between any observation item i and the competency observable behavior index OBj, and is represented by the observation item-OB correlation matrix B,

Among them, b _ij represents the associated attribute between the i-th observation item and the j-th OB, i=1,2,...,m; j=1,2,...,n; when b _ij =1, it means the There is a mapping relationship between the i observation item and the jth OB, otherwise it is 0.

Preferably, in step S3, according to the observation item scoring standard of the training evaluation worksheet, the performance of the observation items when the students complete each subject is checked and scored accordingly, and the measurement results of the observation items are obtained to form the observation vector A,

A=(a _i ) _m×1 ＝(a ₁ ,a ₂ ,...,a _m ) ^T , i=1,2,...,m,

为该观测项的满分值，当所有观测项均取满分时可得到观测向量

Among them, a _i is the score of the i-th observation item, and its maximum value

is the full score value of the observation item, when all observation items take full marks, the observation vector can be obtained

Preferably, the scoring criteria include checked items and key inspection items; checked items: score 0 or 1, used to evaluate whether it is completed, 1 if completed, 0 if not completed; key inspection items: score 3, 2 , 1, 0, respectively corresponding to the completion criteria of specific observation items.

Preferably, in step S4, using the observation vector A and the observation item-OB correlation matrix, the elements in the observation vector A are multiplied by the corresponding rows in the correlation matrix to obtain the competency evaluation matrix Y,

Among them, a _i b _ij represents the contribution level of the i-th observation item to OBj.

Preferably, in step S5, the evaluation model is expressed as follows:

时，根据评估模型可得到OB展现数量f_mny、频度f_ofn的最大值为：When all observations are full marks, the observation vector

, according to the evaluation model, the maximum value of OB display quantity f _mny and frequency f _ofn can be obtained as:

In view of the fact that there are differences in the observation items and completion standard settings of the training assessment worksheets involved in different training institutions and different training courses, and to facilitate the unification of the competency rating standards, the relative norms are introduced as follows, where

和γ₁、γ₂、γ₃，具体如下：Preferably, the optimal solution of the classification threshold is obtained by solving the optimization problem

and γ ₁ , γ ₂ , γ ₃ , as follows:

的分级，P_mny为

划分的等级，

和γ₁、γ₂、γ₃为

的分级阈值，P_OB为根据P_ofn和P_mny基于VENN准则得出的胜任力评级。Among them, P _exa is the examiner's rating of the students, P _exa = {excellent, good, medium, poor} = {4, 3, 2, 1}; P _ofn is

The classification of P _mny is

grades of division,

and γ ₁ , γ ₂ , γ ₃ are

P _OB is the competency rating based on P _ofn and P _mny based on the VENN criteria.

The beneficial effects of the present invention are: the present invention is based on the traditional flight training performance assessment operation mode and the CBTA theory constructed by ICAO, designs typical subject observation item measurement tables and builds competency evaluation matrix, and optimizes the competency rating for each stage of initial flight training A solution is provided, which can be used as a quantitative performance evaluation tool in the practical skills assessment of each stage of initial flight training, seamlessly connected with the current initial flight training assessment mode based on work orders, and the present invention introduces competency-based The training and assessment (CBTA) index system provides basic data support for airlines to implement pilot skill life cycle management (PLM).

Description of drawings

Fig. 1 is a schematic diagram of the framework of the method of the present invention;

Fig. 2 is a schematic diagram of the steps of the method of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Example 1

Competency-based training and assessment (CBTA) emphasizes performance and measurement standards, and conducts training for specific performance standards. In order to ensure the greatest degree of consistency and objectivity in the assessment of trainees' competencies, it is necessary to demonstrate OB's skills from trainees. Competency is assessed in three dimensions: quantity (ie, HOW MANY), frequency (ie, HOW OFTEN), and the threat and error management outcomes that are particularly relevant to competency. Among them, the dimension of "How many" indicates whether the trainee possesses the ability, "How often" indicates the robustness used to judge the ability of the trainee, and "Outcome of TEM" indicates the relationship between the ability and the ability of the trainee. Evidence related to effectiveness as individual and team countermeasures against threats and errors, while the competency assessment (How well) is a combination of the number of demonstrated OBs and their frequency, and the management of threats and errors that are particularly relevant to the assessed competencies combination of corresponding results.

The traditional flight training evaluation method mainly relies on the subjective judgment of the examiners on the quality level of the pilot students' completion of various subjects, lacks quantitative analysis, especially cannot make a more comprehensive quantitative evaluation of the various competency levels and structures of the students, and cannot satisfy The need for full lifecycle management of pilot skills.

The CBTA evaluation system currently constructed by ICAO, such as the EBT manual and the PLM system, is more aimed at mature pilots of airline companies, and is not suitable for the training characteristics based on fixed subjects in the initial flight training stage; the VENN evaluation criteria proposed by ICAO are not There is no quantitative standard for the grading of the two dimensions of OB performance "How many" and "How often" involved in the ability rating, and it is completely dependent on the experience and judgment of the examiner, so it is still subjective. Therefore, for each stage of initial flight training, how to reasonably define the observable behavior (OB) of each competency, and determine the "quantity (How many)" and "frequency (How often)" of the relevant OBs The grading standard is a difficult problem to be solved, which is related to the operability of the initial flight training CBTA.

Therefore, it is necessary to establish an operable competency assessment optimization method based on the ability training requirements at each stage of the initial training, including the observation items and scoring standards of OB, the relationship between observation items and OB, and the "quantity" based on OB (Howmany )”, “How often” data competency rating standards, etc.

The present invention designs a training performance evaluation optimization scheme based on the VENN criterion based on the traditional flight training performance evaluation worksheet mode and the VENN criterion, which can be applied to the hierarchical and quantitative evaluation of the competency performance at each stage of the initial flight training. The specific framework is as follows: As shown in Figure 1, it includes the following steps:

Step 1: Build a training evaluation worksheet

The worksheet includes typical subjects to be investigated, observation items for each subject, scoring criteria and scoring.

Design the initial training evaluation worksheet, and design a unified evaluation worksheet for each inspection item. By standardizing the observation items and completion standards of each subject, a unified quantitative measurement of the mastery of trainees' skills can be achieved. During the assessment process, the flight examiner will score the observation items of each subject according to the completion standard of the training evaluation worksheet, and then the observation vector score of the observation items can be obtained.

According to the flight training practice test standards promulgated by the bureau and the teaching syllabus requirements of each training institution, flight experts design a unified evaluation worksheet for each inspection item, standardize the typical subjects to be investigated, and the observation items and completion of each subject. standard, the examiner checks the completion of the students against the evaluation worksheet to form an observation vector A=(a _i ) _m×1 =(a ₁ ,a ₂ ,…,a _m ) ^T , i=1,2, ..., m, the design of the evaluation worksheet is shown in Table 1.

Table 1 Design of initial flight training evaluation worksheet

Step 2: Construct observation items—OB correlation matrix

Each observation item corresponds to the observable behavior OB of a certain competency. Using the Delphi survey method to solicit the opinions of flight experts, a correlation matrix can be constructed between any observation item i and the observable behavior index OBj of the competency.

The columns of the correlation matrix are observation items, and the rows are the evaluation dimensions of each competency, such as OB1, OB2, etc. The process of obtaining the correlation matrix is shown in Table 2 below.

Table 2 Observation items - OB table design

From Table 2, the correlation matrix can be obtained as B,

Step 3: Obtain the measurement vector A against the training evaluation worksheet

According to the completion standards of the observation items in the evaluation worksheet, the examiners will check and score the performance of the observation items when the students complete each subject, obtain the measurement results of the observation items, and form the observation vector A.

Step 4: Construct the competency assessment matrix Y

According to the VENN evaluation criteria, the competency level of the trainees can be measured by counting the number and frequency of OBs exhibited in the assessment, using the observation vector A and the observation item-OB correlation matrix, the competency evaluation matrix Y can be constructed; and the introduction Based on the concept of norm, the OB presentation frequency (HOW OFTEN) and quantity (HOW MANY) evaluation model based on the competency evaluation matrix can be obtained.

Multiply the elements in the observation vector A with the corresponding rows in the correlation matrix B, and the table shown in Table 3 can be obtained.

Table 3 Table design of the competency assessment matrix

Observation item OB1 ob2 … OB Observation item 1 a₁*b₁₁ a₁*b₁₂ … a₁*b_1n Observation item 2 a₂*b₂₁ a₂*b₂₂ … a₂*b_2n … … … … … … … … … … … … … … … Observation item m a_m*b_m1 a_m*b_m2 … a_m*b_mn

It can be concluded from Table 3 that the competency evaluation matrix Y can be obtained,

Among them, a _i b _ij represents the contribution level of the i-th observation item to OBj.

Step 5: Design Competency Rating Criteria

Refer to the daily teacher's rating of the students' manipulation ability as a reference, construct a competency evaluation model, and construct a rating function by converting it into an optimization problem, and then get the performance rating standard divided by competency.

Example 2

The present invention designs a kind of method based on the flight training quality rating of observable behavior based on the work list mode and VENN criterion of traditional flight training performance appraisal, as shown in Figure 2, comprises the following steps:

Step (1): Design of training and evaluation worksheets based on checks of typical subject observation items

A unified evaluation worksheet is designed for each inspection item. By standardizing the observation items and completion standards of each subject, a unified quantitative measurement of the mastery of the students' skills can be achieved. Design the initial training evaluation worksheet. During the assessment process, the flight examiner will score the observation items of the trainees in each subject according to the completion standard of the training evaluation worksheet.

Scoring criteria include two types, (1) check the item, the score is 0 or 1, which is used to evaluate whether it is completed, if it is completed, it is 1, if it is not completed, it is 0; (2) the key inspection item, the score is 3, 2, 1, 0, respectively corresponding to the completion criteria of specific observation items.

Step (2): Construct observation item-OB correlation matrix

Each observation item corresponds to the observable behavior OB of a certain competency. Using the Delphi survey method to solicit the opinions of flight experts can establish a relationship between any observation item i and the observable behavior index OBj of the competency, and use the correlation Matrix B says,

Among them, b _ij represents the associated attribute between the i-th observation item and the j-th OB, i=1,2,...,m; j=1,2,...,n. When b _ij =1, it means that there is a mapping relationship between the i-th observation item and the j-th OB, otherwise it takes 0.

Step (3): Form the measurement vector A based on the examiner's check

为该观测项的满分值，当所有观测项均取满分时可得到观测向量

According to the flight training practice test standards promulgated by the bureau and the teaching syllabus requirements of each training institution, the observation items and scoring standards of each training subject can be analyzed, and the examiners will check the completion of the students against the evaluation worksheet to get the score. A=(a _i ) _m×1 =(a ₁ ,a ₂ ,…,a _m ) ^T , i=1,2,…,m, where a _i is the score of the i-th observation item , its maximum value

is the full score value of the observation item, when all observation items take full marks, the observation vector can be obtained

表征得到可观察行为OB的展现频度f_ofn和展现数量f_mny的评估模型。Step (4): Construct the competency evaluation matrix Y, and introduce the relative norm according to the competency evaluation matrix Y

The evaluation model of the display frequency f _ofn and the display quantity f _mny of the observable behavior OB is obtained by characterization.

According to the VENN criterion, the competency level of trainees can be measured by counting the number and frequency of OBs exhibited in the assessment. Using the observation vector A and the observation item-OB correlation matrix B, the competency evaluation matrix can be constructed as follows:

where a _i b _ij represents the contribution level of the i-th observation item to OBj, using the attribute of the norm of the vector/matrix to measure the length or size of the vector (or matrix) space, the norm of the Y matrix can be used to characterize the The frequency of presentation f _ofn and the number of presentations f _mny of the observation behavior (OB).

When the OB display frequency is higher than 25% of the maximum value, it is considered that the OB is displayed, otherwise it is not displayed. Calculate the norm of the evaluation matrix to obtain the evaluation model based on the OB display quantity (f _mny ) and frequency (f _ofn ) of the competency evaluation matrix. The evaluation model is as follows:

时，根据评估矩阵可得到OB展现数量(f_mmy)、频度(f_ofn)的最大值为：When all observation items take full marks, that is, the observation vector

, according to the evaluation matrix, the maximum value of OB display quantity (f _mmy ) and frequency (f _ofn ) can be obtained as:

In view of the fact that different training institutions and different training courses involve training assessment worksheets with differences in observation items and completion standard settings, and to facilitate the unification of competency rating standards, the present invention introduces relative norms as follows, wherein

和γ₁、γ₂、γ₃，然后基于胜任力进行评级，得到根据胜任力所划分的评级标准用于评级优化。Step (5): According to the evaluation model, the optimal solution of the threshold is obtained by solving the optimization problem

And γ ₁ , γ ₂ , γ ₃ , and then perform rating based on competency, and obtain the rating criteria divided by competency for rating optimization.

和γ₁、γ₂、γ₃，即可基于胜任力对绩效进行评级。Taking the examiner's rating of the students' manipulative ability as the reference sample data, through the sample parameters, it can be converted to solve the optimization problem and obtain the optimal solution of the threshold

And γ ₁ , γ ₂ , γ ₃ , the performance can be rated based on competency.

P _OB =min(P _ofn ,P _mny ) (11)

的分级，P_mny为

划分的等级，

和γ₁、γ₂、γ₃为OB

的分级阈值，P_OB为根据P_ofn和P_mny，并基于 VENN准则得出的胜任力评级。

和γ₁、γ₂、γ₃可通过求解(8)-(11)式构成的优化问题确定。Wherein, P _exa is the examiner's grade rating for the students, and P _exa = {excellent, good, medium, poor} = {4, 3, 2, 1}. P _ofn is

The classification of P _mny is

grades of division,

and γ ₁ , γ ₂ , γ ₃ are OB

P _OB is the competency rating based on P _ofn and P _mny and based on the VENN criteria.

and γ ₁ , γ ₂ , γ ₃ can be determined by solving the optimization problem constituted by (8)-(11).

The method of the present invention is applicable to the competency assessment of each stage. In view of the different stages of the initial flight training, the requirements for the pilot are different, and there are differences in the subject observation items and scoring standards. The ability of the pilot is cultivated step by step, so the next step can be aimed at other courses Using this method, experts set up evaluation worksheets, evaluation matrices, etc. according to the requirements and characteristics of the course, and establish a performance evaluation model according to the process.

Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments, or perform equivalent replacements for some of the technical features. Within the spirit and principles of the present invention, any modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the present invention.

Claims (7)

1. A flight training quality rating method based on observable behavior is characterized by comprising the following steps:

s1, constructing a training and evaluating work sheet, wherein the work sheet comprises typical subjects to be investigated, observation items of each subject, a scoring standard and a score;

s2, constructing an observation item-OB incidence matrix;

s3, obtaining a measurement vector A by contrasting the training evaluation worksheet;

s4, constructing a competence evaluation matrix Y;

s5, evaluating the matrix Y according to competence, and introducing relative norm

Frequency f of representation of observable behavior OB _ofn And the number f of displays _mny The evaluation model of (2);

s6, obtaining the optimal solution of the threshold value by solving the optimization problem according to the evaluation model

And gamma ₁ 、γ ₂ 、γ ₃ And then carrying out rating based on the competence, and obtaining rating standards divided according to the competence for rating optimization.

2. The method for flight training quality rating based on observable behavior of claim 1, wherein: in step S2, an association is established between any observation item i and the observable behavior index of competency OBj, represented by an observation item-OB association matrix B,

wherein, b _ij Represents the associated attribute of the ith observation item and the jth OB, i =1, 2.·, m; j =1,2,. Ang, n; b _ij If =1, it means that the ith observation item has a mapping relationship with the jth OB, otherwise, it takes 0.

3. The method for observable behavior-based flight training quality rating of claim 1, wherein: in step S3, according to the evaluation standard of the observation item of the training evaluation worksheet, corresponding checking and scoring is carried out on the observation item performance when the trainees finish each subject to obtain the result of measuring the observation item to form an observation vector A,

A＝(a _i ) _m×1 ＝(a ₁ ，a ₂ ，，a _m ) ^T ，i＝1，2，...，m，

wherein, a _i The score of the ith observation item is the maximum value

For the full score value of the observation item, when all the observation items are full scores, the observation vector can be obtained

4. The method for observable behavior-based flight training quality rating of claim 3, wherein: the scoring standard comprises checking items and key investigation items; item selection is performed: the score is 0 or 1 and is used for evaluating whether the completion is 1 or not and the non-completion is 0; the key research items are as follows: the scores are 3,2,1 and 0, which respectively correspond to the completion standard of the specific observation items.

5. The method for flight training quality rating based on observable behavior of claim 1, wherein: in step S4, elements in the observation vector A are multiplied with corresponding rows in the incidence matrix respectively by using the observation vector A and the observation item-OB incidence matrix to construct an evaluation matrix Y of competence,

wherein, a _i b _ij Representing the contribution level of the ith observation to OBj.

6. The method for flight training quality rating based on observable behavior of claim 1, wherein: in step S5, the evaluation model is expressed as follows:

when all observation items are full, namely the observation vector

Then, the OB number f is obtained according to the evaluation model _mny Frequency f _ofn The maximum value of (d) is:

aiming at the condition that training and examination worksheets related to different training institutions and different training courses have differences in the arrangement of observation items and completion standards and are convenient for unifying competency rating standards, relative norms are introduced as follows, wherein

7. The method for observable behavior-based flight training quality rating of claim 1, wherein: solving the optimization problem to obtain a threshold optimal solution

And gamma ₁ 、γ ₂ 、γ ₃ The method comprises the following steps:

P _OB ＝min(P _ofn ，P _mny )

γ ₁ ≤γ ₂ ≤γ ₃ ；

wherein, P _exa Rating of the examiner to the student, P _exa = excellent, good, medium, poor } = {4,3,2,1}; p _ofn Is composed of

Grade of division, P _mny Is composed of

Divided or the likeThe number of stages is such that,

and gamma ₁ 、γ ₂ 、γ ₃ Is composed of

Classification threshold of (P) _OB Is according to P _ofn And P _mny Competency ratings derived based on the VENN criteria.

Images