CN114397876A - Method and apparatus, electronic device, storage medium for evaluating domain controller performance - Google Patents

Abstract

The application relates to the technical field of domain controller testing, and discloses a method for evaluating the performance of a domain controller, which comprises the following steps: obtaining M.N verification results of the detection unit; the N detection units verify the planning unit according to the planning results of the M test samples to obtain M.N verification results; m is an integer greater than 1; determining the credible reference values of the M test samples according to the M & N verification results and the initial parameters; evaluating the performance of the domain controller according to the M credible reference values and the initial parameters; and outputting an evaluation result. In this way, the correct verification result that the detection unit should obtain can be determined due to the trusted reference value. Therefore, the performance of the domain controller is evaluated according to the credible reference value, whether the verification result is correct or not can be determined, and the accuracy of the obtained performance evaluation result of the domain controller is improved. The application also discloses a device, an electronic device and a storage medium for evaluating the performance of the domain controller.

Description

Method and apparatus, electronic device, storage medium for evaluating domain controller performance

technical field

The present application relates to the technical field of domain controller testing, for example, to a method and apparatus for evaluating the performance of a domain controller, an electronic device, and a storage medium.

Background technique

At present, with the rapid development of 5G technology, the planning unit of the domain controller is usually used for network planning to obtain the planning result. The detection unit judges whether the logic of the network link in the planning result is contradictory, whether there is a missing configuration, and whether there is a timing loop. and other problems, so as to obtain the verification results of the detection unit on the planning results. However, the performance of the domain controller is directly evaluated through the verification result. Since it is unknown whether the verification result is correct, the performance evaluation result of the domain controller cannot be accurately obtained.

In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art: directly evaluating the performance of the domain controller through the verification result, the obtained performance evaluation result of the domain controller is less accurate.

SUMMARY OF THE INVENTION

In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be an extensive review, nor to identify key/critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

Embodiments of the present disclosure provide a method and apparatus, an electronic device, and a storage medium for evaluating the performance of a domain controller, so as to improve the accuracy of the obtained performance evaluation result of the domain controller.

In some embodiments, a method for evaluating the performance of a domain controller, the domain controller comprising a planning unit and N detection units, where N is an integer greater than 1, the method comprising: obtaining M·N detection units verification results; wherein, the M·N verification results are obtained by verifying the planning unit according to the planning results of the M test samples by the N detection units; M is an integer greater than 1; according to the M·N verification results and the initial parameters Determine the credible benchmark values of the M test samples; evaluate the performance of the domain controller according to the M credible benchmark values and initial parameters; and output the evaluation results.

In some embodiments, the apparatus for assessing the performance of a domain controller includes a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the above-described method for assessing Methods of Domain Controller Performance.

In some embodiments, the electronic device includes the above-mentioned apparatus for evaluating the performance of a domain controller.

In some embodiments, the storage medium stores program instructions, and when the program instructions are executed, the above-mentioned method for evaluating the performance of a domain controller is executed.

The method and device, electronic device, and storage medium for evaluating the performance of a domain controller provided by the embodiments of the present disclosure can achieve the following technical effects: by obtaining M·N verification results of the detection unit; according to the M·N verification results and The initial parameters determine the credible reference values of the M test samples; evaluate the performance of the domain controller according to the M credible reference values and the initial parameters; and output the evaluation results. In this way, due to the trusted reference value, it is possible to determine the correct verification result that the detection unit should obtain. Therefore, by evaluating the performance of the domain controller according to the trusted reference value, it can be determined whether the verification result is correct, thereby improving the accuracy of the obtained performance evaluation result of the domain controller.

The foregoing general description and the following description are exemplary and explanatory only and are not intended to limit the application.

Description of drawings

One or more embodiments are exemplified by the accompanying drawings, which are not intended to limit the embodiments, and elements with the same reference numerals in the drawings are shown as similar elements, The drawings do not constitute a limitation of scale, and in which:

FIG. 1 is a schematic diagram of a method for evaluating the performance of a domain controller provided by an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of another method for evaluating the performance of a domain controller provided by an embodiment of the present disclosure;

3 is a schematic structural diagram of an evaluation device provided by an embodiment of the present disclosure;

4 is a schematic diagram of a verification result and initial parameters provided by an embodiment of the present disclosure;

5 is a schematic diagram of a verification result and an evaluation result when a test round provided by an embodiment of the present disclosure is 5;

6 is a schematic diagram of a verification result and an evaluation result when a test round provided by an embodiment of the present disclosure is 100;

7 is a schematic diagram of a verification result and an evaluation result when a test round is 1000 according to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of an apparatus for evaluating the performance of a domain controller provided by an embodiment of the present disclosure.

Reference number:

100: Test sample set storage module; 101: Network management/controller; 102: Synchronous network; 103: Planning unit of SD-SCN; 104: Detection unit of SD-SCN; 105: Evaluation Unit; 106: Software-defined synchronization domain controller.

Detailed ways

In order to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are for reference only and are not intended to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, numerous details are provided to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawings.

The terms "comprising" and "having" and any variations thereof in the description and claims of embodiments of the present disclosure and the above-mentioned drawings are intended to cover non-exclusive inclusions.

Unless stated otherwise, the term "plurality" means two or more.

In the embodiment of the present disclosure, the character "/" indicates that the preceding and following objects are in an "or" relationship. For example, A/B means: A or B.

The term "correspondence" may refer to an association relationship or a binding relationship, and the correspondence between A and B refers to an association relationship or a binding relationship between A and B.

The present application may be applied to systems or software for evaluating domain controller performance. Usually, the software-defined synchronization domain controller has two parts: the synchronization network planning unit and the synchronization network detection unit. The synchronization network planning unit can automatically plan the synchronization link path and synchronization port configuration priority according to the network topology connection and resource information. The synchronization network detection unit can perform integrity detection on the actual operation of the synchronization network or the synchronization operation status of the offline planning, including whether there is a missing configuration, whether there is a timing loop, whether there is a synchronization hop number exceeding the limit, etc. Evaluate the performance of the domain controller, that is, evaluate the planning unit or detection unit of the software-defined synchronization domain controller.

Meanwhile, the electronic devices involved in the embodiments of the present invention may include, but are not limited to, mobile phones, tablet computers, personal computers, palmtop computers, servers, and the like.

With reference to FIG. 1 , an embodiment of the present disclosure provides a method for evaluating the performance of a domain controller. The domain controller includes a planning unit and N detection units, where N is an integer greater than 1, and is used for evaluating the performance of the domain controller. Methods include:

Step S101, the electronic device obtains M·N verification results of the detection unit; wherein, the M·N verification results are obtained by the N detection units verifying the planning unit according to the planning results of the M test samples; M is an integer greater than 1 ;

Step S102, the electronic device determines the credible reference values of the M test samples according to the M·N verification results and the initial parameters;

Step S103, the electronic device evaluates the performance of the domain controller according to the M trusted reference values and initial parameters;

Step S104, the electronic device outputs the evaluation result.

By adopting the method for evaluating the performance of a domain controller provided by the embodiments of the present disclosure, M·N verification results of the detection unit are obtained; the credible reference values of M test samples are determined according to the M·N verification results; Trusted baseline values and initial parameters evaluate domain controller performance; output evaluation results. In this way, due to the trusted reference value, it is possible to determine the correct verification result that the detection unit should obtain. Therefore, by evaluating the performance of the domain controller according to the trusted reference value, it can be determined whether the verification result is correct, thereby improving the accuracy of the obtained performance evaluation result of the domain controller.

Optionally, M·N means M times N.

In some embodiments, the electronic device receives the verification result sent by each detection unit of the domain controller; or, the electronic device calls a preset verification result database, and obtains the verification result from the preset verification result database; the preset verification result There are multiple validation results stored in the database.

Optionally, the verification result is 0 or 1; 0 is used to indicate that the verification of the planning result of the detection unit to the planning unit is failed; 1 is used to indicate that the verification of the planning result of the detection unit to the planning unit is passed.

获得测试样本的可信基准值；其中，R_j'(p_k,s_j)是第j个测试样本的可信基准值；w'_i是第i个检测单元对应的初始权重；R(D_i|p_k,s_j)是第i个检测单元对规划单元按照第j个测试样本的规划结果的验证结果；“·”表示乘号。这样，通过初始参数和各检测单元的验证结果能够自动确定各测试样本的可信基准值，提高了获得的域控制器的性能评估结果的效率。Optionally, the initial parameters include initial weights corresponding to the N detection units respectively; determining the credible reference value of the test sample according to the verification result and the initial parameters, including: calculating

Obtain the credible reference value of the test sample; wherein, R _j '(p _k ,s _j ) is the credible reference value of the jth test sample; w' _i is the initial weight corresponding to the ith detection unit; R(D _i |p _k ,s _j ) is the verification result of the i-th detection unit to the planning unit according to the planning result of the j-th test sample; “·” represents a multiplication sign. In this way, the credible reference value of each test sample can be automatically determined through the initial parameters and the verification result of each detection unit, which improves the efficiency of the obtained performance evaluation result of the domain controller.

In some embodiments, R(D _i |p _k ,s _j ) is the verification result of the ith detection unit on the planning result of the planning unit according to the jth test sample. That is, when the test sample is s _j , the planning unit p _k performs planning according to the test sample s _j , and obtains the planning result p _k ,s _j ; the verification result of the planning result p _k ,s _j by the detection unit D _i is R(D _i |p _k ,s _j ).

In some embodiments, R _j '(p _k ,s _j ) is the trusted reference value for the jth test sample. That is, the credible reference value of the _jth test sample in the planning unit pk.

获得测试样本1对应的各检测单元的加权结果，即：0.25×1+0.25×0+0.25×1+0.25×1＝0.75；由于加权结果0.75大于0.5，因此测试样本1的可信基准值为1。In some embodiments, the domain controller includes 4 detection units. The planning result of the planning unit planning according to the test sample 1 is determined as the planning result of the test sample 1; the initial weight of the acquisition detection unit D1 is 0.25, and the verification result of the detection unit D1 on the planning result of the test sample 1 is 0; the acquisition of the detection unit D2 The initial weight is 0.25, and the verification result of the planning result of the test sample 1 by the detection unit D2 is 1; the initial weight of the acquisition detection unit D3 is 0.25, and the verification result of the detection unit D3 on the planning result of the test sample 1 is 1; The initial weight is 0.25, and the verification result of the planning result of test sample 1 by the detection unit D4 is 1. calculate

Obtain the weighted result of each detection unit corresponding to test sample 1, namely: 0.25×1+0.25×0+0.25×1+0.25×1=0.75; since the weighted result 0.75 is greater than 0.5, the credible reference value of test sample 1 is 1.

Optionally, evaluating the performance of the domain controller according to the M credible benchmark values and initial parameters includes: determining the type of each verification result according to the verification result and the credible benchmark value corresponding to the test sample; the types include true negative and true positive. and false positives; count the number of true negatives, true positives and false positives in the verification results corresponding to each detection unit; determine the evaluation value and The corresponding weight of each detection unit. In this way, by determining the type of the verification result and counting the number of types of verification results corresponding to each detection unit, the degree of accuracy of the verification result corresponding to each detection unit can be determined. Therefore, the performance of each detection unit of the domain controller can be quantitatively evaluated, which is beneficial for the user to improve each detection unit according to the evaluation value of each detection unit. At the same time, it is also convenient for users to select an appropriate detection unit for network planning verification.

Optionally, the credible benchmark value is 0 or 1; the type of each verification result is determined respectively according to the verification result and credible benchmark value corresponding to the test sample, including: the credible benchmark value and the verification result corresponding to the test sample are both 1 In the case of 0, the type of the verification result is determined to be a true positive; when the credible reference value corresponding to the test sample and the verification result are both 0, the type of the verification result is determined to be a true negative; When the reference value is 0 and the verification result is 1, the type of the verification result is determined to be a false positive.

获得检测单元准确率；计算

获得检测单元召回率；计算

获得检测单元得分；计算

获得检测单元评估值；计算

获得检测单元对应的权重；其中，precision_rate为检测单元准确率，TP(D_i)为第i个检测单元对应的验证结果中存在的真阳性数量；TN(D_i)为第i个检测单元对应的验证结果中存在的真阴性数量；FP(D_i)为第i个检测单元对应的验证结果中存在的假阳性数量；recall_rate为第i个检测单元召回率；f_score(D_i)为第i个检测单元得分；f_score(D_g)为第g个检测单元得分；P(D_i)为第i个检测单元评估值；P'(D_i)为第i个检测单元初始参数；iter为测试轮次；w_i为第i个检测单元对应的权重。Optionally, the initial parameters also include initial parameters of the detection unit; according to the number of true negatives, the number of true positives, the number of false positives, and the initial parameters, determine the evaluation value of the detection unit and the corresponding weight of the detection unit, including: calculating

Obtain the detection unit accuracy; calculate

Get the detection unit recall; calculate

Get detection unit score; calculate

Obtain the evaluation value of the detection unit; calculate

Obtain the weight corresponding to the detection unit; among them, precision_rate is the accuracy rate of the detection unit, TP(D _i ) is the number of true positives in the verification result corresponding to the i-th detection unit; TN(D _i ) is the corresponding to the i-th detection unit FP(D _i ) is the number of false positives in the verification result corresponding to the ith detection unit; recall_rate is the recall rate of the ith detection unit; f_score(D _i ) is the detection unit score; f_score(D _g ) is the g-th detection unit score; P(D _i ) is the evaluation value of the i-th detection unit; P'(D _i ) is the initial parameter of the i-th detection unit; iter is the test rounds; w _i is the weight corresponding to the i-th detection unit.

Optionally, the test round is greater than or equal to 1.

获得检测单元D3准确率为1；通过计算

获得检测单元D3召回率为1；通过计算

获得检测单元D3检测单元得分为1；通过计算

获得检测单元D3评估值为0.75。In some embodiments, the test round is 1; the initial parameter of the detection unit D3 is 0.5. The planning result of the planning unit planning according to the test sample 3 is determined as the planning result of the test sample 3; the planning result of the planning unit planning according to the test sample 4 is determined as the planning result of the test sample 4. Obtaining the credible benchmark value of test sample 3 is 1; acquiring the credible benchmark value of test sample 4 is 0; acquiring the verification result a of the planning result of test sample 3 by detection unit D3 is 1; acquiring the planning result of test sample 4 by detection unit D3 The verification result b of the result is 1; the verification result c of the acquired detection unit D4 for the planning result of the test sample 3 is 1; the verification result d of the acquisition detection unit D4 for the planned result of the test sample 4 is 0. If the verification result a corresponding to test sample 3 is 1 and the credible benchmark value is 1, then the type of verification result a is a true positive; if the verification result b corresponding to test sample 3 is 1 and the credible benchmark value is 1, then the verification result b is The type of verification result c is true positive; the verification result c corresponding to test sample 4 is 1 and the trusted benchmark value is 0, then the type of verification result c is false positive; the verification result d corresponding to test sample 4 is 0 and the trusted benchmark value is 0 0, then the type of verification result d is true negative. The number of true positives corresponding to the detection unit D3 is 2; the number of false positives corresponding to the detection unit D4 is 1; the number of true negatives corresponding to the detection unit D4 is 1. via caculation

Obtain the detection unit D3 with an accuracy of 1; by calculating

The recall rate of the detection unit D3 is obtained as 1; by calculating

Obtain the detection unit D3, the detection unit score is 1; by calculating

The evaluation value of detection unit D3 was obtained as 0.75.

获得域控制器的规划单元得分；计算

获得域控制器的规划单元评估值；其中，score(p_k)是规划单元得分；P(p_k)为规划单元评估值；P'(p_k)为规划单元初始参数；。这样，通过各验证结果确定测试样本的可信基准值，即，能够确定规划单元对测试样本的规划结果被验证通过或验证不通过。通过计算规划单元对测试样本的规划结果的验证通过情况，即能确定规划单元对应的规划成功情况。从而能够量化评估域控制器的规划单元的性能，有利于用户根据规划单元评估值对各规划单元进行改进。同时，也便于用户选择合适的规划单元进行网络规划。Optionally, the initial parameters further include initial parameters of the planning unit; evaluating the performance of the domain controller according to each credible reference value and the initial parameters further includes: calculating

Obtain planning unit scores for domain controllers; calculate

Obtain the planning unit evaluation value of the domain controller; wherein, score(p _k ) is the planning unit score; P(p _k ) is the planning unit evaluation value; P'(p _k ) is the planning unit initial parameter; In this way, the credible reference value of the test sample is determined by each verification result, that is, it can be determined whether the planning result of the planning unit on the test sample is verified or failed. By calculating the verification pass status of the planning result of the test sample by the planning unit, the planning success status corresponding to the planning unit can be determined. Therefore, the performance of the planning unit of the domain controller can be quantitatively evaluated, which is beneficial for the user to improve each planning unit according to the evaluation value of the planning unit. At the same time, it is also convenient for users to select an appropriate planning unit for network planning.

即：(1+0+1+1+0)/5＝3/5；获得域控制器的规划单元得分为3/5。通过计算

获得规划单元评估值为0.8。In some embodiments, the test round is 1; the initial parameter of the planning unit is 1; the credible benchmark values of 5 test samples are obtained, for example: the credible benchmark value of test sample 1 is 1; the credible benchmark value of test sample 2 is The value is 0; the credible benchmark value of test sample 3 is 1; the credible benchmark value of test sample 4 is 1; the credible benchmark value of test sample 5 is 0. via caculation

That is: (1+0+1+1+0)/5=3/5; the planning unit score for obtaining the domain controller is 3/5. via caculation

The obtained planning unit evaluation value is 0.8.

获得测试样本得分；计算

获得测试样本评估值；其中，score(s_j)为第j个测试样本得分；P(s_j)为第j个测试样本评估值；P'(s_j)为第j个测试样本初始参数。这样，能够对各测试样本进行量化评估，使得用户根据确定测试样本评估值确定各测试样本的难易程度。Optionally, the initial parameters further include initial parameters of the test sample; after evaluating the performance of the domain controller according to the M credible benchmark values and the initial parameters, the method further includes: calculating

Obtain test sample scores; calculate

Obtain the evaluation value of the test sample; wherein, score(s _j ) is the score of the j-th test sample; P(s _j ) is the evaluation value of the j-th test sample; P'(s _j ) is the initial parameter of the j-th test sample. In this way, each test sample can be quantitatively evaluated, so that the user can determine the difficulty level of each test sample according to the determination of the evaluation value of the test sample.

获得测试样本7得分0；计算

获得测试样本7评估值为10。In some embodiments, the test round is 1; the domain controller includes 4 detection units and 1 planning unit. The initial parameter of the test sample 7 is 20; the planning result of the planning unit planning according to the test sample 7 is determined as the planning result of the test sample 7; the initial weight of the acquired detection unit D1 is 0.25, and the verification result of the planning result of the test sample 7 is 1 The initial weight of the acquisition detection unit D2 is 0.25, and the verification result of the planning result of the test sample 7 is 1; the initial weight of the acquisition detection unit D3 is 0.25, and the verification result of the planning result of the test sample 7 is 1; The initial weight is 0.25, and the validation result for the planning result of test sample 7 is 1. calculate

get test sample 7 score 0; calculate

Obtain a test sample 7 evaluation value of 10.

In some embodiments, when the test round is 1, the initial parameters are obtained by searching a preset parameter database; the preset parameter database stores the initial parameters of the detection unit, the initial parameters of the planning unit, the initial parameters of the test sample and the The initial weights corresponding to the detection units respectively.

Optionally, after outputting the evaluation result, the method further includes: obtaining the test round; in the case that the test round reaches the preset round, displaying the evaluation result to the user; in the case that the test round does not reach the preset round , accumulate the test rounds; determine the evaluation result as the initial parameter; re-obtain M·N verification results of the detection unit; determine the credible benchmark values of M test samples according to the M·N verification results; Values and initial parameters to evaluate domain controller performance; output evaluation results. In this way, by iterating over the evaluation results, the domain controller performance can be more accurately evaluated.

With reference to FIG. 2 , an embodiment of the present disclosure provides another method for evaluating the performance of a domain controller. The domain controller includes a planning unit and N detection units, where N is an integer greater than 1, for evaluating the performance of the domain controller methods include:

Step S201, the electronic device obtains M·N verification results of the detection unit; wherein, the M·N verification results are obtained by the N detection units verifying the planning unit according to the planning results of the M test samples; M is an integer greater than 1 ; and then execute step S202.

Step S202, the electronic device determines the credible reference values of the M test samples according to the M·N verification results and the initial parameters; then step S203 is performed.

Step S203, the electronic device evaluates the performance of the domain controller according to the M trusted reference values and the initial parameters; then step S204 is performed.

Step S204, the electronic device outputs the evaluation result; then step S205 is executed.

Step S205, the electronic device acquires the test round; then step S206 is performed.

Step S206, the electronic device determines whether the test round has reached the preset round; in the case that the test round has reached the preset round, execute step S208; in the case that the test round has not reached the preset round, execute step S207 .

In step S207, the electronic device determines the evaluation result as an initial parameter, and then executes step S201.

Step S208, the electronic device displays the evaluation result to the user.

Using the method for evaluating the performance of a domain controller according to the embodiment of the present disclosure, by acquiring the test rounds, if the test rounds do not reach the preset rounds, M·N verification results of the detection unit are obtained; The trusted benchmark value and the initial parameters evaluate the performance of the domain controller; the evaluation result is output, and the evaluation result is determined as the initial parameter until the test round reaches the preset round. In this way, loop iterations are performed on the evaluation results to more accurately evaluate the performance of the domain controller.

Optionally, determining the evaluation result as the initial parameter includes: determining the weight corresponding to the detection unit as the initial weight corresponding to the detection unit; determining the evaluation value of the detection unit as the initial parameter of the detection unit; determining the evaluation value of the planning unit as the planning unit Initial parameters; determine the test sample evaluation value as the test sample initial parameters.

Optionally, displaying the evaluation result to the user includes: pushing the evaluation result to a preset client.

Optionally, displaying the evaluation result to the user includes: sending the evaluation result to a preset display screen, and triggering the display screen to display the evaluation result.

In some embodiments, FIG. 3 is a schematic structural diagram of an evaluation device. As shown in FIG. 3 , the test sample set storage module 100 is connected to the network management or controller 101; the network management or controller 101 is connected to the synchronization network 102 through the network management interface or the southbound interface; the synchronization network 102 is connected to the software-defined synchronization network domain through the southbound interface The planning unit 103 of the controller, the planning unit 103 of the SDSC domain controller is connected to the detection unit 104 of the SDSC domain controller through the northbound interface; the SDSC domain controller 106 is connected to the evaluation unit 105 . Among them, the test sample set storage module stores multiple test samples, the network management or the controller calls the test samples from the test sample set storage module, and loads the test samples as the network management configuration file or the SDN controller configuration file; the network management or the SDN controller configuration file; The controller maps the configuration information corresponding to the network management configuration file or SDN controller configuration file to the synchronization network through the network management interface or southbound interface; the planning unit obtains configuration and operation information from the synchronization network through the southbound interface, and performs network planning to obtain the plan Results: The planning unit sends the planning results to a plurality of detection units, and the detection units verify the planning results to obtain the verification results. The SDS domain controller sends the verification results to the assessment unit. When the evaluation unit runs, it executes the method used to evaluate the performance of the domain controller.

In some embodiments, the evaluation unit 105 is connected to the test sample set storage module 100, and the evaluation unit marks the test samples in the test sample set storage module with corresponding test sample evaluation values.

In some embodiments, the test samples include synchronization device types, bearer device types, quantity and scale, synchronization time/clock source injection point locations, network topology connections, network planning strategies, and the like. Optionally, the synchronization device type includes a time server, a frequency synchronization source, a bearer network synchronization unit, a time allocation unit, and the like. The network management/controller is the network management or controller of the synchronization device/bearer device, and is used to map the configuration information of the test sample to the synchronization network. A synchronization network is an actual physical network or an offline virtual network entity composed of synchronization time/clock sources, bearer devices, etc. The planning unit of the software-defined synchronization network domain controller is the controller unit specified in YD/T3700-2020 "Software-defined synchronization network technical requirements", and has the ability to automatically plan the synchronization network; the software-defined synchronization network domain controller-detection unit is It meets the controller unit specified in YD/T3700-2020 "Technical Requirements for Software-Defined Synchronous Networks", and has the ability to automatically detect synchronous networks.

In some embodiments, domain controller performance is assessed using 13 test samples, a planning unit of 1 domain controller, and a detection unit of 5 domain controllers. First, the planning results of the 13 test samples by the domain controller were obtained, and then the 13 planning results were verified by 5 detection units, and 65 verification results were obtained. Then, based on the 65 verification results and initial parameters, the trusted benchmark values for 13 test samples are determined. Finally, the performance of the controller is evaluated according to the credible reference value and the initial parameters, that is, the evaluation value of the detection unit, the corresponding weight of the detection unit, the evaluation value of the planning unit and the evaluation value of the test sample are obtained. Figure 4 is a schematic diagram of the verification result and initial parameters, as shown in Figure 4, num is the number of the test sample; result_d1 is the verification result of the detection unit 1; result_d2 is the verification result of the detection unit 2; result_d3 is the verification result of the detection unit 3 result_d4 is the verification result of detection unit 4; result_d5 is the verification result of detection unit 5; result_hat is the test sample credible benchmark value; sample_hard_score is the test sample evaluation value; f1_d1 is the detection unit 1 evaluation value; f1_d2 is the detection unit 2 evaluation value f1_d3 is the detection unit 3 evaluation value; f1_d4 is the detection unit 4 evaluation value; f1_d5 is the detection unit 5 evaluation value; weight_d1 is the weight of the detection unit 1; weight_d2 is the weight of the detection unit 2; weight_d3 is the weight of the detection unit 3; weight_d4 is the weight of the detection unit 4; weight_d5 is the weight of the detection unit 5. Wherein, the verification result of testing unit 1 to test sample 2 is 0, the verification result of testing unit 2 to test sample 2 is 0; the verification result of testing unit 3 to test sample 2 is 0; the verification result of testing unit 4 to test sample 2 is 0 The result is 1; the verification result of test sample 2 by detection unit 5 is 0; the initial credible benchmark value of test sample 2 is 0; the initial test sample evaluation value of test sample 2 is 35.736373; detection unit 1, detection unit 2, detection The initial detection unit evaluation values of unit 3, detection unit 4 and detection unit 5 are all 0.2; the initial weights of detection unit 1, detection unit 2, detection unit 3, detection unit 4 and detection unit 5 are all 0.2.

In some embodiments, after the electronic device outputs the evaluation result, it acquires the test round, determines whether the test round reaches the preset round, and re-evaluates the domain controller if the test round does not reach the preset round. performance. In the case of 5 test rounds, as shown in FIG. 5 , the verification result of the testing unit 1 to the test sample 5 is 1, the verification result of the testing unit 2 to the test sample 5 is 1; the detection unit 3 to the test sample 5 The verification result of test sample 5 is 1; the verification result of test unit 4 to test sample 5 is 0; the verification result of test unit 5 to test sample 5 is 1; the credible benchmark value of test sample 5 is 1.0; the test sample evaluation of test sample 5 The evaluation value of detection unit 1 is 0.727273; the evaluation value of detection unit 2 is 0.833333; the evaluation value of detection unit 3 is 0.8; the evaluation value of detection unit 4 is 0.5; the evaluation value of detection unit 5 is 0.8; the evaluation value of detection unit 1 is 0.184376 The weight of detection unit 2 is 0.226845; the weight of detection unit 3 is 0.217727; the weight of detection unit 4 is 0.174943; the weight of detection unit 5 is 0.196109. When the test round is 100 times, the verification results and evaluation results are shown in Figure 6. When the test round is 1000 times, the verification results and evaluation results are shown in Figure 7.

In some embodiments, the performance of the domain controller is comprehensively evaluated using different verification results. For example, the verification result set A is obtained, and there are multiple verification results A stored in the verification result set A; There are multiple test samples; the evaluation result A is obtained according to the verification result set A. The verification result set B is obtained, and there are multiple verification results B stored in the verification result set B; A test sample; the evaluation result B is obtained according to the verification result set B. According to the evaluation result A and the evaluation result B, the performance of the domain controller is comprehensively evaluated. The test samples stored in the test sample set A are not identical or completely different from the test samples stored in the test sample set B.

With reference to FIG. 8 , an embodiment of the present disclosure provides an apparatus for evaluating the performance of a domain controller, including a processor (processor) 800 and a memory (memory) 801 . Optionally, the apparatus may further include a communication interface (Communication Interface) 802 and a bus 803 . The processor 800 , the communication interface 802 , and the memory 801 can communicate with each other through the bus 803 . Communication interface 802 may be used for information transfer. The processor 800 may invoke logic instructions in the memory 801 to perform the method for evaluating the performance of a domain controller of the above-described embodiments.

By adopting the device for evaluating the performance of the domain controller provided by the embodiment of the present disclosure, M·N verification results of the detection unit are obtained; the credible reference values of the M test samples are determined according to the M·N verification results and the initial parameters; Evaluate the performance of the domain controller according to the M trusted benchmark values and initial parameters; output the evaluation result. In this way, due to the trusted reference value, it is possible to determine the correct verification result that the detection unit should obtain. Therefore, by evaluating the performance of the domain controller according to the trusted reference value, it can be determined whether the verification result is correct, thereby improving the accuracy of the obtained performance evaluation result of the domain controller.

In addition, the above-mentioned logic instructions in the memory 801 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

As a computer-readable storage medium, the memory 801 can be used to store software programs and computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 100 executes the program instructions/modules stored in the memory 801 to execute functional applications and data processing, that is, to implement the method for evaluating the performance of the domain controller in the above embodiments.

The memory 801 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. In addition, the memory 801 may include high-speed random access memory, and may also include non-volatile memory.

An embodiment of the present disclosure provides an electronic device, including the above-mentioned apparatus for evaluating the performance of a domain controller.

Optionally, the electronic device includes a computer or a server or the like.

By using the electronic device provided by the embodiment of the present disclosure, M·N verification results of the detection unit are obtained; the trusted reference values of M test samples are determined according to the M·N verification results and the initial parameters; according to the M trusted reference values and initial parameters to evaluate domain controller performance; output evaluation results. In this way, due to the trusted reference value, it is possible to determine the correct verification result that the detection unit should obtain. Therefore, by evaluating the performance of the domain controller according to the trusted reference value, it can be determined whether the verification result is correct, thereby improving the accuracy of the obtained performance evaluation result of the domain controller.

An embodiment of the present disclosure provides a storage medium storing program instructions, and the program instructions execute the foregoing method for evaluating the performance of a domain controller when running.

The aforementioned storage medium can be a non-transitory storage medium, including: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other various A medium that can store program code can also be a transitory storage medium.

The foregoing description and drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The examples represent only possible variations. Unless expressly required, individual components and functions are optional and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for those of other embodiments. Also, the terms used in this application are used to describe the embodiments only and not to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a" (a), "an" (an) and "the" (the) are intended to include the plural forms as well, unless the context clearly dictates otherwise. . Similarly, the term "and/or" as used in this application is meant to include any and all possible combinations of one or more of the associated listings. Additionally, when used in this application, the term "comprise" and its variations "comprises" and/or including and/or the like refer to stated features, integers, steps, operations, elements, and/or The presence of a component does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groupings of these. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, or device that includes the element. Herein, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, reference may be made to the description of the method section for relevant parts.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, operations or steps corresponding to different blocks may also occur in different sequences than those disclosed in the description, and sometimes there is no specific relationship between different operations or steps. order. For example, two consecutive operations or steps may, in fact, be performed substantially concurrently, or they may sometimes be performed in the reverse order, depending upon the functionality involved. Each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or actions, or special purpose hardware implemented in combination with computer instructions.

Claims (10)

1. A method for evaluating the performance of a domain controller, the domain controller comprising a planning unit and N detection units, where N is an integer greater than 1, wherein the method comprises: Obtain M·N verification results of the detection unit; wherein, the M·N verification results are obtained by the N detection units verifying the planning unit according to the planning results of the M test samples; M is an integer greater than 1; Determine the credible benchmark values of M test samples according to M·N verification results and initial parameters; Evaluate domain controller performance according to M trusted benchmarks and initial parameters; Output the evaluation result. 2. The method according to claim 1, wherein the initial parameters comprise initial weights corresponding to N detection units respectively; the credible reference value of the test sample is determined according to the verification result and the initial parameters, comprising: calculate

Obtain a credible benchmark value for the test sample; Among them, R _j '(p _k ,s _j ) is the credible reference value of the jth test sample; w' _i is the initial weight corresponding to the ith detection unit; R(D _i |p _k ,s _j ) is The ith detection unit verifies the planning unit according to the planning result of the jth test sample. 3. The method according to claim 1, wherein evaluating domain controller performance according to M credible reference values and initial parameters, comprising: Determine the type of each verification result according to the verification result and the credible benchmark value corresponding to the test sample; the type includes true negative, true positive and false positive; Count the number of true negatives, true positives and false positives in the verification results corresponding to each detection unit; The evaluation value of each detection unit and the corresponding weight of each detection unit are determined according to the number of true negatives, the number of true positives, the number of false positives and the initial parameters. 4. method according to claim 3, is characterized in that, initial parameter also comprises detection unit initial parameter; According to true negative quantity, true positive quantity, false positive quantity and initial parameter, determine the corresponding weight of detection unit evaluation value and detection unit ,include: calculate

Obtain the detection unit accuracy; calculate

Obtain the recall rate of detection units; calculate

Get the detection unit score; calculate

Obtain the evaluation value of the detection unit; calculate

Obtain the weight corresponding to the detection unit; Among them, precision_rate is the accuracy rate of the detection unit, TP(D _i ) is the number of true positives in the verification result corresponding to the i-th detection unit; TN(D _i ) is the true positives in the verification result corresponding to the i-th detection unit Number of negatives; FP(D _i ) is the number of false positives in the verification result corresponding to the ith detection unit; recall_rate is the recall rate of the ith detection unit; f_score(D _i ) is the score of the ith detection unit; f_score( D _g ) is the score of the g-th detection unit; P(D _i ) is the evaluation value of the i-th detection unit; P'(D _i ) is the initial parameter of the i-th detection unit; iter is the test round; _wi is the ith detection unit Weights corresponding to i detection units. 5. The method according to claim 3, wherein the initial parameter further comprises an initial parameter of the planning unit; and evaluating the performance of the domain controller according to each credible reference value and the initial parameter further comprises: calculate

Obtaining a planning unit score for a domain controller; calculate

Obtain the planning unit evaluation value of the domain controller; Among them, score(p _k ) is the planning unit score; P(p _k ) is the evaluation value of the planning unit; P'(p _k ) is the initial parameter of the planning unit. 6. The method according to claim 3, wherein the initial parameters further comprise initial parameters of the test samples; after evaluating the performance of the domain controller according to the M credible reference values and the initial parameters, the method further comprises: calculating

Get test sample scores; calculate

Obtain the evaluation value of the test sample; Among them, score(s _j ) is the score of the j-th test sample; P(s _j ) is the evaluation value of the j-th test sample; P'(s _j ) is the initial parameter of the j-th test sample. 7. The method according to any one of claims 1 to 6, characterized in that, after outputting the evaluation result, further comprising: Get test rounds; When the test round reaches the preset round, display the evaluation results to the user; If the test rounds do not reach the preset rounds, accumulate the test rounds; determine the evaluation result as the initial parameter; re-obtain M·N verification results of the detection unit; determine M tests according to the M·N verification results The credible benchmark value of the sample; evaluate the performance of the domain controller according to the M credible benchmark values and the initial parameters; output the evaluation result. 8. An apparatus for evaluating the performance of a domain controller, comprising a processor and a memory storing program instructions, wherein the processor is configured to, when executing the program instructions, execute the steps of claims 1 to 1. 7. The method of any one for evaluating domain controller performance. 9. An electronic device, characterized by comprising the apparatus for evaluating the performance of a domain controller as claimed in claim 8. 10 . A storage medium storing program instructions, wherein, when the program instructions are running, the method for evaluating the performance of a domain controller according to any one of claims 1 to 7 is executed. 11 .

Images