CN107733705B

CN107733705B - User experience quality assessment model establishing method and device

Info

Publication number: CN107733705B
Application number: CN201710933787.0A
Authority: CN
Inventors: 李溪林
Original assignee: Ruijie Networks Co Ltd
Current assignee: Ruijie Networks Co Ltd
Priority date: 2017-10-10
Filing date: 2017-10-10
Publication date: 2021-01-15
Anticipated expiration: 2037-10-10
Also published as: CN107733705A

Abstract

The invention discloses a method and equipment for establishing a user experience quality evaluation model, which are used for evaluating the user experience quality of an enterprise-level WLAN. The method comprises the following steps: acquiring a training data sample set, wherein the training data sample set comprises user experience quality labels when network experience is carried out on different positions of a plurality of APs in a WLAN coverage range through different types of application services in different types of user equipment, wireless communication parameters when the APs are communicated with the user equipment and wireless environment parameters of the APs; establishing a two-classification model of each application service according to a training data sample set corresponding to each application service and an algorithm model required to be adopted by each application service; the wireless parameters comprise wireless communication parameters when the AP communicates with the user equipment and/or wireless environment parameters of the AP; and obtaining a user experience quality evaluation model according to the two classification models of each application service.

Description

User experience quality assessment model establishing method and device

Technical Field

The invention relates to the technical field of communication, in particular to a method and equipment for establishing a user experience quality evaluation model.

Background

At present, an enterprise-level Wireless Local Area Network (WLAN) is widely applied to Network construction in places such as shopping malls, enterprises, hospitals, schools, and the like, wherein the number of Wireless Access Point controllers (AC) and Wireless Access Points (AP) included in the enterprise-level WLAN is large. In these locations, the best effect to be achieved is to make the Quality of user Experience (QoE) of the user at each location within the coverage area of the WLAN better, so that after the WLAN network deployment, the Quality of user Experience of each user using a Wireless network (Wireless-FIdelity) needs to be evaluated in real time, and further optimization is performed on the network deployment.

The factors influencing the quality of user experience mainly include three parts, namely the original quality of the service, the quality of a data transmission network and the service presenting capability of the user equipment. The original quality of a service refers to the quality of a service provided by a service provider, for example, when the service is a video service, the original quality refers to the definition of the video, and may be classified into high definition, super definition, blue light super definition, and the like; the data transmission network quality is the quality of the transmission network during the service, for example the quality of the network of the mobile operator; the service presenting capability of the user equipment refers to the performance of the user equipment.

Since the types of the user equipments used by the user are numerous, for example, the user equipments may be a mobile phone, an IPAD, a Personal Computer (PC), and the like, and correspondingly, each type of the user equipment may further include each brand type, and meanwhile, the types of the services experienced by the user are numerous, for example, the user equipment may be a Voice over Internet Protocol (VoIP), a video call, a mobile phone television, a mobile phone game, and the like, and the number of APs in the enterprise-level WLAN is large and the deployment is dense, so that it is obviously not practical to combine the user feedback with manual operation maintenance. Currently, the method for evaluating the user experience quality mainly includes a method for evaluating the user experience quality for a specific service scenario and a method for evaluating the user experience quality for a wired network environment of an operator or a mobile network of the operator. For example, a mobile video quality assessment method, but the method is only suitable for a specific certain service scene and is not suitable for other services; or, the user experience quality evaluation method for the operator wired network environment or the operator mobile network is not suitable for the enterprise WLAN due to the different deployment manners of the operator wired network environment or the operator mobile network and the enterprise WLAN. Therefore, how to evaluate the user experience quality of the enterprise-level WLAN in real time and further improve and optimize the enterprise-level WLAN according to the evaluation result is a technical problem to be solved urgently at present.

Disclosure of Invention

The embodiment of the invention provides a method and equipment for establishing a user experience quality evaluation model, which are used for accurately evaluating the user experience quality of an enterprise-level WLAN.

In a first aspect, a method for establishing a user experience quality assessment model is provided, where the method includes:

acquiring a training data sample set, wherein the training data sample set comprises at least one training data sample, the training data sample set comprises user experience quality labels when network experience is carried out on different positions of a plurality of APs in a WLAN coverage range through different types of application services in different types of user equipment, wireless communication parameters when the APs are communicated with the user equipment and wireless environment parameters of the APs, and one training data sample comprises one user experience quality label, at least one wireless communication parameter and at least one wireless environment parameter;

obtaining a training data sample set corresponding to each application service according to the training data sample set;

establishing a two-classification model of each application service according to a training data sample set corresponding to each application service and an algorithm model required to be adopted by each application service, wherein the two-classification model of each application service is a model for representing a mapping relation between wireless parameters and user experience quality; the wireless parameters comprise wireless communication parameters when the AP communicates with the user equipment and/or wireless environment parameters of the AP;

and obtaining the user experience quality evaluation model according to the two classification models of each application service.

Optionally, obtaining a training data sample set corresponding to each application service according to the training data sample set includes:

carrying out standardization processing on the training data sample set to enable dimensions of all data in the training data sample set to be uniform;

determining a training data sample set of each application service from the training data sample set according to the priority of the application service; the priority of the application service is set according to the network quality required by the application service;

and selecting a preset amount of data in each training data sample according to the correlation between the data in each training data sample included in the training data sample set and the user experience quality label so as to obtain a final training data sample set of each application service.

Optionally, the determining a training data sample set of each application service from the training data samples according to the priority of the application service includes:

adding positive training data samples of other application services having a higher priority than a first application service to a positive training data sample set of the first application service, and adding negative training data samples of other application services having a lower priority than the first application service to a negative training data sample set of the first application service; the added positive training data sample set and the added negative training data sample set form a training data sample set of the first application service; the positive training data sample is data when the satisfaction degree of the user for the network experience represented by the user experience quality label is larger than or equal to a preset satisfaction degree threshold value, and the negative training data sample is data when the satisfaction degree of the user for the network experience represented by the user experience quality label is smaller than the preset satisfaction degree threshold value.

Optionally, the algorithm model required to be used by each application service is an algorithm model with the lowest error rate determined from a pre-stored algorithm model set through a k-fold interleaving algorithm.

Optionally, obtaining the user experience quality assessment model according to the two classification models of each application service includes:

and combining the two classification models of each application service according to the priority of the application service to obtain the user experience quality evaluation model.

Optionally, after fusing the two classification models of each application service to obtain the user experience quality assessment model, the method further includes:

acquiring an update data sample set, wherein the update data sample set comprises at least one update data sample, and the update data sample set comprises wireless communication parameters and wireless environment parameters of an AP (access point) when the AP is communicated with user equipment and uploaded in the running process of the AP; wherein one update data sample comprises at least one wireless communication parameter and at least one wireless environment parameter;

calculating the similarity between any one update data sample in the update data sample set and each training data sample in the training data sample set, and taking the user experience quality label of the training data sample with the highest similarity in the training data sample set as the user experience quality label of the any one update data sample to obtain an update data sample set carrying the user experience quality label;

and updating the user experience quality evaluation model according to the updating data sample set carrying the user experience quality label.

Optionally, the method further includes:

receiving a feedback message with poor user experience quality sent by user equipment;

determining whether the user experience quality evaluation model detects a network fault corresponding to the feedback message;

if it is determined that the user experience quality evaluation model does not detect the network fault corresponding to the feedback message, acquiring operation data corresponding to the feedback message;

and correcting the user experience quality evaluation model according to the operation data.

Optionally, after obtaining the user experience quality assessment model according to the two classification models of each application service, the method further includes:

acquiring current network parameter data of first user equipment during network experience, wherein the network parameter data comprises current wireless communication parameters of an AP (access point) and the first user equipment during communication and current wireless environment parameters of the AP;

and determining the current user experience quality of the first user equipment according to the current network parameter data and the user experience quality evaluation model.

Optionally, after determining, according to the network parameter data, the user experience quality of the user equipment in performing the network experience through the user experience quality evaluation model, the method further includes:

when the determination result shows that the current user experience quality of the first user equipment is poor, calculating the weight of each network parameter data in the current network parameter data through the user experience quality evaluation model; the weight is the influence degree of each network parameter data in the current network parameter data on the determination result;

and determining the reason causing the poor user experience quality of the first user equipment according to the calculated weight of each network parameter data.

In a second aspect, a user experience quality assessment model building device is provided, the device comprising:

a first data obtaining unit, configured to obtain a training data sample set, where the training data sample set includes at least one training data sample, and the training data sample set includes a user experience quality tag when performing network experience at different locations of a plurality of APs in a WLAN coverage area through different types of application services in different types of user equipment, a wireless communication parameter when the APs communicate with the user equipment, and a wireless environment parameter of the APs, where one training data sample includes one user experience quality tag, at least one wireless communication parameter, and at least one wireless environment parameter;

the first data processing unit is used for obtaining a training data sample set corresponding to each application service according to the training data sample set;

the model establishing unit is used for establishing a two-classification model of each application service according to a training data sample set corresponding to each application service and an algorithm model required to be adopted by each application service, wherein the two-classification model of each application service is a model for representing the mapping relation between the wireless parameters and the user experience quality; the wireless parameters comprise wireless communication parameters when the AP communicates with the user equipment and/or wireless environment parameters of the AP;

the model establishing unit is further configured to obtain the user experience quality assessment model according to the two classification models of each application service.

Optionally, the obtaining, by the first data processing unit, a training data sample set corresponding to each application service according to the training data sample set includes:

Optionally, the determining, by the first data processing unit, a training data sample set of each application service from the training data samples according to the priority of the application service includes:

Optionally, the algorithm model that each application service needs to use is an algorithm model with the lowest error rate that is determined by the first data processing unit from a pre-stored algorithm model set through a k-fold interleaving algorithm.

Optionally, the obtaining, by the model establishing unit, the user experience quality assessment model according to the two classification models of each application service includes:

and the model establishing unit combines the two classification models of each application service according to the priority of the application service to obtain the user experience quality evaluation model.

Optionally, the apparatus further includes a first model updating unit;

the data acquisition unit is further configured to acquire an update data sample set, where the update data sample set includes at least one update data sample, and the update data sample set includes wireless communication parameters and wireless environment parameters of an AP when the AP is in communication with a user equipment, which are uploaded in an AP operation process; wherein one update data sample comprises at least one wireless communication parameter and at least one wireless environment parameter;

the first data processing unit is further configured to calculate a similarity between any one of the update data samples in the update data sample set and each of the training data samples in the training data sample set, and use a user experience quality label of a training data sample with a highest similarity in the training data sample set as a user experience quality label of the any one of the update data samples, so as to obtain an update data sample set carrying the user experience quality label;

the first model updating unit is used for updating the user experience quality evaluation model according to the updating data sample set carrying the user experience quality label.

Optionally, the apparatus further includes a second model updating unit;

the data acquisition unit is also used for receiving a feedback message with poor user experience quality sent by the user equipment;

the first data processing unit is further configured to determine whether the user experience quality assessment model detects a network failure corresponding to the feedback message; if it is determined that the user experience quality evaluation model does not detect the network fault corresponding to the feedback message, acquiring operation data corresponding to the feedback message;

and the second model updating unit is used for correcting the user experience quality evaluation model according to the operation data.

Optionally, the method includes:

a second data obtaining unit, configured to obtain current network parameter data of a first user equipment during network experience, where the network parameter data includes a current wireless communication parameter when an AP communicates with the first user equipment and a current wireless environment parameter of the AP;

and the evaluation unit is used for determining the current user experience quality of the first user equipment according to the current network parameter data and the user experience quality evaluation model.

Optionally, the apparatus further comprises a second data processing unit;

the second data processing unit is configured to, when the determination result indicates that the current user experience quality of the first user equipment is poor, calculate, by using the user experience quality evaluation model, a weight of each piece of network parameter data in the current network parameter data; the weight is the influence degree of each network parameter data in the current network parameter data on the determination result; and determining the reason causing the poor user experience quality of the first user equipment according to the calculated weight of each network parameter data.

In a third aspect, a computer apparatus is provided, the apparatus comprising a processor for implementing the steps of the method according to any one of the user experience quality assessment model building method provided in the first aspect and the user experience quality assessment method provided in the second aspect when executing a computer program stored in a memory.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of the user experience quality assessment model building method provided in the first aspect and the user experience quality assessment method provided in the second aspect.

In the embodiment of the invention, the training sample data for establishing the evaluation model not only comprises the running parameters of the AP and the interaction parameters of the AP and the user equipment when the user performs network experience, but also comprises the real experience quality label of the user, so that the established evaluation model is not only the use experience quality of the user predicted by the obtained network service parameters, namely the running parameters of the AP and the interaction parameters of the AP and the user equipment, but is closer to the real experience of the user, and the evaluation result is more accurate. Meanwhile, in the embodiment of the invention, the collected training sample data can comprise user experience parameter data of various user equipment and various application services in different environments, namely, the deviation of user experience quality caused among various user equipment or various application services is considered, so that the evaluation of the invention is not limited to one user equipment or one specific application service, and the application range is wider.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a user experience quality assessment model establishment method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a hierarchical structure of a user experience quality assessment model according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a user experience quality assessment model building device according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.

The technical background of the embodiments of the present invention is described below.

Currently, the method for evaluating the user experience quality mainly includes a method for evaluating the user experience quality for a specific service scenario and a method for evaluating the user experience quality for a wired network environment of an operator or a mobile network of the operator. The method for evaluating the user experience quality for a specific service scene refers to evaluating a specific service, such as a mobile video quality evaluation method, but the method is only applicable to a specific service scene, but is not applicable to other services; the method for evaluating the user experience quality in the wired network environment of the operator or the mobile network of the operator is not suitable for the enterprise-level WLAN due to the different deployment modes of the wired network environment of the operator or the mobile network of the operator and the enterprise-level WLAN. Therefore, how to accurately evaluate the user experience quality of the enterprise-level WLAN and further improve and optimize the enterprise-level WLAN according to the evaluation result is a technical problem to be solved urgently at present.

In view of this, embodiments of the present invention provide a method for evaluating user experience quality, in the method, training sample data used for establishing an evaluation model not only includes an operation parameter of an AP and an interaction parameter of the AP and a user device when a user performs network experience, but also includes a true experience quality label of the user, so that the established evaluation model is not only predicted by using an acquired network service parameter, that is, an operation parameter of the AP and an interaction parameter of the AP and the user device, but is closer to a true experience of the user, and an evaluation result is more accurate. Meanwhile, in the embodiment of the invention, the collected training sample data can comprise user experience parameter data of various user equipment and various application services in different environments, namely, the deviation of user experience quality caused among various user equipment or various application services is considered, so that the evaluation of the invention is not limited to one user equipment or one specific application service, and the application range is wider.

The technical scheme provided by the embodiment of the invention is described below by combining the accompanying drawings.

Referring to fig. 1, an embodiment of the present invention provides a method for establishing a user experience quality assessment model, where the method may be implemented by a user experience quality assessment model establishing device provided in an embodiment of the present invention, and the user experience quality assessment model establishing device may be implemented by a PC, a server, or a monitoring platform of an operation and maintenance worker, and the method includes:

step 101: acquiring a training data sample set, wherein the training data sample set comprises at least one training data sample, the training data sample set comprises user experience quality labels when network experience is carried out on different positions of a plurality of APs in a WLAN coverage range through different types of application services in different types of user equipment, wireless communication parameters when the APs are communicated with the user equipment and wireless environment parameters of the APs, and one training data sample comprises one user experience quality label, at least one wireless communication parameter and at least one wireless environment parameter;

step 102: obtaining a training data sample set corresponding to each application service according to the training data sample set;

step 103: establishing a two-classification model of each application service according to a training data sample set corresponding to each application service and an algorithm model required to be adopted by each application service, wherein the two-classification model of each application service is a model for representing the mapping relation between wireless parameters and user experience quality; the wireless parameters comprise wireless communication parameters when the AP communicates with the user equipment and/or wireless environment parameters of the AP;

step 104: and obtaining a user experience quality evaluation model according to the two classification models of each application service.

In the embodiment of the invention, firstly, a training data sample set for establishing an evaluation model is acquired. Specifically, the training data sample set is composed of a plurality of training data samples, and the training data sample set mainly includes three parts of data, that is, a label for representing good or poor user experience quality, and wireless communication parameter data when the AP communicates with the user equipment, and a wireless environment parameter of the AP, where each training data sample includes the three parts of data.

In the embodiment of the present invention, user equipment and application services that are relatively common in the market may be selected to collect training data samples, for example, the user equipment may be a mobile phone, an IPAD, a PC, or an internet of things device of a mainstream manufacturer, and the application service may be WeChat, VOIP, high definition video, web browsing, and the like, and of course, other possible user equipment or application services may also be used, which is not limited herein.

After the user equipment and the application service required by data acquisition are selected, the user equipment and the application service can be used and experienced at different positions of a plurality of APs within the coverage range of the WLAN through the selected user equipment and application service, and then a user experience quality label in the use and experience process is acquired. The user experience quality label represents the real experience when the WLAN network is used, and when the user experience quality label is collected, evaluation of the collection personnel on experience of the WLAN network can be received, namely the use experience is good or poor. Meanwhile, when the user experience quality label is collected, in order to avoid errors caused by judgment of the subjectivity of a collection person, experience parameter standards can be set according to different application services, and the experience parameter standards are used for helping to judge the quality of the user experience. For example, when the application service is specifically a high-definition video, the experience parameter standard may be set according to the video pause number, the video buffering speed, and the video downloading speed, and when the video pause number is too many or the video downloading speed is slow, and the subjective feeling submitted by the acquiring person is poor at this time, it may be determined that the user experience quality at this time is poor, and the corresponding user experience quality label is 0; if the video is smoothly watched, the video downloading speed is high, and the subjective feeling submitted by the acquisition personnel at the moment is good, the user experience quality at the moment can be determined to be good, and the corresponding user experience quality label is 1. Of course, when the experience parameter standard is greatly different from the user, the judgment can be performed according to the actual situation, for example, when the detected video downloading speed is very slow, but the video watching is always smooth, the judgment can be accurate according to the subjective judgment of the user, that is, the user experience quality is good at this time.

When the user experience quality label is collected, the AP reports the wireless environment parameter of the AP and the wireless communication parameter when the AP communicates with the user equipment, where the wireless environment parameter of the AP and the wireless communication parameter when the AP communicates with the user equipment correspond to the collected user experience quality label, and from a certain angle, the collected user experience quality label may be a label of these parameters. The collected user experience quality label may be the use experience of the collection staff in a period of time, and then the labels of the wireless communication parameters and the wireless environment parameters of the AP, which are uploaded by the AP connected to the collection staff during the communication with the collection staff in the period of time, are the user experience quality labels of the collection staff received in the period of time, that is, the wireless communication parameters and the wireless environment parameters may be bound to the user experience quality labels one by one, so the user experience quality labels may also be uploaded together by uploading the wireless communication parameters and the wireless environment parameters at the AP, or may be corresponding to the user experience quality labels according to the time, the location of the AP, and the like after the wireless communication parameters and the wireless environment parameters are obtained from the server.

Specifically, the wireless environment parameters of the AP may include one or more of a signal-to-noise ratio, a channel utilization rate, a wireless interference rate, and the like; the wireless communication parameters when the AP communicates with the user equipment may include one or more of a delay, a packet loss rate, a received signal strength RSSI, an uplink rate, a downlink rate, a wireless message retransmission rate, a wireless message efficiency, an uplink flow, and a downlink flow, and certainly, may also include other possible parameters, which is not limited in this embodiment of the present invention. The AP reports the parameters to the AC, and the AC sends the parameters to the server side for processing and storage.

After the training data sample set is obtained, a user experience quality evaluation model mapped by wireless communication parameters and wireless environment parameters of the AP when the user experience quality label is communicated with the AP and the user equipment can be established according to the training data sample set.

In the embodiment of the present invention, the training data sample set includes a large number of acquired training data samples, so that the training data sample set may be preprocessed first.

Specifically, each training data sample in the training data sample set includes data of three parts, i.e., a user experience quality tag, a wireless communication parameter when the AP communicates with the user equipment, and a wireless environment parameter of the AP, and since the wireless communication parameter when the AP communicates with the user equipment and the wireless environment parameter of the AP further may include multiple types of data, but dimensions of each data may be different, for example, a dimension of the delay is milliseconds (ms) or seconds(s), and a dimension of the uplink rate is bits per second (bps), and different dimensions make different data not be placed on the same layer for comparison, so that all the dimensions of the data may be removed first for uniform treatment and processing of subsequent data. Specifically, the training data sample set may be normalized by a z-score (z-score), so that dimensions of all data in the training data sample set are uniform, and certainly, dimensions may also be uniform by other normalized methods, which is not limited in the present invention. Since the z-score normalization method is a well-known normalization method, the process falls within the scope of the prior art and is not described herein.

Due to network transmission or data acquisition and other reasons, some sample data with data missing may exist in the acquired training data sample set, for example, data missing of one or more parameters in the wireless environment parameters of the AP in a certain sample data is unavailable, such sample data is required to be removed from the training data sample set, and thus, the sample data with the data missing problem may be removed while or after the unified dimension operation is performed.

In the embodiment of the invention, because the user equipment used by the user is various, the user experience quality cannot be evaluated according to the type of the user equipment, and the user equipment finally needs to perform network experience in an application service mode, the evaluation according to the type of the user equipment has little practical significance compared with the evaluation according to different types of application services. Therefore, after the data is preprocessed, the training data sample set can be divided according to different application services, so as to obtain training data sample sets corresponding to different application services.

In order to make the sample data of each application service richer, the application services for acquiring the training data samples can be divided into different priorities, and the training data samples of different application services are processed according to the priorities.

The priority level is related to the network quality required for experiencing the application service, for example, the higher the network quality required for experiencing the application service, the higher the priority level of the application service, and the priority level may be divided according to experience or experimental results. For example, if the application service includes a high definition video, web browsing, wechat, and VOIP, it can be known from a large number of experimental results that when the user experience quality of the high definition video is good, the user experience quality of other application services is also good, but when the user experience quality of other application services is good, the user experience quality of the high definition video is not necessarily good, so that the network quality required by the high definition video is higher than that of other application services, that is, the priority of the high definition video is highest. The priority obtained according to a large number of experimental results is high-definition video, VOIP, webpage browsing and WeChat in sequence from high to low.

In a specific embodiment, the training data samples may be divided into positive training data samples and negative training data samples, where the positive training data samples are data when the user experience quality label represents that the satisfaction degree of the user for the network experience is greater than or equal to a preset satisfaction degree threshold, for example, the training data samples when the user experience quality label is 1; and the negative training data sample is data when the user experience quality label represents that the satisfaction degree of the user to the network experience is smaller than a preset satisfaction degree threshold, for example, the training data sample is when the user experience quality label is 0. When determining the training data sample set corresponding to each application, the description will be specifically given by taking the first application service as an example. Specifically, a positive training data sample of another application service with a priority higher than that of the first application service may be added to a positive training data sample set originally acquired by the first application service, so as to obtain a positive training data sample set corresponding to the first application service; and adding the negative training data samples of other application services with the priority lower than that of the first application service to the originally acquired negative training data sample set of the first application service to obtain a negative training data sample set corresponding to the first application service, so that the added positive training data sample set and the added negative training data sample set can form the training data sample set of the first application service.

For example, when the first application service is VOIP, since the priority of VOIP is lower than that of the high-definition video, when the user experience quality label of the high-definition video is 1, that is, the user experience quality is good, the user experience quality of VOIP is usually good, so the positive training data sample of the high-definition video can also be used as the positive training data sample of VOIP, that is, the positive training data sample of the high-definition video can be added to the positive training data sample set of VOIP; because the priority of the VOIP is higher than that of the web browsing or the WeChat, when the user experience quality label of the web browsing or the WeChat is 0, that is, the user experience quality is poor, the user experience quality of the VOIP is also poor, so the negative training data sample of the web browsing or the WeChat can also be used as the negative training data sample of the VOIP, that is, the negative training data sample of the web browsing or the WeChat can be added into the negative training data sample set of the VOIP.

In the embodiment of the present invention, in the training data sample set of each application service, not all data plays a decisive factor for the user experience quality, that is, data having a large influence on the user experience quality only includes a few main data, so after the training data sample set of each application service is obtained, the training data sample set of each application service includes a positive training data sample and a negative training data sample, and a predetermined amount of data in each sample data can be selected according to the correlation between the data in each training data sample included in the training data sample set and the user experience quality label, so as to obtain the final training data sample set of each application service.

Specifically, the correlation between each data and the user experience quality may be calculated by a Principal Component Analysis (PCA), and then a predetermined number of data with a large correlation value are screened out as feature data, where the feature data is main data that plays a decisive factor for the user experience quality in the application service, and the predetermined number may be set according to an actual situation, or an optimal number of test results is selected according to the test results, and may be set to 8, for example; and other data with small correlation with the quality of user experience can be deleted, so that information redundancy is avoided, and the accuracy of a final result is influenced.

In the embodiment of the invention, after the final training data sample set of each application service is obtained, a user experience quality evaluation model can be established according to the obtained final training data sample set of each application service.

Specifically, an algorithm model suitable for each application service may be selected from a set of pre-stored algorithm models, and then an algorithm model may be established for each application service. The pre-stored algorithm model set may specifically include a plurality of Machine learning algorithm models, for example, may include a Support Vector Machine algorithm (SVM), a Linear Discriminant Analysis (LDA), a logistic regression Analysis model, and an Artificial Neural network (ans). When specific selection is carried out, the optimal algorithm model for each application service can be selected through a k-fold cross validation method. The K-fold cross validation refers to that 1/K sample data included in a training sample data set of each application service is used as test data, the rest part of data is used as validation data, each algorithm model in an optional algorithm model set is trained for K times according to the test data, then the model obtained by each training is validated through the validation data, finally, the average of the error rates of the K times of validation is taken as the final error rate of the algorithm model, and after the final error rates of all the algorithm models are calculated, the algorithm model with the lowest error rate is selected as the algorithm model of the corresponding application service.

After the algorithm model required to be adopted by each application service is determined, a two-classification algorithm model of each application service can be established according to the training data sample set and the determined algorithm model of each application service, wherein the two-classification algorithm model refers to that the model established by each application service is finally used for carrying out two classifications, namely, the model is used for determining whether the user experience quality is good or poor.

In the embodiment of the invention, after the two-classification algorithm model of each application service is successfully established, the two-classification algorithm models of all the application services can be combined according to the priority relation among the application services to obtain the final user experience quality evaluation model. Specifically, according to the priority relationship of different application services, the two classification algorithm models of the application service with the lowest priority are arranged at the bottom layer, and the two classification algorithm models of the different application services are sorted upwards according to the mode that the priority sorting is from low to high. For example, as shown in fig. 2, the hierarchical structure diagram of the user experience quality evaluation model is shown, in which the algorithm model hierarchical relationship of each application service sequentially includes a high definition video classification algorithm model, a VOIP classification algorithm model, a web browsing classification algorithm model, and a wechat classification algorithm model according to the order of priority. When the user experience quality needs to be determined through the established user experience quality evaluation model, determining from the two-classification algorithm model of the WeChat at the bottommost layer, if the two-classification algorithm model of the WeChat determines that the user experience quality is good, continuing to determine the previous layer until the user experience quality determined by a certain layer is poor or the two-classification algorithm model of the application service with the highest layer number, for example, when the two-classification algorithm model of the VOIP is determined and the user experience quality of the application service is poor, the user experience quality of the application service with the priority lower than the VOIP is good; or when the two-classification algorithm model of the high-definition video is determined, the user experience quality of the application service is still good, which indicates that the user experience quality of all the application services is good.

In the embodiment of the invention, the training data sample for establishing the user experience quality evaluation model only comprises sample data of mainstream user equipment and application services in the market, but other types of user equipment and application services actually exist, and in order to improve the compatibility of the user experience quality evaluation model, the user experience quality evaluation model needs to be further optimized and updated.

In the embodiment of the invention, the update data sample set can be obtained to further optimize and update the user experience quality evaluation model.

In the embodiment of the invention, the running data can be continuously reported to the AC in the actual running process of the AP, and the AC can upload the running data to the server side, so that the updated data sample set can be obtained through the server side. However, since the obtained update data sample set is data of the AP in normal operation, and only includes operating data uploaded by the AP, that is, wireless communication parameters when the AP communicates with the user equipment and wireless environment parameters of the AP, and does not include experience of an acquisition person at this time, that is, does not include a user experience quality label, the update data samples included in the obtained update data sample set are not labeled, and cannot be directly used for further optimization updating of the data user experience quality evaluation models, and therefore, after the update data sample set is obtained, a semi-supervised learning algorithm may be combined to convert the unlabeled update data sample set into the labeled update data sample set.

Specifically, after the update data sample set is obtained, the feature data may be selected, where the feature data may be the feature data determined by the PCA algorithm, the update data samples in the update data sample set are compared with the feature data of the training data samples obtained before, the similarity between the two is calculated, and then the user experience quality label of the training data sample with the highest similarity among the training data samples is used as the label of the corresponding unlabeled update data sample, so that the unlabeled update data sample can be converted into the labeled update data sample, and the labeled update data sample set is obtained.

After the labeled update data sample set is obtained, the established user experience quality evaluation model can be optimized and updated according to the labeled update data sample set. Specifically, the established user experience quality assessment model can be optimized and updated by combining the obtained labeled update data sample set and the collected training data sample set.

In the embodiment of the invention, in order to ensure the accuracy of the established user experience quality evaluation model, the model needs to be verified before the user experience quality evaluation model is put into use, and if the accuracy is found to be low during verification, a reason needs to be found to repair the model. Specifically, the method for verifying the user experience quality assessment model may include, but is not limited to, the following two methods:

(1) and verifying the user experience quality evaluation model through the collected new training data samples. Specifically, the wireless parameter data in the collected new training data sample is used as input, the output is the user experience quality corresponding to the input wireless parameter data, the user experience quality label is used as judgment data, the user experience quality to be output is compared with the user experience quality label actually collected, whether the user experience quality evaluation model is correct or not is further determined, after multiple times of verification, the correct rate of multiple times of verification is calculated, whether the accuracy rate reaches the preset correct rate or not is judged, and if yes, the user experience quality evaluation model can be proved to be high in correct rate and can be put into practical use.

(2) The method comprises the steps of obtaining data reported by an AP when a user actually performs network experience from a server, then performing sampling evaluation, determining corresponding user experience quality of an extracted data sample through a user experience quality evaluation model, then obtaining actual user experience quality of the user corresponding to extracted sample data, comparing the user experience quality determined by the model with the actual user experience quality of the user, and further verifying the accuracy of the user experience quality evaluation model.

In the embodiment of the invention, in the actual user experience quality evaluation process, when the user actually performs network experience, the situation of poor user experience quality may exist, and the user can feed back the situation. Correspondingly, after receiving the feedback message of the user, the user experience quality evaluation model can be determined whether the network fault corresponding to the feedback message of the user is detected, and if the network fault is detected, no response is made; however, if the user experience quality evaluation model does not detect the network fault, it indicates that the current user experience quality evaluation model is not complete, and then the operation data of the network fault fed back by the feedback message can be obtained, and the user experience quality evaluation model is corrected according to the operation data, so that the user experience quality evaluation model is more accurate.

In the embodiment of the invention, after the user experience quality evaluation model verifies that the accuracy reaches the standard which can be put into practical use for many times, the user experience quality evaluation model can be actually used for user experience quality evaluation.

In the embodiment of the present invention, when the first user equipment performs network experience through the enterprise-level WLAN, current network parameter data of the first user equipment during network experience may be obtained, where the current network parameter data may include current wireless communication parameters when the AP communicates with the user equipment and wireless environment parameters of the AP. Specifically, the current network parameter data may be acquired from an AP in communication with the first user equipment; or obtaining current network parameter data from an AC managing the AP; or obtaining the current network parameter data from the server side.

After the user experience quality evaluation model is obtained and the current network parameter data is obtained, the user experience quality corresponding to the current network parameter data can be determined according to the user experience quality evaluation model and the current network parameter data. Specifically, the obtained current network parameter data is input into the user experience quality evaluation model, the determination is performed from the two-classification algorithm model of the application service at the bottommost layer, if the two-classification algorithm model of the application service at the bottommost layer determines that the user experience quality is good, the determination of the previous layer is continued until the two-classification algorithm model of the application service at which the user experience quality determined by a certain layer is poor or the number of layers is the highest, for example, when the two-classification algorithm model of the VOIP is determined as shown in fig. 2, the determination is performed from the two-classification algorithm model of the WeChat at the bottommost layer, if the two-classification algorithm model of the WeChat determines that the user experience quality is good, the determination of the previous layer is continued until the two-classification algorithm model of the application service at which the user experience quality determined by a certain layer is poor or the number of layers is the highest, for example, when the two-classification algorithm model of the VOIP, if the user experience quality of the application service is poor, the user experience quality of the application service with the priority lower than the VOIP is good; or when the two-classification algorithm model of the high-definition video is determined, the user experience quality of the application service is still good, which indicates that the user experience quality of all the application services is good.

In the embodiment of the present invention, when the determination result of the user experience quality evaluation model indicates that the user experience quality of the first user equipment at this time is poor, the weight of each network parameter data in the current network parameter data used for obtaining the determination result may be calculated by the user experience quality evaluation model, where the weight refers to the degree of influence of each network parameter data on the determination result, that is, at the same time, each network parameter data affects the user experience quality of the user. For example, when the network download speed is slow, the user may not watch the video smoothly; or when the time delay is high, the user may watch the video not smoothly, but at the same moment, each network parameter data causes different degrees of influence of poor user experience quality, so that when it is determined that the user experience quality is poor, the degree of influence of each network parameter data can be calculated through the user experience quality evaluation model, and then the network parameter data with the largest degree of influence is determined, which is the main reason for poor user experience quality. And then can further analyze the actual reason that this network parameter data corresponds according to this network parameter data, for example wireless environment disturb one or more in the actual reason such as serious, AP covers not enough, terminal performance is poor, AP power is unreasonable, just so, can help the quick reason that causes the poor user experience quality of finding of backstage operation maintainer, and then repair the network to this reason fast, and then promote user experience quality.

In summary, in the embodiment of the present invention, the training sample data used for establishing the evaluation model not only includes the operation parameters of the AP and the interaction parameters of the AP and the user equipment when the user performs the network experience, but also includes the real experience quality label of the user, so that the established evaluation model is not only the use experience quality of the user predicted by the acquired network service parameters, that is, the operation parameters of the AP and the interaction parameters of the AP and the user equipment, but is closer to the real experience of the user, and the evaluation result is more accurate. Meanwhile, in the embodiment of the invention, the collected training sample data can comprise user experience parameter data of various user equipment and various application services in different environments, namely, the deviation of user experience quality caused among various user equipment or various application services is considered, so that the evaluation of the invention is not limited to one user equipment or one specific application service, and the application range is wider.

The following describes the apparatus provided by the embodiment of the present invention with reference to the drawings.

Referring to fig. 3, based on the same inventive concept of the embodiment shown in fig. 1, an embodiment of the present invention provides a user experience quality assessment model building apparatus 30, which includes:

a first data obtaining unit 301, configured to obtain a training data sample set, where the training data sample set includes at least one training data sample, and the training data sample set includes a user experience quality tag when performing network experience at different locations of multiple APs in a WLAN coverage area through different types of application services in different types of user equipment, a wireless communication parameter when the APs communicate with the user equipment, and a wireless environment parameter of the APs, where one training data sample includes one user experience quality tag, at least one wireless communication parameter, and at least one wireless environment parameter;

a first data processing unit 302, configured to obtain a training data sample set corresponding to each application service according to the training data sample set;

a model establishing unit 303, configured to establish a two-class model of each application service according to the training data sample set corresponding to each application service and the algorithm model that needs to be adopted by each application service, where the two-class model of each application service is a model used for representing a mapping relationship between a wireless parameter and user experience quality; the wireless parameters comprise wireless communication parameters when the AP communicates with the user equipment and/or wireless environment parameters of the AP;

the model establishing unit 303 is further configured to obtain a user experience quality evaluation model according to the two classification models of each application service.

Optionally, the obtaining, by the first data processing unit 302, a training data sample set corresponding to each application service according to the training data sample set includes:

Optionally, the determining, by the first data processing unit 302, a training data sample set of each application service from the training data samples according to the priority of the application service includes:

adding positive training data samples of other application services with higher priority than the first application service to a positive training data sample set of the first application service, and adding negative training data samples of other application services with lower priority than the first application service to a negative training data sample set of the first application service; the added positive training data sample set and the added negative training data sample set form a training data sample set of the first application service; the positive training data sample is data when the user experience quality label represents that the satisfaction degree of the user on the network experience is greater than or equal to a preset satisfaction degree threshold value, and the negative training data sample is data when the user experience quality label represents that the satisfaction degree of the user on the network experience is smaller than the preset satisfaction degree threshold value.

Optionally, the algorithm model required by each application service is the algorithm model with the lowest error rate determined by the first data processing unit 302 from a pre-stored algorithm model set through a k-fold interleaving algorithm.

Optionally, the model establishing unit 303 obtains the user experience quality evaluation model according to the two classification models of each application service, including:

the model establishing unit 303 combines the two classification models of each application service according to the priority of the application service to obtain a user experience quality evaluation model.

Optionally, the apparatus further comprises a first model updating unit 304;

the data acquisition unit is further used for acquiring an update data sample set, wherein the update data sample set comprises at least one update data sample, and the update data sample set comprises wireless communication parameters and wireless environment parameters of the AP when the AP and the user equipment are communicated, and the wireless communication parameters are uploaded in the running process of the AP; wherein one update data sample comprises at least one wireless communication parameter and at least one wireless environment parameter;

the first data processing unit 302 is further configured to calculate a similarity between any one of the updated data samples in the updated data sample set and each of the training data samples in the training data sample set, and use a user experience quality label of a training data sample with a highest similarity in the training data sample set as a user experience quality label of any one of the updated data samples, so as to obtain an updated data sample set carrying the user experience quality label;

the first model updating unit 304 is configured to update the user experience quality assessment model according to the update data sample set carrying the user experience quality label.

Optionally, the apparatus further comprises a second model updating unit 305;

the data acquisition unit is also used for receiving a feedback message which is sent by the user equipment and has poor user experience quality;

the first data processing unit 302 is further configured to determine whether the user experience quality assessment model detects a network failure corresponding to the feedback message; if it is determined that the user experience quality evaluation model does not detect the network fault corresponding to the feedback message, acquiring operation data corresponding to the feedback message;

the second model updating unit 305 is configured to correct the user quality of experience assessment model according to the operation data.

Optionally, the method includes:

a second data obtaining unit 306, configured to obtain current network parameter data of the first user equipment during network experience, where the network parameter data includes a current wireless communication parameter when the AP communicates with the first user equipment and a current wireless environment parameter of the AP;

the evaluation unit 307 is configured to determine the current user experience quality of the first user equipment according to the current network parameter data and the user experience quality evaluation model.

Optionally, the apparatus further comprises a second data processing unit 308;

the second data processing unit 308 is configured to, when the determination result indicates that the current user experience quality of the first user equipment is poor, calculate, through the user experience quality evaluation model, a weight of each network parameter data in the current network parameter data; the weight is the influence degree of each network parameter data in the current network parameter data on the determination result; and determining the reason causing the poor user experience quality of the first user equipment according to the calculated weight of each network parameter data.

The device may be configured to execute the method provided in the embodiment shown in fig. 2, and therefore, for functions and the like that can be realized by each functional module of the device, reference may be made to the description of the embodiment shown in fig. 2, which is not repeated here. Here, since the units 304 to 308 are not indispensable functional modules, they are shown by broken lines in fig. 3.

Referring to fig. 4, an embodiment of the present invention further provides a computer apparatus, where the computer apparatus includes a processor 401, and the processor 401 is configured to implement the steps of the user experience quality evaluation model building method and the user experience quality evaluation method provided by the embodiment of the present invention when executing the computer program stored in the memory.

Optionally, the processor 401 may be specifically a central processing unit, an Application Specific Integrated Circuit (ASIC), one or more Integrated circuits for controlling program execution, a hardware Circuit developed by using a Field Programmable Gate Array (FPGA), or a baseband processor.

Optionally, the processor 401 may include at least one processing core.

Optionally, the computer apparatus further includes a Memory 402, where the Memory 402 may include a Read Only Memory (ROM), a Random Access Memory (RAM), and a disk Memory. The memory 402 is used for storing data required by the processor 401 in operation. The number of the memories 402 is one or more. The memory 402 is also shown in fig. 4, but it should be understood that the memory 402 is not an optional functional block, and is shown in fig. 4 by a dotted line.

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the user experience quality assessment model establishing method and the user experience quality assessment method provided in the embodiments of the present invention.

In the embodiments of the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the described unit or division of units is only one division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical or other form.

The functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be an independent physical module.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the technical solutions of the embodiments of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device, such as a personal computer, a server, or a network device, or a processor (processor) to execute all or part of the steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a Universal Serial Bus flash drive (usb flash drive), a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk.

The above embodiments are only used to describe the technical solutions of the present application in detail, but the above embodiments are only used to help understanding the method of the embodiments of the present invention, and should not be construed as limiting the embodiments of the present invention. Variations or substitutions that may be readily apparent to one skilled in the art are intended to be included within the scope of the embodiments of the present invention.

Claims

1. A method for establishing a user experience quality assessment model is characterized by comprising the following steps:

acquiring a training data sample set, wherein the training data sample set comprises at least one training data sample, the training data sample set comprises user experience quality labels when network experience is carried out on different positions of a plurality of Access Points (APs) in a WLAN coverage range through different types of application services in different types of user equipment, wireless communication parameters when the APs are communicated with the user equipment and wireless environment parameters of the APs, and one training data sample comprises one user experience quality label, at least one wireless communication parameter and at least one wireless environment parameter;

2. The method of claim 1, wherein obtaining a training data sample set corresponding to each application service according to the training data sample set comprises:

3. The method of claim 2, wherein the determining a set of training data samples for each application service from the training data samples according to the priority of the application service comprises:

4. The method of claim 1, wherein the algorithm model required for each application service is the algorithm model with the lowest error rate determined by a k-fold interleaving algorithm from a pre-stored set of algorithm models.

5. The method of claim 1, wherein deriving the user quality of experience assessment model based on a binary model for each application service comprises:

6. The method of claim 5, wherein after combining the two classification models for each application service to obtain a user quality of experience assessment model, the method further comprises:

7. The method of claim 1, wherein the method further comprises:

8. The method of any of claims 1 to 7, wherein after obtaining the user experience quality assessment model according to the binary model of each application service, the method further comprises:

9. The method of claim 8, wherein after determining a quality of user experience of the user device in conducting a network experience via a quality of user experience assessment model based on the network parameter data, the method further comprises:

10. A user experience quality evaluation model creation device, comprising:

a first data obtaining unit, configured to obtain a training data sample set, where the training data sample set includes at least one training data sample, and the training data sample set includes a user experience quality tag when performing network experience at different locations of a plurality of access points AP in a WLAN coverage area through different types of application services in different types of user equipment, a wireless communication parameter when the AP communicates with the user equipment, and a wireless environment parameter of the AP, where one training data sample includes one user experience quality tag, at least one wireless communication parameter, and at least one wireless environment parameter;

11. The apparatus of claim 10, wherein the obtaining, by the first data processing unit, a training data sample set corresponding to each application service according to the training data sample set comprises:

12. The apparatus of claim 11, wherein the first data processing unit determines a set of training data samples for each application service from the training data samples according to the priority of the application service, comprising:

13. The apparatus of claim 10, wherein the algorithm model to be used for each application service is the algorithm model with the lowest error rate determined by the first data processing unit from a pre-stored set of algorithm models by a k-fold interleaving algorithm.

14. The apparatus of claim 10, wherein the model building unit derives the user quality of experience assessment model based on a binary model for each application service, comprising:

15. The apparatus of claim 10, further comprising a first model update unit;

the first data obtaining unit is further configured to obtain an update data sample set, where the update data sample set includes at least one update data sample, and the update data sample set includes wireless communication parameters and wireless environment parameters of an AP when the AP communicates with a user equipment, which are uploaded in an AP operation process; wherein one update data sample comprises at least one wireless communication parameter and at least one wireless environment parameter;

16. The apparatus of claim 10, further comprising a second model updating unit;

the first data acquisition unit is further used for receiving a feedback message with poor user experience quality sent by the user equipment;

17. The apparatus of any of claims 10 to 16, further comprising:

18. The apparatus of claim 17, wherein the apparatus further comprises a second data processing unit;

the second data processing unit is configured to calculate, by using the user experience quality assessment model, a weight of each piece of network parameter data in the current piece of network parameter data when a determination result indicates that the current user experience quality of the first user equipment is poor; the weight is the influence degree of each network parameter data in the current network parameter data on the determination result; and determining the reason causing the poor user experience quality of the first user equipment according to the calculated weight of each network parameter data.

19. A computer arrangement, characterized in that the arrangement comprises a processor for implementing the steps of the method according to any one of claims 1-9 when executing a computer program stored in a memory.

20. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program realizing the steps of the method according to any one of claims 1-9 when executed by a processor.