CN113642503B - Window service scoring method and system based on image and voice recognition - Google Patents

Abstract

The invention belongs to the technical field of image and speech recognition, and provides a window service scoring method and system based on image and speech recognition. First, the expression score is obtained through facial expression recognition, and then the speech text is compared with keywords in a preset database. The voice score is obtained, and finally the window service score is obtained by combining the expression score and the voice score, which improves the accuracy and reliability of the window service score and achieves more accurate and objective feedback on the employee's business level.

Description

Window service scoring method and system based on image and speech recognition

Technical field

The invention belongs to the technical field of image and speech recognition, and in particular relates to a window service scoring method and system based on image and speech recognition.

Background technique

The statements in this section merely provide background technical information related to the present invention and do not necessarily constitute prior art.

The evaluation standard for window service work is user satisfaction. User satisfaction is also called the user satisfaction index. It is the degree of matching between user expectations and user experience. That is, the user compares the perceived effect of a product or business with its expected value. The resulting index.

The traditional window service scoring method is to conduct service satisfaction follow-up visits through phone calls, text messages, on-site button scoring, etc. Users will have unsatisfactory evaluation feedback because the time is too long or there is no time, and it will also cause trouble to users. The traditional service rating method no longer meets the current high-efficiency, fast-paced window office model, and cannot accurately, comprehensively and objectively reflect the business service level of staff.

With the maturity of intelligent facial expression recognition algorithms, the improvement and access to big data platforms, and the continuous updating and upgrading of electronic hardware technology, new window service scoring device software and hardware have sufficient conditions. Therefore, based on images, voice Research on the identification of window service scoring methods is urgent. However, traditional facial expression recognition methods still have certain flaws. They are greatly affected by factors such as noise and lighting, and the recognition accuracy is low.

Contents of the invention

In order to solve the technical problems existing in the above background technology, the present invention provides a window service scoring method and system based on image and voice recognition. First, the expression score is obtained through facial expression recognition, and then the voice text is combined with the keywords in the preset database. Comparison is performed to obtain the voice score, and finally the window service score is obtained by combining the expression score and the voice score, which improves the accuracy and reliability of the window service score and achieves more accurate and objective feedback on the employee's business level.

In order to achieve the above objects, the present invention adopts the following technical solutions:

A first aspect of the present invention provides a window service scoring method based on image and voice recognition, which includes:

Obtain the user's video files and audio files during the window service process;

Convert video files into multiple images to be recognized, perform facial expression recognition, and obtain expression scores based on the recognition results;

Convert audio files into voice text, compare the voice text with keywords in the preset database, and obtain a voice score based on the comparison results;

Based on the expression score and voice score, the window service score is obtained.

Further, the process of facial expression recognition is:

Obtain the image to be recognized and detect the face image;

Preprocess the face image to obtain the face grayscale image;

Locate key points in the grayscale image of the human face, and intercept image blocks of predetermined size centered on each key point;

The LBP algorithm based on mean square error is used to extract the LBP histogram of each image block, and the LBP histograms of all image blocks are connected in a preset order to obtain the LBP texture feature vector of the face grayscale image;

Input the LBP texture feature vector into the classifier to obtain the expression recognition result.

Further, the detection to obtain the face image specifically includes: detecting the image to be recognized through the AdaBoost face detection algorithm, obtaining the face area, and cropping the face image.

Further, the key point positioning adopts a supervised descent algorithm.

Further, the preprocessing includes:

Perform uniform illumination judgment and illumination compensation on the face image to obtain a face image with uniform illumination;

The uniformly illuminated face image is grayscaled to obtain a face grayscale image.

Further, the LBP algorithm based on mean square error is specifically:

Each pixel in the image block is used as the center pixel of the preset sliding window in turn;

Calculate the absolute value of the mean difference between the gray value of each neighborhood pixel and the gray value of the central pixel in the sliding window;

Calculate the mean square error of the gray value of each neighborhood pixel in the sliding window;

The LBP image is obtained based on the absolute value and mean square error;

The histogram of the LBP image is obtained statistically and normalized to obtain the LBP texture feature vector.

Further, the process of obtaining the LBP image based on the absolute value and mean square error is:

When the absolute value is greater than the mean square error, select the average value of the gray value of all neighbor pixels in the sliding window as the threshold; otherwise, select the central pixel gray value as the threshold;

Based on the threshold, the LBP value of each pixel in the image block is calculated to obtain an LBP image.

A second aspect of the present invention provides a window service scoring system based on image and voice recognition, which includes:

A data acquisition module, which is configured to: acquire the user's video files and audio files during the window service process;

An expression score acquisition module, which is configured to: convert video files into multiple images to be recognized, perform facial expression recognition, and obtain expression scores based on the recognition results;

A voice score acquisition module, which is configured to: convert audio files into voice text, compare the voice text with keywords in a preset database, and obtain a voice score based on the comparison results;

The window service rating module is configured to: obtain the window service rating based on the expression rating and the voice rating.

A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the window service scoring method based on image and voice recognition are implemented as described above.

A fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method is implemented based on Steps in the window service scoring method for image and speech recognition.

Compared with the prior art, the beneficial effects of the present invention are:

The invention provides a window service scoring method based on image and voice recognition. The expression score is obtained through facial expression recognition, the voice score is obtained by comparing the voice text with the keywords in the preset database, and finally the expression score and voice score are combined. Obtaining window service ratings improves the accuracy and reliability of window service ratings, achieving more accurate and objective feedback on employees' business levels. It also eliminates the need for additional user return visits, reduces employee workload, and avoids consequences for customers. Harassment.

The present invention provides a window service scoring method based on image and voice recognition. In the process of facial expression recognition, the LBP algorithm based on mean square error is used to extract LBP texture features. The method of calculating the LBP value takes into account the central pixel value and the neighbor value at the same time. The influence of domain pixel values can effectively remove the influence of too large or too small central pixels on LBP values according to the characteristics of domain pixels, reducing the influence of noise points and making the extracted LBP texture features more accurate.

Description of the drawings

The description and drawings that constitute a part of the present invention are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

Figure 1 is a flow chart of a window service scoring method based on image and voice recognition according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terms used herein are for the purpose of describing specific embodiments only, and are not intended to limit the exemplary embodiments according to the present invention. As used herein, the singular forms are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it will be understood that when the terms "comprises" and/or "includes" are used in this specification, they indicate There are features, steps, operations, means, components and/or combinations thereof.

Embodiment 1

As shown in Figure 1, this embodiment provides a window service scoring method based on image and voice recognition, which specifically includes the following steps:

(1) Obtain the video files of the user (the customer receiving the service) during the window service process, that is, when the user accepts the window service, the camera device is turned on to collect the user video files in real time; each user's video is processed to obtain multiple user pictures. image. Then the user image is input into the expression recognition model to perform facial expression recognition, and the expression score is obtained based on the recognition result. Specifically, based on the expression recognition result, it is judged whether the user's expression in the image is happy. When the recognition result is happy, the expression score is plus 1. During the existing window service process, the user service process will be videotaped and audio-recorded, which does not involve infringement of user privacy.

The process of facial expression recognition using the expression recognition model includes:

Step 1: Obtain the image to be recognized (user image) and detect the face image. Specifically: Use the AdaBoost face detection algorithm to detect the image to be recognized, obtain the face area, and crop out the face image.

Step 2: Perform uniform illumination judgment and illumination compensation on the face image to obtain a face image with uniform illumination. The specific steps of illumination uniformity judgment and illumination compensation can be adopted, the method proposed in the patent 201410258412.5 Extraction method of facial image marked by face algorithm.

Step 3: Perform grayscale processing on the uniformly illuminated face image to obtain a grayscale image of the face.

Step 4: Use the facial key point positioning algorithm to locate key points on the grayscale image of the face. The key points include eyebrows, eyes, nose, mouth and other facial positions. The facial key point positioning algorithm uses the supervised descent algorithm (SDM). , Supervised Descent Method).

Step 5: For each key point, intercept an image block of a predetermined size centered on the key point in the face grayscale image. The predetermined size is 16 pixels × 16 pixels.

Step 6: Use the LBP algorithm based on mean square error to extract the LBP texture feature vector of each image block.

The traditional LBP algorithm directly uses the central pixel value as the threshold for calculation, and only considers the influence of the central pixel. When the central pixel is too large or too small, details are easily obscured. Therefore, the present invention proposes an LBP algorithm based on mean square error. The specific process is as follows:

(1) Construct a sliding window, and use each pixel in an image block as the center pixel (x _c , y _c ) of the preset sliding window, c=1,2,...,n, n is the total number of pixels in the image The number of pixels, calculate the gray value g _p (p=1,2,...,P) of each neighborhood pixel in the sliding window with the pixel (x _c , y _c ) as the center pixel and the gray value of the central pixel The absolute value M of the mean difference between g _c is expressed as

As an implementation, the size of the sliding window is 3 pixels × 3 pixels, then P = 8;

(2) Calculate the mean square error S of the gray value of each neighborhood pixel in the sliding window, expressed as

Among them, g is the average value of all neighborhood pixel values g _p .

(3) Determine the size of the absolute value M and the mean square error S. If M ≤ S, select the central pixel gray value g _c as the threshold α; otherwise, select the average value g of all neighbor pixel gray values in the sliding window as Threshold α. Based on the threshold α, calculate the LBP value of each pixel to obtain the LBP image. Specifically:

Among them, (x _c , y _c ) is the coordinate position of the central pixel, and LBP (x _c , y _c ) is the calculated LBP value of the pixel (x _c , y _c ), that is, the pixel (x _c , y ) in the LBP image _c ), g _c is the gray value of the central pixel (x _c , y _c ), g _p is the gray value of the p-th neighbor pixel of the central pixel (x _c , y _c ), f is A symbolic function, expressed as:

(4) Statistically obtain the LBP histogram of the LBP image corresponding to the image block, that is, the frequency of occurrence of each LBP value, and then normalize the histogram of the image block to obtain the LBP texture feature vector of the image block.

It can be seen that the method of calculating the LBP value of the present invention takes into account the influence of the central pixel value and the neighboring pixel value at the same time, can effectively remove the influence of the central pixel that is too large or too small on the LBP value according to the characteristics of the domain pixel, and reduces the noise point. Impact, the extracted LBP texture features are more accurate.

Step 7: Connect the obtained LBP histograms of each image block in a preset order to form a feature vector, which is the LBP texture feature vector of the entire grayscale image.

Before performing expression recognition, the histogram of each image block needs to be integrated and input to the classifier as a whole. The histograms of each image block are connected in a preset order and used as the input of the classifier.

Since there are usually multiple key points obtained, and different key points correspond to different facial positions, the histograms of each image block need to be spliced in a preset order. Taking the key points including eyebrows, eyes, nose, and mouth as an example, the preset order can be eyebrows, eyes, nose, and mouth.

In addition, whether in the training stage of the classifier or in the expression recognition stage using the classifier, the histograms of multiple image blocks extracted from any image are spliced in the same preset order to ensure that the expression The structure of the input data of the recognition model is unified.

Step 8: Input the LBP texture feature vector into the classifier to obtain the expression classification result.

The training of expression recognition models requires a large expression library training model. Currently, there are not many public facial expression databases. The well-known and widely used data set Extended Cohn-Kanada (CK+) in facial expression recognition systems was developed by P. Collected by Lucy. The library contains 327 labeled expressions for 123 objects, divided into seven expressions: normal, angry, contempt, disgust, fear, happy and sad. The expression recognition model of this application uses this database.

(2) Obtain the user audio file during the window service process, convert the user's audio file into voice text, compare the voice text with the keywords in the preset database, and obtain a voice score based on the comparison results.

Specifically, if a keyword in the preset database appears once in the voice text, the voice score will be increased by 1. Keywords in the default database include "thank you", "thank you" and other acknowledgment terms.

(3) Add the expression scores and voice scores of the same user in the same service to obtain the window service score of the user in the service. For example, if the expression score is 10 and the voice score is 3, the user's window service score in this service is (10+3).

The method of the present invention improves the accuracy and reliability of window service scoring, obtains more accurate and objective feedback on employees' business levels, and is conducive to the improvement of a virtuous cycle of window services. Moreover, there is no need to carry out additional user return visits, which reduces the workload of employees and avoids harassment to customers.

Embodiment 2

This embodiment provides a window service scoring system based on image and voice recognition, which specifically includes the following modules:

A data acquisition module, which is configured to: acquire the user's video files and audio files during the window service process;

An expression score acquisition module, which is configured to: convert video files into multiple images to be recognized, perform facial expression recognition, and obtain expression scores based on the recognition results;

A voice score acquisition module, which is configured to: convert audio files into voice text, compare the voice text with keywords in a preset database, and obtain a voice score based on the comparison results;

The window service rating module is configured to: obtain the window service rating based on the expression rating and the voice rating.

It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1, and the specific implementation process is the same, which will not be described again here.

Embodiment 3

This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the window service scoring method based on image and voice recognition as described in the above-mentioned Embodiment 1 are implemented.

Embodiment 4

This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method based on the first embodiment. Steps in the window service scoring method for image and speech recognition.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, etc.) embodying computer-usable program code therein.

The invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. During execution, the process may include the processes of the embodiments of each of the above methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM), etc.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (9)

1. A window service scoring method based on image and speech recognition, which is characterized by: Obtain the user's video files and audio files during the window service process; Convert video files into multiple images to be recognized, perform facial expression recognition, and obtain expression scores based on the recognition results; Convert audio files into voice text, compare the voice text with keywords in the preset database, and obtain a voice score based on the comparison results; Based on the expression score and voice score, the window service score is obtained; The process of facial expression recognition is: Obtain the image to be recognized and detect the face image; Preprocess the face image to obtain the face grayscale image; Locate key points in the grayscale image of the human face, and intercept image blocks of predetermined size centered on each key point; The LBP algorithm based on mean square error is used to extract the LBP histogram of each image block, and the LBP histograms of all image blocks are connected in a preset order to obtain the LBP texture feature vector of the face grayscale image; The LBP algorithm based on mean square error is specifically: Each pixel in the image block is used as the center pixel of the preset sliding window in turn; Calculate the absolute value of the mean difference between the gray value of each neighborhood pixel and the gray value of the central pixel in the sliding window; Calculate the mean square error of the gray value of each neighborhood pixel in the sliding window; The LBP image is obtained based on the absolute value and mean square error; The process of obtaining the LBP image based on the absolute value and mean square error is: When the absolute value is greater than the mean square error, select the average value of the gray value of all neighbor pixels in the sliding window as the threshold; otherwise, select the central pixel gray value as the threshold; Based on the threshold, the LBP value of each pixel in the image block is calculated to obtain an LBP image. 2. The window service scoring method based on image and speech recognition according to claim 1, characterized in that the LBP texture feature vector is input into the classifier to obtain the expression recognition result. 3. The window service scoring method based on image and voice recognition as claimed in claim 2, characterized in that the detected face image is specifically: detecting the image to be recognized through the AdaBoost face detection algorithm to obtain the face area , and crop out the face image. 4. The window service scoring method based on image and speech recognition according to claim 2, characterized in that the key point positioning adopts a supervised descent algorithm. 5. The window service scoring method based on image and speech recognition as claimed in claim 2, characterized in that the preprocessing includes: Perform uniform illumination judgment and illumination compensation on the face image to obtain a face image with uniform illumination; The uniformly illuminated face image is grayscaled to obtain a face grayscale image. 6. The window service scoring method based on image and speech recognition according to claim 2, characterized in that the histogram of the LBP image is statistically obtained and normalized to obtain the LBP texture feature vector. 7. A window service scoring system based on image and voice recognition, which is characterized by: A data acquisition module, which is configured to: acquire the user's video files and audio files during the window service process; An expression score acquisition module, which is configured to: convert video files into multiple images to be recognized, perform facial expression recognition, and obtain expression scores based on the recognition results; A voice score acquisition module configured to: convert audio files into voice text, compare the voice text with keywords in a preset database, and obtain a voice score based on the comparison results; A window service scoring module, which is configured to: obtain a window service score based on expression scoring and voice scoring; The process of facial expression recognition is: Obtain the image to be recognized and detect the face image; Preprocess the face image to obtain the face grayscale image; Locate key points in the grayscale image of the human face, and intercept image blocks of predetermined size centered on each key point; The LBP algorithm based on mean square error is used to extract the LBP histogram of each image block, and the LBP histograms of all image blocks are connected in a preset order to obtain the LBP texture feature vector of the face grayscale image; The LBP algorithm based on mean square error is specifically: Each pixel in the image block is used as the center pixel of the preset sliding window in turn; Calculate the absolute value of the mean difference between the gray value of each neighborhood pixel and the gray value of the central pixel in the sliding window; Calculate the mean square error of the gray value of each neighborhood pixel in the sliding window; The LBP image is obtained based on the absolute value and mean square error; The process of obtaining the LBP image based on the absolute value and mean square error is: When the absolute value is greater than the mean square error, the average value of the gray value of all neighbor pixels in the sliding window is selected as the threshold; otherwise, the gray value of the central pixel is selected as the threshold; Based on the threshold, the LBP value of each pixel in the image block is calculated to obtain an LBP image. 8. A computer-readable storage medium with a computer program stored thereon, characterized in that when the program is executed by a processor, the window service based on image and voice recognition as described in any one of claims 1-6 is implemented. Steps in the Scoring Method. 9. A computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the program, any of claims 1-6 is implemented. A step in the window service scoring method based on image and speech recognition.