CN104077579B - Facial expression recognition method based on expert system - Google Patents

Abstract

The present invention relates to a kind of facial expression recognition method based on expert system, the method makes inferences identification human face expression by setting up the expert system on the basis of facial expression image processing method and traditional computer program function to the image for pre-processing, and the described method comprises the following steps：1) image is caught from video, the user profile in the video is obtained, then by image procossing, image characteristics extraction carries out authentication, obtains the facial expression image characteristic parameter of user, determine user's expression storehouse, set up identification human face expression expert system；2) image procossing and image characteristics extraction are carried out to the image caught from video, obtain characteristic parameter when user's expression degree is maximized, by characteristic parameter and step 1) determine user expression storehouse in expression training sample parameter compare, by the statistics of the reasoning final output Expression Recognition of inference engine of expert system.Compared with prior art, the present invention has the advantages that recognition speed is fast.

Description

Image Recognition Method of Facial Expression Based on Expert System

technical field

The invention relates to an expert system application technology, in particular to an expert system-based facial expression image recognition method.

Background technique

Expert system is a kind of computer intelligent program system with specialized knowledge and experience. By modeling the problem-solving ability of human experts, it uses knowledge representation and knowledge reasoning techniques in artificial intelligence to simulate complex problems that are usually solved by experts. Reach a level of problem-solving ability equivalent to that of an expert. This knowledge-based system design method is centered on knowledge base and inference engine. It separates knowledge from other parts of the system. Expert systems emphasize knowledge rather than methods. There are no algorithm-based solutions for many problems, or the algorithm solutions are too complex. Using expert systems can take advantage of the rich knowledge of human experts, so expert systems are also called knowledge-based systems.

At present, with the development of science and technology, smart beds for rehabilitation care are becoming more and more popular. However, among the patients, a large part of them cannot accurately express their intentions to the outside world through body or language. For each person, different expressions represent different states. When language and body cannot convey, we can recognize the patient's thoughts through facial expressions and complete corresponding operations. Nursing bed operators on the market are all nursing staff, ignoring the patient's own wishes. Based on the people-oriented design concept, through the expert system of expression recognition technology, patients with limited mobility can also operate the nursing bed by themselves.

Contents of the invention

The object of the present invention is to provide a human facial expression image recognition method based on an expert system that has a learning function and can quickly and accurately recognize human facial expressions in order to overcome the above-mentioned defects in the prior art.

The purpose of the present invention can be achieved through the following technical solutions:

A facial expression image recognition method based on an expert system, which uses an expert system for facial expression recognition based on the facial expression image processing method and traditional computer program functions to perform inference on the preprocessed image and recognize the human face Expression, described method comprises the following steps:

1) capture the image from the video, obtain the user information in the video, then through image processing, image feature extraction, identity verification, obtain the user's expression image feature parameters, determine the user expression database, and establish an expert system for recognizing facial expressions;

2) carry out image processing and image feature extraction to the image captured from video, obtain the feature parameter when user's expression degree maximizes, the feature parameter is compared with the expression training sample parameter in the user's expression storehouse that step 1) determines, After the reasoning of the expert system reasoning machine, the statistical results of facial expression recognition are finally output.

Described step 1) in setting up the recognition human facial expression image expert system to specifically include steps:

11) Obtaining facial expression images;

12) Preprocessing the facial expression image;

13) Extracting facial expression image features;

14) Establish a rule base of the facial expression expert system and store the feature parameters extracted in step 13) into the rule base.

Described step 2) in expression recognition, obtain facial expression image recognition result and specifically include steps:

21) Acquiring the user's expression image: after the video information is received, capture an image from the video information to obtain the user's expression image.

22) to the face expression image that obtains in step 21), carry out image preprocessing;

23) carry out feature extraction to the image of eye and mouth that obtains in step 22);

24) facial expression recognition: the feature parameter obtained in step 23) is input in the recognition facial expression image expert system and compares with the facial expression feature parameter stored in the expert system rule base, and outputs through the inference of the inference engine in the expert system reasoning results.

Described step 12) and step 22) and the preprocessing of facial expression image in and specifically include image denoising, scale normalization, grayscale normalization, image segmentation and image binarization;

After the image is denoised, the image g(i, j) obtained after denoising is:

g(i,j)=∑f(i,j)/N, (i,j)∈M

Among them: f(i, j) is a given image containing noise, M is the coordinates of each adjacent pixel in the selected neighborhood, and N is the number of adjacent pixels included in the neighborhood;

After the scale is normalized, the target image g(x, y) is obtained as:

g(x, y)=f(x/a+x ₀ , y/a+y ₀ )

Where: f(x, y) is the image before normalization, (x ₀ , y ₀ ) is the center of gravity of the image target area, a is the scale factor, and the scale factor is:

a=sqrt(T/m)

Among them: m is the area of the target image; T is the area of the image before normalization

The grayscale normalization is a piecewise linear grayscale transformation method;

The image segmentation technology is a serial area segmentation technology, and the image is segmented through direct detection of the mouth and eye target areas.

The image binarization algorithm is a binarization method Otsu algorithm, which uses a gray threshold to divide the gray image into two types: target part and background part.

Described step 13) and 33) in facial expression image feature comprise left eye feature, right eye feature and mouth feature, described human facial expression image feature extraction algorithm specifically comprises the following steps:

201) Calculate the correlation matrix M for each pixel:

in:

Wherein: I _x is the difference in the x direction, I _y is the difference in the y direction, and w(x, y) is a Gaussian function, specifically:

202) Calculate Harris corner response per pixel: R=(AB-CD) ² -k(A+B) ² ;

203) Find the maximum value point within the w*w range, if the Harris corner point response is greater than the threshold value, and the threshold value is the maximum value within the w*w range, it will be regarded as a corner point, and the features of the eyes and mouth will be extracted come out.

After the step 24) is finished, the system adds the task data of this time into the rule base.

The image binarization process specifically includes steps:

101) Calculate the average grayscale u of the image as:

u＝∑i*n(i)/(M*N)

Wherein: M*N is the number of pixels of the image, and n(i) is the number of pixels whose grayscale is i;

102) Determine the grayscale threshold t, t is the segmentation threshold of the foreground and background (the grayscale threshold t is the value that makes the inter-class variance G the largest), and the inter-class variance G is:

G＝w ₁ *(u ₁ -u)*(u ₁ -u)+w ₂ *(u ₂ -u)*(u ₂ -u)

When the variance G is the largest, it can be considered that the difference between the foreground and the background is the largest at this time, that is, the gray level at this time is the optimal threshold.

Among them: w1 is the proportion _of the target pixel in the image, _w2 is the proportion of the background pixel in the image, u1 is the average grayscale _of the target pixel, _u2 is the average grayscale of the background pixel, the proportion of the target pixel in the image w1 and the average gray level u1 _of the target pixel are respectively _:

w ₁ =W ₁ /(M*N) u ₁ =∑i*n(i)/W ₁ , i>t

Among them: W ₁ is the statistical number whose gray value is greater than t,

_The ratio w2 of the background pixel to the image and the average grayscale _u2 of the background pixel are respectively:

w ₂ =W ₂ /(M*N) u ₂ =∑i*n(i)/W ₂ , i<t

Among them: W ₂ is the statistical number whose gray value is less than t;

103) Binarize the image by using pixels with grayscale values greater than t as target pixels and pixels with grayscale values smaller than t as background pixels.

Compared with the prior art, the present invention has the following advantages:

1) The present invention proposes to set up an expert system for identifying human facial expressions. The expert system is a program system with a large amount of specialized knowledge and experience. It fully considers the number of training samples and their category information, and obtains better recognition results. Recognition provides an effective approach.

2) the present invention proposes a kind of fast recognition method of human facial expression, proposes the new human facial expression recognition method that not only has faster speed but also has higher recognition rate for recognizing human facial expression under the video environment.

3) An expert system is built on the basis of facial expression image processing methods and traditional computer program functions. The facial expression image processing expert system does not replace the classic methods and traditional programs that already have strong functions and have reached a fairly high level, but on the basis of making full use of its existing achievements, it focuses on dealing with the part of the problem that is still difficult to solve.

Description of drawings

Fig. 1 is the structural representation of the expert system of a kind of recognition facial expression image provided by the present invention;

Fig. 2 is a three-section linear transformation function diagram in the gray scale normalization;

Figure 3 is a schematic diagram of the sensitivity of the Harris operator to scale.

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

A kind of facial expression image recognition method based on expert system, this method comprises the following steps:

1) capture the image from the video, obtain the user information in the video, then through image processing, image feature extraction, identity verification, obtain the user's expression image feature parameters, determine the user expression database, and establish an expert system for recognizing facial expressions;

2) carry out image processing and image feature extraction to the image captured from video, obtain the feature parameter when user's expression degree maximizes, the feature parameter is compared with the expression training sample parameter in the user's expression storehouse that step 1) determines, After the reasoning of the expert system reasoning machine, the statistical results of facial expression recognition are finally output.

In the step 1), the expert system for identifying facial expression images is set up to specifically include steps:

11) Obtaining facial expression images;

12) Preprocessing the facial expression image;

13) Extracting facial expression image features;

14) Establish a rule base of the facial expression expert system and store the feature parameters extracted in step 13) into the rule base.

In step 2), expression recognition obtains facial expression image recognition result and specifically includes steps:

21) Acquiring the user's expression image: after the video information is received, capture an image from the video information to obtain the user's expression image.

22) to the face expression image that obtains in step 21), carry out image preprocessing;

23) carry out feature extraction to the image of eye and mouth that obtains in step 22);

24) facial expression recognition: the feature parameter obtained in step 23) is input in the recognition facial expression image expert system and compares with the facial expression feature parameter stored in the expert system rule base, and outputs through the inference of the inference engine in the expert system reasoning results.

As shown in Figure 1, it is a schematic diagram of the working structure of the present invention. The expert system of the present invention has a learning function. After step 24), this data is added to the rule base, further increasing the work efficiency and recognition accuracy of the present invention.

The preprocessing of the facial expression image in step 12) and step 22) specifically includes image denoising, scale normalization, grayscale normalization, image segmentation and image binarization.

The facial expression image features in steps 13) and 33) include left eye features, right eye features and mouth features.

The present invention will be further described below in conjunction with example.

1) Establish a facial expression image expert system;

After the video information is received, the image is captured from the video information, and the user information of the video information can be obtained. Through image preprocessing and image feature extraction, identity verification is performed to determine the user's expression library and establish an expert system for recognizing facial expressions. Extract when waiting for facial expression recognition;

11) Facial expression image acquisition

Obtain a static image of a certain facial expression through a camera image capture tool;

12) Facial expression image preprocessing

A image denoising

The ultimate goal of image denoising is to improve a given image and solve the problem of image quality degradation caused by noise interference in actual images. The denoising technology can effectively improve the image quality, increase the signal-to-noise ratio, and better reflect the information carried by the original image. Based on the C# programming language, the image denoising algorithm in the present invention adopts the traditional spatial domain filtering method. Spatial filtering is to directly perform data calculation on the original image, and process the gray value of the pixel. The spatial domain image denoising algorithm adopted in the present invention is a neighborhood averaging method. The neighborhood averaging method is expressed by a mathematical formula: let f(i, j) be a given image containing noise, and the image after neighborhood averaging processing is g(i, j), then g(i, j)=∑ f(i, j)/N, (i, j)∈M, M is the coordinates of each adjacent pixel in the selected neighborhood, and is the number of adjacent pixels contained in the neighborhood. Neighborhood averaging method reduces noise at the cost of image blur, and the larger the template size, the more significant the effect of noise reduction. If f(i, j) is a noise point, and the gray level of its adjacent pixels is very different from it, the neighborhood average method is to replace it with the average value of adjacent pixels, which can significantly weaken the noise point and make the neighborhood The gray scale is close to uniform, which plays a role in smoothing the gray scale.

B scale normalization

Image translation scale normalization refers to eliminating the influence of translation and scaling on the image through transformation. Based on the C# programming language, the method of standard moments is used for scale normalization in the present invention. First, the coordinate origin is moved to the center of gravity of the image, and the center of gravity (x ₀ , y ₀ ) of the target can be obtained from the standard moments. Since the center of gravity of the object is invariant to translation, scale and rotation, placing the origin of the image on the center of gravity of the object solves the problem of translation. Then define a scale factor a to solve the scale problem: a=sqrt(T/m). In fact, if the target pixel value on the binary image is 1, the background has a pixel value of 0, and m is the area of the target, so that the area of the target is a fixed size, so that the scale normalization can be obtained through the following standard transformation Objective: g(x,y)=f(x/a+x ₀ , y/a+y ₀ ). This enlarges the area of the face.

C grayscale normalization

Grayscale normalization is used to improve image quality. Based on the C# programming language, we adopt the most basic piecewise linear grayscale transformation method in the present invention. The piecewise linear transformation is also called grayscale linear stretching, which uses three-section linear transformation. As shown in Figure 2, the grayscale interval [a, b] is expanded, while the grayscale intervals [0, a] and [b, c] are compressed. By carefully adjusting the position of the inflection point of the polyline and controlling the slope of the segmented straight line, any gray scale interval can be expanded and compressed to achieve gray scale normalization. Convert color images of human facial expressions to grayscale images.

D image segmentation

The places that can best reflect the changes of facial expressions in the human face are the mouth and the eyes, so the parts extracted after the image segmentation process in the present invention are the eyes and the mouth. Based on the C# programming language, what we use in the present invention is the serial area segmentation technology, which is the technology of segmenting the image through direct detection of the target area in a serial manner. Its characteristics are: the whole processing process is decomposed into multiple steps in sequence, and the processing of the subsequent steps is determined according to the processing results of the previous steps. We start from the full image and then gradually split into the desired segmentation regions.

E image binarization

Based on the C# programming language, the binarization method in the present invention we adopt is the classic binarization method Otsu algorithm. The basic idea of this algorithm is: Suppose a certain threshold is used to divide the grayscale image into target part and background part according to the gray scale. is the optimal binarization threshold. For an image of N*M pixels, first calculate the average gray level u of the image, and obtain the number u(i) of pixels corresponding to gray level i in all images, so the average gray level value u of the image is =∑i*n(i)/(M*N); List the relevant variables for solving the optimal threshold t, record t as the segmentation threshold between the target and the background, record the proportion of the target pixel (grayscale greater than t) in the image as w ₁ : w ₁ =W ₁ /(M*N), where W ₁ is the statistical number whose gray value is greater than t; record the average gray level of the target pixel as u ₁ : u ₁ =∑i*n(i)/ W ₁ , i>t, similarly, the ratio w ₂ of the background pixels to the image and the average gray level u ₂ of the background pixels are obtained. Finally, to solve the optimal threshold t is to maximize the class difference, that is, to maximize G=w ₁ *(u ₁ *u)*(u ₁ -u)+w ₂ *(u ₂ *u)*(u ₂ -u) . When G is the largest, the optimal threshold is obtained.

13) Feature extraction of facial expression images

The images of eyes and mouth are obtained through image segmentation in the above step E, feature extraction is performed using geometric features, position changes of eyes and mouth are positioned and measured, and characteristics such as size, distance, shape and mutual proportion are determined. Carry out in the present invention in the face expression recognition algorithm of table, need to extract geometric feature, what we adopted is the Harris operator corner point extraction algorithm, is based on C# programming language, but these detection algorithms need some threshold value settings, and every piece of image The number of intersection points is different. Extract the corners of the eyes, manually fix and select the same N points (four points in the left and right corners of the eyes, and the upper and lower eyelids):

The first step is to calculate the correlation matrix M for each pixel. The Harris operator replaces the binary window function with a Gaussian function, and assigns a greater weight to the pixel closer to the center point to reduce the impact of noise.

in,

Wherein: I _x is the difference in the x direction, I _y is the difference in the y direction, and w(x, y) is a Gaussian function, specifically:

The second step computes the Harris corner response per pixel.

R＝(AB-CD) ² -k(A+B) ²

The third step is to find the maximum value point in the range of w*w, as shown in Figure 3, if the Harris corner point response is greater than the threshold value, the threshold value is generally the maximum value in the range of w*w, it is regarded as a corner point. In this way, the features of the eyes and mouth are extracted.

Human beings have six main emotions, and each emotion reflects a unique psychological activity of a person with a unique expression. These six emotions are called basic emotions and consist of anger, smile, sadness, surprise, disgust and fear. The facial expression images representing these six expressions are collected, and after processing in steps 12) and 13), the characteristic parameters representing different expressions are stored in the rule base. The characteristic parameters of each expression are replaced by corresponding letters. For example, the characteristic parameters representing happiness are marked with happiness.

In this way, through the above steps, an expert system for recognizing human facial expressions is established.

(2) Facial expression recognition to obtain facial expression image recognition results

21) Obtain the user's expression image:

After the video information is received, an image is captured from the video information to obtain the user's facial expression image.

22) Image preprocessing:

For the facial expression image obtained in step 21), first through step 12), the image is preprocessed, the face is detected, the face is positioned, then the image is cut, and finally the eyes and mouth of the image are obtained part of the image features.

23) Image feature extraction:

Perform feature extraction on the images of the eyes and mouth obtained in the above step (2), and use the geometric method in the above step (3) to extract the feature parameters of the eyes and mouth.

24) Expression recognition:

Input the characteristic parameters obtained in the step into the expert system for recognizing facial expression images, compare them with the facial expression characteristic parameters stored in the expert system rule base, and then pass the reasoning of the reasoning machine in the expert system, and finally the expert system outputs the reasoning results . For example, if the input is a happy facial expression image, the output result of the expert system is smile.

As shown in Fig. 1, the expert system of the present invention has a learning function. After the recognition is completed, the current data is also added to the rule base. At the same time, experts can also manually update the rule base and programs of the expert system.

Claims (4)

1. A facial expression image recognition method based on an expert system is characterized in that the method carries out reasoning recognition of facial expressions on preprocessed images through the expert system for facial expression recognition which is established on the basis of an expression image processing method and the functions of a traditional computer program, and the method comprises the following steps:

1) capturing an image from the video, obtaining user information in the video, then performing image processing, image feature extraction and identity verification to obtain expression image feature parameters of a user, determining a user expression library, establishing a facial expression recognition expert system,

2) performing image processing and image feature extraction on an image captured from a video to obtain a feature parameter when the expression degree of a user is maximized, comparing the feature parameter with an expression training sample parameter in the user expression library determined in the step 1), and finally outputting a statistical result of expression recognition through reasoning of an expert system reasoning machine;

the expert system for identifying the facial expression image established in the step 1) specifically comprises the following steps:

11) the image of the facial expression is acquired,

12) the pre-processing is carried out on the facial expression image,

13) the characteristics of the facial expression image are extracted,

14) establishing a rule base of the facial expression expert system and storing the characteristic parameters extracted in the step 13) into the rule base;

the step 2) of obtaining the facial expression image recognition result through the expression recognition specifically comprises the following steps:

21) acquiring an expression image of a user: after the video information is received, capturing images from the video information, acquiring expression images of the user,

22) carrying out image preprocessing on the facial expression image acquired in the step 21),

23) extracting the features of the eye and mouth images obtained in step 22),

24) and (3) expression recognition: inputting the characteristic parameters obtained in the step 23) into an expert system for recognizing the facial expression images, comparing the characteristic parameters with facial expression characteristic parameters stored in an expert system rule base, and outputting an inference result through inference of an inference engine in the expert system;

the steps 12) and 22) and the preprocessing of the median facial expression image specifically comprise image denoising, scale normalization, gray level normalization, image segmentation and image binarization,

the image g (i, j) after denoising is obtained as follows:

g(i,j)＝∑f(i,j)/N,(i,j)∈M

wherein: f (i, j) is the given noise-containing image, M is the coordinates of each neighboring pixel in the neighborhood taken, N is the number of neighboring pixels contained in the neighborhood,

obtaining a target image g (x, y) after the scale normalization is as follows:

g(x,y)＝f(x/a+x₀,y/a+y₀)

wherein: f (x, y) is the image before normalization, (x)₀,y₀) The center of gravity of a picture target area is shown, a is a scale factor, and the scale factor is as follows:

a＝sqrt(T/m)

wherein: m is the area of the target image; t is the area of the image before normalization;

the gray scale is normalized to a piecewise linear gray scale transform method,

the image segmentation technique is a serial region segmentation technique, which segments the image by direct detection of the target region of the mouth and eyes,

the image binarization algorithm is a binarization method Otsu algorithm, and a gray level image is divided into a target part and a background part by using a gray level threshold value.

2. The expert system-based facial expression image recognition method according to claim 1, wherein the facial expression image features in steps 13) and 33) include left-eye features, right-eye features and mouth features, and the facial expression image feature extraction algorithm specifically includes the following steps:

201) calculating a correlation matrix M for each pixel:

$M=\left[\begin{array}{cc}A& D\\ C& B\end{array}\right]$

wherein:

wherein: i is_xIs the difference in the x direction, I_yFor the difference in the y direction, w (x, y) is a gaussian function, specifically:

$w(x,y)=\frac{1}{2{\mathrm{\&pi;\&sigma;}}^2}{e}^{-(x^2+y^2)/2{\mathrm{\&sigma;}}^2}$

202) calculating Harris angular point response of each pixel point: r ═ AB-CD)²-k(A+B)²；

203) And searching a maximum value point in the w-w range, and if the Harris angular point response is greater than a threshold value which is the maximum value in the w-w range, regarding the Harris angular point response as an angular point, and extracting the characteristics of the eyes and the mouth.

3. The expert system based facial expression image recognition method as claimed in claim 1, wherein after the step 24) is finished, the system adds the task data to the rule base.

4. The expert system-based facial expression image recognition method as claimed in claim 1, wherein the image binarization process specifically comprises the steps of:

101) the average gray level u of the image is calculated as:

u＝∑i*n(i)/(M*N)

wherein: m × N is the number of pixels of the image, and N (i) is the number of pixels with the gray level of i;

102) determining a gray threshold t, wherein the gray threshold t is a value which enables the inter-class variance G to be maximum, and the inter-class variance G is as follows:

G＝w₁*(u₁-u)*(u₁-u)+w₂*(u₂-u)*(u₂-u)

wherein: w is a₁Is the proportion of the target pixel in the image, w₂Is the proportion of background pixels to the image, u₁Is the average gray level of the target pixel, u₂The average gray scale of the background pixel is the proportion w of the target pixel to the image₁And the average gray level u of the target pixel₁Respectively as follows:

w₁＝W₁/(M*N) u₁＝Σi*n(i)/W₁,i＞t

wherein: w₁Is a statistic of gray values greater than t,

the proportion w of the background pixels to the image₂And average gray level u of background pixel₂Respectively as follows:

w₂＝W₂/(M*N) u₂＝Σi*n(i)/W₂,i＜t

wherein: w₂The statistics of the gray value less than t;

103) and taking the pixel with the gray value larger than t as a target pixel, and taking the pixel with the gray value smaller than t as a background pixel to carry out binarization on the image.