CN109948433A - An embedded face tracking method and device - Google Patents

Abstract

The invention discloses a kind of embedded human face tracing method and devices, face image processing module carries out processing to the facial image that infrared camera acquires and obtains final face window, horizontal departure computing module calculates the horizontal departure of the central point of final face window center point abscissa and infrared camera acquisition image, and the rotation of PWM module control flaps machine head drives infrared camera rotation amendment deviation.The present invention follows technology using infrared camera, and the human face region detected can be made to be constantly in acquisition picture centre, and driver's face is avoided to cause missing inspection beyond acquisition image range.

Description

An embedded face tracking method and device

technical field

The invention belongs to the safe driving technology of automobiles, and in particular relates to a face tracking technology and device for detecting driver fatigue.

Background technique

With the increasing application level of science and technology, new technologies represented by "Internet +", big data, and Internet of Vehicles are changing people's traffic behavior and mode of transportation. Cars have become an indispensable transportation in people's life and work. tool. While the number of cars in our country is soaring, traffic accidents are also increasing. According to the traffic statistics of the Ministry of Public Security, excluding traffic accidents caused by subjective factors such as overloading of trucks and speeding, fatigue driving accounted for 14% of the causes of traffic accidents in 2018, which is a major "culprit" in objectively causing traffic accidents. At present, researches on fatigue detection of driver facial features emerge in an endless stream, and face detection is a key step in fatigue detection of facial features. Due to the movement of the face during the driving of the car, the face detection is faced with low real-time performance and is prone to false detection or missed detection.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide an embedded face tracking method and device, which can track the driver's face in real time, realize real-time detection, have high accuracy, and avoid false detection.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions: an embedded face tracking method, comprising the following steps:

Step 1: Collect the working image of the driver through the infrared camera, and convert the image into a grayscale image;

Step 2: Obtain a homogenized image by counting the occurrence times of each gray-level pixel of the gray-scale image generated in step 1, and using gray-scale homogenization processing;

Step 3: Use the frontal face detector to perform face detection on the homogenized image generated in step 2, obtain multiple primary face candidate windows, and apply non-maximum suppression NMS to merge highly overlapping primary face candidate windows;

Step 4: Divide the primary face candidate window processed by the non-maximum suppression NMS into two equal parts in the vertical direction, intercept the upper half of the image and input it into the trained deep convolutional neural network to judge whether the human eye exists, and reserve the detection The face candidate window to the eyes is used as the final face window;

Step 5: After steps 1-4, the final face window is obtained, the coordinates of the center point of the final face window are calculated, and the horizontal deviation between the abscissa of the center point of the final face window and the center point of the image captured by the infrared camera is calculated;

The horizontal deviation is the difference between the abscissa x of the center point of the candidate face window and the abscissa w ₁ of the center point of the image captured by the infrared camera, which is calculated by normalization, and its formula is:

Δ=((xw ₁ )/w ₁ -a ₁ )*a ₂ (8)

Among them, x represents the abscissa value of the center point of the candidate face window, w ₁ represents the abscissa value of the center point of the image captured by the infrared camera, when a ₁ is 0.5 and a ₂ is 2, the value of Δ will be normalized to [- 1, 1] range;

Step 6: Send the horizontal deviation obtained in Step 5 to the Arduino Leonardo microcontroller embedded in the LattePanda control board through serial communication, and the Arduino Leonardo microcontroller controls the steering gear gimbal to rotate the steering gear gimbal under the infrared camera through PWM to correct the deviation;

The correction deviation of the servo gimbal is to minimize the normalized horizontal deviation, thereby updating the rotation angle of the servo gimbal. The formula is:

angle=ag+T*Δ (9)

Among them, angle is the rotation angle of the servo gimbal after the update, ag is the original rotation angle of the servo gimbal, T is the rotation factor, and Δ is the horizontal deviation between the abscissa of the center point of the candidate face and the center point of the infrared acquisition image.

Optionally, the window position information is denoted as (x ₁ , y ₁ , w, h), where x ₁ , y ₁ represent the upper left, top and bottom coordinate information of the final face window, and w, h represent the length and width of the final face window , calculate the coordinates of the center point of the final face window, the formula is:

Optionally, T=0.85.

Optionally, in the grayscale uniformization process in step 2, firstly count the number of pixels of each grayscale level of the input grayscale image, and the formula is as follows:

h(r _k )=n _k ,k=0,1,2,...,255 (1)

Among them, r _k represents the k-th gray level, and n _k represents the number of pixel values corresponding to the k-th gray level in the image;

Then calculate the gray level probability value, and its formula is:

Among them, n _k represents the number of pixel values corresponding to the k-th grayscale in the image, and n represents the total number of pixel points in the image;

Finally, the gray value of each gray level is updated, and its formula is:

S=255*s(r _k ) (4)

Among them, p(r _k ) is the gray level probability value of the k-th level, s(r _k ) represents the sum of the probabilities of the first k gray levels, and S represents the gray level after gray level equalization.

Optionally, the deep convolutional neural network in step 4 has a structure consisting of 1 data layer, 4 convolution layers, 3 pooling layers, and 1 output layer; The image training data of different states of the human eye with a rate of 20*20 pixels is extracted, and the main features of the training data are extracted. The formula is:

Where X represents the training data, the matrix size is 16000×400, each row represents a training sample, and C represents the covariance matrix of the training data; the covariance matrix is a symmetric matrix, and its characteristic matrix is calculated. The formula is:

P=(e ₁ e ₂ ... e _n ) ^T (6)

Among them, e _i represents the feature column vector, and λ _i is the feature value; by extracting the main contour features of the training image, the amount of calculation is reduced and the recognition accuracy is improved.

The invention also provides an embedded face tracking device, comprising a base, a LattePanda control board and a steering gear pan-tilt fixed on the base, a camera bracket connected with the steering gear pan-tilt, and an infrared camera mounted on the camera bracket, The infrared camera is connected with the LattePanda control board through USB, and the LattePanda control board is provided with a face image processing module, a horizontal deviation calculation module and a PWM module, and the face image processing module processes the face image collected by the infrared camera to obtain In the final face window, the horizontal deviation calculation module calculates the horizontal deviation between the abscissa of the center point of the final face window and the center point of the image captured by the infrared camera, and the PWM module controls the rotation of the steering gear pan and tilt to drive the infrared camera to rotate to correct the deviation.

The present invention adopts the above technical scheme, and obtains the facial feature image of the driver through the image gray level homogenization technology, the frontal face detection technology, the deep convolutional neural network image recognition technology and the infrared camera following technology. Has the following advantages:

1. The infrared camera is used to ensure that clear images can be collected during the day or night for face detection.

2. The grayscale equalization technology is used to process the collected images, which is beneficial to increase the image contrast and improve the accuracy of frontal face detection.

3. Through the trained deep convolutional neural network to identify the existence of eyes, the face candidate window that can be detected is correct, which reduces the false detection rate of face detection.

4. Using the infrared camera follow-up technology, the detected face area can always be in the center of the collected image, so as to avoid missed detection caused by the driver's face exceeding the range of the collected image.

The specific technical solutions of the present invention and the beneficial effects thereof will be described in detail in the following specific embodiments in conjunction with the accompanying drawings.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments:

Fig. 1 is the schematic flow chart of face detection of the present invention;

2 is a schematic structural diagram of a deep convolutional neural network of the present invention;

Figure 3 is a schematic diagram of the structure of the device of the present invention;

In Figure 3: 1. Infrared camera, 2. Camera bracket, 3. LattePanda control board, 4. Servo head, 5. Base.

Detailed ways

Example 1

Referring to Figure 1, an embedded face tracking method includes the following steps:

Step 1: Collect the driver's working image through the infrared camera, convert the image into a grayscale image, and enter it as the following steps;

Further, there are 16 infrared LEDs around the lens of the infrared camera, the resolution of the collected image is 200*200 pixels, and the image content includes the part above the driver's shoulder.

Step 2: By counting the occurrences of each gray-level pixel of the gray-scale image generated in step 1, and using the gray-scale homogenization process, a homogenized image is obtained, which is input as the following steps;

Further, in the grayscale uniformization process, firstly, the number of pixels of each grayscale level of the input grayscale image is counted, and the formula is as follows:

h(r _k )=n _k ,k=0,1,2,...,255 (10)

Among them, r _k represents the k-th gray level, and n _k represents the number of pixel values corresponding to the k-th gray level in the image; then calculate the gray level probability value, and its formula is:

Among them, n _k represents the number of pixel values corresponding to the k-th gray level in the image, and n represents the total number of pixel points in the image; finally, the gray value of each gray level is updated, and the formula is:

S=255*s(r _k ) (13)

Among them, p(r _k ) is the gray level probability value of the k-th level, s(r _k ) represents the probability sum of the first k gray levels, and S represents the gray level after gray leveling.

Step 3: Use the frontal face detector to perform face detection on the homogenized image generated in step 2, obtain multiple primary face candidate windows, and apply non-maximum suppression NMS to merge highly overlapping primary face candidate windows;

Further, the described frontal face detector is to input the Haar-Like feature extracted from the image into the trained AdaBoost cascade classifier for identification, wherein the Haar-Like feature represents the change of the gray value of the image; The suppression NMS merges the face candidate windows detected by the frontal face detector according to the rules. The rule is: when the detected candidate face window overlap is higher than 50%, delete the face candidate window with low confidence. ; When it is detected that a face candidate window is a sub-window of another face candidate window, delete the sub-window.

Step 4: Divide the primary face candidate window processed by the non-maximum suppression NMS into two equal parts in the vertical direction, intercept the upper half of the image and input it into the trained deep convolutional neural network to judge whether the human eye exists, and reserve the detection The face candidate window to the eyes is used as the final face window;

Further, the described deep convolutional neural network, as shown in Figure 2, has a structure consisting of 1 data layer, 4 convolutional layers, 3 pooling layers, and 1 output layer; in the training process, 16000 The image training data of different states of the human eye with a resolution of 20*20 pixels is extracted, and the main features of the training data are extracted. The formula is:

Where X represents the training data, the matrix size is 16000×400, each row represents a training sample, and C represents the covariance matrix of the training data; the covariance matrix is a symmetric matrix, and its characteristic matrix is calculated. The formula is:

P=(e ₁ e ₂ ... e _n ) ^T (6)

where e _i represents the eigencolumn vector, and λ _i is the eigenvalue.

Step 5: After steps 1-4, the final face window is obtained, the coordinates of the center point of the final face window are calculated, and the horizontal deviation between the abscissa of the center point of the final face window and the center point of the image captured by the infrared camera is calculated;

The window position information is recorded as (x ₁ , y ₁ , w, h), where x ₁ , y ₁ represent the top left, top and bottom coordinate information of the final face window, w, h represent the length and width of the final face window, and calculate the final face window. The coordinates of the center point of the face window, the formula is:

The horizontal deviation is calculated by normalizing the difference w ₁ between the abscissa x of the center point of the candidate face window and the abscissa of the center point of the image captured by the infrared camera. The formula is:

Δ=((xw ₁ )/w ₁ -a ₁ )*a ₂ (5)

Among them, x represents the abscissa value of the center point of the candidate face window, and w ₁ represents the abscissa of the center point of the image collected by the infrared camera. For the convenience of calculation, the present invention adopts normalization to deal with the horizontal deviation, and only when a ₁ =0.5, a ₂ = 2, the horizontal deviation data range can be compressed to [-1, 1].

Step 6: Send the horizontal deviation obtained in Step 5 to the Arduino Leonardo MCU embedded in the LattePanda control board through serial communication. The Arduino Leonardo MCU controls the rotation of the servo pan/tilt under the infrared camera to correct the deviation through PWM.

Further, the steering gear gimbal correction deviation in the step 6 is to minimize the horizontal deviation, thereby updating the steering gear gimbal rotation angle, and its formula is:

angle=ag+T*Δ (6)

Among them, angle is the rotation angle of the servo gimbal after the update, ag is the original rotation angle of the servo gimbal, T is the rotation factor, and Δ is the horizontal deviation between the abscissa of the center point of the candidate face and the center point of the infrared acquisition image.

When there is a horizontal deviation between the center point of the final face window and the center point of the collected image, when T=0.85, the calculation and update of the pan-tilt angle angle can quickly respond to the deviation.

The method of the invention has simple steps, strong operability, and can realize a good face detection function.

Embodiment 2

Referring to Figure 3, an embedded face tracking device includes a base, a LattePanda control board and a steering gear gimbal fixed on the base, a camera bracket connected to the steering gear gimbal, and an infrared camera mounted on the camera bracket. The infrared camera and the LattePanda control board are connected via USB, and the LattePanda control board is provided with a face image processing module, a horizontal deviation calculation module and a PWM module, and the face image processing module processes the face image collected by the infrared camera The final face window is obtained, and the horizontal deviation calculation module calculates the horizontal deviation between the abscissa of the center point of the final face window and the center point of the image captured by the infrared camera, and the PWM module controls the rotation of the steering gear pan and tilt to drive the infrared camera to rotate to correct the deviation.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the above specific embodiments. Any modifications that do not depart from the functional and structural principles of the present invention are intended to be included within the scope of the claims.

Claims (6)

1. a kind of embedded human face tracing method, it is characterised in that include the following steps:

Step 1: driver's work image being acquired by infrared camera, and converts the image into grayscale image；

Step 2: through each gray-level pixels frequency of occurrence for the grayscale image that statistic procedure 1 generates, at Histogram equalization Reason obtains homogenization image；

Step 3: Face datection being carried out to the homogenization image that step 2 generates using obverse face detection device, obtains multiple primary Face candidate window, and application non-maximum restraining NMS merges the primary face candidate window of high superposed；

Step 4: will be halved in the longitudinal direction by the primary face candidate window of non-maximum restraining NMS processing, in interception Trained depth convolutional neural networks carry out judging that human eye whether there is the input of half part area image, and reservation detects The face candidate window of eyes is as final face window；

Step 5: passing through step 1-4, obtain final face window, calculate the coordinate of final face window center point, Yi Jiji Calculate the horizontal departure of the central point of final face window center point abscissa and infrared camera acquisition image；Wherein horizontal departure It is by the central point abscissa w of candidate face window center point abscissa x and infrared camera acquisition image₁Difference, by returning One changes calculating and obtains, formula are as follows:

Δ=((x-w₁)/w₁-a₁)*a₂ (8)

Wherein, x represents candidate face window center point abscissa value, w₁Infrared camera acquisition image center abscissa is represented, Work as a₁Value 0.5, a₂When value 2, Δ value will be normalized to [- 1,1] range；

Step 6: the obtained horizontal departure of step 5 being sent to LattePanda control panel embeds by serial communication Arduino Leonardo single-chip microcontroller, Arduino Leonardo single-chip microcontroller rotate infrared photography by PWM control flaps machine head Steering engine cloud platform rotation under head corrects deviation；

Steering engine holder amendment deviation is that horizontal departure will minimize after normalizing, to update steering engine holder corner, formula are as follows:

Angle=ag+T* Δ (9)

Wherein, angle is steering engine holder corner after updating, and ag is the original corner of steering engine holder, and T is twiddle factor, and Δ is candidate Face center abscissa and infrared collecting image center abscissa horizontal departure.

2. a kind of embedded human face tracing method according to claim 1, it is characterised in that: window position information is denoted as (x₁,y₁, w, h), wherein x₁,y₁Indicate that bottom coordinate information is pushed up in final face window upper left, w, h indicate the length of final face window And width, calculate the coordinate of final face window center point, formula are as follows:

3. a kind of embedded human face tracing method according to claim 1, it is characterised in that: T=0.85.

4. a kind of embedded human face tracing method according to claim 1, it is characterised in that: the gray scale of the step 2 is equal Processing is homogenized, each gray-level pixels number of the gray scale picture of input is counted first, formula are as follows:

h(r_k)=n_k, k=0,1,2 ..., 255 (1)

Wherein, r_kIndicate kth grade gray level, n_kIndicate the number of pixel value corresponding to kth grade gray scale in image；

Then gray level probability value, formula are calculated are as follows:

Wherein, n_kIndicate the number of pixel value corresponding to kth grade gray scale in image, n indicates the total number of pixel in image；

The gray value of each gray level of final updating, formula are as follows:

S=255*s (r_k) (4)

Wherein, p (r_k) it is kth grade gray probability value, s (r_k) indicating preceding k gray level probability summation, S is indicated after being homogenized into gray scale Gray level.

5. a kind of embedded human face tracing method according to claim 1, it is characterised in that: the depth of the step 4 is rolled up Product neural network, structure is by 1 layer data layer, 4 layers of convolutional layer, 3 layers of pond layer, 1 layer of output layer composition；In the training process, Use 16000 resolution ratio for the human eye different conditions image training data of 20*20 pixel, and it is mainly special to extract training data Sign, formula are as follows:

Wherein X indicates that training data, matrix size are 16000 × 400, and every a line indicates that a training sample, C indicate training number According to covariance matrix；It is symmetrical matrix using covariance matrix, calculates its eigenmatrix, formula are as follows:

P=(e₁ e₂ ... e_n)^T (6)

Wherein e_iIndicate feature column vector, λ_iIt is characterized value；It is extracted, is subtracted by carrying out principal outline feature to training image Few calculation amount, improves recognition accuracy.

6. a kind of embedded human face tracking device, it is characterised in that: controlled including pedestal, the LattePanda being fixed on pedestal Plate and steering engine holder, the camera bracket being connect with steering engine holder, the infrared camera being installed on camera bracket, it is described red Outer camera is connect with LattePanda control panel by USB, and the LattePanda control panel is equipped with face image processing mould Block, horizontal departure computing module and PWM module, the facial image that the face image processing module acquires infrared camera Carry out processing and obtain final face window, the horizontal departure computing module calculate final face window center point abscissa with it is red The horizontal departure of the central point of outer camera collection image, the PWM module control flaps machine head rotation drive infrared camera Rotation amendment deviation.