CN110728206A - Fatigue driving detection method and device, computer readable storage medium and terminal - Google Patents
Fatigue driving detection method and device, computer readable storage medium and terminal Download PDFInfo
- Publication number
- CN110728206A CN110728206A CN201910904247.9A CN201910904247A CN110728206A CN 110728206 A CN110728206 A CN 110728206A CN 201910904247 A CN201910904247 A CN 201910904247A CN 110728206 A CN110728206 A CN 110728206A
- Authority
- CN
- China
- Prior art keywords
- state information
- expression state
- matching
- fatigue driving
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/175—Static expression
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application discloses a fatigue driving detection method, a device, a computer readable storage medium and a terminal, wherein the fatigue driving detection method comprises the following steps: the method comprises the steps of collecting facial images of a user, processing the facial images based on a trained model to obtain expression state information of the user, matching the expression state information with preset expression state information, and if matching is successful, giving an alarm and prompting fatigue driving. The scheme of the application can improve the accuracy rate of fatigue driving detection.
Description
Technical Field
The application relates to the field of image processing, in particular to a fatigue driving detection method and device, a computer readable storage medium and a terminal.
Background
With the development of economy, cars have become a travel tool for many people. In modern society, more and more vehicles run on a highway, and the increase of the number of the vehicles causes more danger; the physical and psychological health of the driver must also affect the safety of automobile driving and accident emergencies. According to research, fatigue driving has become a main factor of traffic accidents in recent years, so that detection of fatigue driving is of great importance for safe travel.
In the related art, the vehicle-mounted camera and the fatigue driving detection device are mainly matched for detecting the fatigue driving, so that the vehicle must be provided with the vehicle-mounted camera and the fatigue driving detection device, and the fatigue driving detection device can be started manually. However, the manual activation of the fatigue driving detection device has disadvantages of being cumbersome to operate and being easily forgotten.
Disclosure of Invention
The embodiment of the application provides a fatigue driving detection method, a fatigue driving detection device, a computer readable storage medium and a terminal.
The embodiment of the application provides a fatigue driving detection method, which comprises the following steps:
collecting a face image of a user;
processing the facial image based on the trained model to obtain expression state information of the user;
matching the expression state information with preset expression state information;
and if the matching is successful, giving an alarm and prompting fatigue driving.
Correspondingly, this application embodiment still provides a driver fatigue detection device, includes:
the acquisition unit is used for acquiring a face image of a user;
the processing unit is used for processing the facial image based on the trained model to obtain expression state information of the user;
the matching unit is used for matching the expression state information with preset expression state information;
and the prompting unit is used for giving an alarm and prompting fatigue driving if the matching is successful.
Accordingly, the present application also provides a computer-readable storage medium, which stores a plurality of instructions, where the instructions are suitable for being loaded by a processor to execute the steps in the fatigue driving detection method as described above.
Correspondingly, the embodiment of the application also provides a terminal, which comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the fatigue driving detection method.
According to the scheme, the facial image of the user is collected, the facial image is processed based on the trained model, the expression state information of the user is obtained, the expression state information is matched with the preset expression state information, and if the matching is successful, an alarm is given and fatigue driving is prompted. The scheme of the application can improve the accuracy rate of fatigue driving detection.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a first fatigue driving detection method according to an embodiment of the present application.
Fig. 2 is a schematic flowchart of a second fatigue driving detection method according to an embodiment of the present application.
Fig. 3 is a block diagram of a first fatigue driving detection method according to an embodiment of the present application.
Fig. 4 is a block diagram of a second fatigue driving detection method according to an embodiment of the present application.
Fig. 5 is a schematic image acquisition diagram of a fatigue driving detection method according to an embodiment of the present application.
Fig. 6 is a schematic view of face recognition of a fatigue driving detection method according to an embodiment of the present application.
Fig. 7 is a schematic structural diagram of a terminal according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Based on the above problems, embodiments of the present application provide a method and an apparatus for detecting fatigue driving, a computer-readable storage medium, and a terminal, which can effectively improve the accuracy of detecting fatigue driving. The following are detailed below. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a first fatigue driving detection method according to an embodiment of the present disclosure. The fatigue driving detection method may be applied to mobile terminals such as mobile phones, tablet computers, notebook computers, palm computers, Portable Media Players (PMPs), and fixed terminals such as desktop computers. The specific flow of the fatigue driving detection method can be as follows:
101. an image of the user's face is captured.
Specifically, the image of the face of the user can be captured by the camera device on the terminal. Wherein, the user can be a driver, and the facial image can be a facial graph of the driver. For example, the facial area of the user can be photographed by a terminal camera, so that the facial image of the user is acquired.
In some embodiments, before the step "acquiring the face image of the user", the following process may be further included:
receiving an image acquisition request;
opening a camera according to the image acquisition request;
and monitoring the face image through the camera.
Wherein the image acquisition request may be triggered by a user. Specifically, the user may trigger the image capturing request through third-party software, where the third-party software may be pre-installed on the terminal. For example, the user may trigger the image capture request by opening third-party software and performing a corresponding operation.
Specifically, the terminal may start the camera after receiving the image acquisition request. Specifically, the terminal device may be a mobile phone, and the opened camera may be a front camera or a rear camera. And after the terminal starts the camera, monitoring the face image of the user.
In some embodiments, the step of "monitoring the facial image by the camera" may include the following steps:
detecting whether a face image exists in the camera monitoring picture, wherein the face image comprises a complete face area;
if so, starting to monitor the face image;
if not, prompting that no face image exists.
And after the terminal opens the camera, detecting whether a face image exists in a camera monitoring picture. The face image may include a complete face region. For example, a complete face region may include a face contour, and eyes, nose, mouth, etc. that are in the face contour. Specifically, whether the face image exists in the monitoring picture can be detected by detecting whether the complete face area exists in the monitoring picture.
For example, when the terminal detects that the camera monitoring picture has a complete face contour, it can be determined that the monitoring picture has a face image; when the terminal detects that part of face contour exists in the camera picture, the fact that the face image does not exist in the monitoring picture can be judged. The facial image is detected in real time by starting the camera equipment, so that the facial expression state of the user can be effectively monitored.
Specifically, when the terminal detects that a face image exists, the terminal can start to monitor the face image. For example, the terminal may collect each frame of the face in the monitoring picture, so that the current face image may be processed. When the terminal detects that the face image does not exist, the terminal can prompt the user of a corresponding message. For example, the prompt message may be "face image not detected". After the user sees the prompting message, corresponding operations can be performed, for example, the user can adjust the position of the camera or adjust the position of the user, and the like.
In some embodiments, before the step "acquiring the face image of the user", the following process may be further included:
acquiring face images of a plurality of target fatigue driving users to obtain a plurality of sample images;
adding the sample image to a set of training samples;
and training a preset neural network model according to the training sample set to obtain the trained model.
Specifically, the face images of the multiple target fatigue driving users may be obtained through an image capturing device, for example, the image capturing device may be a camera, a mobile phone, a computer, or the like. The target fatigue driving user may be determined to be in a fatigue driving state through other ways. For example, the way of determining fatigue driving may be subjectively evaluated by other users, or may be checked by other detection devices.
For example, multiple face sample images can be collected by shooting a large number of face images of fatigue driving users, shooting multiple face image images of the same fatigue driving user, and the like; alternatively, the face images of a plurality of fatigue driving users and the like can be acquired by searching on the internet or from a face image database.
The acquired face images of a plurality of target fatigue driving users can be taken as sample images, and then the sample images are added to the training sample set. By acquiring the face images of a plurality of target fatigue driving users, the training accuracy can be improved.
Specifically, a preset neural network model is trained according to a training sample set to obtain the trained model.
The preset Neural Network model may be set according to a requirement of an actual application, and taking the structure as a Convolutional Neural Network (CNN) as an example, the structure may include a Convolutional layer (Convolutional Layers) and a Fully Connected layer (FC).
In particular, Neural Networks (NN) are complex network systems formed by a large number of simple processing units (called neurons) widely connected to each other, reflect many basic features of human brain functions, and are highly complex nonlinear dynamical learning systems.
A Convolutional Neural Network (CNN) is a feed-forward Neural Network whose artificial neurons can respond to a portion of the surrounding cells within the coverage, and consists of one or more Convolutional layers and an apical fully-connected layer (corresponding to a classical Neural Network), and also includes associated weights and pooling layers (Poolinglayer). This structure enables the convolutional neural network to utilize a two-dimensional structure of the input data.
The convolutional layer and each convolutional layer in the convolutional neural network are composed of a plurality of convolutional units, and parameters of each convolutional unit are obtained through optimization of a back propagation algorithm. The convolution operation aims to extract different input features, the convolution layer at the first layer can only extract some low-level features such as edges, lines, angles and other levels, and more layers of networks can iteratively extract more complex features from the low-level features. And the full connection layer is used for connecting all the characteristics and sending the output value to a classifier (such as a softmax classifier).
Specifically, the sample image is input into the convolutional layer neural network model and is processed to obtain a trained model. The post-training model may be a facial expression package that may include various facial emotions, such as happy, sad, angry, and the like.
For example, a facial image of a target fatigue driving user is input into the convolutional layer neural network model for training, so that a facial expression packet representing the fatigue state of the user can be obtained. The training of the neural network model is carried out by acquiring a large number of face sample images to obtain a trained model, so that the efficiency of facial expression recognition can be improved.
In some embodiments, the step of "acquiring a facial image of a user" may comprise the following process:
at least three face images to be processed are collected based on a preset time interval, and the face images to be processed are the same user.
Specifically, the collecting of the at least three face images to be processed based on the preset time interval may be that the terminal obtains the three face images to be processed through the camera and stores the three face images in the preset time period, or the terminal obtains the face images based on the preset time interval and stores the three face images to be processed which are continuously obtained.
For example, the preset time period may be 5 seconds, and the terminal may store three face images arbitrarily acquired within 5 seconds; the preset time interval can be 1 second, the terminal can acquire one face image in the first second, then acquire one face image every other second, and at least acquire three face images and store the face images. By continuously acquiring a plurality of human face images to detect the human face images, the method can avoid the contingency of detection results and improve the detection accuracy.
102. And processing the facial image based on the trained model to obtain the expression state information of the user.
Specifically, the trained model is used for processing the acquired face image, so that the face expression packet corresponding to the face image can be obtained. For example, the facial expression package may be expression 1, expression 2, etc. as shown in fig. 6, wherein expression 1 may express happy mood, etc.
In some embodiments, there may be an error in acquiring the facial image of a single user to identify the facial emotion of the user, and in order to improve the accuracy of identifying the facial emotion of the user, the facial images of multiple users may be acquired, and information of the multiple facial images is fused to determine the expression state of the end user. That is, the step of processing the facial image based on the trained model to obtain the expression state information of the user may include the following steps:
processing the at least three facial images to be processed based on the trained model to obtain expression state information corresponding to each facial image to be processed;
weighting the expression state information corresponding to each facial image to be processed to obtain weighted expression state information;
and fusing the expression state information after weighting to obtain the expression state information of the user.
Specifically, the at least three acquired facial images to be processed are processed based on the trained model, and expression state information corresponding to each facial image to be processed is obtained. The expression state information may be probabilities of various expressions included in each to-be-processed facial image. Specifically, the facial image may include a variety of expressions, such as happy, sad, surprised, horror, angry, disgust, neutral, and so forth.
And obtaining expression state information corresponding to the facial image to be processed after the facial image to be processed is processed. For example, the acquired face image may be a picture a, and the picture a may be processed based on the trained model, and the expression state information of the picture a is obtained as expression state information a, which may be: happy 10%, sad 20%, surprised 10%, fear 10%, angry 20%, disgust 10%, neutral 10%, etc.
Specifically, after determining the expression state information corresponding to the facial image to be processed, weighting processing may be performed on the expression state information. The weighting process sets weights (i.e., coefficients) for the acquired face images. For example, the number of the acquired pictures may be three, including picture a, picture B, and picture C, wherein a weight may be set for each face image according to the completeness of each picture (i.e., whether the complete face organ of the user is included). Picture a may be weighted 30%, picture B30%, picture C40%, etc.
Specifically, the expression state information a after weighting the expression state information a may be: happy 0.03, sad 0.06, surprised 0.03, fear 0.03, anger 0.06, disgust 0.03, neutral 0.03.
Specifically, after determining the expression state information of all the facial images to be processed, weighting all the expression state information, and performing fusion processing on all the weighted expression state information to obtain the expression state information of the user.
For example, the facial image to be processed may include a facial image a, a facial image B, and a facial image C, the expression state information a of the facial image a may be 10% happy, 20% sad, 10% surprised, 10% fear, 20% angry, 10% disgust, and 10% neutral, and the weighted expression state information a obtained by weighting the expression state information a may be: happy 0.03, sad 0.06, surprised 0.03, fear 0.03, anger 0.06, disgust 0.03, neutral 0.03; the expression state information B of the face image B may be 10% happy, 10% sad, 10% surprised, 10% fear, 30% angry, 10% disgust, and 10% neutral, and the weighted expression state information B obtained by weighting the expression state information B may be: happy 0.03, sad 0.03, surprised 0.03, fear 0.03, angry 0.09, disgust 0.03, neutral 0.03; the expression state information C of the face image C may be 20% happy, 10% sad, 10% surprised, 10% fear, 20% angry, 10% disgust, and 10% neutral, and the weighted expression state information a obtained by weighting the expression state information C may be: happy 0.08, sad 0.04, surprised 0.04, fear 0.04, angry 0.08, disgust 0.04, neutral 0.04.
Further, the probabilities of the expressions in the expression state information of the facial image a, the facial image B, and the facial image C are superimposed, and the expression state information of the user can be obtained as follows: happy 0.14, sad 0.13, surprised 0.1, fear 0.1, angry 0.23, disgust 0.1, neutral 0.1, wherein in the expression state information, the proportion of angry is the largest, then the current expression state of the user can be determined to be angry.
103. And matching the expression state information with preset expression state information.
In some embodiments, the step of "matching the expression state information with the preset expression state information" may include the following processes:
matching the expression state information with the preset expression state information to obtain a matching degree;
comparing the matching degree with a preset threshold value;
when the matching degree is smaller than a preset threshold value, the matching is successful;
and when the matching degree is greater than or equal to a preset threshold value, the matching fails.
Specifically, the expression state information is matched with preset expression state information. The expression state information can comprise various facial features, and the expression state of the facial image can be determined based on the algorithm library according to the acquired expression state information.
The expression state refers to a facial expression package obtained by processing the acquired facial image based on the trained model, and the facial expression package can express various facial emotions of the user, such as happiness, anger and the like. The preset expression refers to a predefined facial expression package, and the predefined facial expression package can represent facial emotion conforming to a fatigue state.
And matching the expression state with a preset expression state, and matching the characteristics of the expression state with the preset expression state to obtain a matching degree. Specifically, the matching degree of the expression state and the preset expression state is compared with a preset threshold. For example, the matching degree of the expression state and the preset expression state may be 80%, and the preset threshold may be 60%, and then the matching degree of the expression state and the preset expression state is greater than the preset threshold.
Specifically, the matching degree between the expression state and the preset expression state may be greater than a preset threshold, or less than a preset threshold, or equal to a preset threshold, or the like. When the matching degree of the expression state and the preset expression state is smaller than a preset threshold value, the expression state and the preset expression state can be successfully matched; when the matching degree of the expression state and the preset expression state is greater than or equal to the preset threshold, the expression state and the preset expression state may be matching failure.
104. And if the matching is successful, giving an alarm and prompting fatigue driving.
Specifically, after the expression state information is successfully matched with the preset expression state information, it can be determined that the face of the current user is in a fatigue state, that is, it can be determined that the current user is in fatigue driving. The terminal can send out alarm information, and the alarm mode can be various, for example, the user can be reminded of the current fatigue driving state in a ringing mode or a voice prompt mode.
In some embodiments, before the step "issue an alarm and prompt fatigue driving", the following process may be further included:
judging whether the emotion state information to be processed is successfully matched with the preset expression state information or not, wherein the emotion state to be processed is acquired based on the face image to be processed;
if yes, an alarm is given and fatigue driving is prompted.
Specifically, whether the to-be-processed expression state information is successfully matched with the preset expression state information or not is judged, and the terminal extracts corresponding expression state information from at least 3 to-be-processed face images. For example, the terminal may acquire 3 images of the face to be processed, the 3 two images may be image a, image B, and image C, the expression state information of image a may be information a, the expression state information of image B may be information B, and the expression state information of image C may be information C.
For example, the information a may be matched with preset expression state information, the information B may be matched with preset expression state information, the information C may be matched with preset expression state information, and when the information a, the information B, and the information C are successfully matched with the preset expression state information, respectively, it may be determined that the face state of the current user is in a fatigue state, and then the step "issue an alarm and prompt fatigue driving" may be performed.
In some embodiments, after the step "alarm and prompt fatigue driving", the following process may be further included:
acquiring a communication database, and searching a target communicator from the communication database;
and transmitting fatigue driving information of the user to the target communicator.
Specifically, a communication database is obtained, and a target correspondent is searched from the communication database, wherein the communication database can be a contact list in a terminal address book; the target correspondent person can be an emergency contact person preset by the user, namely, when the target correspondent person is in a dangerous state, the terminal can send related information to the emergency contact person.
For example, when the terminal detects that the user is in a fatigue driving state, the terminal may find the target contact from the address book and then send a message to the target contact. Wherein, the sent message can be 'the current user is in a fatigue driving state' and the like, and the risk of dangerous driving can be reduced.
According to the scheme, the facial image of the user is collected, the facial image is processed based on the trained model, expression state information corresponding to the facial image is obtained, the expression state information is matched with preset expression state information, and if the matching is successful, an alarm is given and fatigue driving is prompted. According to the scheme, the expression state of the driver is judged according to the currently acquired image information by acquiring the facial image information of the driver, so that whether the driver is in fatigue driving or not is detected, the risk of dangerous driving can be reduced, and the accuracy of fatigue driving detection is improved.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a second fatigue driving detection method according to an embodiment of the present application. In this embodiment, the specific application scenarios of the fatigue driving detection method may be as follows:
201. and the terminal enters a driving mode and triggers an image acquisition instruction.
In this embodiment, the user may turn on the third party detection software on the terminal when the user begins driving the vehicle. The third party software may be downloaded from the network via the terminal. After the third party detection software is turned on, the user may select a driving mode, at which point an image capture instruction will be triggered.
202. And the terminal starts a camera according to the image acquisition instruction and starts to monitor the shooting picture in real time.
Specifically, the terminal opens the camera according to the received image acquisition instruction. For example, the terminal device may have a plurality of cameras, and the user may select a camera to be turned on by himself. After the camera is opened at the terminal, the user can place the terminal equipment at a proper position so as to monitor a complete face image.
As shown in fig. 5, fig. 5 is a schematic image acquisition diagram of a fatigue driving detection method according to an embodiment of the present application. The face area of the user can be completely located in the shooting area of the camera, so that the terminal can conveniently acquire the face image.
203. And the terminal detects whether a face image exists in the current monitoring picture. If yes, go to step 204; if not, go to step 205.
Specifically, after the camera is opened at the terminal, the camera can be monitored in real time to shoot the picture, and whether the shot picture has a face image or not is detected. The face image can be a face image of a user, and refers to a complete face image of the user. For example, a complete face image may include the entire face contour, as well as five organs, etc.
When the terminal detects that the face image exists in the shooting picture, step 204 can be executed; when the terminal detects that the face image does not exist in the shot picture, for example, the terminal detects that only a partial face contour exists in the shot picture, and a complete five-officer does not appear, the step 205 can be executed.
204. The terminal acquires at least 3 face images of the driver based on a preset time interval.
Specifically, the terminal acquires at least 3 face images of the driver based on a preset time interval. The preset time interval may be set according to actual conditions, for example, the preset time interval may be 10 seconds. After the terminal detects the face image, the face image of the user may be acquired based on a preset time interval.
For example, the preset time interval may be 10 seconds, the terminal may acquire a first face image within 1 st second, then acquire one face image every 10 seconds, and may sequentially acquire and store face images of three users.
205. The terminal prompts the user that the face image is not detected.
When the terminal does not detect the face image from the shot picture, the user may be prompted with a relevant message, for example, the prompt message may be "no face detected, please adjust the shooting angle", and the like. The user can adjust the terminal position and other operations according to the prompt message of the terminal.
206. And the terminal processes the acquired face image by using an algorithm library to obtain a face emotion expression package.
Specifically, the terminal processes the acquired face image by using an algorithm library. The algorithm library can be a Kaggle algorithm and belongs to an AI artificial intelligence machine learning algorithm. The Kaggle algorithm library is trained and integrated with a plurality of basic emotions of the human face, for example, the basic emotions can be as follows: happiness, sadness, surprise, fear, anger, disgust, neutrality, etc. As shown in fig. 6, fig. 6 is a schematic view of a face recognition of a fatigue driving detection method according to an embodiment of the present application. The terminal can process the face image by using a Kaggle algorithm library through the collected face image, and can obtain a corresponding expression.
For example, the facial image in fig. 6 may be processed by the terminal to obtain an expression 1, an expression 2, an expression 3, an expression 4, an expression 5, an expression 6, an expression 7, or the like. Wherein expression 1 may be expressed as happy emotion, expression 2 may be expressed as surprised emotion, expression 3 may be expressed as sad emotion, expression 4 may be expressed as horror emotion, expression 5 may be expressed as angry emotion, expression 6 may be expressed as disgusting emotion, and expression 7 may be expressed as disgusting emotion. By utilizing the Kaggle algorithm to classify different expressions of the face image, the current emotion state of the user can be effectively identified, and whether the user is in a fatigue driving state or not is judged.
207. And the terminal matches the facial emotion expression packet with a preset facial emotion expression packet and judges whether the matching is successful. If the matching is successful, go to step 208; if the matching fails, step 209 is performed.
Specifically, after the terminal determines the facial emotion expression package of the current user. The facial emotion expression package is an expression obtained by processing a facial image by using a Kaggle algorithm. And matching the expression with a preset expression package, wherein the preset expression package can be an expression representing the fatigue state of the user.
The terminal extracts the characteristics of the facial emotion expression packet and the preset facial emotion expression packet, and matches the characteristics of the facial emotion expression packet with the characteristics of the preset facial emotion expression packet to obtain the matching degree. The terminal compares the matching degree with a preset matching degree.
For example, the preset matching degree may be 60%, and the matching degree of the facial emotion expression packet features and the preset facial emotion expression packet may be 80%, 60%, 50%, and the like. If the matching degree is 80%, the facial emotion expression packet can be judged to be successfully matched with the preset facial emotion expression packet; if the matching degree is 60%, the facial emotion expression packet can be judged to be successfully matched with the preset facial emotion expression packet; if the matching degree is 50%, it can be judged that the matching of the facial emotion expression package and the preset facial emotion expression package fails.
When the facial emotion expression packet is successfully matched with the preset facial emotion expression packet, step 208 can be executed; when the matching of the facial emotion expression package and the preset facial emotion expression package fails, step 209 may be executed.
208. The terminal initiates an alarm message and prompts the driver to be in a fatigue driving state.
Specifically, after the terminal determines that the current user is in a fatigue state, it may be determined that the current user is in fatigue driving, and the terminal may send an alarm message to prompt the driver that the driver is in fatigue driving currently and there is a driving risk.
209. No operation is performed.
According to the scheme, the facial image of the user is collected, the facial image is processed based on the trained model, expression state information corresponding to the facial image is obtained, the expression state information is matched with preset expression state information, and if the matching is successful, an alarm is given and fatigue driving is prompted. According to the scheme, the expression state of the driver is judged according to the currently acquired image information by acquiring the facial image information of the driver, so that whether the driver is in fatigue driving or not is detected, the risk of dangerous driving can be reduced, and the accuracy of fatigue driving detection is improved.
In order to better implement the fatigue driving detection method provided by the embodiment of the application, the embodiment of the application also provides a device based on the fatigue driving detection. The term is the same as that in the fatigue driving detection method, and specific implementation details can be referred to the description in the method embodiment.
Referring to fig. 3, fig. 3 is a block diagram of a fatigue driving detection apparatus according to an embodiment of the present application, including:
an acquisition unit 301 for acquiring a face image of a user;
the processing unit 302 is configured to process the facial image based on the trained model to obtain expression state information corresponding to the facial image;
a matching unit 303, configured to match the expression state information with preset expression state information;
and the prompting unit 304 is used for giving an alarm and prompting fatigue driving if the matching is successful.
In some embodiments, as shown in fig. 4, fig. 4 is a block diagram of a second fatigue driving detection apparatus provided in the embodiments of the present application. The matching unit 303 may include:
a matching subunit 3031, configured to match the expression state information with the preset expression state information to obtain a matching degree;
a comparing subunit 3032, configured to compare the matching degree with a preset threshold, and when the matching degree is smaller than the preset threshold, the matching is successful; and when the matching degree is greater than or equal to a preset threshold value, the matching fails.
In some embodiments, the fatigue driving detection apparatus may further include:
the device comprises an acquisition unit, a display unit and a display unit, wherein the acquisition unit is used for acquiring face images of a plurality of target fatigue driving users to obtain a plurality of sample images;
an adding unit, configured to add the sample image to a training sample set;
and the training unit is used for training a preset neural network model according to a training sample set to obtain the trained model.
In some embodiments, the acquisition unit 301 may include:
the face image processing device comprises a collecting subunit, a processing unit and a processing unit, wherein the collecting subunit is used for collecting at least three face images to be processed based on a preset time interval, and the face images to be processed are the same user.
In some embodiments, the fatigue driving detection apparatus may further include:
the judging unit is used for judging whether the emotion state information to be processed is successfully matched with the preset expression state information or not, and the emotion state to be processed is acquired based on the face image to be processed;
and the alarm unit is used for giving an alarm and prompting fatigue driving.
In some embodiments, the fatigue driving detection apparatus may further include:
the searching unit is used for acquiring a communication database and searching a target communicator from the communication database;
and the sending unit is used for sending fatigue driving information of the user to the target communicator.
According to the scheme, the facial image of the user is collected, the facial image is processed based on the trained model, the expression state information of the user is obtained, the expression state information is matched with the preset expression state information, and if the matching is successful, an alarm is given and fatigue driving is prompted. According to the scheme, the expression state of the driver is judged according to the currently acquired image information by acquiring the facial image information of the driver, so that whether the driver is in fatigue driving or not is detected, the risk of dangerous driving can be reduced, and the accuracy of fatigue driving detection is improved.
The embodiment of the application also provides a terminal. As shown in fig. 7, the terminal may include Radio Frequency (RF) circuitry 601, memory 602 including one or more computer-readable storage media, input unit 603, display unit 604, sensor 605, audio circuitry 606, Wireless Fidelity (WiFi) module 607, processor 608 including one or more processing cores, and power supply 609. Those skilled in the art will appreciate that the terminal structure shown in fig. 7 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:
the RF circuit 601 may be used for receiving and transmitting signals during the process of transmitting and receiving information, and in particular, for processing the received downlink information of the base station by one or more processors 608; in addition, data relating to uplink is transmitted to the base station. In general, the RF circuit 601 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuit 601 may also communicate with networks and other devices via wireless communications.
The memory 602 may be used to store software programs and modules, and the processor 608 executes various functional applications and data processing by operating the software programs and modules stored in the memory 602. The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and the like. Further, the memory 602 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 608 and the input unit 603 access to the memory 602.
The input unit 603 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. In particular, in one particular embodiment, input unit 603 may include a touch-sensitive surface as well as other input devices. The touch-sensitive surface, also referred to as a touch display screen or a touch pad, may collect touch operations by a user (e.g., operations by a user on or near the touch-sensitive surface using a finger, a stylus, or any other suitable object or attachment) thereon or nearby, and drive the corresponding connection device according to a predetermined program. The input unit 603 may include other input devices in addition to the touch-sensitive surface. In particular, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 604 may be used to display information input by or provided to the user and various graphical user interfaces of the server, which may be made up of graphics, text, icons, video, and any combination thereof. The display unit 604 may include a display panel, and optionally, the display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch-sensitive surface may overlay the display panel, and when a touch operation is detected on or near the touch-sensitive surface, the touch operation is transmitted to the processor 608 to determine the type of touch event, and the processor 608 then provides a corresponding visual output on the display panel according to the type of touch event. Although in FIG. 7 the touch-sensitive surface and the display panel are two separate components to implement input and output functions, in some embodiments the touch-sensitive surface may be integrated with the display panel to implement input and output functions.
The terminal may also include at least one sensor 605, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel according to the brightness of ambient light, and a proximity sensor that turns off the display panel and the backlight when the server moves to the ear.
WiFi belongs to short-distance wireless transmission technology, and the terminal can help a user to receive and send e-mails, browse webpages, access streaming media and the like through the WiFi module 607, and provides wireless broadband internet access for the user. Although fig. 7 shows the WiFi module 607, it is understood that it does not belong to the essential constitution of the terminal, and may be omitted entirely as needed within the scope of not changing the essence of the application.
The processor 608 is a control center of the terminal, connects various parts of the entire handset using various interfaces and lines, and performs various functions of the server and processes data by operating or executing software programs and modules stored in the memory 602 and calling data stored in the memory 602, thereby performing overall monitoring of the handset. Optionally, processor 608 may include one or more processing cores; preferably, the processor 608 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 608.
The terminal also includes a power supply 609 (e.g., a battery) for powering the various components, which may preferably be logically connected to the processor 608 via a power management system that may be used to manage charging, discharging, and power consumption. The power supply 609 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
Specifically, in this embodiment, the processor 608 in the terminal loads the executable file corresponding to the process of one or more application programs into the memory 602 according to the following instructions, and the processor 608 runs the application programs stored in the memory 602, thereby implementing various functions:
collecting a face image of a user;
processing the facial image based on the trained model to obtain expression state information of the user;
matching the expression state information with preset expression state information;
and if the matching is successful, giving an alarm and prompting fatigue driving.
According to the scheme, the facial image of the user is collected, the facial image is processed based on the trained model, the expression state information of the user is obtained, the expression state information is matched with the preset expression state information, and if the matching is successful, an alarm is given and fatigue driving is prompted. According to the scheme, the expression state of the driver is judged according to the currently acquired image information by acquiring the facial image information of the driver, so that whether the driver is in fatigue driving or not is detected, the risk of dangerous driving can be reduced, and the accuracy of fatigue driving detection is improved.
It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.
To this end, the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the fatigue driving detection methods provided by the embodiments of the present application. For example, the instructions may perform the steps of:
the method comprises the steps of collecting facial images of a user, processing the facial images based on a trained model to obtain expression state information of the user, matching the expression state information with preset expression state information, and if matching is successful, giving an alarm and prompting fatigue driving.
The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.
Wherein the computer-readable storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the fatigue driving detection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the fatigue driving detection methods provided in the embodiments of the present application can be achieved, which are detailed in the foregoing embodiments and will not be described again here.
The fatigue driving detection method, the fatigue driving detection device, the computer-readable storage medium and the terminal provided by the embodiments of the present application are introduced in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the embodiments is only used to help understanding the method and the core concept of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A fatigue driving detection method, characterized by comprising:
collecting a face image of a user;
processing the facial image based on the trained model to obtain expression state information of the user;
matching the expression state information with preset expression state information;
and if the matching is successful, giving an alarm and prompting fatigue driving.
2. The method of claim 1, wherein the matching the expression state information with preset expression state information comprises:
matching the expression state information with the preset expression state information to obtain a matching degree;
comparing the matching degree with a preset threshold value;
when the matching degree is smaller than a preset threshold value, the matching is successful;
and when the matching degree is greater than or equal to a preset threshold value, the matching fails.
3. The method of claim 1, prior to acquiring the image of the face of the user, further comprising:
acquiring face images of a plurality of target fatigue driving users to obtain a plurality of sample images;
adding the sample image to a set of training samples;
and training a preset neural network model according to the training sample set to obtain the trained model.
4. The method of claim 1, wherein the capturing of the facial image of the user comprises:
acquiring at least three face images to be processed based on a preset time interval, wherein the face images to be processed are the same user;
processing the facial image based on the trained model to obtain expression state information of the user, including:
processing the at least three facial images to be processed based on the trained model to obtain expression state information corresponding to each facial image to be processed;
weighting the expression state information corresponding to each facial image to be processed to obtain weighted expression state information;
and fusing the expression state information after weighting to obtain the expression state information of the user.
5. The method of claim 1, further comprising, prior to issuing the alert and prompting fatigue driving:
judging whether the emotion state information to be processed is successfully matched with the preset expression state information or not, wherein the emotion state to be processed is acquired based on the face image to be processed;
if yes, an alarm is given and fatigue driving is prompted.
6. The method of claim 1, further comprising, after said alerting and prompting fatigue driving:
acquiring a communication database, and searching a target communicator from the communication database;
and transmitting fatigue driving information of the user to the target communicator.
7. A fatigue driving detecting device, characterized by comprising:
the acquisition unit is used for acquiring a face image of a user;
the processing unit is used for processing the facial image based on the trained model to obtain expression state information of the user;
the matching unit is used for matching the expression state information with preset expression state information;
and the prompting unit is used for giving an alarm and prompting fatigue driving if the matching is successful.
8. The apparatus of claim 7, wherein the matching unit comprises: a matching subunit, a comparing subunit;
the matching subunit is configured to match the expression state information with the preset expression state information to obtain a matching degree;
the comparison subunit is configured to compare the matching degree with a preset threshold, and when the matching degree is smaller than the preset threshold, the matching is successful; and when the matching degree is greater than or equal to a preset threshold value, the matching fails.
9. A computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the fatigue driving detection method according to any one of claims 1 to 6.
10. A terminal, characterized in that it comprises a processor and a memory, said memory storing a plurality of instructions, said processor loading said instructions to perform the steps in the fatigue driving detection method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910904247.9A CN110728206A (en) | 2019-09-24 | 2019-09-24 | Fatigue driving detection method and device, computer readable storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910904247.9A CN110728206A (en) | 2019-09-24 | 2019-09-24 | Fatigue driving detection method and device, computer readable storage medium and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110728206A true CN110728206A (en) | 2020-01-24 |
Family
ID=69218331
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910904247.9A Pending CN110728206A (en) | 2019-09-24 | 2019-09-24 | Fatigue driving detection method and device, computer readable storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110728206A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112633247A (en) * | 2021-01-04 | 2021-04-09 | 珠海研果科技有限公司 | Driving state monitoring method and device |
| CN113469023A (en) * | 2021-06-28 | 2021-10-01 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining alertness |
| CN115810252A (en) * | 2021-09-16 | 2023-03-17 | 小米科技(武汉)有限公司 | Fatigue driving early warning method, device, equipment, system and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN106205055A (en) * | 2016-09-13 | 2016-12-07 | 珠海格力电器股份有限公司 | Control method and device of intelligent terminal |
| CN106650633A (en) * | 2016-11-29 | 2017-05-10 | 上海智臻智能网络科技股份有限公司 | Driver emotion recognition method and device |
| CN107856536A (en) * | 2017-09-09 | 2018-03-30 | 深圳市赛亿科技开发有限公司 | A kind of fatigue driving monitoring device and monitoring method |
| CN108664947A (en) * | 2018-05-21 | 2018-10-16 | 五邑大学 | A kind of fatigue driving method for early warning based on Expression Recognition |
| CN109190468A (en) * | 2018-07-26 | 2019-01-11 | 深圳市赛亿科技开发有限公司 | A kind of fatigue driving monitoring method and system |
-
2019
- 2019-09-24 CN CN201910904247.9A patent/CN110728206A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN106205055A (en) * | 2016-09-13 | 2016-12-07 | 珠海格力电器股份有限公司 | Control method and device of intelligent terminal |
| CN106650633A (en) * | 2016-11-29 | 2017-05-10 | 上海智臻智能网络科技股份有限公司 | Driver emotion recognition method and device |
| CN107856536A (en) * | 2017-09-09 | 2018-03-30 | 深圳市赛亿科技开发有限公司 | A kind of fatigue driving monitoring device and monitoring method |
| CN108664947A (en) * | 2018-05-21 | 2018-10-16 | 五邑大学 | A kind of fatigue driving method for early warning based on Expression Recognition |
| CN109190468A (en) * | 2018-07-26 | 2019-01-11 | 深圳市赛亿科技开发有限公司 | A kind of fatigue driving monitoring method and system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112633247A (en) * | 2021-01-04 | 2021-04-09 | 珠海研果科技有限公司 | Driving state monitoring method and device |
| CN112633247B (en) * | 2021-01-04 | 2024-10-18 | 珠海研果科技有限公司 | Driving state monitoring method and device |
| CN113469023A (en) * | 2021-06-28 | 2021-10-01 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining alertness |
| CN115810252A (en) * | 2021-09-16 | 2023-03-17 | 小米科技(武汉)有限公司 | Fatigue driving early warning method, device, equipment, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110706179B (en) | Image processing method and electronic device | |
| CN107784114A (en) | Recommended method, device, terminal and storage medium for emoticon images | |
| CN107623778B (en) | Call answering method and mobile terminal | |
| CN112149589B (en) | Method and device for identifying behavior of driver in vehicle | |
| CN107451449A (en) | Bio-identification unlocking method and Related product | |
| CN108491775A (en) | A kind of image correcting method and mobile terminal | |
| CN109215683B (en) | A prompt method and terminal | |
| CN111402866A (en) | Semantic recognition method and device and electronic equipment | |
| CN106815557A (en) | A kind of evaluation method of face features, device and mobile terminal | |
| CN108366220A (en) | A kind of video calling processing method and mobile terminal | |
| CN110728206A (en) | Fatigue driving detection method and device, computer readable storage medium and terminal | |
| CN110881105A (en) | A shooting method and electronic device | |
| CN111614544B (en) | Message processing method and electronic equipment | |
| CN108197558A (en) | Face recognition method and device, storage medium and electronic equipment | |
| CN108765522B (en) | Dynamic image generation method and mobile terminal | |
| KR20190009201A (en) | Mobile terminal and method for controlling the same | |
| CN114333774B (en) | Speech recognition method, device, computer equipment and storage medium | |
| CN107895108B (en) | Operation management method and mobile terminal | |
| CN114065168A (en) | Information processing method, intelligent terminal and storage medium | |
| CN115203411A (en) | Training method of text classification model, and text classification method and device | |
| CN109040427A (en) | Split-screen processing method, device, storage medium and electronic equipment | |
| CN107944242A (en) | A kind of bio-identification function prohibited method and mobile terminal | |
| CN107437070A (en) | Iris living body recognition method and related products | |
| CN118824294A (en) | Emotion regulation method, device, vehicle and storage medium | |
| CN116860913A (en) | Voice interaction method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200124 |
|
| RJ01 | Rejection of invention patent application after publication |