CN104083160A - Sleep state monitoring method and device based on machine vision - Google Patents

Abstract

The invention discloses a sleep state monitoring method and device based on machine vision. The method includes: respectively detecting and characterizing the visual behavior characteristics and physiological parameter characteristics of a testee through eye video images, lip video images and cheek video images : The above parameters are fused and judged to obtain the test result of the sleep state of the test subject. The device includes: the embedded system is used to extract the frequency of the subject's mouth opening and closing, the head position and its maintenance time, the visual characteristics of the frequency of transposition, and the physiological parameters of the subject such as pulse rate, Carry out fusion processing and judgment alarm on these feature information; when the sleep state of the subject is found to be abnormal, an alarm signal is output to the alarm device, and an abnormal signal is output to parents or related guardians at the same time. The invention does not affect the behavior of the testee, obtains more information including the physiological parameters of the testee, and greatly improves the reliability of monitoring the sleep state of the testee.

Description

A sleep state monitoring method and device based on machine vision

technical field

The invention relates to the field of machine vision, in particular to a machine vision-based sleep state monitoring method and device.

Background technique

Adequate and deep sleep is the basic guarantee of physical and mental health, but there are many factors that lead to poor sleep quality. In order to diagnose the cause of poor sleep quality, many researchers have proposed various sleep monitoring devices. Then, there are many deficiencies in the prior art: such as Yu Mengsun's Chinese invention patent (publication number: CN1584529) "Electrodeless Sleep State and Respiratory Disorder Event Sensing Device", which uses the airbag in the mattress to monitor heartbeat, body movement and respiration. change, low reliability; Tang Yiping's Chinese invention patent (publication number: CN1803089) "unfettered, non-perceptual sleep disorder measuring device and method thereof", the pickup and microprocessor etc. are placed in the pillow, for the The tester makes a snoring sound to record and judge after falling asleep, and the reliability is also poor; the Chinese invention patent (public number: CN1923132) "Method, device and application for detecting respiratory effort during sleep" proposed by Yang Fusheng, through pressure micro-motion sensing The device obtains the micro-movement signals of the heart beating, respiration and body movement of the monitored person during sleep, but there is still a problem of poor reliability; the Chinese invention patent (public number: CN101044981) "Sleep Breathing State Detection Device" proposed by Zhou Changan is also for the monitored person. The tester makes a snoring sound to record and judge after falling asleep, and the reliability is also poor; the Chinese invention patent (public number: CN102188233A) "A Device for Monitoring Sleep Apnea" proposed by Zhang Kaixun uses the measurement developed by the giant stress impedance effect of amorphous wire The waveform of human chest and abdomen breathing movement needs to be wrapped in the body of infants and young children, and can only monitor the breathing of infants and young children; the Chinese invention patent (publication number: CN102274022) "A sleep state monitoring based on EEG signal" proposed by Meng Jun Method", the collection of EEG will definitely affect the sleep of the monitor, and it is difficult to use it for a long time; the Chinese invention patent (public number: CN102415879A) "A Sleep Monitoring Device Based on Piezoelectric Film Sensor" proposed by Du Yan uses piezoelectric Thin-film sensor measures the airbag pressure in the mattress to monitor heartbeat, body movement and respiration changes, and the reliability is low; the Chinese invention patent (publication number: CN102499656A) "A Wristband Sleep Monitoring Device" proposed by Zhang Kaisheng, through the sensor of the wristband Collecting the pulse signal and temperature signal of the sleeper has low reliability and is easy to affect the sleep of the wearer; the Chinese invention patent (public number: CN102648845A) proposed by Shen Jinpeng "a wireless automatic monitoring and early warning system for heartbeat and breathing during sleep" requires Putting electrodes on the subject to measure the ECG and respiratory signals has a great impact on the subject's sleep. The Chinese invention patent (publication number: CN102835951A) "A Mobile Wristband Device and Its Working Method" proposed by Liu Xin monitors and records the user through the acceleration sensor and gyroscope, infrared inspection circuit and blood oxygen inspection circuit in the wristband Daily activity data and sleep information have low reliability and have a great impact on the sleep of the subjects. The Chinese invention patent (publication number: CN102973273A) proposed by Jiao Teng "A Sleep Respiratory Function Monitoring System Based on Infrared Radiation Detection" has the advantage of non-contact measurement, but it can only measure respiratory signals, and the reliability is not high enough. Chinese invention patent (publication number: CN103167849A) "Method and device for the diagnosis and/or treatment of sleep apnea and related dysfunctions" proposed by M. Youngs, through an inflatable device applied to the outer surface of the chest and/or abdomen (Vest) To measure the breathing parameters of the testee, the testee needs to wear a special vest, which has a great impact on the sleep of the testee, and can only measure the breathing signal, and the reliability is not high enough. Lu Xiaodong's Chinese invention patent (publication number: CN1751652) "A Method and Device for Detecting Cardiac and Respiratory Cycles of the Wrist" is greatly affected by the movements of the subject during sleep. The Chinese invention patent (public number: CN1860987) "Sleep and Dream Monitoring and Intervention System and Processing Method" proposed by Shen Zheng needs to place electrodes on the testee to measure the EEG signal, which has a great impact on the sleep of the testee . The Chinese invention patent (publication number: CN103263260A) proposed by Song Jun et al. "Physiological parameter detection system and sleep depth monitoring system using comb filter" is essentially to transmit signals through air mattresses, and the sleep posture of the monitored person will be seriously affected. Affects the reliability of signal detection. The Chinese invention patent proposed by Lin Shanggang (public number: CN102648845A) "Abdominal monitoring and early warning system for wireless transmission of sleep and breathing information" adopts contact measurements such as abdominal sensor belts and breathing sensors to affect the sleep of the subjects. very big. The Chinese invention patent (publication number: CN103006223A) "a household non-contact sleep monitoring device and method" proposed by Lu Guohua and others adopts microwave technology, and the subject is exposed to microwave radiation for a long time. The Chinese invention patent (publication number: CN103271741A) "a kind of sleep posture monitor" proposed by Xia Li can only monitor the sleeping posture of the testee. The Chinese invention patent (publication number: CN103445777A) proposed by Hu Wendong et al. "Sleep and fatigue monitoring watch device and monitoring method for normal small devices" uses skin resistance sensors, temperature sensors, atmospheric pressure sensors, and acceleration sensors and can only be placed on the wrist. It is difficult to measure the information to assess the quality of sleep.

All in all, the existing technologies for monitoring the subject either cannot monitor the subject's heart rate, breathing and other important physiological parameters, or can only be measured by contact sensors, or use microwave radiation technology to achieve.

Contents of the invention

The present invention provides a sleep state monitoring method and device based on machine vision. The present invention obtains more information including the physiological parameters of the subject, and greatly improves the reliability of the sleep state monitoring of the subject. See the description below for details:

A kind of sleep state monitoring method based on machine vision, described method comprises the following steps:

Obtain video images of the subject's face through the two cameras on the upper left and right sides of the subject's pillow;

Cleaning the video image to obtain the cleaned video image;

Segment the cleaned video images to obtain eye video images, lip video images and cheek video images;

Through the eye video image, lip video image and cheek video image, respectively detect and characterize the visual behavior characteristics and physiological parameter characteristics of the testee:

The above parameters are fused and judged to obtain the sleep state detection result of the test subject.

The steps of detecting and characterizing the visual behavior characteristics and physiological parameter characteristics of the testee respectively through the eye video image, lip video image and cheek video image are specifically:

1) Detect the frequency of the subject's mouth opening and closing through the lip video image, detect the subject's head position and the time it is maintained, and the visual characteristics of the transposition frequency through the eye video image;

2) Obtain the heart rate, breathing frequency and heart rate variation sequence of the subject by analyzing the video image of the cheek.

When both cameras are infrared cameras, the steps of obtaining the testee's heart rate, number of breaths and heart rate variation sequence thereof by analyzing the cheek video images are specifically:

(a) average the pixels of the R channel in the cheek region, and there are N data in the R channel;

(b) Perform normalization, whitening processing and independent component analysis on the data in the R channel in sequence;

(c) Perform Fourier transformation on the data after each independent component analysis to obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the frequency corresponding to the second maximum amplitude of the power spectrum Frequency, that is, the number of breaths of the testee;

(d) Calculating the heart rate variation value of the test subject through the test subject's heart rate and breathing rate.

When both cameras are color cameras, the steps of obtaining the testee's heart rate, number of breaths and heart rate variation sequence thereof by analyzing the cheek video images are specifically:

(a) Separate the RGB channels of the cheek region, and average the pixels of each channel;

(b) Save the data into 3 channels, each with N data;

(c) Perform normalization, whitening processing and independent component analysis on the data of each channel in turn;

(d) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the subject;

(e) Calculating the heart rate variation value of the testee through the heart rate and respiration rate of the testee in a single channel, and then obtaining the heart rate variation sequence through the calculation of multiple channels.

When the two cameras are respectively an infrared camera and a color camera, the steps of obtaining the testee's heart rate, number of breaths and heart rate variation sequence thereof by analyzing the cheek video images are specifically:

(a) For the cheek video image obtained by the color camera, separate the RGB channels of the cheek region, and average the pixels of each channel;

(b) averaging the pixels of the R channel of the cheek region in the cheek video image acquired by the infrared camera;

(c) Perform normalization, whitening processing and independent component analysis on the data in the four channels in turn;

(d) save the data into 4 channels, each channel has N data;

(e) performing normalization, whitening processing and independent component analysis on the data of each channel in sequence;

(f) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the subject;

(g) Calculate the heart rate variation value of the testee through the heart rate and breathing rate of the testee in a single channel, and then obtain the heart rate variation sequence through the calculation of multiple channels;

A kind of sleep state monitoring device based on machine vision, described sleep state monitoring device comprises: 2 cameras, coder, display, embedded system, alarm,

The camera is used to record the image of the subject's head and face in real time and input it to the encoder;

The encoder is used to encode the image information of the camera into a digital image signal and send it to the embedded system;

The embedded system is used to extract the visual characteristics of the frequency of the testee's mouth opening and closing, the head position and its maintenance time, the frequency of transposition, and the physiological parameter characteristics of the testee such as pulse rate. These feature information are fused and processed and judged and alarmed; when the sleep state of the subject is found to be abnormal, the alarm signal is output to the alarm, and the abnormal signal is output to the parents or related guardians at the same time;

The alarm is used to sound an alarm;

The display is used to display the physiological parameters of the subject.

The camera is: an infrared camera and/or a color camera.

In order to obtain higher-precision video images, the color camera is: ANC Core HD1080P, and the infrared camera is Samsung SCO-2080RP.

Further, in order to increase the processing speed, the embedded system is: one of i.MX6 processor, OMAP processor and TigerSHARC processor.

The beneficial effect of the technical solution provided by the present invention is that the present invention provides a safe and real-time monitoring method and device in order to ensure the safety of the subject. The present invention not only detects the physiological parameter characteristics of the subject, but also detects the The visual characteristics of the tester's head and face, and only using the camera to obtain the above information, not only retains the non-contact measurement that does not affect the behavior of the tester and increases the wearing burden of the tester, but also obtains the More information, including physiological parameters, greatly improves the reliability of the sleep state monitoring of the subjects.

Description of drawings

Fig. 1 is a flow chart of a sleep state monitoring method based on machine vision provided by the present invention;

Fig. 2 is a flow chart of extracting the physiological parameter characteristics of the subject such as pulse rate and respiration based on image processing provided by the present invention;

Fig. 3 is another flow chart of a kind of sleep state monitoring method based on machine vision provided by the present invention;

Fig. 4 is another flowchart of extracting the physiological parameter characteristics of the subject such as pulse rate and respiration based on image processing provided by the present invention;

Fig. 5 is another flow chart of a kind of sleep state monitoring method based on machine vision provided by the present invention;

Fig. 6 is another flowchart of extracting the physiological parameter characteristics of the subject such as pulse rate and respiration based on image processing provided by the present invention;

Fig. 7 is a structural diagram of a sleep state monitoring device based on machine vision provided by the present invention.

In the accompanying drawings, the list of parts represented by each label is as follows:

1: Camera; 2: Encoder;

3: Display; 4: Embedded system;

5: Alarm.

Wherein, the camera 1 is an infrared camera, a color camera, and a combination of an infrared camera and a color camera in the three device embodiments.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the implementation manners of the present invention will be further described in detail below.

Example 1

A kind of sleep state monitoring method based on machine vision, adopts 2 infrared cameras when acquiring video image, referring to Fig. 1 and Fig. 2, this method comprises the following steps:

101: Obtain video images of the subject's face through the two infrared cameras on the upper left and upper right of the subject's pillow;

Wherein, in order to improve the accuracy of the acquired video image, the infrared camera used in this step adopts Samsung SCO-2080RP, and it can also be selected according to the needs in practical applications during specific implementation, which is not limited in the embodiment of the present invention.

102: Clean the video image, and obtain the cleaned video image;

Among them, the cleaning method can use image clipping and screening, image grayscale and histogram equalization, image normalization, video frame matching, image whitening processing, and removal of singular images to optimize the data set according to the difference of the collected video images. and other processing methods. For example, preprocessing steps such as histogram equalization, extraction of image feature points and normalization can be used to remove the influence of ambient light intensity changes. The detailed operation of this step is well known to those skilled in the art, and will not be repeated in this embodiment of the present invention.

103: Segment the cleaned video images to obtain eye video images, lip video images, and cheek video images;

Among them, the method of segmenting the cleaned video image is as follows: according to the detected video image of the human face, the area of the human face is divided according to the "three courts and five eyes" principle of the human face structure, so as to segment the eye area. video images of the upper body, lips, and cheeks. The principle of "three courts and five eyes" is well known to those skilled in the art, and will not be described in detail in the embodiment of the present invention.

104: Through eye video images, lip video images and cheek video images, respectively detect and characterize the visual behavior characteristics and physiological parameter characteristics of the testee:

1) Detect the frequency of the subject's mouth opening and closing through the lip video image, detect the subject's head position and the time it is maintained, and the visual characteristics of the transposition frequency through the eye video image;

For example: if the frequency of changing the position of the head of a person is too high, it can be determined to some extent that the person is dozing off according to some experience and data standards.

2) Obtain the heart rate, respiration rate and heart rate variation sequence of the subject by analyzing the cheek video image (shown in FIG. 2 ).

The step 2) is specifically:

(a) average the pixels of the R channel in the cheek region, and there are N data in the R channel;

(b) Perform normalization, whitening processing and independent component analysis on the data in the R channel in sequence;

Wherein, the normalization, whitening processing and independent component analysis used in the processing of step (b) are several technologies well known to those skilled in the art, which will not be described in detail in the embodiment of the present invention.

(c) Perform Fourier transformation on the data after each independent component analysis to obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the frequency corresponding to the second maximum amplitude of the power spectrum Frequency, that is, the number of breaths of the testee;

(d) Calculating the heart rate variation value of the test subject through the test subject's heart rate and breathing rate.

Wherein, this step adopts the principle of Lomb periodogram in the field, and this embodiment of the present invention will not repeat it in specific implementation.

105: Perform fusion judgment on the above parameters to obtain a sleep state detection result of the test subject.

Wherein, the fusion judgment described in this step can be:

1) By characterizing the visual behavior characteristics of the tested subject, the sleep detection result of the tested subject can be directly obtained; or,

2) The detection result of the sleep state of the subject can be directly obtained through the heart rate variation sequence; or,

3) By characterizing the visual behavior characteristics of the test subject and the weighted sum of the heart rate variation sequence, the detection result of whether the test subject enters the sleep state is obtained.

It is about to compare the weighting of the visual behavior characteristics and the heart rate variation sequence representing the subject to the preset threshold, and if it is greater than the threshold, it is directly judged that the subject has entered a sleep state. Wherein, the weight coefficient and the threshold are set according to requirements in practical applications, which are not limited in this embodiment of the present invention.

Example 2

A kind of sleep state monitoring method based on machine vision, adopts 2 color cameras when acquiring video images, referring to Fig. 3 and Fig. 4, this method comprises the following steps:

201: Obtain video images of the subject's face through the two color cameras on the upper left and upper right of the subject's pillow;

Among them, in order to improve the accuracy of the acquired video image, the color camera used in this step adopts ANC Core, which can also be selected according to the needs of practical applications during specific implementation, which is not limited in the embodiment of the present invention.

202: Clean the video image, and obtain the cleaned video image;

203: Segment the cleaned video images to obtain eye video images, lip video images and cheek video images;

Wherein, the operation steps of step 202 and step 203 are the same as those in embodiment 1, which will not be described in detail in this embodiment of the present invention.

204: Through eye video images, lip video images and cheek video images, respectively detect and characterize the visual behavior characteristics and physiological parameter characteristics of the testee:

1) Detect the frequency of the subject's mouth opening and closing through the lip video image, detect the subject's head position and the time it is maintained, and the visual characteristics of the transposition frequency through the eye video image;

For example: if the frequency of changing the position of the head of a person is too high, it can be determined to some extent that the person is dozing off according to some experience and data standards.

2) Obtain the heart rate, number of breaths and heart rate variation sequence thereof (referring to Fig. 4) of the testee by analyzing the cheek video image (see Figure 4). This step 2) is specifically:

(a) Separate the RGB channels of the cheek region, and average the pixels of each channel;

(b) Save the data into 3 channels, each with N data;

(c) Perform normalization, whitening processing and independent component analysis on the data of each channel in turn;

Among them, the correlation between the three channels is eliminated through whitening processing, and the interference among the channels is prevented. The normalization, whitening processing and independent component analysis used in the processing of step (c) are several technologies well known to those skilled in the art, which will not be described in detail in the embodiment of the present invention.

(d) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the subject;

(e) Calculating the heart rate variation value of the testee through the heart rate and respiration rate of the testee in a single channel, and then obtaining the heart rate variation sequence through the calculation of multiple channels.

Wherein, this step adopts the principle of Lomb periodogram in the field, and this embodiment of the present invention will not repeat it in specific implementation.

205: Perform fusion judgment on the above parameters to obtain a sleep state detection result of the test subject.

Wherein, the operation steps in step 205 are the same as those in Embodiment 1, which will not be repeated in this embodiment of the present invention.

Example 3

A kind of sleep state monitoring method based on machine vision, adopts 1 infrared camera and 1 color camera when acquiring video image, see Fig. 5 and Fig. 6, this method comprises the following steps:

301: Obtain a video image of the subject's face through the infrared camera and the color camera set on the upper left and right sides of the subject's pillow;

Among them, in order to improve the accuracy of the video image obtained, the infrared camera used in this step adopts Samsung SCO-2080RP, and the color camera adopts ANC Core. During specific implementation, it can also be selected according to the needs in practical applications. There is no limit to this. That is, in this step, a video image with RGB three channels and a video image with only R channel are acquired.

302: Clean the two video images respectively, and obtain the two video images after cleaning;

303: Segment the cleaned two video images to obtain eye video images, lip video images, and cheek video images respectively;

Wherein, the operation steps of step 302 and step 303 are the same as those in embodiment 1, and are not described in detail in this embodiment of the present invention.

304: Detect and characterize the visual behavior characteristics and physiological parameter characteristics of the testee respectively through the eye video images, lip video images and cheek video images:

1) Detect the frequency of the subject's mouth opening and closing through the lip video image, detect the subject's head position and the time it is maintained, and the visual characteristics of the transposition frequency through the eye video image;

For example: if the frequency of changing the position of the head of a person is too high, it can be determined to some extent that the person is dozing off according to some experience and data standards.

2) Obtain the heart rate, breathing frequency and heart rate variation sequence of the subject by analyzing the cheek video image,

Referring to Fig. 6, the step 2) is specifically:

(a) For the cheek video image obtained by the color camera, separate the RGB channels of the cheek region, and average the pixels of each channel;

(b) averaging the pixels of the R channel of the cheek region in the cheek video image acquired by the infrared camera;

(c) Perform normalization, whitening processing and independent component analysis on the data in the four channels (2 R channels, 1 G channel and 1 B channel);

(d) save the data into 4 channels, each channel has N data;

(e) performing normalization, whitening processing and independent component analysis on the data of each channel in turn;

Among them, the correlation between the four channels is eliminated through whitening processing, and the interference between the channels is prevented. The normalization, whitening, and independent component analysis used in the processing of step (e) are several technologies well known to those skilled in the art, which will not be described in detail in this embodiment of the present invention.

(f) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the subject;

(g) Calculating the heart rate variation value of the testee through the heart rate and respiration rate of the testee in a single channel, and then obtaining the heart rate variation sequence through the calculation of multiple channels.

Wherein, this step adopts the principle of Lomb periodogram in the field, and this embodiment of the present invention will not repeat it in specific implementation.

305: Perform fusion judgment on the above parameters to obtain a sleep state detection result of the testee.

Wherein, the operation steps in step 305 are the same as those in Embodiment 1, which will not be repeated in this embodiment of the present invention.

Example 4:

A sleep state monitoring device based on machine vision, corresponding to the method in Embodiment 1, referring to Fig. 7, the monitoring device includes: 2 infrared cameras 1, encoder 2, display 3, embedded system 4, alarm 5,

Two infrared cameras 1 located directly in front of and above the subject record real-time images of the subject's head and face and input them to the encoder 2, and the encoder 2 encodes the image information of the infrared camera 1 into digital image signals and sends them to the embedded System 4, in the embedded system 4, the frequency of opening and closing the mouth of the subject, the head position and its maintenance time, the visual features of the frequency of transposition, and the physiological parameters of the subject such as pulse rate are extracted , these feature information are fused and processed and judged and alarmed; when the sleep state of the subject is found to be abnormal, the alarm 5 sends out an alarm sound, and the embedded system 4 simultaneously outputs an abnormal signal to parents or related guardians.

Wherein, the infrared camera 1 adopts Samsung SCO-2080RP; the embedded system adopts OMAP produced by TI Company, and the display 3 can be an LCD display and the like.

Example 5

A sleep state monitoring device based on machine vision, corresponding to the method in Embodiment 2, referring to Fig. 7, the monitoring device includes: 2 color cameras 1, encoder 2, display 3, embedded system 4, alarm 5,

Two color cameras 1 located directly in front of and above the subject record real-time images of the subject's head and face and input them to the encoder 2, and the encoder 2 encodes the image information of the color camera 1 into digital image signals and sends them to the embedded System 4, in the embedded system 4, the frequency of opening and closing the mouth of the subject, the head position and its maintenance time, the visual features of the frequency of transposition, and the physiological parameters of the subject such as pulse rate are extracted , these feature information are fused and processed and judged and alarmed; when the sleep state of the subject is found to be abnormal, the alarm 5 sends out an alarm sound, and the embedded system 4 simultaneously outputs an abnormal signal to parents or related guardians.

Wherein, the color camera 1 adopts ANC Core HD1080P, the embedded system adopts OMAP produced by TI, and the display 3 can be an LCD display.

Embodiment 6:

A sleep state monitoring device based on machine vision, corresponding to the method in Embodiment 3, referring to Fig. 7, the monitoring device includes: an infrared camera and a color camera 1, an encoder 2, a display 3, an embedded system 4, and an alarm device 5,

The infrared camera and the color camera 1, which are directly in front of the subject, respectively record the images of the subject's head and face in real time and input them to the encoder 2, and the encoder 2 encodes the image information of the infrared camera and the color camera 1 into a digital image The signal is sent to the embedded system 4, and in the embedded system 4, the frequency of opening and closing the mouth of the subject, the position of the head and its maintenance time, the visual features of the frequency of transposition, and the pulse rate are extracted from the tested person. According to the physiological parameter characteristics of the testee, these characteristic information are fused and processed and judged and alarmed; when the sleep state of the tested person is found to be abnormal, the alarm 5 sends out an alarm sound, and the embedded system 4 simultaneously outputs an abnormal signal to the parents or related guardians.

Among them, the color camera adopts ANC Core HD1080P, the infrared camera adopts Samsung SCO-2080RP; the embedded system adopts TigerSHARC produced by Analog Devices, and the display 3 can be an LCD display.

In the embodiments of the present invention, unless otherwise specified, the models of the devices are not limited, as long as they can complete the above functions.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the above-mentioned embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (9)

1. a sleep state monitoring method based on machine vision, is characterized in that, described method comprises the following steps: Obtain video images of the subject's face through the two cameras on the upper left and right sides of the subject's pillow; Cleaning the video image to obtain the cleaned video image; Segment the cleaned video images to obtain eye video images, lip video images and cheek video images; Through eye video images, lip video images and cheek video images, respectively detect and characterize the visual behavior characteristics and physiological parameter characteristics of the testee: The above parameters are fused and judged to obtain the sleep state detection result of the test subject. the 2. a kind of sleep state monitoring method based on machine vision according to claim 1, is characterized in that, described by eye video image, lip video image and cheek video image, respectively detects the vision that characterizes the testee. The steps of behavioral characteristics and physiological parameter characteristics are as follows: 1) Detect the frequency of the testee's mouth opening and closing through the lip video image, and detect the visual characteristics of the testee's head position and its maintenance time and transposition frequency through the eye video image; 2) Obtain the heart rate, breathing frequency and heart rate variation sequence of the subject by analyzing the video image of the cheek. the 3. a kind of sleep state monitoring method based on machine vision according to claim 2, is characterized in that, when two cameras are all infrared cameras, the heart rate, The specific steps of respiratory frequency and its heart rate variability sequence are as follows: (a) Average the pixels of the R channel in the cheek region, and there are N data in the R channel; (b) Perform normalization, whitening processing and independent component analysis on the data in the R channel in turn; (c) Perform Fourier transformation on the data after each independent component analysis to obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the frequency corresponding to the second maximum amplitude of the power spectrum Frequency, i.e. the number of breaths of the person being tested; (d) Calculating the heart rate variation value of the test subject through the test subject's heart rate and breathing rate. the 4. a kind of sleep state monitoring method based on machine vision according to claim 2, is characterized in that, when two cameras are all color cameras, the heart rate, The specific steps of respiratory frequency and its heart rate variability sequence are as follows: (a) Separate the RGB channels of the cheek region, and average the pixels of each channel; (b) Save the data into 3 channels, each channel has N data; (c) Perform normalization, whitening processing and independent component analysis on the data of each channel in turn; (d) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the testee; (e) Calculating the heart rate variation value of the testee through the heart rate and respiration rate of the testee in a single channel, and then obtaining the heart rate variation sequence through the calculation of multiple channels. the 5. A kind of sleep state monitoring method based on machine vision according to claim 2, is characterized in that, when two cameras are infrared camera and color camera respectively, described by the video image analysis of the opposite cheek to obtain the person under test The specific steps of heart rate, respiration rate and heart rate variation sequence are as follows: (a) For the cheek video image obtained by the color camera, separate the RGB channels of the cheek region, and average the pixels of each channel; (b) average the pixels of the R channel of the cheek region in the cheek video image obtained by the infrared camera; (c) Perform normalization, whitening processing and independent component analysis on the data in the four channels in turn; (d) save the data into 4 channels, each channel has N data; (e) Perform normalization, whitening processing and independent component analysis on the data of each channel in turn; (f) Perform Fourier transformation on the data after the independent component analysis of each channel, obtain the power spectrum, and obtain the frequency corresponding to the maximum amplitude of the power spectrum, that is, the heart rate of the subject, and obtain the second maximum amplitude of the power spectrum corresponding to The frequency, that is, the number of breaths of the subject; (g) Calculating the heart rate variation value of the testee through the heart rate and respiration rate of the testee in a single channel, and then obtaining the heart rate variation sequence through the calculation of multiple channels. 6. A sleep state monitoring device based on machine vision, said sleep state monitoring device comprising: 2 cameras, encoder, display, embedded system, alarm, characterized in that, The camera is used to record the image of the subject's head and face in real time and input it to the encoder; The encoder is used to encode the image information of the camera into a digital image signal and send it to the embedded system; The embedded system is used to extract the visual characteristics of the frequency of the testee's mouth opening and closing, the head position and its maintenance time, the frequency of transposition, and the physiological parameter characteristics of the testee such as pulse rate. These feature information are fused and processed and judged and alarmed; when the sleep state of the subject is found to be abnormal, an alarm signal is output to the alarm, and an abnormal signal is output to parents or related guardians at the same time; The alarm is used to sound an alarm; The display is used to display the physiological parameters of the subject. the 7. A kind of sleep state monitoring device based on machine vision according to claim 6, is characterized in that, described camera is: Infrared camera and/or color camera. the 8. a kind of sleep state monitoring device based on machine vision according to claim 7, is characterized in that, The color camera is: ANC Core HD1080P, The infrared camera is Samsung SCO-2080RP. the 9. a kind of sleep state monitoring device based on machine vision according to claim 6, is characterized in that, described embedded system is: One of i.MX6 processor, OMAP processor and TigerSHARC processor. the