CN117617889A - A strabismus degree detection system based on covering test - Google Patents

Abstract

The invention discloses a system for detecting the degree of strabismus based on a coverage test, which comprises a head rest, a calibration point, a display, a camera, a light source, a voice broadcasting device and a control device, wherein the head rest is connected with the display; the head rest is arranged opposite to the display, and planes of the head rest and the display are parallel to each other; the calibration points are arranged right above the head support through a bracket consisting of two symmetrical upright posts and a cross beam on the upright posts, and the calibration points are arranged in the middle of the cross beam; the camera and the light source are respectively arranged on the display and are positioned in the same plane, the plane is parallel to the plane of the display, and the view field of the camera covers the head rest; the display, the camera and the voice broadcasting device are all connected with the controller. The invention is not affected by kappa angle, can accurately measure the strabismus degree, and solves the defects that the traditional strabismus degree detection technology is subjective and needs professional doctors to participate in measurement.

Description

A strabismus degree detection system based on covering test

Technical field

The invention belongs to the technical field of ophthalmic disease detection, and specifically relates to a strabismus degree detection system based on covering test.

Background technique

When a normal person looks at a target, both eyes look at the same time. However, for patients with strabismus, usually only one eye on both sides can be aligned with the target, and the other eye is deflected. This phenomenon of the inability of both eyes to focus on the target at the same time is medically called strabismus. Strabismus will have a serious impact on the patient's vision, and will also hinder the patient's life, work and psychology. If strabismus is not detected and treated promptly in childhood, it can easily lead to loss of direction recognition or amblyopia in adulthood.

At present, common clinical strabismus specialist examination methods include corneal photoreflection, homotopia, cover test, etc. Corneal photoreflection requires the examiner to sit opposite the patient. The examiner holds the written test flashlight at eye level with the patient, and asks the patient to look at the light source of the written test flashlight in front of him. The examiner then observes the position of the light source's reflective point on the cornea straight ahead to determine whether there is eye deviation. Corneal photometry cannot accurately obtain the degree of strabismus and can only be used for rough judgment. If the reflection point is located at the edge of the pupil, the degree of strabismus is about 10° to 15°. When it is located between the edge of the cornea and the edge of the pupil, the degree of strabismus is about 25° to 30°. When it is located at the edge of the cornea, the degree of strabismus is about 45°. At the same time, this method cannot eliminate the influence of kappa angle. Synopsis machine is a large-scale photoelectric instrument that combines light, mechanics and electricity. When measuring the subject's strabismus angle with a synoptic machine, first fix the spectacle tube at 0°, and the subject looks at the picture through the lens tube. The doctor alternately turns off the light source of the picture and observes the movement of the eyeballs. Adjust the lens tube wall on one side so that when the left and right eyes look at the picture alone, they will no longer move. At this time, the degree pointed by the lens tube arm is the strabismus degree. However, the price of the homotrope machine is relatively expensive, and the use of the homotrope machine to measure the degree of strabismus requires the participation of a professional physician. . The cover test can measure the overt and hidden degree of strabismus and is simple, easy to perform and highly reproducible. The covering test is divided into unilateral covering test, covering and uncovering test and alternating covering test. The unilateral cover test is used to check for dominant strabismus, the cover-to-cover test is used to check for covert strabismus, and the alternating cover test can check for both types of strabismus. However, cover testing is subject to physician subjective factors and cannot quantify the degree of strabismus.

It can be seen that there is an urgent need to develop an intelligent detection technology that can achieve objective and accurate strabismus degree.

Contents of the invention

The purpose of the present invention is to provide a strabismus degree detection system and detection method based on cover testing to solve the problem that traditional strabismus degree detection technology is relatively subjective and requires professional doctors to participate in measurement.

In order to achieve the above objects, the present invention adopts the following technical solutions to solve them:

A strabismus degree detection system based on covering test, including a headrest, a calibration point, a display, a camera, a light source, a voice announcer and a control device; wherein, the headrest and the display are arranged oppositely and both the headrest and the display are located The planes are parallel to each other; the calibration point is installed directly above the headrest through a bracket composed of two symmetrical columns and a beam on the column, and the calibration point is installed in the middle of the beam; the camera and light source are respectively installed on the display And the two are located in the same plane, which plane is parallel to the plane where the display is located, and the field of view of the camera covers the headrest; the display, camera and voice announcer are all connected to the controller.

Further, the calibration points are black circular points.

Further, the controller includes a system calibration module, a pupil and reflective point positioning module, a calibration module and a cover detection module, wherein:

The system calibration module is used to implement the following process: Step 11, establish a world coordinate system with the lower left corner of the display as the origin, the height of the display is SH, the width is SW, and the resolution is RWxRH; Step 12, control the camera to collect images; Step 13, based on the collected image, obtain the conversion ratio ratio from the plane pixel of the headrest to the distance, and manually measure the coordinates of the calibration point center in the world coordinate system (X _b , Y _b , Z _b );

The pupil and reflective point positioning module is used to collect human eye images, and obtain the pixel coordinates of the pupil centers of both eyes, the coordinates of the pupil center in the world coordinate system, and the pixels of the reflective point centers in the pupils based on the collected human eye images. coordinate;

The calibration module is used to call the pupil and reflective point positioning module to calibrate the person being measured and calculate the Hesburgh ratio;

The cover detection module is used to call the pupil and reflective point positioning module, and control the voice announcer to prompt the person being tested to use the cover test method to measure the degree of strabismus.

Further, step 13 includes the following operations:

Convert the collected image into a grayscale image, use Gaussian filtering or other methods to denoise the image; use adaptive threshold segmentation or other methods to threshold segment the denoised image; use morphological opening operations to remove the noise after threshold segmentation Isolated points, burrs, etc. in the image are obtained from the image after morphological operation; use canny edge detection or other methods to detect the edges of the image after morphological operation; use Hough transform or other circular detectors to detect the image obtained by the above processing For the calibration point in the image, the pixel coordinates (x _b , y _b ) of the calibration point and the pixel radius r _b of the calibration point on the image are obtained; the calculation formula for the conversion ratio from the plane pixel of the headrest to the distance is calculated as ratio= R/r _b , where R is the true radius of the calibration point; the coordinates (X _b , Y _b , Z _b ) of the calibration point center in the world coordinate system are obtained through a tape measure or other distance measurement tools.

Further, the implementation process of the pupil and reflective point positioning module is as follows:

Step 21, control the camera to continuously collect human eye images at a collection rate of 60fps, directly collect images containing human eyes, and then flip the images horizontally;

Step 22: Based on the image obtained in step 21, first locate the position of both eyes through the human eye detection method, and then use pupil positioning technology in the two areas to obtain the pixel coordinates of the two pupil centers; and calculate the position of the pupil center in the system calibration module. Coordinates in the established world coordinate system;

Step 23: Take the left eye pupil center (x _L , y _L ) as the center of the circle, and use about 1.5 times the left eye pupil radius R _L as the radius to make a circular area. Find the center of the reflective point in the left eye pupil in this area. Pixel coordinates (x _g , y _g ); similarly, obtain the pixel coordinates of the center of the reflective point in the pupil of the right eye.

Further, the specific process of step 22 is as follows:

(1) The human eye detection method adopts any of the following methods: (1) manually determine the position of the eyes based on multiple collected human eye images; (2) collect and mark multiple positive and negative samples, and train Haar-cascade classifier, the trained classifier can obtain the position of the eyes of the image; (3) Human eye detection is performed through deep learning method training;

(2) Pupil position detection uses the segmentation network model in deep learning. After inputting the image, the segmentation probability map is obtained, and then the binary image is obtained through threshold segmentation. Connected domain analysis is performed in the binary image, and the centroid of the largest spot is used as the left eye. The pixel coordinates of the pupil center (x _L , y _L ), take the long side of the circumscribed rectangular frame of the largest spot as the left eye pupil diameter 2*R _L , R _L is the pupil radius;

(3) Calculate the left eye pupil center in the world coordinate system based on the world coordinate system established in step 11 and the coordinates (X _b , Y _b , Z _b ) of the calibration point measured in step 12 in the world coordinate system The coordinates are (X _L ,Y _L ,Z _L ); where, Z _L =Z _b , where r _b is the pixel radius of calibration point 3 on the image, R is the real radius of the calibration point (cm), (x _b , y _b ) is the pixel coordinate of the calibration point center;

(4) In the same way, perform steps (2) and (3) for the right eye to obtain the pixel coordinates (x _R , y _R ) of the pupil center of the right eye, the pupil radius R _R , and the pupil center of the right eye in the world coordinate system. The coordinates (X _R , Y _R , Z _R ).

Further, the implementation process of the calibration module is as follows:

Step 31, initialize 9 gaze points with known coordinates: P ₁ (X ₁ ,Y ₁ ,0), P ₂ (X ₂ ,Y ₂ ,0)...P ₉ (X ₉ ,Y ₉ ,0); The pixel coordinates of gaze point P ₀ are (x ₀ , y ₀ ), and its world coordinates are (X ₀ , Y ₀ , 0), then X ₀ =x ₀ *SW/RW, Y ₀ = (RH-y ₀ ) *SW/RW; respectively calculate the offset gaze angles from the gaze point P ₀ to the gaze points P ₁ , P ₂ ...P ₉ RH is the height of the monitor (cm), RW is the width of the monitor (cm);

Step 32: Display the gaze point P ₀ (X ₀ , Y ₀ , 0) on the display, control the voice announcer to prompt the person to be tested to place their head on the headrest, and remind the person to watch the coordinate point P ₀ ; then At least two gaze points with known positions are displayed on the display in sequence as calibration points, and the voice announcer is controlled to remind the subject to gaze at these calibration points in sequence. During this process, the system calls the pupil and reflective point positioning module to compare the pupil and the internal Reflective point positioning; each calibration point is displayed for 5 seconds. When the subject's gaze lasts for 2 seconds or more, the current calibration point is considered stable; calculate the mean and standard deviation of the pupil center position within 2 seconds. If the pupil center position coordinates within 2 seconds If the standard deviation is less than 1.5 times, the gaze is considered stable. When the current gaze point is stable, record the pixel coordinates of the pupil center and the pixel coordinates of the reflective points in all image frames during the last 2 seconds of each gaze point display;

Step 33: For the pixel coordinates of the pupil center and the pixel coordinates of the reflective points recorded in the last 2 seconds of each fixation point display, eliminate the data with eyes closed;

Calculate the mean value of the abscissa coordinate of the remaining pupil center pixel coordinates after removing the data The variance of the abscissa Dx _p and the mean of the ordinate/> Vertical coordinate variance Dy _p ; if the abscissa coordinate of a certain pupil center is greater than/> Or the ordinate is greater than If the calibration fails, otherwise the calibration is successful; similarly, calculate the mean value of the abscissa of the center pixel coordinate of the reflective point/> The variance of the abscissa Dx _g and the mean of the ordinate/> Vertical coordinate variance Dy _g ; if the abscissa coordinate of a certain pupil center is greater than/> Or the ordinate is greater than/> If the calibration fails, otherwise the calibration is successful;

After the calibration fails, return to step 31, that is, re-enter the calibration module; if all nine fixation points are successfully calibrated, enter step 34;

Step 34: Calculate the horizontal pixel distance d ₀ ~ d ₉ from the pupil center to the center of Purchin's spot when observing points P ₀ ~ P ₉ respectively, where Calculate the horizontal pixel distance from the pupil center of observation points P ₁ to P ₉ to the center of Purchin's spot and the offset t ₁ to t ₉ at point P ₀ , where _ti =d _i -d ₀ (i=1, 2... …9);

Step 35: After obtaining (t ₁ ,A ₁ ), (t ₂ ,A ₂ )...(t ₉ ,A ₉ ), use the least squares method to perform straight line fitting on them, A=H ₀ t+b ₀ , The Hesburgh ratio H ₀ and the intercept b ₀ of the fitted straight line are calculated.

Further, in the covering detection module, the covering test method selects a unilateral covering test method, a covering and uncovering test method or an alternating covering method.

Furthermore, the covering and de-covering method is used in the covering detection module. The specific implementation process is as follows:

Step 41: Control the voice announcer to prompt the person being tested to start covering the test. After starting the test, the system calls the pupil and reflective point positioning module to continuously collect images and obtain the positions of the eyes and the pixel coordinates of the pupil and reflective point centers; at this time, if the detection If both eyes are not detected, then re-enter step 41 and control the voice announcer to prompt the subject to adjust the position until both eyes are detected. If both eyes and the center of the pupil and the reflective point are detected, then enter step 42;

Step 42, monitor the eye status of both eyes: first, the eyes are in the initial state s1 when monitoring starts, and then the eyes are in the ready state s2, and the voice announcer is controlled to prompt the subject to cover the right eye; the system calls the pupil and reflective point positioning The module detects the state of the right eye. If the number of consecutive frames of the eye pupil cannot be detected within fifty frames, the right eye is in the blinking state s0; when the right eye is in the blinking state and the right eye pupil is re-detected, the right eye state returns to ready. State s2; when the number of consecutive frames in which the pupil cannot be detected reaches fifty frames or more, the right eye enters the occlusion state s3, and then enters step 43;

Step 43: Control the voice announcer to prompt the subject to remove the baffle from the right eye and cover the left eye. The system records the moment when the baffle is removed from the right eye. The first frame that can simultaneously detect the reflective point and the pupil is the key frame. This When the right eye is in the instant state s4, the system calculates and saves the difference p ₀ between the pupil pixel coordinates of the frame and the pixel coordinates of the reflective point; after the right eye enters the instant state s4 for more than 2 seconds, the system calculates the pupil pixel coordinates within 60 frames. The mean and variance of the components. If the variance of the coordinate components is less than the set variance threshold T _d , the eye enters the steady state s5; calculate the pixel coordinates of the pupil center and the pixel center of the reflective point within 60 frames when entering the steady state. Difference p ₁ ; control the voice announcer to prompt the subject that the covering test is over, and enter step 44;

Step 44, calculate the degree of strabismus: Calculate the offset of the right eye t = p ₁ - p ₀ , and calculate the degree of strabismus of the right eye A = H ₀ t + b ₀ ; similarly, cover the left eye first and then the right eye, The offset and strabismus of the left eye were calculated.

Compared with the existing technology, the present invention has the following technical effects:

1. The present invention is not affected by kappa angle and can accurately measure strabismus. The Hesburgh ratio H is obtained by calibrating the subject, and then using the image algorithm to measure the offset T1 and T2 from the center of the subject's pupil to the center of the reflective point during the cover test, and calculate the strabismus degree A0 = H (T1 -T2). The process of measuring the offset eliminates the kappa angle error. Moreover, the present invention calculates the Hesberg ratio H for each subject during the calibration stage instead of using the average value. The mean value of H is 12.5, and the variability among subjects is ±20% of the mean, so measuring the Hesberg ratio for each subject makes the final result more accurate.

2. The devices used in this method include cameras, infrared light sources, computer hosts and computer monitors, and there is no need to use professional optical instruments.

3. The present invention defines the eye state, including initial state, ready state, blinking state, occlusion state, instantaneous state and stable state. In the instantaneous state and stable state, the image information is automatically saved to calculate the degree of strabismus. Therefore, the present invention does not require the participation of professional doctors and can intelligently measure the degree of strabismus. At the same time, the present invention uses a voice announcer to interact with the user, and the voice prompts the subject to perform a covering test, and can automatically detect the occlusion state of the subject's eyes, and obtain strabismus detection results after the covering test.

4. The present invention combines the unilateral covering test to measure whether the subject has overt strabismus. If so, the degree of strabismus is given. Otherwise, the alternating covering test is performed to determine whether the patient has invisible strabismus. If so, the degree of strabismus is given. Otherwise, the degree of strabismus is given. The subjects were normal. Therefore, the present invention can measure both the obviousness and recessiveness of strabismus and the accurate degree of strabismus.

Description of drawings

Figure 1 is a schematic structural diagram of the strabismus detection system of the present invention.

Figure 2 is a diagram of the eye state during the working process of the strabismus monitoring system of the present invention.

Figure 3 is a schematic diagram of human eye detection rectangular frame.

Figure 4 is a schematic diagram of pupil and reflective point detection.

The meanings of the symbols in the figure are as follows:

1. Head rest; 2. Eye plane; 3. Calibration point; 4. Eyes; 5. World coordinate system; 6. Gaze point; 7. Display; 8. Camera; 9. Infrared light source.

Detailed ways

The basic principle of the present invention is that the near-infrared light emitted by the near-infrared light source will form high-brightness reflection points on the cornea of the user's eyes, becoming Purkin spots. When the head is fixed, the horizontal offset of Purchin's spot to the pupil center (mm) is linearly related to the gaze offset angle (°), and the slope is called the Hirschberg ratio. First, calibrate the camera, then calibrate the Hesburgh ratio H, and then conduct a covering and removing experiment on the subject. During the experiment, the camera collects eye images, and uses an algorithm to determine the image frame frame 1 at the moment when the strabismus eye is covered, and the image frame frame 2 after the eye is stabilized. Use an image processing algorithm to obtain the horizontal offsets T1 and T2 of the two needles. Calculate the strabismus degree A0=H(T1-T2) through H.

Principle of the invention: The offset of the displacement of Purchin's spot to the pupil center is proportional to the offset of the gaze angle. Intelligent detection of strabismus degree is completed through image processing algorithm and eye state judgment.

The strabismus degree detection system based on covering test of the present invention includes a headrest 1, a calibration point 3, a display 7, a camera 8, a light source 9, a voice announcer and a control device; wherein, the headrest 1 is arranged opposite to the display 7 and the head The planes of the support 1 and the monitor 7 are parallel to each other, and the distance between the monitor 7 and the plane of the head support 1 is 30cm to 60cm; the calibration point 3 is installed on the head through a bracket composed of two symmetrical columns and beams on the columns. Directly above the support 1, the calibration point 3 is installed in the middle of the beam. The distance between the two columns is 18-25cm, allowing an adult to place his head while facing the monitor 7. The camera 8 and the light source 9 are respectively installed on the display and are located in the same plane, which is parallel to the plane of the display 7. The field of view of the camera 8 covers the headrest 1 to ensure that the eye image of the person being tested can be captured. The distance between the camera 8 and the infrared light source 9 is no more than 15cm. The display 7, the camera 8 and the voice announcer are all connected to the controller.

Preferably, calibration point 3 is a black circular point.

Specifically, the controller includes a system calibration module, a pupil and reflective point positioning module, a calibration module and a cover detection module. in:

System calibration module is used to implement the following processes:

Step 11: Establish a world coordinate system with the lower left corner of the display 7 as the origin. The height of the monitor is SH, the width is SW, and the resolution is RWxRH. Example: The height SH of a 24-inch monitor is 29.9cm, the width SW is 53.15cm, and the resolution is 1920x1080.

Step 12, control the camera 8 to collect images (set the image acquisition format to YUY2, and the image acquisition rate to 60fps);

Step 13: According to the collected image, obtain the conversion ratio ratio (mm/pixel) of the plane pixel of the headrest 1 to the distance, and manually measure the coordinates of the calibration point center in the world coordinate system (X _b , Y _b , Z _b ).

Specifically, step 13 includes the following operations: convert the collected image into a grayscale image, and use Gaussian filtering or other methods to denoise the image; use adaptive threshold segmentation or other methods to perform threshold segmentation on the denoised image; use The morphological opening operation removes isolated points, burrs, etc. in the image after threshold segmentation to obtain the image after morphological operation; uses canny edge detection or other methods to detect the edges of the image after morphological operation; uses Hough transform or other circular shapes The detector detects the calibration point 3 in the image obtained by the above processing, and obtains the pixel coordinates (x _b , y _b ) of the calibration point 3 and the pixel radius r _b of the calibration point 3 on the image; the headrest 1 is calculated The formula for calculating the conversion ratio from plane pixels to distance is ratio=R/r _b , where R is the true radius of the calibration point. The coordinates (X _b , Y _b , Z _b ) of the calibration point center in the world coordinate system can be obtained through a tape measure or other distance measurement tools.

Pupil and reflective point positioning module is used to implement the following processes:

Step 21, control the camera 8 to continuously collect human eye images at a collection rate of 60fps, directly collect images containing human eyes, and then flip the images horizontally. The purpose of horizontal flipping is to make the left eye in the image correspond to the patient's left eye, and the right eye in the image correspond to the patient's right eye for easy observation.

Step 22, based on the image obtained in step 21, first locate the position of the eyes through the human eye detection method (marked by two rectangular boxes, as shown in Figure 3 and Figure 4), and then use the pupil positioning technology in the two areas to obtain Pixel coordinates of the centers of the two pupils. And calculate the coordinates of the pupil center in the world coordinate system established in the system calibration module.

(1) The human eye detection method can adopt any of the following methods: (1) Manually determine the position of the eyes based on multiple collected human eye images. Since after the system is fixed, the heads of different subjects are fixed on the headrest. In the collected images, the positions of the two human eyes are roughly the same relative to the positions of the entire image. Therefore, the two human eyes can be manually selected. A rectangular frame with fixed length, width and position defines the area of the two eyes. (2) Collect and label multiple positive samples (including pictures of human eyes) and negative samples (pictures that do not include human eyes), train the Haar-cascade classifier, and obtain the binocular positions of the image through the trained classifier . (3) Human eye detection is performed through deep learning method training. For example, using the yolo model, collect and label several positive and negative samples through labelimg software, and then generate a txt sample file in yolo format. Download the pre-trained model to speed up model training. After training, save the model parameters, and use the trained classifier to obtain the binocular positions of the image.

(2) Pupil position detection uses segmentation network models in deep learning, such as Unet. Label the collected images with pupils, using the pupil area in the image as the foreground and other areas as the background to create labels. Binary cross entropy loss and elliptical fitting error loss are used as loss items for network training, and the model parameters are saved after training. When used, the model parameters are loaded, the segmentation probability map is obtained after inputting the image, and then the binary image is obtained through threshold segmentation. Connected domain analysis is performed in the binary image, using the centroid of the largest spot as the pixel coordinate (x _L , y _L ) of the left eye pupil center, and the long side of the rectangular frame surrounding the largest spot as the left eye pupil diameter 2*R _L (R _L is the pupil radius).

(3) Calculate the left eye pupil center in the world coordinate system based on the world coordinate system established in step 11 and the coordinates (X _b , Y _b , Z _b ) of the calibration point measured in step 12 in the world coordinate system The coordinates are (X _L , Y _L , Z _L ). in, Z _L =Z _b . Among them, r _b is the pixel radius of the calibration point 3 on the image, R is the real radius of the calibration point (cm), and (x _b , y _b ) is the pixel coordinate of the calibration.

(4) In the same way, perform steps (2) and (3) for the right eye to obtain the pixel coordinates (x _R , y _R ) of the pupil center of the right eye, the pupil radius R _R , and the pupil center of the right eye in the world coordinate system. The coordinates (X _R , Y _R , Z _R ).

Step 23: Take the center of the left eye pupil (x _L , y _L ) as the center of the circle, and use about 1.5 times the radius of the left eye pupil R _L as the radius to make a circular area, and find the center of the reflective point in the left eye pupil in this area. First, threshold segmentation is performed on the area. Since the brightness of the reflective points is high, the segmentation threshold is set to about 220 (the range of image pixel values is 0 to 255). Connected domain analysis is performed on the binary image obtained after segmentation, and the centroid of the largest spot is used as the pixel coordinate (x _g , y _g ) of the center of the reflective point in the pupil of the left eye. In the same way, the pixel coordinates of the center of the reflective point in the pupil of the right eye are obtained.

The calibration module is used to calibrate the person being measured and calculate the Hirschberg ratio.

Step 31: Initialize 9 gaze points with known coordinates: P ₁ (X ₁ ,Y ₁ ,0), P ₂ (X ₂ ,Y ₂ ,0)...P ₉ (X ₉ ,Y ₉ ,0). The pixel coordinates of the gaze point P ₀ are (x ₀ , y ₀ ), and its world coordinates are (X ₀ , Y ₀ , 0), then X ₀ =x ₀ *SW/RW, Y ₀ = (RH-y ₀ ) *SW/RW. Calculate the offset gaze angles from the gaze point P ₀ to the gaze points P ₁ , P ₂ ...P ₉ respectively. RH is the height of the monitor in cm.

Step 32: Display the gaze point P ₀ (X ₀ , Y ₀ , 0) on the display, control the voice announcer to prompt the person to be tested to place their head on the headrest 1, and remind the person being tested to gaze at the coordinate point P ₀ . Then, at least two fixation points with known positions are displayed on the display in sequence as calibration points, and the voice announcer is controlled to remind the subject to fixate on these calibration points in sequence. During this process, the system calls the pupil and reflective point positioning module to determine the pupil and inner Position the reflective point (i.e. repeat step 2). Each calibration point is displayed for 5 seconds. When the subject's gaze lasts longer than or equal to 2 seconds, the current calibration point is considered stable. By default, the subject is observing the calibration point, and the mean and standard deviation of the pupil center position within 2 s are calculated. If the pupil center position coordinates within 2 s are less than 1.5 times the standard deviation, the gaze is considered stable. When the current fixation point is stable, the pixel coordinates of the pupil center and the pixel coordinates of the reflective point in all image frames during the last 2 seconds of each fixation point display are recorded. When the pupil center is not detected in the human eye gaze image, it is considered that the eyes are closed, and the pixel coordinates of the pupil are recorded as (-1, -1) to facilitate elimination. The same applies to reflective points. When no reflective points are detected, record the coordinates of the reflective points as (-1, -1) for easy elimination.

Step 33: For the pixel coordinates of the pupil center and the pixel coordinates of the reflective points recorded in the last 2 seconds of each fixation point display, eliminate the data with eyes closed (that is, the situation where the coordinates are (-1, -1));

Calculate the mean value of the abscissa of the remaining pupil center pixel coordinates after removing the data The variance of the abscissa Dx _p , the mean of the ordinate/> Vertical coordinate variance Dy _p . If the abscissa of a certain pupil center is greater than/> Or the ordinate is greater than The calibration fails, otherwise the calibration is successful. In the same way, calculate the mean value of the abscissa coordinate of the center pixel of the reflective point/> The variance of the abscissa Dx _g and the mean of the ordinate/> Vertical coordinate variance Dy _g . If the abscissa of a certain pupil center is greater than/> Or the ordinate is greater than/> The calibration fails, otherwise the calibration is successful.

After the calibration fails, return to step 31, that is, re-enter the calibration module. If all nine fixation points are calibrated successfully, proceed to step 34;

Step 34: Calculate the horizontal pixel distance d ₀ ~ d ₉ from the pupil center to the center of Purchin's spot when observing points P ₀ ~ P ₉ respectively, where Calculate the horizontal pixel distance from the pupil center of observation points P ₁ to P ₉ to the center of Purchin's spot and the offset t ₁ to t ₉ at point P ₀ . Where _ti =d _i -d ₀ (i=1, 2...9).

Step 35: After obtaining (t ₁ ,A ₁ ), (t ₂ ,A ₂ )...(t ₉ ,A ₉ ), use the least squares method to perform straight line fitting on them, A=H ₀ t+b ₀ , The Hesburgh ratio H ₀ and the intercept b0 of the fitted straight line are calculated.

Cover detection module is used to implement the following process:

The voice announcer is controlled to prompt the person being tested to measure the degree of strabismus using a covering test method. The covering test method can select a unilateral covering test, a covering-uncovering test or an alternating covering method. The following uses the masking and unmasking method as an example for explanation.

Step 41: Control the voice announcer to prompt the person being tested to start the covering test. After starting the test, the system calls the pupil and reflective point positioning module to continuously collect images and obtain the pixel coordinates of the reflective point centers of both pupils. If both eyes cannot be detected at this time, step 41 will be re-entered to control the voice announcer to prompt the subject to adjust the position. Until both eyes are detected, if the center of the pupils of both eyes and the reflective point is detected, step 42 is entered.

Step 42: Monitor the eye status of both eyes. As shown in Figure 2, in the present invention, the eye state is designed to include an initial state (s1), a ready state (s2), an occlusion state (s3), a transient state (s4), a steady state (s5), and a blinking state. (s0). First, the eyes are in the initial state (s1) when monitoring starts, and then (generally 2 seconds after the initial state) the eyes are in the ready state (s2), and the voice announcer is controlled to prompt the subject to cover the right eye. The system calls the pupil and reflective point positioning module to detect the status of the right eye. If the number of consecutive frames of the eye's pupil cannot be detected within fifty frames, the right eye is in the blinking state (s0). When the right eye is in the blinking state and the right eye pupil is re-detected, the right eye state returns to the ready state (s2). When the number of consecutive frames in which the pupil cannot be detected reaches fifty frames or more, the right eye enters the occlusion state (s3), and step 43 is entered.

Step 43: Control the voice announcer to prompt the subject to remove the baffle from the right eye and cover the left eye. The system records the moment when the baffle moves away from the right eye. The first frame that can detect the reflective point and the pupil at the same time is the key frame. At this time, the right eye is in the instant state (s4). The system calculates and saves the pupil pixel coordinates and reflective points of this frame. The difference p ₀ of pixel coordinates. After the right eye enters the instantaneous state (s4) for more than 2s, calculate the mean and variance of the pupil pixel coordinate components within 60 frames. If the variances of the coordinate components are less than the set variance threshold T _d , the eye enters a stable state. (s5). Calculate the difference p ₁ between the pixel coordinates of the pupil center and the pixel coordinates of the reflective point center within 60 frames when entering the steady state. Control the voice announcer to prompt the subject that the covering test is over and proceed to step 44.

Step 44: Calculate the degree of strabismus: calculate the offset of the right eye t=p ₁ -p ₀ , and calculate the degree of strabismus of the right eye A=H ₀ t+b ₀ .

In the same way, you can also cover the left eye first and then the right eye to calculate the offset and strabismus of the left eye.

Claims (9)

1. A strabismus degree detection system based on covering test, characterized by including a headrest (1), calibration point (3), display (7), camera (8), light source (9), voice announcer and Control device; wherein, the headrest (1) and the display (7) are arranged oppositely and the planes of the headrest (1) and the display (7) are parallel to each other; the calibration point (3) is set by two A bracket composed of a symmetrical column and a beam on the column is installed directly above the headrest (1), and the calibration point (3) is installed in the middle of the beam; the camera (8) and the light source (9) are respectively installed on the display and both are located in the same plane, which plane is parallel to the plane where the display (7) is located, and the field of view of the camera (8) covers the headrest (1); the display (7), camera (8) and voice announcer are all connected controller. 2. The strabismus degree detection system based on covering test according to claim 1, characterized in that the calibration point (3) is a black circular point. 3. The strabismus detection system based on covering test as claimed in claim 1, characterized in that the controller includes a system calibration module, a pupil and reflective point positioning module, a calibration module and an covering detection module, wherein: The system calibration module is used to implement the following process: Step 11, establish a world coordinate system with the lower left corner of the display (7) as the origin, the height of the display (7) is SH, the width is SW, and the resolution is RWxRH; Step 12, Control the camera (8) to collect images; step 13, according to the collected images, obtain the conversion ratio ratio from the plane pixels of the headrest (1) to the distance, and manually measure the coordinates of the calibration point center in the world coordinate system (X _b ,Y _b ,Z _b ); The pupil and reflective point positioning module is used to collect human eye images, and obtain the pixel coordinates of the pupil centers of both eyes, the coordinates of the pupil center in the world coordinate system, and the pixels of the reflective point centers in the pupils based on the collected human eye images. coordinate; The calibration module is used to call the pupil and reflective point positioning module to calibrate the person being measured and calculate the Hesburgh ratio; The cover detection module is used to call the pupil and reflective point positioning module, and control the voice announcer to prompt the person being tested to use the cover test method to measure the degree of strabismus. 4. The strabismus degree detection system based on covering test as claimed in claim 3, characterized in that the step 13 includes the following operations: Convert the collected image into a grayscale image, use Gaussian filtering or other methods to denoise the image; use adaptive threshold segmentation or other methods to threshold segment the denoised image; use morphological opening operations to remove the noise after threshold segmentation Isolated points, burrs, etc. in the image are obtained from the image after morphological operation; use canny edge detection or other methods to detect the edges of the image after morphological operation; use Hough transform or other circular detectors to detect the image obtained by the above processing Calibration point (3) in, get the pixel coordinates (x _b , y _b ) of the calibration point (3) and the pixel radius rb of the calibration point (3) on the image; calculate the plane of the headrest (1) The calculation formula for the conversion ratio from pixels to distance is ratio=R/r _b , where R is the real radius of the calibration point; the coordinates of the center of the calibration point in the world coordinate system (X _b , Y _b , Z _b ) are measured using a tape measure or other ranging tools. 5. The strabismus detection system based on covering test as claimed in claim 4, characterized in that the implementation process of the pupil and reflective point positioning module is as follows: Step 21, control the camera (8) to continuously collect human eye images at a collection rate of 60fps, directly collect images containing human eyes, and then flip the images horizontally; Step 22: Based on the image obtained in step 21, first locate the position of both eyes through the human eye detection method, and then use pupil positioning technology in the two areas to obtain the pixel coordinates of the two pupil centers; and calculate the position of the pupil center in the system calibration module. Coordinates in the established world coordinate system; Step 23: Take the left eye pupil center (x _L , y _L ) as the center of the circle, and use about 1.5 times the left eye pupil radius R _L as the radius to make a circular area. Find the center of the reflective point in the left eye pupil in this area. Pixel coordinates (x _g , y _g ); similarly, obtain the pixel coordinates of the center of the reflective point in the pupil of the right eye. 6. The strabismus detection system based on covering test as claimed in claim 5, characterized in that the specific process of step 22 is as follows: (1) The human eye detection method adopts any of the following methods: (1) manually determine the position of the eyes based on multiple collected human eye images; (2) collect and mark multiple positive and negative samples, and train Haar-cascade classifier, the trained classifier can obtain the position of the eyes of the image; (3) Human eye detection is performed through deep learning method training; (2) Pupil position detection uses the segmentation network model in deep learning. After inputting the image, the segmentation probability map is obtained, and then the binary image is obtained through threshold segmentation. Connected domain analysis is performed in the binary image, and the centroid of the largest spot is used as the left eye. The pixel coordinates of the pupil center (x _L , y _L ), take the long side of the rectangular frame surrounding the largest spot as the left eye pupil diameter 2*R _L , R _L is the pupil radius; (3) Calculate the left eye pupil center in the world coordinate system based on the world coordinate system established in step 11 and the coordinates (X _b , Y _b , Z _b ) of the calibration point measured in step 12 in the world coordinate system The coordinates are (X _L , Y _L , Z _L ); where, Z _L =Z _b , where r _b is the pixel radius of calibration point 3 on the image, R is the real radius of the calibration point (cm), (x _b , y _b ) is the pixel coordinate of the calibration point center; (4) In the same way, perform steps (2) and (3) for the right eye to obtain the pixel coordinates (x _R , y _R ) of the pupil center of the right eye, the pupil radius R _R , and the pupil center of the right eye in the world coordinate system. The coordinates (X _R , Y _R , Z _R ). 7. The strabismus detection system based on covering test as claimed in claim 6, characterized in that the implementation process of the calibration module is as follows: Step 31, initialize 9 gaze points with known coordinates: P ₁ (X ₁ , Y ₁ , 0), P ₂ (X ₂ , Y ₂ , 0)...P ₉ (X ₉ , Y ₉ , 0); The pixel coordinates of gaze point P ₀ are (x ₀ , y ₀ ), and its world coordinates are (X ₀ , Y ₀ , 0), then X ₀ =x ₀ *SW/RW, Y ₀ = (RH-y ₀ ) *SW/RW; calculate the offset gaze angles from the gaze point P ₀ to the gaze points P ₁ , P ₂ ...P ₉ respectively. RH is the height of the monitor (cm), RW is the width of the monitor (cm); Step 32: Display the fixation point P ₀ (X ₀ , Y ₀ , 0) on the display, control the voice announcer to prompt the person to be tested to place their head on the headrest (1), and remind the person being tested to fixate on the coordinate point P ₀ ; then display at least two gaze points with known positions as calibration points in sequence on the display (7), and control the voice announcer to remind the subject to gaze at these calibration points in sequence. During this process, the system calls the pupil and reflective point positioning The module locates the pupil and the reflective point within it; each calibration point is displayed for 5 seconds. When the subject's gaze lasts for 2 seconds or more, the current calibration point is considered stable; the mean and standard deviation of the pupil center position within 2 seconds are calculated. If If the pupil center position coordinates within 2 seconds are less than 1.5 times the standard deviation, the gaze is considered stable. When the current gaze point is stable, record the pixel coordinates of the pupil center and the pixel coordinates of the reflective points in all image frames in the last 2 seconds of each gaze point display. ; Step 33: For the pixel coordinates of the pupil center and the pixel coordinates of the reflective points recorded in the last 2 seconds of each fixation point display, eliminate the data with eyes closed; Calculate the mean value of the abscissa of the remaining pupil center pixel coordinates after removing the data The variance of the abscissa Dx _p , the mean of the ordinate/> Vertical coordinate variance Dy _p ; if the abscissa coordinate of a certain pupil center is greater than/> Or the ordinate is greater than/> If the calibration fails, otherwise the calibration is successful; similarly, calculate the mean value of the abscissa of the center pixel coordinate of the reflective point/> The variance of the abscissa Dx _g and the mean of the ordinate/> Vertical coordinate variance Dy _g ; if the abscissa coordinate of a certain pupil center is greater than/> Or the ordinate is greater than/> If the calibration fails, otherwise the calibration is successful; After the calibration fails, return to step 31, that is, re-enter the calibration module; if all nine fixation points are successfully calibrated, enter step 34; Step 34: Calculate the horizontal pixel distance d ₀ ~ d ₉ from the pupil center to the center of Purchin's spot when observing points P ₀ ~ p ₉ respectively, where Calculate the horizontal pixel distance from the pupil center of observation point P ₁ to p ₉ to the center of Purchin's spot and the offset t ₁ to t ₉ at point P ₀ , where _ti =d _i -d ₀ (i=1, 2... …9); Step 35: After obtaining (t ₁ , A ₁ ), (t ₂ , A ₂ )...(t ₉ , A ₉ ), use the least squares method to perform straight line fitting on them, A=H ₀ t+b ₀ , The Hesburgh ratio H ₀ and the intercept b ₀ of the fitted straight line are calculated. 8. The strabismus degree detection system based on covering test as claimed in claim 1, characterized in that, in the covering detection module, the covering test method selects a unilateral covering test method, a covering and uncovering test method or an alternating covering method. . 9. The strabismus degree detection system based on covering test as claimed in claim 8, characterized in that the covering and removing method is adopted in the covering detection module, and the specific implementation process is as follows: Step 41: Control the voice announcer to prompt the person being tested to start covering the test. After starting the test, the system calls the pupil and reflective point positioning module to continuously collect images and obtain the positions of the eyes and the pixel coordinates of the pupil and reflective point centers; at this time, if the detection If both eyes are not detected, then re-enter step 41 and control the voice announcer to prompt the subject to adjust the position until both eyes are detected. If both eyes and the center of the pupil and the reflective point are detected, then enter step 42; Step 42, monitor the eye status of both eyes: first, the eyes are in the initial state s1 when monitoring starts, and then the eyes are in the ready state s2, and the voice announcer is controlled to prompt the subject to cover the right eye; the system calls the pupil and reflective point positioning The module detects the state of the right eye. If the number of consecutive frames of the eye pupil cannot be detected within fifty frames, the right eye is in the blinking state s0; when the right eye is in the blinking state and the right eye pupil is re-detected, the right eye state returns to ready. State s2; when the number of consecutive frames in which the pupil cannot be detected reaches fifty frames or more, the right eye enters the occlusion state s3, and then enters step 43; Step 43: Control the voice announcer to prompt the subject to remove the baffle from the right eye and cover the left eye. The system records the moment when the baffle is removed from the right eye. The first frame that can simultaneously detect the reflective point and the pupil is the key frame. This When the right eye is in the instant state s4, the system calculates and saves the difference p ₀ between the pupil pixel coordinates of the frame and the pixel coordinates of the reflective point; after the right eye enters the instant state s4 for more than 2 seconds, the system calculates the pupil pixel coordinates within 60 frames. The mean and variance of the components. If the variance of the coordinate components is less than the set variance threshold T _d , the eye enters the steady state s5; calculate the pixel coordinates of the pupil center and the pixel center of the reflective point within 60 frames when entering the steady state. Difference p ₁ ; control the voice announcer to prompt the subject that the covering test is over, and enter step 44; Step 44, calculate the degree of strabismus: Calculate the offset of the right eye t = p ₁ - p ₀ , and calculate the degree of strabismus of the right eye A = H ₀ t + b ₀ ; similarly, cover the left eye first and then the right eye, The offset and strabismus of the left eye were calculated.