CN110427101A - Calibration method, device, equipment and the storage medium of eyeball tracking - Google Patents
Calibration method, device, equipment and the storage medium of eyeball tracking Download PDFInfo
- Publication number
- CN110427101A CN110427101A CN201910612124.8A CN201910612124A CN110427101A CN 110427101 A CN110427101 A CN 110427101A CN 201910612124 A CN201910612124 A CN 201910612124A CN 110427101 A CN110427101 A CN 110427101A
- Authority
- CN
- China
- Prior art keywords
- calibration
- eye
- target position
- position information
- mark
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000005252 bulbus oculi Anatomy 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 49
- 210000001508 eye Anatomy 0.000 claims abstract description 153
- 238000004422 calculation algorithm Methods 0.000 claims description 30
- 230000003287 optical effect Effects 0.000 claims description 30
- 238000009434 installation Methods 0.000 claims description 17
- 239000000284 extract Substances 0.000 claims description 6
- 210000001742 aqueous humor Anatomy 0.000 claims description 5
- 210000004087 cornea Anatomy 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 239000013598 vector Substances 0.000 description 30
- 238000010586 diagram Methods 0.000 description 10
- 238000005457 optimization Methods 0.000 description 8
- 238000000605 extraction Methods 0.000 description 6
- 210000001747 pupil Anatomy 0.000 description 6
- 230000005291 magnetic effect Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 210000000744 eyelid Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 210000002189 macula lutea Anatomy 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Eye Examination Apparatus (AREA)
Abstract
The embodiment of the invention discloses a kind of calibration method of eyeball tracking, device, equipment and storage mediums, wherein this method comprises: forming calibration mark according to preset target position information projection calibrating light；Obtain eyes image when user watches calibration mark attentively；Target alignment coefficient is determined according to eyes image and target position information.The embodiment of the present invention calibrates the calibration factor in the eyeball phantom for realizing eye tracking by the calibration mark of predetermined position, the position of calibration mark is not limited to display, therefore calibration can be realized without relying on external equipments such as displays, the calibration of Eye-tracking devices is quickly and conveniently realized, and then improves the efficiency of calibration.
Description
Technical field
The present embodiments relate to eyes control technology field more particularly to a kind of calibration methods of eyeball tracking, dress
It sets, equipment and storage medium.
Background technique
Eyeball tracking, alternatively referred to as Eye-controlling focus, be by measurement eye motion situation come estimate eyes sight and/
Or the technology of blinkpunkt.With the development of intelligent terminal, may be implemented by eyeball tracking equipment a kind of contactless man-machine
Interaction.
Core is calculated by built-in infrared light source, optical sensor, image processor and viewpoint in eyeball tracking equipment
Point estimation is watched attentively to realize, specific working principle is as follows: obtaining the eyes image of user, from the eyes image got
Eye feature is extracted, is carried out watching point estimation attentively according to eye feature.Existing eyeball tracking equipment is needed by calibrating come to note
The result of viewpoint estimation is adjusted, but is usually to require user using a set of calibration procedure, shows calibration over the display
The completions such as image calibration, that is, existing calibrating mode at least requires to work together with display, needs external equipment to participate in, nothing
Method meets the demand of the convenience of user.
Summary of the invention
The embodiment of the invention provides a kind of calibration method of eyeball tracking, device, equipment and storage mediums, can solve
The certainly technical problem of calibration method convenience deficiency in the prior art.
In a first aspect, the embodiment of the invention provides a kind of calibration methods of eyeball tracking, comprising:
According to preset target position information projection calibrating light, calibration mark is formed；
Obtain eyes image when user watches the calibration mark attentively；
Target alignment coefficient is determined according to the eyes image and the target position information.
Second aspect, the embodiment of the invention also provides a kind of calibrating installation of eyeball tracking, which includes:
Mark module is calibrated, for forming calibration mark according to preset target position information projection calibrating light；
Image module, for obtaining eyes image when user watches the calibration mark attentively；
Calibration module, for determining target alignment coefficient according to the eyes image and the target position information.
Further, described image module is specifically used for:
The user images that user watches the calibration mark attentively are acquired by image capture module；
Extract the eyes image in the user images.
Further, described image module is specifically used for:
Eyes image when user watches the calibration mark attentively is obtained based on default voice prompting.
Further, the target alignment coefficient include corneal curvature radius, cornea aqueous humor refractive index, eye optical axis with
At least one of the distance of the angle of the optical axis and corneal curvature center to pupil center location.
Further, the calibration module includes:
Feature extraction unit, for extracting the eye feature information in the eyes image；
Calibration factor unit, for based on the eye feature information, the target position information and the calibration of initialization
Coefficient determines target alignment coefficient.
Further, the calibration factor unit is specifically used for:
It is calculated based on the eye feature information, the target position information, the calibration factor of initialization and setting calibration
Method determines error on line of sight；
Target alignment coefficient is determined according to the error on line of sight.
Further, the device further include:
Photocontrol module is calibrated, for controlling the calibration light by calibration photocontrol instruction, the calibration photocontrol refers to
It enables including out code or open command, the out code is used for after the setting time for forming calibration mark described in closing
Calibrate light.
The third aspect, the embodiment of the invention also provides a kind of equipment, the equipment includes:
One or more processors；
Projection module is used for projection calibrating light；
Image capture module, for acquiring user images；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of places
Reason device realizes the calibration method of eyeball tracking as described above.
Fourth aspect, the embodiment of the invention also provides a kind of computer readable storage mediums, are stored thereon with computer
Program, the program realize the calibration method of eyeball tracking as described above when being executed by processor.
The embodiment of the present invention forms calibration mark according to preset target position information projection calibrating light, obtains user's note
Depending on eyes image when calibration mark, and target alignment coefficient is determined according to eyes image and target position information.The present invention is real
Example is applied to calibrate the calibration factor in the eyeball phantom for realizing eye tracking by the calibration mark of predetermined position, it should
The position of calibration mark is not limited to display, therefore calibration can be realized without relying on external equipments such as displays, more
The calibration of Eye-tracking devices is easily realized, and then improves the efficiency of calibration.
Detailed description of the invention
Fig. 1 is the flow chart of the calibration method of the eyeball tracking in the embodiment of the present invention one；
Fig. 2 is the schematic diagram of the calibration method of the eyeball tracking in the embodiment of the present invention one；
Fig. 3 is the flow chart of the calibration method of the eyeball tracking in the embodiment of the present invention two；
Fig. 4 is the schematic diagram of the calibration mark in the embodiment of the present invention two；
Fig. 5 is the structural schematic diagram of the calibrating installation of the eyeball tracking in the embodiment of the present invention three；
Fig. 6 is the structural schematic diagram of the equipment in the embodiment of the present invention four.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limiting the invention.It also should be noted that in order to
Convenient for describing, only the parts related to the present invention are shown rather than entire infrastructure in attached drawing.
Embodiment one
Fig. 1 is the flow chart of the calibration method of the eyeball tracking in the embodiment of the present invention one, and the present embodiment is applicable to pair
The eyeball phantom of eyeball tracking equipment realizes the case where calibration, and this method can be executed by the calibrating installation of eyeball tracking, the dress
Setting can be realized by the way of software and/or hardware, which is configured in equipment, such as the equipment may include intelligence
The mechanical, electrical brain of expert and tablet computer etc..
Fig. 2 is the schematic diagram of the calibration method of the eyeball tracking in the embodiment of the present invention one, and the schematic diagram in Fig. 2 is only
A kind of example.The calibrating installation of eyeball tracking in the present embodiment can be configured in the eyeball tracking equipment 11 in figure with reality
The calibration method of existing eyeball tracking.The eyeball tracking equipment 11 is placed in horizontal table top 12 in figure, and A point is eyeball tracking
Calibrating installation is incident upon the mark of the calibration in horizontal table top 12, passes through the direction vector and user's eye to user eyeball 14 and A point
Error between the practical direction of visual lines vector of ball 14 is minimized, and then realizes the calibration of calibration factor.Box 13 in figure
For the image acquisition region of the calibrating installation of eyeball tracking.
As shown in Figure 1, this method can specifically include:
S110, according to preset target position information projection calibrating light, form calibration mark.
Wherein, calibration light can be the visible light that the light source module being built in the calibrating installation of eyeball tracking provides, excellent
Choosing can be visible laser.For visible laser since wave spread is very narrow, color is extremely pure, and monochromaticjty is good, this implementation
The visible lasers such as the green laser of 532nm or the red laser of 650nm can be used as calibration light in example, to improve user
Visual impact.Since the feux rouges in visible light is relatively easy to cause the vision of eyes, it can be used in the present embodiment
Feux rouges is also suitable as calibration light, other light sources for meeting condition.Target position information is set as the letter of the position on objective plane
Breath, which can be the surface of arbitrary plane object, such as the horizontal table top in Fig. 2, since the geometry in plane closes
System is easy determination, and not needing manpower intervention can determine spatial relationship, and therefore, the target position in the present embodiment is set in flat
On face.If the calibrating installation of eyeball tracking and the spatial relationship of target position predefine, which is not limited to
In plane.
Specifically, can be according to preset target position by the projection module being built in the calibrating installation of eyeball tracking
Confidence ceases projection calibrating light, forms calibration mark.The quantity for calibrating mark can be to be multiple, and calibration mark is not limited to a mark
Know point, is also possible to measurement image and calibration picture that one is played mark action.For example, the calibration in the present embodiment identifies
Shape can be for round, rectangular, linear and other irregular shapes etc., concrete shape can carry out according to the actual situation
Setting.The target position can be determined based on the space coordinate of the local coordinate system of the calibrating installation of eyeball tracking.
S120, acquisition user watch eyes image when calibration mark attentively.
Specifically, obtaining eyes image when user watches calibration mark attentively based on default voice prompting.The school of eyeball tracking
Sound module can be set in standard apparatus, after calibration mark is formed, default voice prompting is played by sound module, it should
Default voice prompting is for prompting user to watch calibration mark attentively.It is adopted simultaneously by the image in the calibrating installation of eyeball tracking
Collection module acquisition user watches eyes image when calibration mark attentively.It is understood that when the quantity of calibration mark is multiple,
The default voice prompting can play repeatedly, until completing calibration process.
S130, target alignment coefficient is determined according to eyes image and target position information.
Wherein, calibration factor can in eye tracking model for carrying out sight estimation it needs to be determined that parameter, this
Target alignment coefficient in embodiment may include the folder of corneal curvature radius, cornea aqueous humor refractive index, eye optical axis and the optical axis
Angle and corneal curvature center are at least one of to parameters such as the distances of pupil center location.The calibration factor of initialization is root
According to the calibration factor that priori knowledge determines, the calibration factor is different for different users, it is therefore desirable to be calibrated.
It may include: extraction eye specifically, determining target alignment coefficient according to eyes image and target position information
Eye feature information in image；Based on eye feature information, target position information and the calibration factor of initialization, mesh is determined
Mark calibration factor.Wherein, eye feature information may include pupil center location, pupil shape, iris position, iris shape,
Eyelid position, canthus position, hot spot (also referred to as Purkinje image) position etc., with pupil center location and hot spot in the present embodiment
It is illustrated for position.
Further, based on eye feature information, target position information and the calibration factor of initialization, target school is determined
Quasi- coefficient may include: to be calculated based on eye feature information, target position information, the calibration factor of initialization and setting calibration
Method determines error on line of sight；Target alignment coefficient is determined according to error on line of sight.
Error on line of sight can identify the angle between geometry sight for user eyeball Real line-of-sight and user's eye to calibration
Difference.Wherein, setting calibration algorithm can be set according to the actual situation, be with corneal reflection calibration algorithm in the present embodiment
Example is illustrated.Based on eye feature information, target position information, the calibration factor of initialization and setting calibration algorithm, determine
Error on line of sight may include: to obtain eye view based on eye feature information, target position information and the calibration factor of initialization
Direction vector of the axis direction vector sum eye to calibration mark；Based on angular error formula, eye boresight direction vector sum eye
To the direction vector of calibration mark, the error of eye boresight direction is obtained, the error of eye boresight direction is determined as sight and is missed
Difference.Wherein eye boresight direction vector is the direction vector of the Real line-of-sight of user eyeball.Referring to fig. 2, wherein user eyeball 14
Sight to B point is the Real line-of-sight of user eyeball, the geometry sight and family eyeball of user eyeball 14 to calibration mark A point
There are differential seat angles, i.e. error on line of sight between Real line-of-sight.
Further, eye feature information includes pupil center location and facula position；Based on eye feature information, mesh
Cursor position information and the calibration factor of initialization obtain the direction vector that eye boresight direction vector sum eye is identified to calibration,
It may include: that corneal curvature center is obtained according to the corneal curvature radius in facula position and the calibration factor of initialization；Root
According to the angle of eye optical axis and the optical axis in the calibration factor of pupil center location, corneal curvature center and initialization, eye is obtained
Portion's boresight direction vector；According to target position and corneal curvature center, the direction vector that eye is identified to calibration is obtained.Wherein,
Eye optical axis be pupil center location reach foveal region of retina line, the eye optical axis be by blinkpunkt outside eye by node with
The line of macula lutea forms 4 ° to 5 ° of angle between eye optical axis and the eye optical axis.According to pupil center location and corneal curvature
The available eye optical axis direction vector in center, according to the angle of eye optical axis and the optical axis in the calibration factor of initialization and
The available eye boresight direction vector of eye optical axis direction vector.
Further, target alignment coefficient is determined according to error on line of sight, may include: to optimize to calculate based on preset error
Method determines the minimal error of error on line of sight；The corresponding calibration factor of minimal error is determined as the target alignment coefficient after calibration.
Wherein error optimization algorithm is to calculate the smallest algorithm of error, is not construed as limiting in the present embodiment to specific error optimization algorithm,
Such as error optimization algorithm can be sequential quadratic programming (Sequence Quadratic Program, SQP) algorithm, sequence two
Secondary planning algorithm is to solve for a kind of efficient algorithm of medium and small plan constraint optimization problem.
It can also include: extraction eye in addition, determining target alignment coefficient according to eyes image and target position information
Eye feature information in image；Sight track is determined based on eye feature information, and eye feature information includes pupil center position
It sets and facula position；Based on the calibration factor of sight track, target position information and initialization, target alignment coefficient is determined.Its
Middle target position information may include the location information of each point in predetermined path of movement.Further, it is based on eye feature information
It determines sight track, may include: that pupil center location and facula position are obtained into the multiple of sight by sight calculations formula
The position of blinkpunkt；Calculate the average and standard deviation of the position of multiple blinkpunkts；Choose less than three standard deviations of anomaly mean value
Multiple blinkpunkts as the first reference point；Sight track is determined according to the position of the first reference point.Further, by right
The calibration factor of initialization is adjusted, and is matched sight track and the predetermined path of movement of calibration mark, then can be determined energy
The matched calibration factor of predetermined path of movement of the sight track and calibration mark that enough make user is the target alignment after calibrating
Coefficient, to complete to calibrate.
The technical solution of the present embodiment forms calibration mark, obtains according to preset target position information projection calibrating light
Eyes image when calibration mark is watched at family attentively is taken, and target alignment coefficient is determined according to eyes image and target position information.
The present embodiment carries out school to the calibration factor in the eyeball phantom for realizing eye tracking by the calibration mark of predetermined position
The position of standard, calibration mark is not limited to display, therefore school can be realized without relying on external equipments such as displays
Standard, quickly and conveniently realizes the calibration of Eye-tracking devices, and then improves the efficiency of calibration.
Embodiment two
Fig. 3 is the flow chart of the calibration method of the eyeball tracking in the embodiment of the present invention two.The present embodiment is in above-mentioned implementation
On the basis of example, further materialization has been carried out to the calibration method of above-mentioned eyeball tracking.Correspondingly, as shown in figure 3, originally
The method of embodiment specifically includes:
S210, according to preset target position information projection calibrating light, form calibration mark.
Wherein, target position information is set as the location information on objective plane, which can be arbitrary plane
The surface of object.
Target position information can be set on objective plane, which can be the surface of arbitrary plane object.
The quantity for calibrating mark can be to be multiple, and calibration mark is not limited to an identification point, is also possible to one and plays mark action
Measurement image and calibration picture.Referring to fig. 4, Fig. 4 is the schematic diagram of the calibration mark in the embodiment of the present invention two, includes in figure
Multiple calibrating patterns.
Optionally, it according to preset target position information projection calibrating light, is formed after calibration mark, can also include:
By calibration photocontrol instruction control calibration light, calibration photocontrol instruction includes out code or open command, and out code is used
Light is calibrated in closing after the setting time for forming calibration mark.Wherein, detect user to the trigger action of setting key
When, it can indicate to receive calibration photocontrol instruction；Alternatively, can indicate to receive school when receiving the setting gesture of user
Quasi-optical control instruction；Alternatively, can indicate to receive calibration photocontrol instruction, setting condition can wrap when meeting setting condition
Include the setting time etc. to form calibration mark.Pass through the closing of control calibration light, it is possible to reduce the power consumption of calibration module, section
Save the energy.
S220, the user images that user watches calibration mark attentively are acquired by image capture module.
Wherein, user images can be include the image of face and do not include face image.
Sound module can be set in the calibrating installation of eyeball tracking, after calibration mark is formed, pass through sound module
Default voice prompting is played, the default voice prompting is for prompting user to watch calibration mark attentively.Pass through eyeball tracking simultaneously
Image capture module acquisition user in calibrating installation watches user images when calibration mark attentively.
Eyes image in S230, extraction user images.
Collect user watch attentively calibration mark user images after, in user images carry out face lookup, if with
Face is not found in the image of family, then returns to reacquisition user images, until people can be found in the user images obtained
Face.The method searched in the present embodiment face is not construed as limiting, such as detects people by haar-like human-face detector algorithm
Face carries out Face datection to user images by face characteristic (such as colour of skin or nose etc.), according in user images
Face characteristic and the matching value of standard feature determine in user images whether to include face.
Further, the eyes image in user images is extracted by eye detection algorithm, wherein to eye detection algorithm
Also it is not construed as limiting, such as eye detection algorithm may include method or Statistics-Based Method etc. based on template matching.This
By being illustrated for Statistics-Based Method in embodiment, by a large amount of samples including eyes image and not including
The sample of eye image is trained study and obtains a group model parameter, is then based on model construction classifier or filter
To detect the eyes image in user images.After detecting eyes image in user images, the eyes image is extracted.
Eye feature information in S240, extraction eyes image.
Wherein, eye feature information may include pupil center location, pupil shape, iris position, iris shape, eye
Skin position, canthus position, hot spot (also referred to as Purkinje image) position etc., in the present embodiment with pupil center location in image and
It is illustrated for facula position.
Specifically, this step can handle eyes image to obtain the gradient value of the gray scale on direction initialization, it can
It, then can be to pupil the maximum of gradients present position of the gray scale to be determined as to the pupil edge feature of the eyes image
Edge feature is fitted, and determines the center of fitting image, and the center can be determined as to pupil in image
Center.It is understood that be not defined to the means of determining pupil center location herein.It further, can be with
It is general that gray level image after handling eyes image can obtain the higher region of gray value by Spot recognition algorithm
You admire spot, the position for obtaining the Purkinje image is facula position.
S250, based on eye feature information, target position information, initialization calibration factor and setting calibration algorithm, really
Determine error on line of sight.
Error on line of sight can identify the angle between geometry sight for user eyeball Real line-of-sight and user's eye to calibration
Difference.Wherein, setting calibration algorithm can be set according to the actual situation, be with corneal reflection calibration algorithm in the present embodiment
Example is illustrated.Based on eye feature information, target position information, the calibration factor of initialization and setting calibration algorithm, determine
Error on line of sight may include: to obtain eye view based on eye feature information, target position information and the calibration factor of initialization
Direction vector of the axis direction vector sum eye to calibration mark；Based on angular error formula, eye boresight direction vector sum eye
To the direction vector of calibration mark, the error of eye boresight direction is obtained, the error of eye boresight direction is determined as sight and is missed
Difference.
Further, eye feature information includes pupil center location and facula position；Based on eye feature information, mesh
Cursor position information and the calibration factor of initialization obtain the direction vector that eye boresight direction vector sum eye is identified to calibration,
It may include: that corneal curvature center is obtained according to the corneal curvature radius in facula position and the calibration factor of initialization；Root
According to the angle of eye optical axis and the optical axis in the calibration factor of pupil center location, corneal curvature center and initialization, eye is obtained
Portion's boresight direction vector；According to target position and corneal curvature center, the direction vector that eye is identified to calibration is obtained.Wherein
Pupil center location is eye pupil center.
Wherein, corneal curvature center can be indicated using C, and pupil center location can be indicated using P, eye optical axis side
V can be used to vectorSIt indicates, the direction vector of eye to calibration mark can use VtIt indicates, error on line of sight can use e
It indicates.Corneal curvature center can pass through C=f1(gi,Li, R) and it determines, wherein giIndicate that facula position, the facula position can be with
For the corresponding multiple facula positions of multiple infrared lights, LiIndicate that the coordinate of light source in the calibrating installation of eyeball tracking, R indicate
Corneal curvature radius.Pupil center location can pass through P=f2(p, C, K, n) is determined, wherein p indicates pupil in eyes image
Center position coordinates, K indicate corneal curvature center to pupil center location distance, n expression cornea aqueous humor refractive index.
Eye optical axis direction vector can pass throughIt determines, whereinIndicate eye light
The spherical coordinate of axis direction unit vector indicates.Eye boresight direction vector can pass throughIt determines, wherein (α, β) indicates eye optical axis and optical axis angle.Eye is identified to calibration
Direction vector can pass throughIt determines, wherein T=(Tx,Ty,Tz) indicate the target location coordinate that calibration identifies.
Determine VSAnd VtLater, error on line of sight can pass through e=arccos θ (VS·Vt) calculate both direction vector angle.Work as school
When the quantity that fiducial mark is known is N, the quantity for the calibration image that each calibration mark uses is M, then error on line of sight passes throughIt determines.
S260, target alignment coefficient is determined according to error on line of sight.
Specifically, the minimal error of error on line of sight is determined based on preset error optimization algorithm, minimal error is corresponding
Calibration factor is determined as the target alignment coefficient after calibration.Wherein error optimization algorithm is to calculate the smallest algorithm of error, this reality
It applies in example and specific error optimization algorithm is not construed as limiting, such as error optimization algorithm can be sequential quadratic programming
(Sequence Quadratic Program, SQP) algorithm, it is optimal that sequential quadratic programming algorithm is to solve for medium and small plan constraint
A kind of efficient algorithm of change problem.
Calibration factor can pass throughIt determines, wherein E indicates error on line of sight,Indicate the R, K, n, α, β when calculating E minimum.
The technical solution of the present embodiment passes through image capture module according to preset target position information projection calibrating light
Acquisition user watches the user images of calibration mark attentively, extracts the eyes image in user images, extracts the eye in eyes image
Characteristic information determines view based on eye feature information, target position information, the calibration factor of initialization and setting calibration algorithm
Line error determines target alignment coefficient according to error on line of sight.The present embodiment is by the calibration mark of predetermined position to realization eye
Calibration factor in the eyeball phantom of ball tracking is calibrated, and the position of calibration mark is not limited to display, therefore does not have to
Calibration can be realized dependent on external equipments such as displays, quickly and conveniently realize the calibration of Eye-tracking devices, Jin Erti
The high efficiency of calibration.
Embodiment three
Fig. 5 is the structural schematic diagram of the calibrating installation of the eyeball tracking in the embodiment of the present invention three, and the present embodiment is applicable
The case where calibrating is realized in the eyeball phantom to eyeball tracking equipment.The calibration of eyeball tracking provided by the embodiment of the present invention
The calibration of eyeball tracking provided by any embodiment of the invention can be performed in device, has the corresponding functional module of execution method
And beneficial effect.The device specifically includes calibration mark module 310, image module 320 and calibration module 330, in which:
Mark module 310 is calibrated, for forming calibration mark according to preset target position information projection calibrating light；
Image module 320, for obtaining eyes image when user watches calibration mark attentively；
Calibration module 330, for determining target alignment coefficient according to eyes image and target position information.
The embodiment of the present invention forms calibration mark according to preset target position information projection calibrating light, obtains user's note
Depending on eyes image when calibration mark, and target alignment coefficient is determined according to eyes image and target position information.The present invention is real
Example is applied to calibrate the calibration factor in the eyeball phantom for realizing eye tracking by the calibration mark of predetermined position, it should
The position of calibration mark is not limited to display, therefore calibration can be realized without relying on external equipments such as displays, more
The calibration of Eye-tracking devices is easily realized, and then improves the efficiency of calibration.
Further, image module 320 is specifically used for:
The user images that user watches calibration mark attentively are acquired by image capture module；
Extract the eyes image in user images.
Further, image module 320 is specifically used for:
Eyes image when user watches calibration mark attentively is obtained based on default voice prompting.
Further, target alignment coefficient includes corneal curvature radius, cornea aqueous humor refractive index, eye optical axis and the optical axis
Angle and corneal curvature center to pupil center location at least one of distance.
Further, calibration module 330 includes:
Feature extraction unit, for extracting the eye feature information in eyes image；
Calibration factor unit, for based on eye feature information, target position information and the calibration factor of initialization, really
Set the goal calibration factor.
Further, calibration factor unit is specifically used for:
Based on eye feature information, target position information, the calibration factor of initialization and setting calibration algorithm, view is determined
Line error；
Target alignment coefficient is determined according to error on line of sight.
Further, the device further include:
Photocontrol module is calibrated, for by calibration photocontrol instruction control calibration light, calibration photocontrol instruction to include closing
Instruction or open command are closed, out code is used to close calibration light after the setting time for forming calibration mark.
The calibrating installation of eyeball tracking provided by the embodiment of the present invention can be performed provided by any embodiment of the invention
The calibration method of eyeball tracking has the corresponding functional module of execution method and beneficial effect.
Example IV
Fig. 6 is the structural schematic diagram of the equipment in the embodiment of the present invention four.Fig. 6, which is shown, to be suitable for being used to realizing the present invention
The block diagram of the example devices 412 of embodiment.The equipment 412 that Fig. 6 is shown is only an example, should not be to of the invention real
The function and use scope for applying example bring any restrictions.
As shown in fig. 6, equipment 412 is showed in the form of common apparatus.The component of equipment 412 can include but is not limited to:
One or more processor 416, storage device 428 connect different system components (including storage device 428 and processor
416) bus 418.
Bus 418 indicates one of a few class bus structures or a variety of, including storage device bus or storage device control
Device processed, peripheral bus, graphics acceleration port, processor or the local using any bus structures in a variety of bus structures
Bus.For example, these architectures include but is not limited to industry standard architecture (Industry Subversive
Alliance, ISA) bus, microchannel architecture (Micro Channel Architecture, MAC) bus is enhanced
Isa bus, Video Electronics Standards Association (Video Electronics Standards Association, VESA) local are total
Line and peripheral component interconnection (Peripheral Component Interconnect, PCI) bus.
Equipment 412 typically comprises a variety of computer system readable media.These media can be it is any can be by equipment
The usable medium of 412 access, including volatile and non-volatile media, moveable and immovable medium.
Storage device 428 may include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (Random Access Memory, RAM) 430 and/or cache memory 432.Equipment 412 can be further
Including other removable/nonremovable, volatile/non-volatile computer system storage mediums.Only as an example, it stores
System 434 can be used for reading and writing immovable, non-volatile magnetic media, and (Fig. 6 do not show, commonly referred to as " hard drive
Device ").Although being not shown in Fig. 6, the disk for reading and writing to removable non-volatile magnetic disk (such as " floppy disk ") can be provided
Driver, and to removable anonvolatile optical disk, such as CD-ROM (Compact Disc Read-Only Memory,
CD-ROM), digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical mediums)
The CD drive of read-write.In these cases, each driver can be by one or more data media interfaces and total
Line 418 is connected.Storage device 428 may include at least one program product, which has one group (for example, at least one
It is a) program module, these program modules are configured to perform the function of various embodiments of the present invention.
Program/utility 440 with one group of (at least one) program module 442 can store in such as storage dress
It sets in 428, such program module 442 includes but is not limited to operating system, one or more application program, other programs
It may include the realization of network environment in module and program data, each of these examples or certain combination.Program mould
Block 442 usually executes function and/or method in embodiment described in the invention.
Equipment 412 can also be with one or more external equipments 414 (such as keyboard, direction terminal, display 424 etc.)
Communication, can also be enabled a user to one or more terminal interact with the equipment 412 communicate, and/or with make the equipment
The 412 any terminals (such as network interface card, modem etc.) that can be communicated with one or more of the other computing terminal are led to
Letter.This communication can be carried out by input/output (I/O) interface 422.Also, equipment 412 can also pass through Network adaptation
Device 420 and one or more network (such as local area network (Local Area Network, LAN), wide area network (Wide Area
Network, WAN) and/or public network, such as internet) communication.As shown in fig. 6, network adapter 420 passes through bus 418
It is communicated with other modules of equipment 412.It should be understood that although not shown in the drawings, other hardware can be used with bonding apparatus 412
And/or software module, including but not limited to: microcode, terminal driver, redundant processor, external disk drive array, magnetic
Disk array (Redundant Arrays of Independent Disks, RAID) system, tape drive and data are standby
Part storage system etc..
Processor 416 by the program that is stored in storage device 428 of operation, thereby executing various function application and
Data processing, such as realize the calibration method of eyeball tracking provided by the embodiment of the present invention, this method comprises:
According to preset target position information projection calibrating light, calibration mark is formed；
Obtain eyes image when user watches calibration mark attentively；
Target alignment coefficient is determined according to eyes image and target position information.
Embodiment five
The embodiment of the present invention five additionally provides a kind of computer readable storage medium, is stored thereon with computer program, should
The calibration method of the eyeball tracking as provided by the embodiment of the present invention is realized when program is executed by processor, this method comprises:
According to preset target position information projection calibrating light, calibration mark is formed；
Obtain eyes image when user watches calibration mark attentively；
Target alignment coefficient is determined according to eyes image and target position information.
The computer storage medium of the embodiment of the present invention, can appointing using one or more computer-readable media
Meaning combination.Computer-readable medium can be computer-readable signal media or computer readable storage medium.Computer can
Reading storage medium for example may be-but not limited to-the system of electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, device
Or device, or any above combination.The more specific example (non exhaustive list) of computer readable storage medium wraps
It includes: there is the electrical connection of one or more conducting wires, portable computer diskette, hard disk, random access memory (RAM), read-only
Memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, the read-only storage of portable compact disc
Device (CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.In this document, computer
Readable storage medium storing program for executing can be any tangible medium for including or store program, which can be commanded execution system, device
Either device use or in connection.
Computer-readable signal media may include in a base band or the data as the propagation of carrier wave a part are believed
Number, wherein carrying computer-readable program code.The data-signal of this propagation can take various forms, including but
It is not limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be calculating
Any computer-readable medium other than machine readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit
For by the use of instruction execution system, device or device or program in connection.
The program code for including on computer-readable medium can transmit with any suitable medium, including --- but not
It is limited to wireless, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereof
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
It further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
Fully execute, partly execute on the user computer on the user computer, being executed as an independent software package,
Part executes on the remote computer or executes on remote computer or terminal completely on the user computer for part.In
Be related in the situation of remote computer, remote computer can pass through the network of any kind --- including local area network (LAN) or
Wide area network (WAN)-be connected to subscriber computer, or, it may be connected to outer computer (such as mentioned using Internet service
It is connected for quotient by internet).
Note that the above is only a better embodiment of the present invention and the applied technical principle.Those skilled in the art can manage
Solution, the invention is not limited to the specific embodiments described herein, is able to carry out various apparent changes for a person skilled in the art
Change, readjust and substitutes without departing from protection scope of the present invention.Therefore, although by above embodiments to the present invention
It is described in further detail, but the present invention is not limited to the above embodiments only, the case where not departing from present inventive concept
Under, it can also include more other equivalent embodiments, and the scope of the invention is determined by the scope of the appended claims.
Claims (10)
1. a kind of calibration method of eyeball tracking characterized by comprising
According to preset target position information projection calibrating light, calibration mark is formed；
Obtain eyes image when user watches the calibration mark attentively；
Target alignment coefficient is determined according to the eyes image and the target position information.
2. the method according to claim 1, wherein obtaining eye figure when user watches the calibration mark attentively
Picture, comprising:
The user images that user watches the calibration mark attentively are acquired by image capture module；
Extract the eyes image in the user images.
3. method according to claim 1 or 2, which is characterized in that obtain eye when user watches the calibration mark attentively
Image, comprising:
Eyes image when user watches the calibration mark attentively is obtained based on default voice prompting.
4. method according to claim 1 or 2, which is characterized in that the target alignment coefficient include corneal curvature radius,
Cornea aqueous humor refractive index, eye optical axis at a distance from the angle of the optical axis and corneal curvature center to pupil center location at least
One of.
5. the method according to claim 1, wherein true according to the eyes image and the target position information
Set the goal calibration factor, comprising:
Extract the eye feature information in the eyes image；
Based on the eye feature information, the target position information and the calibration factor of initialization, target alignment coefficient is determined.
6. according to the method described in claim 5, it is characterized in that, being believed based on the eye feature information, the target position
Breath and the calibration factor of initialization, determine target alignment coefficient, comprising:
Based on the eye feature information, the target position information, the calibration factor of initialization and setting calibration algorithm, determine
Error on line of sight；
Target alignment coefficient is determined according to the error on line of sight.
7. method according to claim 1, which is characterized in that according to preset target position information projection calibrating light, formed
After calibration mark, further includes:
The calibration light is controlled by calibrating photocontrol instruction, the calibration photocontrol instruction includes that out code or unlatching refer to
It enables, the out code is used to close the calibration light after the setting time for forming calibration mark.
8. a kind of calibrating installation of eyeball tracking characterized by comprising
Mark module is calibrated, for forming calibration mark according to preset target position information projection calibrating light；
Image module, for obtaining eyes image when user watches the calibration mark attentively；
Calibration module, for determining target alignment coefficient according to the eyes image and the target position information.
9. a kind of equipment, which is characterized in that the equipment includes:
One or more processors；
Projection module is used for projection calibrating light；
Image capture module, for acquiring user images；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
The now calibration method of the eyeball tracking as described in any in claim 1-7.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The calibration method of the eyeball tracking as described in any in claim 1-7 is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910612124.8A CN110427101A (en) | 2019-07-08 | 2019-07-08 | Calibration method, device, equipment and the storage medium of eyeball tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910612124.8A CN110427101A (en) | 2019-07-08 | 2019-07-08 | Calibration method, device, equipment and the storage medium of eyeball tracking |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110427101A true CN110427101A (en) | 2019-11-08 |
Family
ID=68409020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910612124.8A Pending CN110427101A (en) | 2019-07-08 | 2019-07-08 | Calibration method, device, equipment and the storage medium of eyeball tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110427101A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857333A (en) * | 2020-06-29 | 2020-10-30 | 维沃移动通信有限公司 | Eye movement tracking method and device and electronic equipment |
| CN112232255A (en) * | 2020-10-26 | 2021-01-15 | 上海鹰瞳医疗科技有限公司 | Identity authentication method and device based on multimodal data |
| CN112987910A (en) * | 2019-12-17 | 2021-06-18 | 七鑫易维(深圳)科技有限公司 | Testing method, device, equipment and storage medium of eyeball tracking equipment |
| CN113035355A (en) * | 2021-05-27 | 2021-06-25 | 上海志听医疗科技有限公司 | Video head pulse test sensor post-correction method, system, electronic device and storage medium |
| CN113491502A (en) * | 2020-04-03 | 2021-10-12 | 北京七鑫易维信息技术有限公司 | Eyeball tracking calibration inspection method, device, equipment and storage medium |
| CN113495613A (en) * | 2020-03-18 | 2021-10-12 | 北京七鑫易维信息技术有限公司 | Eyeball tracking calibration method and device |
| CN113534945A (en) * | 2020-04-17 | 2021-10-22 | 北京七鑫易维信息技术有限公司 | Method, device and equipment for determining eyeball tracking calibration coefficient and storage medium |
| CN115546884A (en) * | 2022-12-01 | 2022-12-30 | 广州视景医疗软件有限公司 | Eyeball tremor identification method, device and equipment |
| WO2024230660A1 (en) * | 2023-05-08 | 2024-11-14 | 北京七鑫易维信息技术有限公司 | Processing method and apparatus for eyeball tracking, device, and storage medium |
| WO2025016179A1 (en) * | 2023-07-20 | 2025-01-23 | 北京字跳网络技术有限公司 | Method for selecting calibration image, method for estimating line of sight, apparatus, and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102496005A (en) * | 2011-12-03 | 2012-06-13 | 辽宁科锐科技有限公司 | Eye characteristic-based trial auxiliary study and judging analysis system |
| CN107358217A (en) * | 2017-07-21 | 2017-11-17 | 北京七鑫易维信息技术有限公司 | A kind of gaze estimation method and device |
| CN108369744A (en) * | 2018-02-12 | 2018-08-03 | 香港应用科技研究院有限公司 | 3D gaze point detection through binocular homography mapping |
| CN109558012A (en) * | 2018-12-26 | 2019-04-02 | 北京七鑫易维信息技术有限公司 | Eyeball tracking method and device |
| CN109696954A (en) * | 2017-10-20 | 2019-04-30 | 中国科学院计算技术研究所 | Eye-controlling focus method, apparatus, equipment and storage medium |
| CN109885169A (en) * | 2019-02-25 | 2019-06-14 | 清华大学 | Eyeball parameter calibration and gaze direction tracking method based on 3D eyeball model |
-
2019
- 2019-07-08 CN CN201910612124.8A patent/CN110427101A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102496005A (en) * | 2011-12-03 | 2012-06-13 | 辽宁科锐科技有限公司 | Eye characteristic-based trial auxiliary study and judging analysis system |
| CN107358217A (en) * | 2017-07-21 | 2017-11-17 | 北京七鑫易维信息技术有限公司 | A kind of gaze estimation method and device |
| CN109696954A (en) * | 2017-10-20 | 2019-04-30 | 中国科学院计算技术研究所 | Eye-controlling focus method, apparatus, equipment and storage medium |
| CN108369744A (en) * | 2018-02-12 | 2018-08-03 | 香港应用科技研究院有限公司 | 3D gaze point detection through binocular homography mapping |
| CN109558012A (en) * | 2018-12-26 | 2019-04-02 | 北京七鑫易维信息技术有限公司 | Eyeball tracking method and device |
| CN109885169A (en) * | 2019-02-25 | 2019-06-14 | 清华大学 | Eyeball parameter calibration and gaze direction tracking method based on 3D eyeball model |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112987910A (en) * | 2019-12-17 | 2021-06-18 | 七鑫易维(深圳)科技有限公司 | Testing method, device, equipment and storage medium of eyeball tracking equipment |
| CN112987910B (en) * | 2019-12-17 | 2023-04-07 | 七鑫易维(深圳)科技有限公司 | Testing method, device, equipment and storage medium of eyeball tracking equipment |
| CN113495613B (en) * | 2020-03-18 | 2023-11-21 | 北京七鑫易维信息技术有限公司 | Eyeball tracking calibration method and device |
| CN113495613A (en) * | 2020-03-18 | 2021-10-12 | 北京七鑫易维信息技术有限公司 | Eyeball tracking calibration method and device |
| CN113491502A (en) * | 2020-04-03 | 2021-10-12 | 北京七鑫易维信息技术有限公司 | Eyeball tracking calibration inspection method, device, equipment and storage medium |
| CN113534945A (en) * | 2020-04-17 | 2021-10-22 | 北京七鑫易维信息技术有限公司 | Method, device and equipment for determining eyeball tracking calibration coefficient and storage medium |
| CN113534945B (en) * | 2020-04-17 | 2024-11-05 | 北京七鑫易维信息技术有限公司 | Method, device, equipment and storage medium for determining eye tracking calibration coefficient |
| CN111857333A (en) * | 2020-06-29 | 2020-10-30 | 维沃移动通信有限公司 | Eye movement tracking method and device and electronic equipment |
| CN111857333B (en) * | 2020-06-29 | 2022-09-13 | 维沃移动通信有限公司 | Eye movement tracking method and device and electronic equipment |
| CN112232255A (en) * | 2020-10-26 | 2021-01-15 | 上海鹰瞳医疗科技有限公司 | Identity authentication method and device based on multimodal data |
| CN112232255B (en) * | 2020-10-26 | 2022-11-01 | 北京鹰瞳科技发展股份有限公司 | Identity authentication method and equipment based on multi-mode data |
| CN113035355A (en) * | 2021-05-27 | 2021-06-25 | 上海志听医疗科技有限公司 | Video head pulse test sensor post-correction method, system, electronic device and storage medium |
| CN115546884A (en) * | 2022-12-01 | 2022-12-30 | 广州视景医疗软件有限公司 | Eyeball tremor identification method, device and equipment |
| WO2024230660A1 (en) * | 2023-05-08 | 2024-11-14 | 北京七鑫易维信息技术有限公司 | Processing method and apparatus for eyeball tracking, device, and storage medium |
| WO2025016179A1 (en) * | 2023-07-20 | 2025-01-23 | 北京字跳网络技术有限公司 | Method for selecting calibration image, method for estimating line of sight, apparatus, and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110427101A (en) | Calibration method, device, equipment and the storage medium of eyeball tracking | |
| CN106133649B (en) | It is tracked using the eye gaze that binocular gaze constrains | |
| AU2022271466A1 (en) | Accumulation and confidence assignment of iris codes | |
| CN107609383B (en) | 3D face identity authentication method and device | |
| CN103106401B (en) | Mobile terminal iris recognition device with human-computer interaction mechanism | |
| WO2020125499A1 (en) | Operation prompting method and glasses | |
| Baek et al. | Eyeball model-based iris center localization for visible image-based eye-gaze tracking systems | |
| Nishino et al. | Corneal imaging system: Environment from eyes | |
| US11693475B2 (en) | User recognition and gaze tracking in a video system | |
| CN103870796B (en) | Eye sight evaluation method and device | |
| US20030169907A1 (en) | Facial image processing system | |
| EP4088173B1 (en) | Eye tracking device calibration | |
| CN104978548A (en) | Visual line estimation method and visual line estimation device based on three-dimensional active shape model | |
| TW200422971A (en) | Statistical facial feature extraction method | |
| CN106407772A (en) | Human-computer interaction and identity authentication device and method suitable for virtual reality equipment | |
| CN110555426A (en) | Sight line detection method, device, equipment and storage medium | |
| TWI533234B (en) | Control method based on eye's motion and apparatus using the same | |
| CN104089606A (en) | Free space eye tracking measurement method | |
| CN112446322B (en) | Eyeball characteristic detection method, device, equipment and computer readable storage medium | |
| EP3987380A1 (en) | Systems and methods for determining one or more parameters of a user's eye | |
| CN117274383A (en) | Viewpoint prediction method and device, electronic equipment and storage medium | |
| Jankó et al. | Improving human-computer interaction by gaze tracking | |
| CN110347257A (en) | Calibration method, device, equipment and the storage medium of eyeball tracking equipment | |
| CN112287909B (en) | Double-random in-vivo detection method for randomly generating detection points and interactive elements | |
| Schnieders et al. | Reconstruction of display and eyes from a single image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191108 |
|
| RJ01 | Rejection of invention patent application after publication |