CN107657589B - Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration - Google Patents
Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration Download PDFInfo
- Publication number
- CN107657589B CN107657589B CN201711138756.2A CN201711138756A CN107657589B CN 107657589 B CN107657589 B CN 107657589B CN 201711138756 A CN201711138756 A CN 201711138756A CN 107657589 B CN107657589 B CN 107657589B
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- slave device
- slave
- coordinates
- local coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000006073 displacement reaction Methods 0.000 claims description 16
- 230000003993 interaction Effects 0.000 abstract description 9
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
- G06T3/604—Rotation of whole images or parts thereof using coordinate rotation digital computer [CORDIC] devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
本发明提供一种基于三基准点标定的手机AR定位坐标轴同步方法，包括以下步骤：在各参与联机的设备上分别用自身的位置以及陀螺仪的角度建立各参与联机设备的本地坐标系；选取任意一个参与联机的设备作为主参考设备，其余为从属设备，将主参考设备的本地坐标系作为参考坐标系，并在主参考设备中显示三个基准点；各从属设备根据所显示的三个基准点调整各自本地坐标系的位置，获取本地坐标系与参考坐标系之间的偏差；将参考坐标系中虚拟物体的坐标转换成各从属设备本地坐标系中的坐标，并在各从属设备上显示。本发明能够在各台手机间形成统一的坐标系，便于进行方便和统一的联机互动；方法简单易行，有助于推动AR虚拟互动在低成本手机上的实现。
The invention provides a mobile phone AR positioning coordinate axis synchronization method based on three reference point calibration, comprising the following steps: establishing the local coordinate system of each participating device on the device using its own position and the angle of the gyroscope respectively; Select any device participating in the connection as the master reference device, the rest are slave devices, use the local coordinate system of the master reference device as the reference coordinate system, and display three reference points in the master reference device; Each reference point adjusts the position of each local coordinate system, and obtains the deviation between the local coordinate system and the reference coordinate system; converts the coordinates of the virtual object in the reference coordinate system into the coordinates in the local coordinate system of each slave device, and displays the coordinates in the local coordinate system of each slave device. displayed above. The invention can form a unified coordinate system among mobile phones, which facilitates convenient and unified online interaction; the method is simple and easy to implement, and helps to promote the realization of AR virtual interaction on low-cost mobile phones.
Description
Technical Field
The invention belongs to the field of augmented reality, and particularly relates to a coordinate axis position and direction synchronization method based on calibration of three reference points when AR virtual interaction is carried out among multiple mobile phones.
Background
The existing AR augmented reality technology has huge development in numerous fields such as games, education, entertainment and the like, and the interestingness of the technology is different from a VR virtual reality fully-immersive experience mode and is achieved by combining with real scene content and interaction between virtual objects. With the release of the ARKIT development kit carried by the IOS11 system, the technical fields of AR and MR are newly expanded, and the indoor space positioning function which can be realized only in expensive equipment originally can be realized conveniently. The traditional AR technology uses card or real lamp markers as reference points to attach 3D virtual images around them, while the more advanced AR technology today uses gravity sensing and the auxiliary data of a gyro sensor apart from these markers, and uses a spatial positioning method instead of the identification of the reference points, thereby enabling the virtual and real integration in a wider range. However, this new method will cause the formation of independent spatial coordinate systems between the devices, and therefore, the online interaction cannot be conveniently and uniformly performed.
Therefore, in order to solve the above problems, it is an urgent need of those skilled in the art to develop a method suitable for conveniently and uniformly performing online interaction among a plurality of mobile phones within a certain range and realizing AR virtual interaction.
Disclosure of Invention
In order to solve the problems, the invention discloses a mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration.
In order to achieve the purpose, the invention provides the following technical scheme:
a mobile phone AR positioning coordinate axis synchronization method based on three-datum point calibration comprises the following steps:
(1) establishing a local coordinate system of each online participating device by using the position of the online participating device and the angle of a gyroscope on each online participating device;
(2) selecting any one device participating in online as a main reference device, taking the rest devices as slave devices, taking a local coordinate system of the main reference device as a reference coordinate system, and displaying three reference points in the main reference device;
each slave device adjusts the position of each local coordinate system according to the three displayed reference points, and obtains the deviation between the local coordinate system and the reference coordinate system;
(3) and converting the coordinates of the virtual object in the reference coordinate system into the coordinates in the local coordinate system of each slave device, and displaying the coordinates on each slave device.
Further, the following steps: the three reference points displayed in the main reference device in step (2) can form an isosceles right triangle.
Further, the Z axis of the local coordinate system of each participating online device in step (1) is vertically upward.
Further, the specific method of step (2) is:
(2-a) selecting any one device participating in online as a main reference device, taking the other devices as slave devices, taking a local coordinate system of the main reference device as a reference coordinate system, and displaying three circular reference points on the main reference device;
(2-b) displaying the same three circular reference points on the slave device as well;
(2-c) the slave device scanning a display screen of the master reference device by using the camera, and adjusting the position of the slave device until the three circular reference points displayed on the slave device can be completely coincided with the three circular reference points displayed on the master reference device;
(2-d) obtaining the position (x) of the slave device in the reference coordinate system according to the position of the reference coordinate system and the orientation of the master reference device1,y1,z1)；
(2-e) acquiring euler rotation angles and coordinate positions of the orientations of the master reference device and the slave device in local coordinate systems thereof, respectively;
(2-f) setting an initial vector (1,0,0), and obtaining a new vector (x, y, z) through Euler rotation;
(2-g) obtaining a rotation angle of the slave device on the XY plane through the new vector;
(2-h) calculating a rotation angle difference theta between the slave device and the master reference device;
(2-i) the slave device reversely rotating its local coordinate system according to the rotation angle difference θ, so that the local coordinate system of the slave device and the reference coordinate system are consistent in directions of X, Y, Z three coordinate axes;
(2-j) new coordinates (x) rotated by the local coordinate system of the slave device2,y2,z2) With its coordinates (x) in the reference coordinate system1,y1,z1) Obtaining the coordinate origin displacement between the slave equipment and the master reference equipment by the difference value, and recording the displacement as delta x, delta y and delta z;
and (2-k) repeating the steps (2-b) to (2-j) to respectively obtain the coordinate origin displacement between each slave device and the master reference device.
Further, the specific method of step (3) is:
(3-a) transmitting the relevant object coordinates in the AR environment to each slave device with the master reference device as a data distribution center;
(3-b) each slave device respectively displacing the received object coordinates in X, Y, Z three-axis directions according to the displacement of the origin of coordinates;
and (3-c) rotating the object coordinate after the displacement in the step (3-b) by an angle theta by taking the Z axis as a rotating axis to obtain the slave device coordinate in the master reference coordinate system on each slave device.
Compared with the prior art, the invention can form a uniform coordinate system among all mobile phones, thereby facilitating convenient and uniform online interaction; the method is simple and easy to implement, and is beneficial to promoting the realization of AR virtual interaction on a low-cost mobile phone.
Drawings
FIG. 1 is a flow chart of the present invention.
Detailed Description
The technical solutions provided by the present invention will be described in detail below with reference to specific examples, and it should be understood that the following specific embodiments are only illustrative of the present invention and are not intended to limit the scope of the present invention.
Example 1:
the invention provides a mobile phone AR positioning coordinate axis synchronization method based on three-datum point calibration, which comprises the following steps:
(1) and respectively establishing a local coordinate system of each online participating device by using the position of each online participating device and the angle of the gyroscope, wherein the Z axis of the local coordinate system of each online participating device is vertically upward.
(2-a) selecting any one device participating in online as a main reference device, taking the other devices as slave devices, taking a local coordinate system of the main reference device as a reference coordinate system, and displaying three circular reference points on the main reference device, wherein the three displayed circular reference points can form an isosceles right triangle;
(2-b) displaying the same three circular reference points on the slave device as well;
(2-c) the slave device scanning a display screen of the master reference device by using the camera, and adjusting the position of the slave device until the three circular reference points displayed on the slave device can be completely coincided with the three circular reference points displayed on the master reference device;
(2-d) obtaining the position (x) of the slave device in the reference coordinate system according to the position of the reference coordinate system and the orientation of the master reference device1,y1,z1)；
(2-e) acquiring euler rotation angles and coordinate positions of the orientations of the master reference device and the slave device in local coordinate systems thereof, respectively;
(2-f) setting an initial vector (1,0,0), and obtaining a new vector (x, y, z) through Euler rotation;
(2-g) obtaining a rotation angle of the slave device on the XY plane through the new vector;
(2-h) calculating a rotation angle difference theta between the slave device and the master reference device;
(2-i) the slave device reversely rotating its local coordinate system according to the rotation angle difference θ, so that the local coordinate system of the slave device and the reference coordinate system are consistent in directions of X, Y, Z three coordinate axes;
(2-j) new coordinates (x) rotated by the local coordinate system of the slave device2,y2,z2) With its coordinates (x) in the reference coordinate system1,y1,z1) Obtaining the coordinate origin displacement between the slave equipment and the master reference equipment by the difference value, and recording the displacement as delta x, delta y and delta z;
and (2-k) repeating the steps (2-b) to (2-j) to respectively obtain the coordinate origin displacement between each slave device and the master reference device.
(3-a) transmitting the relevant object coordinates in the AR environment to each slave device with the master reference device as a data distribution center;
(3-b) each slave device respectively displacing the received object coordinates in X, Y, Z three-axis directions according to the displacement of the origin of coordinates;
and (3-c) rotating the object coordinate after the displacement in the step (3-b) by an angle theta by taking the Z axis as a rotating axis to obtain the slave device coordinate in the master reference coordinate system on each slave device.
Example 2:
first, a master reference device and a slave device are initialized at arbitrary positions, and reference object coordinate positions are set to r11(0,1,0) and r21(0,1,0), respectively.
Secondly, the slave device scans the display screen of the master reference device by using a camera, the position of the slave device is adjusted until three circular reference points displayed on the slave device can be completely coincided with three circular reference points displayed on the master reference device, two new coordinates r12(-0.16035,0.98610, -0.04343) and r22(0.80003,0.15726, -0.57898) are respectively obtained, x and z components of the respective coordinates are respectively substituted into the following formula, and finally a difference value delta theta of two rotation angles is obtained, wherein the angle is the rotation angle of the master device relative to the slave device:
when z is greater than 0, theta is actg (x/z)
When z <0, θ ═ actg (x/z) + π
When z is 0, x is >0, θ is 0; x <0, θ ═ pi, x ═ 0, no solution.
The calculated delta theta was-0.58244 radians.
Substituting the measured parameters into the following formula to obtain the displacement of the coordinate origin of the master reference device relative to the slave device as follows:
ΔT(-0.12688,-0.0780,-0.02480)，
θ21＝θ2+Δθ，
r＝sqrt(pow(r22.x,2)+pow(r22.z,2))，
Δx＝r*cos(θ21)–r12.x，
Δz＝r*sin(θ21)–r12.z，
Δy＝r22.y。
fourthly, an object is placed at the position P1(0,0,0) of the reference coordinate system, and the new coordinate position obtained by the translation and rotation formulas of the following formulas is P2(-0.11961, -0.07805, 0.04907);
x＝P1.x+Δx，
y＝P1.y+Δy，
z＝P2.z+Δz，
r＝sqrt(pow(x,2)+pow(z,2))，
the angle theta 3 is obtained through the step three,
θ4＝θ3–Δθ，
P2.x＝cos(θ4)*r，
P2.y＝y2，
P2.z＝sin(θ4)*r。
finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the technical solutions, and those skilled in the art should understand that modifications or equivalent substitutions can be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all the modifications and equivalent substitutions should be covered by the claims of the present invention.
Claims (4)
1. A mobile phone AR positioning coordinate axis synchronization method based on three reference point calibration is characterized in that: the method comprises the following steps:
(1) establishing a local coordinate system of each online participating device by using the position of the online participating device and the angle of a gyroscope on each online participating device;
(2) selecting any one device participating in online as a main reference device, and the rest of the devices are slave devices, taking a local coordinate system of the main reference device as a reference coordinate system, and displaying three reference points in the main reference device, wherein the displayed three reference points can form an isosceles right triangle;
each slave device adjusts the position of each local coordinate system according to the three displayed reference points, and obtains the deviation between the local coordinate system and the reference coordinate system;
(3) and converting the coordinates of the virtual object in the reference coordinate system into the coordinates in the local coordinate system of each slave device, and displaying the coordinates on each slave device.
2. The mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration according to claim 1, characterized in that: and (2) the Z axis of the local coordinate system of each piece of online equipment in the step (1) is vertically upward.
3. The mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration according to claim 1, characterized in that: the specific method of the step (2) is as follows:
(2-a) selecting any one device participating in online as a main reference device, taking the other devices as slave devices, taking a local coordinate system of the main reference device as a reference coordinate system, and displaying three circular reference points on the main reference device;
(2-b) displaying the same three circular reference points on the slave device as well;
(2-c) the slave device scanning a display screen of the master reference device by using the camera, and adjusting the position of the slave device until the three circular reference points displayed on the slave device can be completely coincided with the three circular reference points displayed on the master reference device;
(2-d) obtaining the position (x) of the slave device in the reference coordinate system according to the position of the reference coordinate system and the orientation of the master reference device1,y1,z1)；
(2-e) acquiring euler rotation angles and coordinate positions of the orientations of the master reference device and the slave device in local coordinate systems thereof, respectively;
(2-f) setting an initial vector (1,0,0), and obtaining a new vector (x, y, z) through Euler rotation;
(2-g) obtaining a rotation angle of the slave device on the XY plane through the new vector;
(2-h) calculating a rotation angle difference theta between the slave device and the master reference device;
(2-i) the slave device reversely rotating its local coordinate system according to the rotation angle difference θ, so that the local coordinate system of the slave device and the reference coordinate system are consistent in directions of X, Y, Z three coordinate axes;
(2-j) new coordinates (x) rotated by the local coordinate system of the slave device2,y2,z2) With its coordinates (x) in the reference coordinate system1,y1,z1) Obtaining the coordinate origin displacement between the slave equipment and the master reference equipment by the difference value, and recording the displacement as delta x, delta y and delta z;
and (2-k) repeating the steps (2-b) to (2-j) to respectively obtain the coordinate origin displacement between each slave device and the master reference device.
4. The mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration according to claim 1, characterized in that: the specific method of the step (3) is as follows:
(3-a) transmitting the relevant object coordinates in the AR environment to each slave device with the master reference device as a data distribution center;
(3-b) each slave device respectively displacing the received object coordinates in X, Y, Z three-axis directions according to the displacement of the origin of coordinates;
and (3-c) rotating the object coordinate after the displacement in the step (3-b) by an angle theta by taking the Z axis as a rotating axis to obtain the slave device coordinate in the master reference coordinate system on each slave device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711138756.2A CN107657589B (en) | 2017-11-16 | 2017-11-16 | Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711138756.2A CN107657589B (en) | 2017-11-16 | 2017-11-16 | Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107657589A CN107657589A (en) | 2018-02-02 |
| CN107657589B true CN107657589B (en) | 2021-05-14 |
Family
ID=61121734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711138756.2A Expired - Fee Related CN107657589B (en) | 2017-11-16 | 2017-11-16 | Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107657589B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108389264B (en) * | 2018-02-07 | 2022-03-29 | 杭州易现先进科技有限公司 | Coordinate system determination method and device, storage medium and electronic equipment |
| CN108682282A (en) * | 2018-05-09 | 2018-10-19 | 北京航空航天大学青岛研究院 | A kind of exchange method of the augmented reality version periodic table of chemical element based on ARKit frames |
| CN109087399B (en) * | 2018-07-17 | 2024-03-01 | 上海游七网络科技有限公司 | Method for rapidly synchronizing AR space coordinate system through positioning map |
| CN109976523B (en) * | 2019-03-22 | 2021-05-18 | 联想(北京)有限公司 | Information processing method and electronic device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102147658A (en) * | 2011-02-12 | 2011-08-10 | 华为终端有限公司 | Method and device for realizing interaction of augment reality (AR) and mobile terminal |
| CN102884490A (en) * | 2010-03-05 | 2013-01-16 | 索尼电脑娱乐美国公司 | Maintaining multiple views on a shared stable virtual space |
| WO2014074465A1 (en) * | 2012-11-06 | 2014-05-15 | Stephen Latta | Cross-platform augmented reality experience |
| CN106568381A (en) * | 2016-10-25 | 2017-04-19 | 西北工业大学 | Calibration method for linear radar measurement system based on standard plane |
| CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for generating virtual and augmented reality |
| CN107111996A (en) * | 2014-11-11 | 2017-08-29 | 本特图像实验室有限责任公司 | The augmented reality experience of Real-Time Sharing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120086630A1 (en) * | 2010-10-12 | 2012-04-12 | Sony Computer Entertainment Inc. | Using a portable gaming device to record or modify a game or application in real-time running on a home gaming system |
-
2017
- 2017-11-16 CN CN201711138756.2A patent/CN107657589B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102884490A (en) * | 2010-03-05 | 2013-01-16 | 索尼电脑娱乐美国公司 | Maintaining multiple views on a shared stable virtual space |
| CN102147658A (en) * | 2011-02-12 | 2011-08-10 | 华为终端有限公司 | Method and device for realizing interaction of augment reality (AR) and mobile terminal |
| WO2014074465A1 (en) * | 2012-11-06 | 2014-05-15 | Stephen Latta | Cross-platform augmented reality experience |
| CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for generating virtual and augmented reality |
| CN107111996A (en) * | 2014-11-11 | 2017-08-29 | 本特图像实验室有限责任公司 | The augmented reality experience of Real-Time Sharing |
| CN106568381A (en) * | 2016-10-25 | 2017-04-19 | 西北工业大学 | Calibration method for linear radar measurement system based on standard plane |
Non-Patent Citations (2)
| Title |
|---|
| Massively Multiplayer Online Worlds as a Platform for;Tobias Lang,et al;《2008 IEEE Virtual Reality Conference》;20080404;正文第67-70页 * |
| 基于Android平台的增强现实太极拳教学软件研究与开发;李宇放;《中国优秀硕士学位论文全文数据库 信息科技辑》;20150315;I138-2733 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107657589A (en) | 2018-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107657589B (en) | Mobile phone AR positioning coordinate axis synchronization method based on three-datum-point calibration | |
| CN105556742B (en) | The acquisition methods and equipment and system of antenna works parameter | |
| US9355451B2 (en) | Information processing device, information processing method, and program for recognizing attitude of a plane | |
| US8907983B2 (en) | System and method for transitioning between interface modes in virtual and augmented reality applications | |
| CN105279750B (en) | An Equipment Display Navigation System Based on IR-UWB and Image Moment | |
| CN107729707B (en) | Engineering construction lofting method based on mobile augmented reality technology and BIM | |
| CN107240156B (en) | A kind of high-precision outdoor augmented reality spatial information display system and method | |
| JP5575758B2 (en) | Spatial prediction approximation | |
| JP2013517579A (en) | Augmented reality system | |
| KR100789100B1 (en) | Mobile Augmented Reality Service System and Method | |
| WO2021053805A1 (en) | Information terminal device and positional recognition sharing method | |
| CN106767775B (en) | A kind of localization method based on imaging sensor and inertial navigation sensor | |
| CN106546172B (en) | Three-dimensional coordinate measurement method based on non-orthogonal axis laser total station | |
| CN111197980B (en) | Intelligent adsorption method for AR measuring room | |
| CN107945116B (en) | Coordinate axis synchronization method for AR indoor positioning between multiple devices | |
| CN110599432B (en) | Image processing system and image processing method | |
| CN111080704A (en) | Method and device for enhancing reality of video | |
| JP5644379B2 (en) | Electronic bulletin board system, server device, and portable terminal | |
| CN107958491A (en) | Mobile augmented reality virtual coordinates and construction site coordinate matching method | |
| CN206627122U (en) | Non-visible laser calibrates indoor positioning navigation system | |
| Wu et al. | An omnidirectional spatial monocular visual localization and tracking method for indoor unmanned aerial vehicles based on the two-axis rotary table | |
| CN104269081B (en) | Light-duty geodesic system analog training device and light-duty geodesic system simulation training method | |
| CN106230518A (en) | Multidrop topology based on WiFi Yu sound wave approaches indoor orientation method | |
| Rajagopal et al. | Demo abstract: welcome to my world: demystifying multi-user AR with the cloud | |
| CN109147055B (en) | Augmented reality display method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210514 |