CN114612557B - Viewpoint alignment method, device, storage medium and electronic device - Google Patents
Viewpoint alignment method, device, storage medium and electronic device Download PDFInfo
- Publication number
- CN114612557B CN114612557B CN202210234161.1A CN202210234161A CN114612557B CN 114612557 B CN114612557 B CN 114612557B CN 202210234161 A CN202210234161 A CN 202210234161A CN 114612557 B CN114612557 B CN 114612557B
- Authority
- CN
- China
- Prior art keywords
- target
- screen
- adjustment
- image
- adjusting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Studio Devices (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
本申请公开了一种视点对齐方法、装置、存储介质及电子设备。该方法包括：确定靶标在屏幕对应人眼视点的投影位置；根据投影位置显示特征点；调整摄像设备的位置，使得所述摄像设备透过所述屏幕拍摄所述靶标得到的靶标图像中的靶标与特征点对齐。本申请可以提高摄像设备进行摄像头与屏幕之间的位置关系的标定的准确性。
The present application discloses a viewpoint alignment method, device, storage medium and electronic device. The method comprises: determining the projection position of a target on a screen corresponding to a human eye viewpoint; displaying feature points according to the projection position; and adjusting the position of a camera device so that the target and the feature points in a target image obtained by the camera device shooting the target through the screen are aligned. The present application can improve the accuracy of the camera device in calibrating the positional relationship between the camera and the screen.
Description
Technical Field
The application belongs to the technical field of electronics, and particularly relates to a viewpoint alignment method, a viewpoint alignment device, a storage medium and electronic equipment.
Background
For an optical transmission type display device, if AR development is required, a camera needs to be added, and the position relationship between the camera and the screen of the optical transmission type display device needs to be calibrated, that is, a user needs to wear the optical transmission type display device to align some cursors on the screen with objects in the real world multiple times. Because the manual calibration mode is low in efficiency, the related technology proposes that the camera equipment replaces human eyes to calibrate the position relationship between the camera and the screen, but the calibration mode is low in accuracy.
Disclosure of Invention
The embodiment of the application provides a viewpoint alignment method, a viewpoint alignment device, a storage medium and electronic equipment, which can improve the accuracy of calibrating the position relationship between a camera and a screen by camera equipment.
In a first aspect, an embodiment of the present application provides a viewpoint alignment method, including:
determining the projection position of a target on a screen corresponding to a human eye viewpoint;
Displaying the characteristic points according to the projection positions;
And adjusting the position of the image pickup device so that the target in the target image obtained by shooting the target through the screen by the image pickup device is aligned with the characteristic point.
In a second aspect, an embodiment of the present application provides a viewpoint alignment apparatus, including:
the determining module is used for determining the projection position of the target on the screen corresponding to the eye viewpoint;
The display module is used for displaying the characteristic points according to the projection positions;
The adjusting module is used for adjusting the position of the image pickup device so that the target in the target image obtained by shooting the target through the screen by the image pickup device is aligned with the characteristic point.
In a third aspect, an embodiment of the present application provides a storage medium having stored thereon a computer program which, when executed on a computer, causes the computer to perform the viewpoint alignment method provided by the embodiment of the present application.
In a fourth aspect, an embodiment of the present application further provides an electronic device, including a memory, and a processor, where the processor is configured to execute the viewpoint alignment method provided by the embodiment of the present application by calling a computer program stored in the memory.
According to the embodiment of the application, the alignment of the camera and the human eye viewpoint can be realized by determining the projection position of the target at the human eye viewpoint corresponding to the screen, displaying the characteristic point according to the projection position, and adjusting the position of the camera so that the target in the target image obtained by shooting the target through the screen by the camera is aligned with the characteristic point, thereby improving the calibration accuracy of the camera and the screen by the camera.
Drawings
The technical solution of the present application and its advantageous effects will be made apparent by the following detailed description of the specific embodiments of the present application with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a first viewpoint alignment method according to an embodiment of the present application.
Fig. 2 is a view of an arrangement of multiple targets provided by an embodiment of the application.
Fig. 3 is a schematic diagram of a target provided by an embodiment of the present application.
Fig. 4 is a second flowchart of a viewpoint alignment method according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of a viewpoint alignment device according to an embodiment of the present application.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
It should be noted that the terms "first," "second," and "third," etc. are used herein to distinguish between different objects and not to describe a particular order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to the particular steps or modules listed and certain embodiments may include additional steps or modules not listed or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The embodiment of the application provides a viewpoint alignment method, a viewpoint alignment device, a storage medium and electronic equipment, wherein an execution subject of the viewpoint alignment method can be the viewpoint alignment device provided by the embodiment of the application or the electronic equipment integrated with the viewpoint alignment device, and the viewpoint alignment device can be realized in a hardware or software mode. Wherein the electronic device may be a transmissive display device.
The transmissive display device may include transmissive head-mounted display devices such as AR glasses and AR helmets, and the like. The transmission type head-mounted display device is a display device which can be worn on the head of a user and can be conveniently used without two hands. The transmission type head-mounted display device is provided with a semi-transparent and semi-reflective optical system which can transmit external ambient light like ordinary glasses on one hand, so that a user wearing the transmission type head-mounted display device can see the real world in front of eyes, and on the other hand, images from the micro-display can be reflected and overlapped into the field of vision of a person, namely, the user wearing the transmission type head-mounted display device can see virtual images and the displayed world through a screen. Thus, the fusion of the virtual information and the real environment can be realized by utilizing the perspective type head-mounted display device.
Referring to fig. 1, fig. 1 is a schematic flow chart of a first viewpoint alignment method according to an embodiment of the present application, where the flow chart may include:
101. and determining the projection position of the target on the screen corresponding to the eye point.
In this embodiment, the target may be set in advance within a preset range, such as 5 meters, where the transmissive display device is facing the field of view. When the eyes are required to be aligned with the view points of the camera equipment, the user can wear the transmission type display equipment, the eyes observe the preset target through the screen of the transmission type display equipment, and the positions of the target are marked on the screen, so that the projection positions of the target corresponding to the view points of the eyes on the screen can be obtained by the transmission type display equipment.
102. And displaying the characteristic points according to the projection positions.
In this embodiment, after obtaining the projection position of the target on the screen corresponding to the eye point, the transmissive display device may display the feature point at the projection position. For example, the projection position is in the upper left corner of the screen, and then the feature point may be displayed in the upper left corner of the screen. For another example, the projection position is in the lower right corner of the screen, and then the feature point may be displayed in the lower right corner of the screen.
It will be understood that in the present embodiment, the feature points are points displayed on the screen, which can be seen by the human eye or photographed by the image pickup apparatus. As for the type and size of the feature points, the embodiment of the present application is not limited, and the actual requirement is based.
103. The position of the image pickup device is adjusted so that the target in the target image obtained by shooting the target through the screen by the image pickup device is aligned with the characteristic point.
In this embodiment, the transmissive display device may control the adjustment device to adjust the position of the image capturing device, so that the target in the target image obtained by capturing the target through the screen by the image capturing device is aligned with the feature point.
In this embodiment, when the overlapping percentage of the target and the feature point in the target image is greater than a preset percentage, it is determined that the target and the feature point in the target image are aligned. The preset percentage may be set according to practical situations, and is not particularly limited herein. For example, when alignment accuracy is required to be high, the preset percentage may be set high, for example, 99.8%.
It can be understood that when the target in the target image obtained by photographing the target through the screen by the image pickup apparatus is aligned with the feature point, since the feature point on the screen observed by the human eye is also aligned with the target observed through the screen, the viewpoint alignment of the human eye with the image pickup apparatus is achieved.
According to the embodiment of the application, the alignment of the camera and the human eye viewpoint can be realized by determining the projection position of the target at the human eye viewpoint corresponding to the screen, displaying the characteristic point according to the projection position, and adjusting the position of the camera so that the target in the target image obtained by shooting the target through the screen by the camera is aligned with the characteristic point, thereby improving the calibration accuracy of the camera and the screen by the camera.
In some embodiments, before adjusting the position of the image capturing apparatus, it may further include:
acquiring a first target image obtained by shooting a target through a screen by using camera equipment;
adjusting the position of the image capturing apparatus may include:
the position of the image capturing apparatus is adjusted according to the first target image.
In this embodiment, the feature points are obtained by the user observing the disposed target through the eyes, that is, the feature points and the target are aligned when the user views the feature points and the target through the eyes. Then, when the target in the target image obtained by the image pickup apparatus photographing the target through the screen is not aligned with the feature point, the eye is also not aligned with the viewpoint of the image pickup apparatus, and when the target in the target image obtained by the image pickup apparatus photographing the target through the screen is aligned with the feature point, the eye is also aligned with the viewpoint of the image pickup apparatus, and therefore, the transmissive display apparatus can adjust the position of the image pickup apparatus according to the target image obtained by the image pickup apparatus photographing the target through the screen.
For example, the transmissive display device may control the image capturing device to capture a first target image through a screen of the transmissive display device. The exposure display device can acquire the first target image and adjust the position of the camera device according to the first target image.
It can be understood that when the target in the first target image is aligned with the feature point, the process can be directly ended without adjusting the position of the image capturing device, or the position of the image capturing device can be adjusted according to the first target image, so that the overlapping percentage of the target in the target image obtained by the image capturing of the target through the screen by the image capturing device after the position adjustment is larger than the overlapping percentage of the target in the first target image and the feature point, and the alignment precision of human eyes and the image capturing device is improved. When the target in the first target image is not aligned with the feature point, the position of the camera device can be adjusted directly according to the first target image.
In some embodiments, adjusting the position of the image capturing device according to the first target image may include:
determining a first adjustment amount and a first adjustment direction of the camera device according to positions of the target and the feature points in the first target image;
The position of the image capturing apparatus is adjusted according to the first adjustment amount and the first adjustment direction.
For example, the positions of the target and the feature point may include coordinates of the target and the feature point. The transmission type display device determines the distance between the target and the characteristic point according to the coordinates of the target and the characteristic point in the first target image. The transmissive display device may further determine a relative direction of the target and the feature point in the first target image, and determine a first adjustment amount and a first adjustment direction of the image capturing device according to the distance and the relative direction. After the first adjustment amount and the first adjustment direction are determined, the transmission type display device can control the adjustment device to adjust the position of the image pickup device according to the first adjustment amount and the first adjustment direction, so that the target in the target image obtained by the image pickup device through the screen shooting the target is aligned with the characteristic point.
For example, a first mapping relationship between a distance and an adjustment amount, and a second mapping relationship between a relative direction and an adjustment direction may be preset and built in the transmissive display device. After obtaining the distance between the target and the feature point in the first target image and the relative direction between the target and the feature point, the transmissive display device may determine a first adjustment amount according to the distance and the first mapping relationship, and determine a first adjustment direction according to the relative direction and the second mapping relationship.
The first adjustment direction and the first adjustment amount are adjustment directions and adjustment amounts that enable the imaging apparatus after adjusting the position of the imaging apparatus according to the first adjustment direction and the first adjustment amount to align a target in a target image obtained by photographing the target through a screen with the feature point. The first adjustment may include a translational amount and/or a rotational amount, and the first adjustment direction may include a translational direction of an x, y, or z axis, or a rotational direction of a pitch, yaw, roll.
In some embodiments, adjusting the position of the image capturing apparatus may include:
Receiving an input adjustment amount and an input adjustment direction;
The position of the image pickup apparatus is adjusted according to the input adjustment amount and adjustment direction.
In this embodiment, the transmissive display device may receive the adjustment amount and the adjustment direction input by the user, and control the adjustment device to adjust the position of the image capturing device according to the adjustment amount and the adjustment direction input by the user, so that the target in the target image obtained by capturing the target by the image capturing device through the screen is aligned with the feature point.
In some embodiments, after adjusting the position of the image capturing apparatus according to the input adjustment amount and the adjustment direction, it may further include:
When a target in a target image obtained by shooting the target through a screen by the camera equipment is aligned with the feature points, outputting alignment prompt information.
It can be understood that, when the adjustment device is controlled to adjust the position of the image capturing device according to the adjustment amount and the adjustment direction input by the user, the transmissive display device can acquire the target image obtained by the image capturing device after the adjustment of the position through the screen capturing the target. The transmissive display device may detect whether a target in the target image is aligned with a feature point. If the target in the target image is aligned with the feature point, the transmission type display device can output alignment prompt information to prompt the user that human eyes are aligned with the viewpoint of the camera device, and the adjustment amount and the adjustment direction do not need to be continuously input. If the target in the target image is not aligned with the feature point, the transmissive display device may output misalignment prompt information to prompt the user that the eyes of the user are not aligned with the viewpoint of the image capturing device, and then the user may continue to input the adjustment amount and the adjustment direction to adjust the position of the image capturing device until the target in the target image obtained by the image capturing device after the position adjustment captures the target through the screen is aligned with the feature point.
In this embodiment, the output mode of the alignment prompt information is not particularly limited, and is based on actual requirements. For example, the alignment prompt may be played and/or displayed, or other devices may be controlled to play and/or display the alignment prompt. The output mode of the misalignment prompt information is not particularly limited, and the actual requirement is met. For example, the misalignment prompt may be played and/or displayed, or other devices may be controlled to play and/or display the misalignment prompt.
In some embodiments, after adjusting the position of the image capturing apparatus according to the input adjustment amount and the adjustment direction, it may further include:
When a target in a target image obtained by shooting the target through a screen by the camera equipment is not aligned with the feature points, obtaining a second target image obtained by shooting the target through the screen by the camera equipment;
Determining a second adjustment amount and a second adjustment direction of the image pickup device according to the positions of the target and the feature points in the second target image;
and outputting adjustment prompt information according to the second adjustment quantity and the second adjustment direction.
When the adjusting device is controlled to adjust the position of the image capturing device according to the adjustment amount and the adjustment direction input by the user, the transmission type display device can acquire a second target image obtained by the image capturing device after the position is adjusted through the screen shooting target. The transmissive display device may detect whether the target in the second target image is aligned with the feature point. If the target in the second target image is not aligned with the feature point, the transmission type display device can determine a second adjustment amount and a second adjustment direction of the image pickup device according to the second target image, and output adjustment prompt information according to the second adjustment amount and the second adjustment direction, so that a user can input the adjustment amount and the adjustment direction according to the prompt information, and further, the eye alignment of eyes and the view point of the image pickup device can be realized more rapidly.
For example, the positions of the target and the feature point may include coordinates of the target and the feature point. The transmission type display device determines the distance between the target and the characteristic point according to the coordinates of the target and the characteristic point in the second target image. The transmissive display device may further determine a relative direction of the target and the feature point in the second target image, and determine a second adjustment amount and a second adjustment direction of the image capturing device according to the distance and the relative direction. After the second adjustment amount and the second adjustment direction are determined, the transmission type display device can control the adjustment device to adjust the position of the image pickup device according to the second adjustment amount and the second adjustment direction, so that the target in the target image obtained by the image pickup device through the screen shooting the target is aligned with the characteristic point.
For example, a second mapping relationship between the distance and the adjustment amount, and a second mapping relationship between the relative direction and the adjustment direction may be preset and built into the transmissive display device. After obtaining the distance between the target and the feature point in the second target image and the relative direction between the target and the feature point, the transmissive display device may determine a second adjustment amount according to the distance and the second mapping relationship, and determine a second adjustment direction according to the relative direction and the second mapping relationship.
Wherein the second adjustment direction and the second adjustment amount are adjustment directions and adjustment amounts that can be such that, after adjusting the position of the image capturing apparatus in accordance with the second adjustment direction and the second adjustment amount, the image capturing apparatus after adjusting the position captures a target through the screen, the target in the target image being aligned with the feature point. The second adjustment may include a translational amount and/or a rotational amount, and the second adjustment direction may include a translational direction of the x, y, or z axis, or a rotational direction of the pitch, yaw, roll.
In this embodiment, the output mode of the adjustment prompt information is not particularly limited, and the actual requirement is satisfied. For example, the adjustment prompt may be played and/or displayed, or other devices may be controlled to play and/or display the adjustment prompt. For example, the first adjustment amount and the first adjustment direction may be directly voice-broadcast, or the first adjustment amount and the first adjustment direction may be directly displayed, or other devices may be controlled to display the first adjustment amount and the first adjustment direction.
In some embodiments, the targets are multiple, the multiple targets being at different distances from the screen.
For example, multiple targets may be displayed within a predetermined range, such as 5 meters, of the field of view of the transmissive display device, wherein the multiple targets are spaced from the screen. In particular, the plurality of targets may be disposed at different depths and different heights within a predetermined range of the transmissive display device facing the field of view. For example, a scenario of placement of multiple targets may be as shown in fig. 2. Wherein the target may be a checkerboard target as shown in fig. 3. It will be appreciated that the target may be other forms of target, without specific limitation.
It will be appreciated that a human eye includes a left eye and a right eye, and a transmissive display device typically includes a first screen corresponding to the left eye and a second screen corresponding to the right eye, and therefore, when performing viewpoint alignment of the human eye with an image pickup device, it is necessary to perform viewpoint alignment of the left eye and one image pickup device based on the first screen and viewpoint alignment of the right eye and the other image pickup device based on the second screen, respectively.
The following description will be made taking, as an example, a case where a transmissive display device is AR glasses, an adjustment device is a calibration device, an imaging device is an industrial camera, and a left eye is aligned with a viewpoint of the industrial camera. The industrial camera can be an undistorted industrial camera or an industrial camera subjected to distortion correction, so that imaging is ensured to be undistorted, and the alignment precision of human eyes and the industrial camera is improved.
The calibration device can comprise a glasses fixing clamp and a six-axis turntable (three-axis rotation pitch, yaw, roll; three-axis translation x, y, z), wherein the fixing clamp is used for fixing the AR glasses, and the six-axis turntable is used for setting the industrial camera, so that the industrial camera is adjustable in six degrees of freedom. AR glasses employ augmented reality (Augmented Reality, AR) technology. Among them, augmented reality is a technology of increasing a user's perception of the real world through information provided by a computer system, which superimposes computer-generated virtual objects, scenes, or content objects such as system hint information into the real scene to augment or modify the perception of the real world environment or data representing the real world environment.
Firstly, fixing the AR glasses through a fixing clamp, and then keeping the calibration device motionless, wherein the checkerboard targets are arranged at different depths and different heights within the range of 5 meters of the visual field of the AR glasses, as shown in fig. 2.
The user drills the head into the calibration device, wears the AR glasses normally, the left eye observes a target in a scene through the AR glasses, and clicks a target position arranged on a screen corresponding to the left eye of the AR glasses by using a mouse to record, so that the AR glasses can obtain a projection position of the target at a viewpoint corresponding to the left eye of the screen. The AR glasses may then display the feature points in the projected positions. After recording the target position, the user may drill the head out of the calibration device.
The industrial camera may then be positioned at a location of the six-axis turret corresponding to the left eye, i.e., to maintain the position of the industrial camera substantially coincident with the position of the user's left eye. Then, the AR glasses may control the image capturing device to capture the target image through the screen. The AR glasses may acquire the target image and detect whether the target in the target image is aligned with the feature point. Because the position of the industrial camera cannot be completely consistent with the position of the left eye of the user, the industrial camera is also different from the human eye, and therefore, the target in the target image is not aligned with the characteristic point, and then the six-axis machine can be controlled to adjust the position of the industrial camera, so that the target in the target image obtained by shooting the target through the screen by the industrial camera is aligned with the characteristic point. In some cases, if the target and the feature point in the target image obtained by the industrial camera are aligned before the position of the industrial camera is adjusted, the position of the industrial camera may not be adjusted, or the position of the industrial camera may be adjusted, so that the overlapping percentage of the target and the feature point in the target image obtained by the industrial camera is higher.
Based on the above flow, the left eye is aligned with the viewpoint of the industrial camera. Reference is also made to the above procedure for the alignment of the right eye with the viewpoint of the industrial camera, and the description thereof will be omitted. In addition, after the left eye is aligned with the viewpoint of the industrial camera, the industrial camera may be fixed so that the industrial camera does not move, thereby avoiding the problem that the left eye is not aligned with the viewpoint of the industrial camera due to the movement of the industrial camera. When the right eye is aligned with the viewpoint of the industrial camera, another industrial camera may be used for the right eye to be aligned with the viewpoint of the industrial camera, and after the alignment, the other industrial camera is fixed.
It can be understood that by the viewpoint alignment method provided by the embodiment of the application, the transmission type display equipment, such as the imaging equipment used in the production of AR glasses, such as an industrial camera, can be used for replacing human eyes to perform large-scale rapid calibration, and meanwhile, good precision is ensured.
In some embodiments, considering that the pupil distance and the eye size of each person are different, for example, the pupil distance of a large person is larger than the pupil distance of a child, the eye sizes of the large person are also larger than the eye sizes of the child, so that the eyes of different persons can be classified according to the pupil distance and/or the eye sizes to be classified into a plurality of classes, and then the eyes of each class are aligned with the view point of the image pickup device based on the eyes of each class, so that AR eyes suitable for different crowds with higher accuracy can be calibrated.
In some embodiments, the method for aligning the view point can also be applied to virtual-real combination calibration of an on-vehicle AR-HUD, namely, a display surface of the front windshield is a screen, a user records a projection position of a real object target in front of eyes and the front windshield on the front windshield at a seat, a characteristic point can be displayed at the projection position, an industrial camera is used for aligning the real object target point and the characteristic point, at the moment, the position of the industrial camera can be used for replacing the view point position of eyes, the virtual-real combination of the HUD is completed for large-batch quick calibration, and time-consuming operation that the HUD of each automobile needs artificial calibration is omitted.
Referring to fig. 4, fig. 4 is a schematic flow chart of a second viewpoint alignment method according to an embodiment of the present application, where the flow chart may include:
201. And determining the projection position of the target on the screen corresponding to the eye point.
In this embodiment, the target may be set in advance within a preset range, such as 5 meters, where the transmissive display device is facing the field of view. When the eyes are required to be aligned with the view points of the camera equipment, the user can wear the transmission type display equipment, the eyes observe the preset target through the screen of the transmission type display equipment, and the positions of the target are marked on the screen, so that the projection positions of the target corresponding to the view points of the eyes on the screen can be obtained by the transmission type display equipment.
202. And displaying the characteristic points according to the projection positions.
In this embodiment, after obtaining the projection position of the target on the screen corresponding to the eye point, the transmissive display device may display the feature point at the projection position. For example, the projection position is in the upper left corner of the screen, and then the feature point may be displayed in the upper left corner of the screen. For another example, the projection position is in the lower right corner of the screen, and then the feature point may be displayed in the lower right corner of the screen.
It will be understood that in the present embodiment, the feature points are points displayed on the screen, which can be seen by the human eye or photographed by the image pickup apparatus. As for the type and size of the feature points, the embodiment of the present application is not limited, and the actual requirement is based.
203. And acquiring a first target image obtained by shooting a target through a screen by the camera equipment.
In this embodiment, the feature points are obtained by the user observing the disposed target through the eyes, that is, the feature points and the target are aligned when the user views the feature points and the target through the eyes. Then, when the target in the target image obtained by the image pickup apparatus photographing the target through the screen is not aligned with the feature point, the eye is also not aligned with the viewpoint of the image pickup apparatus, and when the target in the target image obtained by the image pickup apparatus photographing the target through the screen is aligned with the feature point, the eye is also aligned with the viewpoint of the image pickup apparatus, and therefore, the transmissive display apparatus can adjust the position of the image pickup apparatus according to the target image obtained by the image pickup apparatus photographing the target through the screen.
For example, the transmissive display device may control the image capturing device to capture a first target image through a screen of the transmissive display device. The first target image may be acquired by a display device.
204. According to the positions of the target and the feature points in the first target image, a first adjustment amount and a first adjustment direction of the image pickup device are determined.
For example, the positions of the target and the feature point may include coordinates of the target and the feature point. The transmission type display device determines the distance between the target and the characteristic point according to the coordinates of the target and the characteristic point in the first target image. The transmissive display device may further determine a relative direction of the target and the feature point in the first target image, and determine a first adjustment amount and a first adjustment direction of the image capturing device according to the distance and the relative direction.
For example, a first mapping relationship between a distance and an adjustment amount, and a second mapping relationship between a relative direction and an adjustment direction may be preset and built in the transmissive display device. After obtaining the distance between the target and the feature point in the first target image and the relative direction between the target and the feature point, the transmissive display device may determine a first adjustment amount according to the distance and the first mapping relationship, and determine a first adjustment direction according to the relative direction and the second mapping relationship.
The first adjustment direction and the first adjustment amount are adjustment directions and adjustment amounts that enable the imaging apparatus after adjusting the position of the imaging apparatus according to the first adjustment direction and the first adjustment amount to align a target in a target image obtained by photographing the target through a screen with the feature point. The first adjustment may include a translational amount and/or a rotational amount, and the first adjustment direction may include a translational direction of an x, y, or z axis, or a rotational direction of a pitch, yaw, roll.
205. According to the first adjustment amount and the first adjustment direction, the position of the image pickup device is adjusted, so that the target in the target image obtained by shooting the target through the screen by the image pickup device is aligned with the characteristic point.
For example, after the first adjustment amount and the first adjustment direction are determined, the transmissive display device may control the adjustment device to adjust the position of the image capturing device according to the first adjustment amount and the first adjustment direction, so that the target in the target image obtained by capturing the target through the screen by the image capturing device is aligned with the feature point.
For example, assuming that the first adjustment amount is 5mm and the first adjustment direction is the x-axis direction, the transmissive display device may control the adjustment device to move the image pickup device 5mm in the x-axis direction so that the target in the target image obtained by photographing the target through the screen by the image pickup device is aligned with the feature point.
According to the embodiment of the application, the alignment of the camera and the human eye viewpoint can be realized by determining the projection position of the target at the human eye viewpoint corresponding to the screen, displaying the characteristic point according to the projection position, and adjusting the position of the camera so that the target in the target image obtained by shooting the target through the screen by the camera is aligned with the characteristic point, thereby improving the calibration accuracy of the camera and the screen by the camera.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a viewpoint alignment device according to an embodiment of the present application. The viewpoint alignment device 300 includes a determination module 301, a display module 302, and an adjustment module 303.
The determining module 301 is configured to determine a projection position of the target on the screen corresponding to the eye viewpoint;
A display module 302, configured to display the feature points according to the projection positions;
An adjusting module 303, configured to adjust a position of an image capturing device, so that a target in a target image obtained by the image capturing device capturing the target through the screen is aligned with a feature point.
In some embodiments, the adjusting module 303 may be configured to acquire a first target image obtained by the image capturing device capturing the target through the screen, and adjust a position of the image capturing device according to the first target image.
In some embodiments, the adjusting module 303 may be configured to determine a first adjustment amount and a first adjustment direction of the image capturing apparatus according to the positions of the target and the feature point in the first target image, and adjust the position of the image capturing apparatus according to the first adjustment amount and the first adjustment direction.
In some embodiments, the adjustment module 303 may be configured to receive an input adjustment amount and an adjustment direction and adjust a position of the image capturing apparatus according to the input adjustment amount and the adjustment direction.
In some embodiments, the viewpoint alignment device 300 may further include an output module, where the output module may be configured to output alignment prompt information when a target in a target image obtained by shooting the target through the screen by the image capturing device is aligned with a feature point after adjusting the position of the image capturing device according to the input adjustment amount and the adjustment direction.
In some embodiments, the output module may be configured to obtain a second target image obtained by the image capturing device capturing the target through the screen when the target and the feature point in the target image obtained by the image capturing device capturing the target through the screen are not aligned, determine a second adjustment amount and a second adjustment direction of the image capturing device according to the positions of the target and the feature point in the second target image, and output adjustment prompt information according to the second adjustment amount and the second adjustment direction.
In some embodiments, the targets are a plurality of, the plurality of targets being at different distances from the screen.
It should be noted that, the viewpoint alignment device provided in the embodiment of the present application and the viewpoint alignment method in the above embodiment belong to the same concept, and detailed implementation processes of the viewpoint alignment device are shown in the above embodiment, and are not repeated herein.
According to the viewpoint alignment device 200 provided by the embodiment of the application, the projection position of the target at the corresponding human eye viewpoint of the screen is determined through the determination module 301, the characteristic points are displayed through the display module 302 according to the projection position, and the position of the camera device is adjusted through the adjustment module 303, so that the target in the target image obtained by shooting the target through the screen by the camera device is aligned with the characteristic points, the alignment of the camera device and the human eye viewpoint can be realized, and the accuracy of calibrating the position relationship between the camera and the screen by the camera device can be improved.
An embodiment of the present application provides a storage medium having stored thereon a computer program which, when executed on a processor of an electronic device provided in an embodiment of the present application, causes the processor of the electronic device to perform any of the steps in the above method for aligning a viewpoint suitable for an electronic device. The storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.
The application also provides an electronic device, referring to fig. 6, the electronic device 400 includes a processor 401, a memory 402, and other components. It will be appreciated by those skilled in the art that the electronic device structure shown in fig. 6 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. For example, the electronic device 400 may also include a screen for displaying feature points according to projection positions, and so on.
The processor 401 in an embodiment of the present application may be a general purpose processor, such as an ARM architecture processor.
The memory 402 has stored therein a computer program, which may be a high speed random access memory, or may be a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device, etc. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402. The processor 401 is configured to execute, by executing a computer program in the memory 402:
determining the projection position of a target on a screen corresponding to a human eye viewpoint;
Displaying the characteristic points according to the projection positions;
And adjusting the position of the image pickup device so that the target in the target image obtained by shooting the target through the screen by the image pickup device is aligned with the characteristic point.
In some embodiments, before the processor 401 executes the adjusting the position of the image capturing apparatus, it may also be executed to acquire a first target image obtained by the image capturing apparatus capturing the target through the screen, and when the processor 401 executes the adjusting the position of the image capturing apparatus, it may be executed to adjust the position of the image capturing apparatus according to the first target image.
In some embodiments, when the processor 401 executes the adjustment of the position of the image capturing apparatus according to the first target image, the adjustment of the position of the image capturing apparatus according to the first adjustment amount and the first adjustment direction may be executed by determining the first adjustment amount and the first adjustment direction of the image capturing apparatus according to the positions of the targets and the feature points in the first target image.
In some embodiments, the processor 401 may perform the adjustment of the position of the image capturing apparatus by receiving an input adjustment amount and an adjustment direction and adjusting the position of the image capturing apparatus according to the input adjustment amount and adjustment direction.
In some embodiments, after the processor 401 executes the adjustment amount and the adjustment direction according to the input and adjusts the position of the image capturing apparatus, it may also execute to output an alignment prompt message when the target in the target image obtained by capturing the target through the screen by the image capturing apparatus is aligned with the feature point.
In some embodiments, after the processor 401 executes the adjustment of the position of the image capturing device according to the input adjustment amount and the adjustment direction, it may further execute the steps of acquiring a second target image obtained by the image capturing device capturing the target through the screen when the target and the feature point in the target image obtained by the image capturing device capturing the target through the screen are not aligned, determining a second adjustment amount and a second adjustment direction of the image capturing device according to the position of the target and the feature point in the second target image, and outputting adjustment prompt information according to the second adjustment amount and the second adjustment direction.
In some embodiments, the targets are a plurality of targets that are different distances from the screen.
The electronic equipment provided by the embodiment of the application can realize the alignment of the camera equipment and the eye point by determining the projection position of the target at the eye point corresponding to the screen, displaying the characteristic point according to the projection position, and adjusting the position of the camera equipment so that the target in the target image obtained by shooting the target through the screen by the camera equipment is aligned with the characteristic point, thereby improving the calibration accuracy of the position relationship between the camera and the screen by the camera equipment.
The above detailed description of the viewpoint alignment method, apparatus, storage medium and electronic device provided by the present application has been provided, and specific examples are provided herein to illustrate the principles and embodiments of the present application, and the above description of the examples is only for aiding in understanding the method and core concept of the present application, and meanwhile, for those skilled in the art, according to the concept of the present application, there are variations in the specific embodiments and application scope, so the disclosure should not be interpreted as limiting the application.
Claims (10)
1. A viewpoint alignment method, comprising:
Determining the projection position of a target obtained by observing the target through a screen by human eyes on a screen corresponding to the eye point of the human eyes;
displaying feature points at the projection positions, the feature points being points that can be photographed by an image pickup apparatus;
And adjusting the position of the image pickup device so that a target in a target image obtained by shooting the target through the screen by the image pickup device is aligned with a characteristic point in the target image.
2. The viewpoint alignment method according to claim 1, further comprising, before the adjusting the position of the image capturing apparatus:
Acquiring a first target image obtained by shooting the target through the screen by the camera equipment;
The adjusting the position of the image capturing apparatus includes:
and adjusting the position of the image pickup device according to the first target image.
3. The viewpoint alignment method according to claim 2, wherein the adjusting the position of the image capturing apparatus according to the first target image includes:
determining a first adjustment amount and a first adjustment direction of the camera device according to positions of targets and feature points in the first target image;
And adjusting the position of the image capturing apparatus according to the first adjustment amount and the first adjustment direction.
4. The viewpoint alignment method according to claim 1, wherein the adjusting the position of the image capturing apparatus includes:
Receiving an input adjustment amount and an input adjustment direction;
And adjusting the position of the image capturing apparatus according to the input adjustment amount and the adjustment direction.
5. The viewpoint alignment method according to claim 4, wherein after the adjusting the position of the image capturing apparatus according to the inputted adjustment amount and adjustment direction, further comprising:
When a target in a target image obtained by shooting the target through the screen by the camera equipment is aligned with the characteristic points, outputting alignment prompt information.
6. The viewpoint alignment method according to claim 4, wherein after the adjusting the position of the image capturing apparatus according to the inputted adjustment amount and adjustment direction, further comprising:
when a target in a target image obtained by shooting the target through the screen by the camera equipment is not aligned with the characteristic points, obtaining a second target image obtained by shooting the target through the screen by the camera equipment;
Determining a second adjustment amount and a second adjustment direction of the camera device according to the positions of the target and the feature points in the second target image;
and outputting adjustment prompt information according to the second adjustment amount and the second adjustment direction.
7. The viewpoint alignment method according to any one of claims 1 to 6, wherein the targets are plural, and the plural targets are different in distance from the screen.
8. A viewpoint alignment apparatus, comprising:
the determining module is used for determining the projection position of a target obtained by observing the target through a screen by human eyes on a screen corresponding to the human eye viewpoint;
A display module for displaying feature points at the projection positions, the feature points being points that can be photographed by an image pickup apparatus;
the adjusting module is used for adjusting the position of the camera equipment so that a target in a target image obtained by shooting the target through the screen by the camera equipment is aligned with a characteristic point in the target image.
9. A storage medium having stored therein a computer program which, when run on a computer, causes the computer to perform the viewpoint alignment method of any of claims 1 to 7.
10. An electronic device comprising a processor and a memory, the memory having stored therein a computer program, the processor being operable to perform the viewpoint alignment method of any of claims 1 to 7 by invoking the computer program stored in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210234161.1A CN114612557B (en) | 2022-03-10 | 2022-03-10 | Viewpoint alignment method, device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210234161.1A CN114612557B (en) | 2022-03-10 | 2022-03-10 | Viewpoint alignment method, device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114612557A CN114612557A (en) | 2022-06-10 |
| CN114612557B true CN114612557B (en) | 2024-12-31 |
Family
ID=81863232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210234161.1A Active CN114612557B (en) | 2022-03-10 | 2022-03-10 | Viewpoint alignment method, device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114612557B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782499A (en) * | 2019-10-23 | 2020-02-11 | Oppo广东移动通信有限公司 | A calibration method, calibration device and terminal device for augmented reality equipment |
| CN110874867A (en) * | 2018-09-03 | 2020-03-10 | 广东虚拟现实科技有限公司 | Display method, display device, terminal equipment and storage medium |
| CN113902796A (en) * | 2021-08-31 | 2022-01-07 | 杭州易现先进科技有限公司 | Calibration method and system of image-based head display equipment |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100416336C (en) * | 2003-06-12 | 2008-09-03 | 美国西门子医疗解决公司 | Calibrating real and virtual views |
| CN101452117B (en) * | 2004-11-24 | 2011-01-26 | 寇平公司 | Binocular display system and method thereof |
| US9494800B2 (en) * | 2014-01-21 | 2016-11-15 | Osterhout Group, Inc. | See-through computer display systems |
| CN108022306B (en) * | 2017-12-30 | 2021-09-21 | 华自科技股份有限公司 | Scene recognition method and device based on augmented reality, storage medium and equipment |
| WO2019136588A1 (en) * | 2018-01-09 | 2019-07-18 | 深圳前海达闼云端智能科技有限公司 | Cloud computing-based calibration method, device, electronic device, and computer program product |
| CN109801379B (en) * | 2019-01-21 | 2023-02-17 | 视辰信息科技(上海)有限公司 | Universal augmented reality glasses and calibration method thereof |
| GB2588774B (en) * | 2019-11-05 | 2024-05-15 | Arspectra Sarl | Augmented reality headset for medical imaging |
-
2022
- 2022-03-10 CN CN202210234161.1A patent/CN114612557B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110874867A (en) * | 2018-09-03 | 2020-03-10 | 广东虚拟现实科技有限公司 | Display method, display device, terminal equipment and storage medium |
| CN110782499A (en) * | 2019-10-23 | 2020-02-11 | Oppo广东移动通信有限公司 | A calibration method, calibration device and terminal device for augmented reality equipment |
| CN113902796A (en) * | 2021-08-31 | 2022-01-07 | 杭州易现先进科技有限公司 | Calibration method and system of image-based head display equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114612557A (en) | 2022-06-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11928838B2 (en) | Calibration system and method to align a 3D virtual scene and a 3D real world for a stereoscopic head-mounted display | |
| Grubert et al. | A survey of calibration methods for optical see-through head-mounted displays | |
| EP2966863B1 (en) | Hmd calibration with direct geometric modeling | |
| US11577159B2 (en) | Realistic virtual/augmented/mixed reality viewing and interactions | |
| JP6860488B2 (en) | Mixed reality system | |
| US7369101B2 (en) | Calibrating real and virtual views | |
| CN100416336C (en) | Calibrating real and virtual views | |
| Uchiyama et al. | MR Platform: A basic body on which mixed reality applications are built | |
| US20160267720A1 (en) | Pleasant and Realistic Virtual/Augmented/Mixed Reality Experience | |
| CN113252309B (en) | Testing method, testing device and storage medium for near-eye display equipment | |
| CN112166397B (en) | Apparatus, system and method for accelerating position tracking of head mounted displays | |
| JP2002268621A (en) | Image composing device | |
| CN113902796A (en) | Calibration method and system of image-based head display equipment | |
| Cutolo et al. | Optical see-through head-mounted displays with short focal distance: Conditions for mitigating parallax-related registration error | |
| CN103517061A (en) | Method and device for display control of terminal device | |
| JP2022501876A (en) | Reduced bandwidth stereo distortion compensation for fisheye lenses in head-mounted displays | |
| TWI501193B (en) | Computer graphics using AR technology. Image processing systems and methods | |
| CN114612557B (en) | Viewpoint alignment method, device, storage medium and electronic device | |
| CN113767272B (en) | Camera holder for economical and simplified test calibration | |
| TWI825982B (en) | Method for providing visual content, host, and computer readable storage medium | |
| CN116205980A (en) | Method and device for positioning and tracking virtual reality in mobile space | |
| CN116309854A (en) | Method, device, equipment, system and storage medium for calibrating augmented reality equipment | |
| CN118967796B (en) | Virtual display calibration method and device for extended reality device and extended reality device | |
| Hua et al. | Calibration of an HMPD-based augmented reality system | |
| JP2005122233A (en) | Information processor and method, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |