CN105812776A - Stereoscopic display system based on soft lens and method - Google Patents

Abstract

The invention belongs to the field of medical technology, and provides a stereoscopic display system and method based on a soft mirror. The system includes: a display unit, a light splitting unit, a tracking device, and an image capturing unit. The light splitting unit is used to display the image space of the display unit. The upper part is divided into a left view and a right view, the tracking device is used to obtain the position information of the first target object, the image capture unit is used to capture the second target object, the stereoscopic display system based on the soft mirror also includes an image playback processing device, according to The position information of the first target object, the grating parameters of the spectroscopic unit and the display parameters of the display unit are processed in real time on the received stereoscopic image captured by the image capture unit, and then sent to the display unit for real-time display after processing. Compared with the prior art, the image playback speed of the present invention is greatly improved, can meet the requirement of real-time stereoscopic display, and has the advantages of being convenient for doctors to operate and assisting doctors to improve the success rate of operations.

Description

Stereoscopic display system and method based on soft mirror

technical field

The invention relates to the technical field of medical equipment, in particular to a soft mirror-based stereoscopic display system and display method applied in clinical medicine.

Background technique

An endoscope is a tube equipped with a light that is passed through the mouth into the stomach or through other openings in the body. The endoscope can be used to see lesions that cannot be seen on X-rays, so it is very useful for doctors. For example, with the help of an endoscope, doctors can observe ulcers or tumors in the stomach and formulate the best treatment plan accordingly. Endoscope is an optical instrument, which is composed of cold light source lens, fiber optic wire, image transmission system, screen display system, etc. It can expand the surgical field of view. The prominent feature of using endoscope is that the surgical incision is small, the scar of the incision is not obvious, the postoperative reaction is mild, the time of bleeding, bruising and swelling can be greatly reduced, and the recovery is faster than traditional surgery, which is very in line with the requirements of beauty and no scars in cosmetic surgery.

Endoscopes are divided into hard and soft materials, called medical hard mirrors and soft mirrors. The first endoscopes were made of rigid tubes and were invented more than 100 years ago. Although they have gradually improved, they are still not widely used. Later, in the 1950s, endoscopes were made of flexible tubes that could easily bend around the corners of the body.

Medical flexible mirrors include gastroscopes, colonoscopes, duodenoscopes, bronchoscopes, nasopharyngoscopes, choledochoscopes, etc.

At present, a few existing medical endoscopes on the market have the function of displaying 3D images. However, it is limited to the auxiliary 3D display of wearing 3D glasses. The technology it uses is to make the left and right eyes inject different polarized light to generate parallax and form a three-dimensional perception. The disadvantage of this technology is that you need to wear polarizer glasses. On the one hand, the use of polarizer glasses reduces the light entering the doctor's eyes to less than half of the original, wasting the very precious light information in the cavity itself, and reducing the relatively low light intensity in the cavity. information recognition rate in dark environments; on the other hand, for doctors who do not wear glasses in daily life, wearing polarized glasses during surgery is very likely to cause discomfort, and wearing a mask at the same time is easy Fog is generated on the surface, which has a great impact on surgical safety.

Therefore, how to overcome the above-mentioned problems has become a major technical problem that the medical field is currently facing.

Contents of the invention

The purpose of the present invention is to provide a stereoscopic display system and display method based on a soft mirror, aiming to solve one or more of the above-mentioned technical problems caused by the limitations and shortcomings of the prior art.

A stereoscopic display system based on a soft mirror provided by the present invention includes: a display unit, a light splitting unit, a tracking device, and an image capture unit, the light splitting unit is located on the display side of the display unit, and is used to display the display unit The image space is divided into a left view and a right view, the tracking device is used to obtain the position information of the first target object, and the image capture unit is used to capture the second target object, wherein the stereoscopic display system based on the soft mirror It also includes an image playback processing device, which is respectively connected to the tracking device, the display unit, and the image capturing unit, and the image playback processing device is based on the position information of the first target object and the grating parameters of the spectroscopic unit and the display parameters of the display unit, process the received stereoscopic image captured by the image capturing unit in real time, and send the processed stereo image to the display unit for real-time display.

The present invention also provides a method for stereoscopic display based on a soft mirror, said method comprising the following steps: S0 taking a stereoscopic image of a second target object, and sending the information of said stereoscopic image in real time; S1 obtaining the position information of the first target object ; S2 acquires the grating parameters of the spectroscopic unit of the display device and the display parameters of the display unit of the display device; S3 captures the received image capturing unit according to the position information, the grating parameters and the display parameters The stereoscopic image is processed in real time; S4 displays the image to be played.

The stereoscopic display system and display method based on the soft mirror provided by the present invention greatly improves the image playback speed compared with the prior art, can meet the requirements of real-time stereoscopic display, and has the advantages of being convenient for doctors to operate and assisting doctors to improve the success rate of surgery.

Description of drawings

FIG. 1 shows a schematic structural diagram of a stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention.

FIG. 2 is a schematic structural diagram of a specific example of a stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention.

FIG. 3 shows a schematic structural diagram of the image playback processing unit in FIG. 2 .

FIG. 4 is a schematic structural diagram of bonding the light splitting unit and the display unit in the stereoscopic display system based on the soft mirror according to Embodiment 1 of the present invention.

FIG. 5 shows a schematic structural diagram of a preferred embodiment of a tracking device in a stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention.

FIG. 6 shows a schematic structural diagram of the acquisition unit in FIG. 4 .

FIG. 7 shows a schematic structural diagram of a first modified example of the reconstruction unit in FIG. 4 .

FIG. 8 shows a schematic structural diagram of a second modified example of the reconstruction unit in FIG. 4 .

FIG. 9 shows a schematic structural diagram of a third modified example of the reconstruction unit in FIG. 4 .

FIG. 10 shows a schematic structural diagram of a positioning bracket for setting a marker point corresponding to a first target object in the tracking device of FIG. 4 .

FIG. 11 is a schematic flowchart of a stereoscopic display method based on a soft mirror according to Embodiment 2 of the present invention.

FIG. 12 is a schematic flowchart of S1 in FIG. 11 .

FIG. 13 is a schematic flow chart of S12 in FIG. 12 .

FIG. 14 is a schematic flowchart of a first modified example of S13 in FIG. 11 .

FIG. 15 is a schematic flowchart of a second modified example of S13 in FIG. 11 .

FIG. 16 is a schematic flowchart of a third modified example of S13 in FIG. 11 .

FIG. 17 is a schematic flowchart of S3 in FIG. 11 .

detailed description

In order to understand the above-mentioned purpose, features and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described here. Therefore, the protection scope of the present invention is not limited by the specific details disclosed below. Implementation limitations.

Implementation Mode 1

Please refer to FIG. 1 . FIG. 1 is a schematic structural diagram of a stereoscopic display system based on a soft mirror according to the present invention. As shown in FIG. 1 , the stereoscopic display system based on the soft mirror of the present invention includes: an image capturing unit 10 , a tracking device 30 , a spectroscopic unit 50 and a display unit 40 . The image capture unit 10 is used to capture a second target object, and send the captured image of the second target object to the image playback unit in real time. The tracking device 30 is used to obtain the position information of the first target object. The light splitting unit 50 is located on the display side of the display unit 40 and is used to spatially divide the image displayed by the display unit 40 into a left view and a right view. The stereoscopic display system based on the soft mirror also includes an image playback processing device 20, which is respectively connected to the tracking device 30 and the display unit 40, and the image playback processing device 20 is based on the position information of the first target object, the The raster parameters and display parameters of the display unit 40 process the image to be played in real time, and send it to the display unit 40 for display after processing. In addition, the image capture unit 10 is set on the soft mirror, and when the soft mirror enters the body of a human or animal, it can capture images inside the human or animal, which is convenient for doctors to observe in real time and operate in time.

Since the tracking device 30 and the display unit 40 are directly connected to the image playback processing device 20, the image playback processing device 20 obtains the position information, raster parameters and display parameters of the first target object in time, and performs image processing accordingly, eliminating the need In the existing technology, the processing process of the central processing unit is required, so the speed of image playback is greatly improved compared with the existing technology, which can meet the requirements of real-time stereoscopic display, facilitate the doctor's operation and assist the doctor to improve the success rate of the operation. This is because the doctor can obtain accurate stereoscopic images in real time during the operation, and perform the operation in time without the problems mentioned in the background art. The grating parameters above mainly include the pitch of the grating, the inclination angle of the grating relative to the display panel, and the placement distance of the grating relative to the display panel. These grating parameters can be directly stored in the memory of the image playback processing device, or other detection devices can detect and acquire the grating parameters of the spectroscopic unit in real time, and send the raster parameter values to the image playback processing device 20 . The above parameters of the display unit include parameters such as size of the display unit, screen resolution of the display unit, arrangement sequence and arrangement structure of sub-pixels in the pixel unit of the display unit. The sub-pixel arrangement order means whether the sub-pixels are arranged in RGB or RBG, or in BGR, or in other orders; the sub-pixel arrangement structure is whether the sub-pixels are arranged vertically or horizontally, such as from top to bottom according to Arranged circularly in the way of RGB, or arranged circularly in the way of RGB from left to right.

The image capture unit 10 is used to capture a second target object, and send the captured image of the second target object to the image playback unit in real time. The second target object here mainly refers to various scenes shot and recorded by the camera, such as a scene of an operation, an internal image of a patient, and the like. The stereoscopic image is captured in real time by the image capture unit 10, and the captured stereoscopic image is displayed on the display unit in real time, without additional image processing, and various scenes captured are displayed in a timely and true manner, satisfying the user's demand for real-time display needs and improve the user experience. The image capture unit 10 may include at least one of a monocular camera, a binocular camera or a multi-camera.

When the image capturing unit 10 includes a monocular camera, the stereoscopic image of the second target object is captured and acquired according to the monocular camera. Preferably, the monocular camera may use a liquid crystal lens imaging device or a liquid crystal microlens array imaging device. In a specific embodiment, the monocular camera obtains two digital images of the measured object from different angles at different times, and restores the three-dimensional geometric information of the object based on the parallax principle, and reconstructs the three-dimensional outline and position of the object.

When the image capture unit 10 includes a binocular camera, it includes two cameras or one camera has two cameras, and the second target object is captured by the binocular camera to form a stereoscopic image. Specifically, the binocular camera is mainly based on the principle of parallax and obtains three-dimensional geometric information of objects from multiple images. The binocular stereo vision system generally obtains two digital images of the measured object (second target object) from different angles simultaneously by dual cameras, and restores the three-dimensional geometric information of the object based on the parallax principle, and reconstructs the three-dimensional outline and position of the object.

When the image capture unit 10 includes a multi-eye camera, that is, more than three (including three) cameras, these cameras are arranged in a matrix for acquiring stereoscopic images. Simultaneously acquire multiple digital images of the second target object from different angles by the above three or more cameras, restore the three-dimensional geometric information of the object based on the parallax principle, and reconstruct the three-dimensional outline and position of the object.

The image capture unit 10 also includes a capture unit, which is used to capture a stereo image of the second target object, and extract left view information and right view information from the stereo image. One end of the acquisition unit is connected to the above-mentioned monocular camera, binocular camera or the above-mentioned multi-eye camera, and the other end is connected to the image playback processing device 20 . The acquisition unit extracts the left view information and the right view information of the stereoscopic image while shooting the stereoscopic image, which improves the speed of image processing and ensures the display effect of real-time stereoscopic display.

The above-mentioned tracking device 30 may be a camera and/or an infrared sensor, and is mainly used to track the position of the first target object, such as the position of the eyes of the person or the head or the face of the person or the position of the upper body of the person. The number of cameras or infrared sensors is not limited, and may be one or more. A camera or an infrared sensor can be mounted on the frame of the display unit, or placed separately at a position where it is easy to track the first target object. In addition, if an infrared sensor is used as the tracking device, an infrared transmitter can also be installed at the position corresponding to the first target object, and by receiving the infrared positioning signal sent by the infrared transmitter, the relative positional relationship between the infrared transmitter and the first target object can be used , to calculate the location information of the first target object.

Specifically, the tracking device 30 includes a camera, and the camera shoots the first target object. The number of cameras can be one or more, and can be set on the display unit or set separately.

The tracking device 30 includes an infrared receiver. Correspondingly, an infrared emitter is arranged corresponding to the first target object. On the fixed object, the infrared receiver receives the infrared signal sent from the infrared emitter corresponding to the first target object. The positioning of the first target object is realized by a common infrared positioning method.

In addition, the tracking device 30 may also use a GPS positioning module, and the GPS positioning module sends positioning information to the image playback processing device 20 .

The above-mentioned spectroscopic unit 50 is arranged on the light-emitting side of the display unit 40, and sends the left view and right view with parallax displayed by the display unit 40 to the left and right eyes of the person respectively, and synthesizes a stereoscopic image according to the left and right eyes of the person, Make people watch the effect of stereoscopic display. Preferably, the light splitting unit may be a parallax barrier or a lens grating. The parallax barrier can be a liquid crystal slit or a solid slit grating sheet or an electrochromic slit grating sheet, and the lens grating can be a liquid crystal lens or a solid liquid crystal lens grating. The solid liquid crystal lens grating mainly solidifies the liquid crystal on the sheet by ultraviolet light to form a solid lens, which splits the light and emits it to the left and right eyes of the person. Preferably, the above-mentioned display unit 40 and the spectroscopic unit 50 are used as an integrated display device 60, which is the display part of the entire soft mirror-based stereoscopic display system, and can be assembled with the aforementioned image playback processing device and tracking device. together, or as an independent part. For example, according to viewing needs, the display device 60 can be placed separately at a convenient viewing position, while the image playback processing device 20 and the tracking device 30 can be devices with independent functions. Stereoscopic display function can be. For example, the image playback processing device 20 may be a VMR3D playback device, which itself has a 3D playback processing function, and is used to assemble it into the stereoscopic display system based on the soft mirror of the present invention, and establish a connection with other devices.

Please refer to FIG. 2 . FIG. 2 is a schematic structural diagram of a specific example of a stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention. As shown in Figure 2, in the stereoscopic display system based on the soft mirror in Embodiment 1 of the present invention, the image playback processing device 20 further includes: an image playback processing unit 22 and a storage unit 23, and the image playback processing unit 22 is mainly based on the first The location information of the target object, the grating parameters of the spectroscopic unit and the display parameters of the display unit are used to perform real-time layout processing on the received stereoscopic images, and then send the processed images to the display unit for real-time display. The storage unit 23 is used for storing images transmitted from the image capture unit 10 . When it is necessary to play a stereoscopic image, the image playback processing unit 22 invokes the stereoscopic image stored in the storage unit 23 to perform image layout processing.

Further, the image playback processing device 20 further includes: a signal processing unit 21, the signal processing unit 21 is respectively connected to the storage unit 23 and the image playback processing unit 22, wherein the signal processing unit 21 is mainly a unit for capturing images received 10 to process the signal of the stereoscopic image taken, including image format conversion, image compression and other processing. The compressed image is stored in the storage unit 23 . The signals processed by the signal processing unit 21 can output image signals corresponding to left view and right view information respectively, or output them to the image playback processing unit 22 together. The image playback processing unit 22 performs real-time layout processing on the received stereoscopic image processed by the signal processing unit 21 according to the position information of the first target object and the grating parameters of the spectroscopic unit, and sends the processed image to the display unit real-time display. Here, the image playback processing unit 22 can directly call and decompress the stereoscopic images in the storage unit 23, or directly receive the stereoscopic images processed by the signal processing unit 21, and then perform image layout processing.

Referring to Fig. 3, the image playback processing unit 22 further includes:

The layout parameter determination module 201 calculates the layout parameters on the display unit according to the acquired position information of the first target object, the grating parameters of the spectroscopic unit and the display parameters of the display sheet;

The parallax image arrangement module 202 is configured to arrange the parallax images on the display unit according to the layout parameters; the parallax images are generated by spatially dividing the left-eye image and the right-eye image.

The parallax image playing module 203 plays the parallax image. After receiving the arranged parallax images, they are played, and the viewer sees the displayed stereoscopic images in real time on the display unit.

Further, the image playback processing unit 22 also includes: a stereoscopic image acquisition module 204, which acquires the stereoscopic image information captured by the image capturing unit 10, that is, the left view and right view information of the stereoscopic image. A stereoscopic image includes a left view and a right view. Therefore, for a stereoscopic image to be played, the image information of the left view and the right view needs to be obtained before image layout processing can be performed.

Please continue to refer to FIG. 2 , the tracking device 30 in the present invention further includes a tracking and positioning processing unit 31 and a tracking unit 32 . The tracking unit 32 is used to track the real-time image of the first target object, which mainly refers to a type of equipment such as a camera and an infrared receiver that can accurately capture human eye or head video signals. The tracking and positioning processing unit 31 calculates the spatial coordinates of the first target object by extracting feature points of the first target object according to the real-time image of the first target object tracked by the tracking unit 32 . Specifically, for example, a camera is used to record a face in real time, and the tracking and positioning processing unit 31 performs feature point extraction on the face to calculate the spatial coordinates of the human eye; point, such as an infrared emitting device, and then capture a real-time image of the feature point through a camera, and the tracking and positioning processing unit 31 finally calculates the spatial coordinates of the human eye.

In addition, the tracking and positioning processing unit 31 can quickly track the position of the human eye in real time when the position of the human eye moves, give the spatial coordinates of the human eye, and provide the coordinate information to the image playback processing unit 22 .

In addition, the above-mentioned tracking unit 32 may include a camera or an infrared receiver. When the tracking unit 32 includes a camera, the camera tracks the position change of the feature point corresponding to the first target object. When the tracking unit 32 includes an infrared receiver, the infrared receiver receives an infrared positioning signal sent from an infrared transmitter corresponding to the first target object and serving as the feature point.

Through the tracking and positioning processing unit 31 and the tracking unit 32, the viewing effect of the stereoscopic display is improved, the stereoscopic display is automatically adjusted with the movement of the person, and the optimal stereoscopic display effect is provided in real time.

Example 1

In Embodiment 1 of the present invention, in order to obtain a better real-time stereoscopic display effect, it is necessary to carry out optical design on the light splitting unit and the display unit according to the grating parameters of the light splitting unit and the display parameters of the display unit, and the optical design is based on the following formula:

$<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; IPD</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}$

$<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; pitch</span>}}{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; pitch</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}}$

(3) m*t=p-pitch

In the above formula, F is the distance between the spectroscopic unit and the display unit (that is, the placement distance of the grating in the above grating parameters relative to the display panel), L is the distance between the viewer and the display unit, and IPD is the matching pupil distance. The distance between the pupils, for example, generally takes a value of 62.5mm, l-pitch is the pitch (pitch) of the light splitting unit, p-pitch is the layout pitch of the pixels on the display unit, n is the number of stereoscopic views, m is the number of pixels covered by the light-splitting unit, and p is the dot pitch of the display unit. The dot pitch here mainly refers to the size of a pixel unit (a kind of display parameter), and the pixel unit usually includes R, G, B three sub-pixels. In order to eliminate moiré, the light splitting unit is generally rotated at a certain angle when it is attached (that is, the light splitting unit has a certain inclination angle compared to the display unit). Therefore, the actual pitch of the light splitting unit is given by the following formula:

(4) W _lens = l-pitch*sinθ

Wherein, W _lens is the actual pitch of the light splitting unit, and θ is the inclination angle of the light splitting unit relative to the display panel (that is, one of the above grating parameters).

As mentioned above, for the distance F between the light splitting unit and the display unit, when the medium between the display unit and the light splitting unit is air, F is equal to the actual distance between the light splitting unit and the display unit; when the display unit and the light splitting unit When the medium in between is a transparent medium with a refractive index n (n is greater than 1), F is equal to the actual distance between the spectroscopic unit and the display unit divided by the refractive index n; when there are different media between the display unit and the spectroscopic unit , and the refractive index of the medium is n1, n2, n3... (the refractive index is greater than or equal to 1), F=s1/n1+s2/n2+s3/n3..., where s1, s2, s3 ··· is the thickness of the corresponding medium.

By setting the spectroscopic unit and the display unit through the above optical calculation formula, the moiré pattern can be reduced, and the stereoscopic display effect of real-time viewing can be improved.

In addition, in a modified embodiment, a bonding unit is provided between the light splitting unit and the display unit, please refer to FIG. 4 , which shows the relationship between the light splitting unit and the display unit in the stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention. Schematic diagram of the fitting structure. As shown in FIG. 4, a bonding unit is provided between the light splitting unit 50 and the display unit 40, and the three are similar to a "sandwich structure". The bonding unit includes a first substrate 42 and a second substrate 43, and a 42 and the air layer 41 between the second substrate 43. The air layer 41 is in a sealed state between the first substrate 42 and the second substrate 43 to prevent air from escaping. The first substrate 42 is bonded to the display panel, and may be made of a transparent glass material, or a transparent resin material or the like. The second substrate 43 is disposed opposite to the first substrate 42 , and the side of the second substrate 43 facing away from the first substrate 42 is used for attaching the spectroscopic unit 50 . Since the bonding unit is arranged between the light splitting unit 50 and the display unit 40, and the bonding unit adopts the above-mentioned structure, for a large-screen stereoscopic display device, the flatness of grating bonding is guaranteed, and the weight of the entire stereoscopic display device is reduced. Weight, to prevent the risk of screen cracking due to excessive weight when pure glass is used.

Example 2

In Embodiment 2, the tracking device 30 includes a camera, and the camera shoots the first target object. The number of cameras can be one or more, and can be set on the display unit or set separately. Also, the camera may be a monocular camera, a binocular camera or a multi-eye camera.

In addition, the tracking device 30 may also include an infrared receiver. Correspondingly, an infrared emitter is provided corresponding to the first target object. On an object with a relatively fixed position on a target object, the infrared receiver receives an infrared signal sent from an infrared emitter corresponding to the first target object. The positioning of the first target object is realized by a common infrared positioning method.

In addition, the above-mentioned tracking device 30 may also use a GPS positioning module, and the GPS positioning module sends positioning information to the image playback processing unit 20 .

Example 3

Please refer to FIG. 5 , which shows a schematic structural diagram of a preferred embodiment of a tracking device in a stereoscopic display system based on a soft mirror according to Embodiment 1 of the present invention. As shown in Figure 5, Embodiment 3 of the present invention also proposes another tracking device 30, which includes:

Marking point setting unit 1 is used to set the marking point corresponding to the spatial position of the first target object; the marking point here can be set on the first target object, or can not be set on the first target object, but be set on the first target object. The target object has a relative positional relationship, and it can also be on an object that moves synchronously with the first target object. For example, if the first target object is the human eye, marking points can be set around the eye sockets of the human eye; On the ear of a person whose positional relationship is relatively fixed. The marking point can be various components such as infrared emission sensors that send signals, LED lights, GPS sensors, laser positioning sensors, etc., or other physical signs that can be captured by the camera, such as objects with shape characteristics and/or color characteristics . Preferably, in order to avoid the interference of external stray light and improve the robustness of marker point tracking, it is preferable to use infrared LED lights with a relatively narrow spectrum as marker points, and use a corresponding infrared camera that can only pass through the spectrum used by infrared LEDs to Mark points to snap to. Considering that the external stray light is mostly irregular in shape and the brightness distribution is uneven, the marking points can be set to emit regular-shaped light spots, with high luminous intensity and uniform brightness. In addition, multiple marking points can be set, each marking point corresponds to a light spot, and each marking point forms a regular geometric shape, such as a triangle, a quadrilateral, etc., so that it is easy to track the marking point, obtain the spatial position information of the marking point, and improve the light spot. Extraction accuracy.

Acquisition unit 2 is used to obtain the position information of the marking point; this can be by receiving the signal sent by the marking point to determine the position information of the marking point, and it can also be to use a camera to shoot an image containing the marking point, and to determine the position information of the marking point in the image mark points for extraction. The location information of the marked points is obtained through image processing algorithms.

The reconstructing unit 3 is configured to reconstruct the spatial position of the first target object according to the position information of the marker point. After obtaining the position information of the mark point, the spatial position of the mark point is reconstructed, and then according to the relative positional relationship between the mark point and the first target object, the spatial position of the mark point is converted to the spatial position of the first target object ( For example, the spatial position of the left and right eyes of a person).

The tracking device 30 in the embodiment of the present invention reconstructs the spatial position of the first target object by acquiring the position information of the marker point corresponding to the first target object and according to the position information. Compared with the existing technology that uses a camera as a human eye capture device and needs to perform feature analysis on a two-dimensional image to obtain the position of the human eye or use other human eye capture devices that use the reflection effect of the iris of the human eye to obtain the position of the human eye, it has good stability. , has the advantages of high accuracy, low cost, and no requirement on the distance between the tracking device and the first target object.

Please refer to FIG. 6 , which shows a schematic structural diagram of the acquisition unit in FIG. 5 . The aforementioned acquisition unit further includes:

The preset module 21 is used to preset a standard image, the reference mark point is set in the standard image, and the spatial coordinates and plane coordinates of the reference mark point are obtained; the standard image can be, for example, a collected by an image acquisition device Standard image, obtain the image coordinates of the reference mark points, and use other accurate stereo measurement equipment such as laser scanners, structured light scanners (such as Kinect, etc.) to obtain the spatial coordinates and plane coordinates of the reference mark points in the standard image.

An acquisition module 22, configured to acquire a current image including the first target object and the marker point, and the plane coordinates of the marker point in the current image;

A matching module 23, configured to match the marker points in the current image with the reference marker points in the standard image. Here, it is first necessary to establish a corresponding relationship between the plane coordinates of the marker points in the current image and the plane coordinates of the reference marker points in the standard image, and then match the marker points with the reference marker points.

By setting the standard image and the reference mark point, it is convenient to have a reference object when acquiring the spatial position of the current image, which further ensures the stability and accuracy of the target tracking device in the embodiment of the present invention.

Further, the tracking device 30 also includes:

a collection unit, configured to collect the marker points;

The screening unit is configured to screen target markers from the markers.

Specifically, when there are multiple marker points, a camera is used to collect all marker points corresponding to the first target object, and the marker point most relevant to the first target object is selected from all marker points, and then corresponding image processing is used to The algorithm extracts the marked points on the image, and the extraction needs to be carried out according to the characteristics of the marked points. Generally speaking, the method of extracting the features of the marked points is to use the feature extraction function H on the image I to obtain the feature score of each point in the image, and to filter out the marked points with sufficiently high feature values. Here it can be summed up by the following formula:

S(x,y)=H(I(x,y))

F={arg _{(x, y)} (S(x, y)>s0)}

In the above formula, H is the feature extraction function, I(x, y) is the image value corresponding to each pixel (x, y), which can be the gray value or the color energy value of the three channels, etc., S(x, y ) is the feature score of each pixel (x, y) after feature extraction, s0 is a feature score threshold, S(x, y) greater than s0 can be considered as a marker point, and F is a set of marker points. Preferably, the embodiment of the present invention uses infrared marker points and the energy characteristics of the image formed by the infrared camera are relatively obvious. Due to the use of narrow-band LED infrared lamps and corresponding infrared cameras, most of the pixels of the image formed by the camera have low energy, and only the pixels corresponding to the marking points have high energy. Therefore, the corresponding function H(x, y) can perform region growth on the image B(x, y) after using the threshold segmentation operator to obtain several sub-images, and extract the center of gravity of the obtained sub-images. At the same time, according to the stray light that can be imaged in the infrared camera in the ambient light, we can add constraints such as the area of the spot formed by the marker point and the positional relationship of the marker point in the two-dimensional image to the extraction process of the infrared marker point. mark points for filtering.

When the number of cameras is greater than 1, it is necessary to perform marker matching on images acquired by different cameras at or near the same time, so as to provide conditions for subsequent 3D reconstruction of markers. The method of mark point matching needs to be determined according to the feature extraction function H. We can use some classic feature point extraction operators based on image grayscale gradient maps and matching matching methods such as Harris, SIFT, FAST and other methods to obtain and match marker points. You can also use limit constraints, prior conditions for markers, etc. to match markers. The method of matching and screening using limit constraints here is: according to the principle that the projections of the same point on two different camera images are on the same plane, for a certain marker point p0 in a certain camera c0, we can use it in other cameras An epipolar line equation is calculated in c1, and the marker point p0 corresponds to the marker point p1 on the other camera c1 in accordance with the following relationship:

[p1;1] ^TF [p0;1]=0

In the above formula, F is the fundamental matrix of camera c0 to camera c1. By using the above relationship, we can greatly reduce the number of candidates for the marker point p1 and improve the matching accuracy.

In addition, we can use the prior conditions of the markers to be the spatial order of the markers, the dimensions of the markers, etc. For example, according to the mutual positional relationship of the two cameras, each pair of two pixels corresponding to the same spatial point on the captured image is equal in a certain dimension such as the y-axis. This process is also called image rectification. At this time, the matching of the marked points can also be performed according to the x-axis order of the marked points, that is, the smallest x corresponds to the smallest x, and so on, and the largest x corresponds to the largest x.

The object tracking device of the present invention will be introduced in detail below according to the number of cameras used for tracking.

Please refer to FIG. 7 , which shows a schematic structural diagram of the reconstruction unit in FIG. 5 . As shown in FIG. 7 , in this embodiment, the first target object tracked by the tracking device 30 corresponds to no more than four marker points, and when a monocular camera is used to obtain the position information of the marker points, the reconstruction unit further includes :

The first calculation module 31 is configured to calculate the current position according to the plane coordinates of the marker points in the current image, the plane coordinates of the reference marker points in the standard image, and the assumed conditions of the scene where the first target object is located. The homography transformation relationship between the image and the standard image; match the marker points of the current image with the reference marker points in the standard image, and calculate the homography between the current image and the standard image according to their respective plane coordinates transform relationship. The so-called homography transformation refers to the homography in corresponding geometry, and it is a transformation method often used in the field of computer vision.

The first reconstruction module 32 is used to calculate the rigid transformation from the spatial position of the marker point at the moment when the standard image is taken to the spatial position at the current moment according to the homography transformation relationship, and then calculate the rigid transformation of the marker point at the current moment The spatial position at the moment, and calculate the current spatial position of the first target object according to the spatial position of the marker point at the current moment.

Specifically, for the assumed conditions of the scene, we can assume that the value of a certain dimension does not change during the rigid transformation of the marked points in the scene. For example, in a three-dimensional space scene, the spatial coordinates are x, y, z, and x and y are respectively It is parallel to the x-axis and y-axis in the image coordinates (plane coordinates) of the camera, and the z-axis is perpendicular to the image of the camera. The assumption can be that the coordinates of the marked point on the z-axis are unchanged, or that the marked point is on the x-axis and /or the coordinates on the y-axis are unchanged. Different scenario assumptions require different inference methods. For another example, under another assumption, assuming that the rotation angle between the orientation of the first target object and the orientation of the camera remains constant during use, the distances and The ratio of the distances between the marker points on the standard image estimates the current spatial position of the first target object.

Through the above calculation method, the spatial position of the first target object can be reconstructed when the number of marker points of the monocular camera is not more than four, the operation is simple, and the tracking result is also relatively accurate. reduces the cost of tracking the first target object.

In the above-mentioned method of recovering the three-dimensional coordinates of an object by collecting images with a single camera, since the acquired image information is less, it is necessary to increase the number of marker points to provide more image information to calculate the three-dimensional coordinates of the object. According to machine vision theory, to deduce the stereoscopic information of the scene from a single image, at least five calibration points in the image need to be determined. Therefore, the monocular solution increases the number of marking points and the complexity of the design, but at the same time, only one camera is needed, which reduces the complexity of image acquisition and reduces the cost.

Please refer to FIG. 8 , which shows a schematic structural diagram of a second modified embodiment of the reconstruction unit in FIG. 5 . As shown in FIG. 7, in this embodiment, when the number of the marked points is more than five, and a monocular camera is used to obtain the position information of the marked points, the reconstruction unit further includes:

The second calculation module 33 is used to calculate the homography between the current image and the standard image according to the plane coordinates of the marker points in the current image and the plane coordinates of the reference marker points in the standard image transform relationship.

The second reconstruction module 34 is used to calculate the rigid transformation from the spatial position of the marker point at the moment when the standard image is taken to the spatial position at the current moment according to the homography transformation relationship, and then calculate the rigid transformation of the marker point at the current moment The spatial position at the moment, and calculate the current spatial position of the first target object according to the spatial position of the marker point at the current moment.

First, a standard image is collected, and the spatial position of the reference mark point is measured using an accurate depth camera or laser scanner, and the two-dimensional image coordinates (ie, plane coordinates) of the reference mark point at this time are obtained.

During use, the camera continuously captures the two-dimensional image coordinates of all marker points in the current image containing the first target object, and calculates the current state based on the two-dimensional image coordinates and the two-dimensional coordinates of the standard image reference marker points at this time. The rigid transformation between the marked point and the marked point when shooting the standard image, assuming that the relative position between the marked points remains unchanged, and then calculate the spatial position transformation of the marked point relative to the standard image at this time, so that Calculate the spatial position of the current marker.

Here, the space position rigid transformation [R|T] between the current marker point and the marker point when shooting the standard image can be calculated by using more than five points. Preferably, the five or more points are not on the same plane, and the camera’s The projection matrix P is calibrated in advance. The specific way to calculate [R|T] is as follows:

The homogeneous coordinates of each marked point in the standard image and the current image are X0, Xi respectively. Both satisfy the limit constraint, that is, X ₀ P ^-1 [R|T]P=X _i . All marked points form a system of equations whose unknown parameter is [R|T]. When the number of marked points is greater than 5, [R|T] can be solved; when the number of marked points is greater than 6, the optimal solution can be found for [R|T], and the method can use singular value decomposition SVD, and/or use The iterative method computes the nonlinear optimal solution. After calculating the spatial position of the marker point, we can infer the spatial position of the first target object (eg human eye) according to the mutual positional relationship between the pre-marked marker point and the first target object (eg human eye).

In this embodiment, only one camera and five or more marking points can be used to accurately construct the spatial position of the first target object, which is not only easy to operate, but also low in cost.

Please refer to FIG. 9 , which shows a schematic structural diagram of a third modified embodiment of the reconstruction unit in FIG. 5 . As shown in FIG. 9 , this embodiment uses two or more cameras and one or more marking points. When using a binocular camera or a multi-eye camera to obtain the position information of the marker point, the reconstruction unit further includes:

The third calculation module 35 uses binocular or multi-eye three-dimensional reconstruction principles to calculate the spatial position of each marker point at the current moment; the so-called binocular or tri-eye reconstruction principles can use the following methods, for example, using the marks matched by the left and right cameras The parallax between points calculates the spatial position of each marker point at the current moment. Or use other existing common methods to achieve.

The third reconstruction module 36 calculates the current spatial position of the first target object according to the current spatial position of the marker point.

Specifically, firstly, a multi-camera calibration method is used to calibrate the mutual positional relationship between each camera. Then in the process of use, extract the coordinates of the marker points from the images acquired by each camera, and match each marker point, that is, obtain its corresponding marker point in each camera, and then use the matched marker points and between cameras The mutual positional relationship calculates the spatial position of the marker points.

In a specific example, multi-cameras (that is, the number of cameras is greater than or equal to 2) are used to shoot marked points to achieve stereoscopic reconstruction. Knowing the coordinate u of a marker point on the image captured by a certain camera and the parameter matrix M of the camera, we can calculate a ray, and the marker point is located on this ray in space.

α ^j u ^j = M ^j Xj = 1...n (where n is a natural number greater than or equal to 2)

Similarly, according to the above formula, the marking point can also calculate the ray corresponding to the other camera on other cameras. Theoretically, these two rays converge on a point, which is the spatial position of this marked point. In fact, due to the digitization error of the camera, the calibration error of the internal and external parameters of the camera, etc., these rays cannot converge at one point, so it is necessary to use the method of triangulation to approximate the spatial position of the marker point. For example, the least square criterion can be used to determine the point closest to all rays as the object point.

${\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}}\mathrm{<span\; class="notranslate">\; min</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>$

After calculating the spatial position of the marker point, we can infer the spatial position of the first target object (human eye) according to the mutual positional relationship between the pre-marked marker point and the first target object (such as the human eye).

Among the above methods for realizing stereo reconstruction using multi-cameras, a better method is to use binocular cameras for calculation. The principle is the same as the aforementioned multi-camera reconstruction principle, which is to calculate the spatial position of the marker point based on the mutual positional relationship between the two cameras and the two-dimensional coordinates of the marker point imaged by the two cameras. The slight difference is that the binocular cameras are placed in parallel, and the images of the two cameras are calibrated as described above after simple calibration, so that the two matching two-dimensional marker points are equal on the y (or x) axis , then the depth from the marker point to the camera can be calculated from the distance between the calibrated two-dimensional marker points on the x (or y) axis. This method can be regarded as a special method for multi-eye stereo reconstruction in the case of binoculars, which simplifies the steps of stereo reconstruction and is easier to implement on the device hardware.

Example 4

Please refer to FIG. 10 . FIG. 10 shows a schematic structural diagram of a positioning bracket for setting a marker point corresponding to a first target object in the tracking device in FIG. 5 . As shown in Figure 10, the present invention provides a positioning bracket, the positioning bracket is located in front of the human eye (the first target object), the structure is similar to glasses, and its wearing is similar to glasses, including: a beam 11, a fixing part 12, a supporting part 13 and the control part 14, the beam 11 is provided with a marking point 111; the support part 13 is provided on the beam 11; the fixing part 12 is pivotally connected to the end of the beam 11. The position of the marking point 111 corresponds to the position of the human eye (the first target object), and the spatial position information of the human eye is calculated by acquiring the spatial position information of the marking point 111 . When the head of the person moves, correspondingly, the marker point 111 corresponding to the human eye also moves, the camera tracks the movement of the marker point 111, and then uses the target object tracking method of the first embodiment to obtain the image of the marker point 111 The spatial location information uses the relative spatial location relationship between the marker point 111 and the human eye to reconstruct the spatial location (ie, the three-dimensional coordinates in space) of the human eye (the first target object).

In this embodiment, the crossbeam 11 is long and has a certain curvature, which is similar to the curvature of the forehead of a person, so that it is convenient to use. The beam 11 includes an upper surface 112 , a lower surface opposite thereto, a first surface 114 and a second surface disposed between the upper surface 112 and the lower surface.

In this embodiment, the marking points 111 are three LED lights, which are evenly spaced on the first surface 114 of the beam 11 . It can be understood that there may be one, two or more marking points 111, and may be any light source, including LED lamps, infrared lamps, or ultraviolet lamps. Moreover, the arrangement and setting position of the marking points 111 can also be adjusted as required.

It can be understood that the beam 11 can also be designed as a straight line or other shapes as required.

In this embodiment, there are two fixing parts 12, which are respectively pivotally connected to the two ends of the beam 11, and the two fixing parts 12 can be relatively inwardly folded, and at the same time, the two fixing parts 12 can be respectively expanded to the The beam 11 has an inner angle of about 100°, specifically, the size of the inner angle can be adjusted according to actual operation requirements. It can be understood that there may also be one fixing part 12 .

An end of the fixing part 12 away from the beam 11 is bent along the extending direction of the supporting part 13 for fixing the end of the fixing part 12 on the ear of a person.

In this embodiment, the supporting portion 13 is in the shape of a strip, disposed in the middle of the lower surface 113 of the beam 11 and extending downward. Further, a nose pad 131 is provided at the end of the supporting part 13 far away from the cross beam 11, for fitting the positioning device with the bridge of the nose, and setting the positioning device above the human eyes. It can be understood that, in other embodiments, if the nose pad 131 is not provided, the support part 13 can be arranged in an inverted "Y" shape and extend downward along the middle of the beam 11 to match the positioning device with the bridge of the nose, and The positioning device is arranged above the human eyes.

The control part 14 is in the form of a rectangular parallelepiped with rounded corners and is disposed on the fixing part 12 . The control unit 14 is used to provide power to the LED lamp, infrared lamp or ultraviolet lamp, and/or control the use state of the LED lamp, infrared lamp or ultraviolet lamp, which includes a power switch 141, a power indicator light and a charging indicator light . It can be understood that the shape of the control unit 14 is not limited, it can be in any shape, and can also be an integrated chip. Moreover, the control part 14 can also be arranged at other positions, such as on the beam 11 .

During use, turn on the power switch 141, the power indicator shows that the LED is in the power supply state, and the LED light is lit; The LED light turns off.

Since the interpupillary distance ranges from 58mm to 64mm, it can be approximately considered as a fixed value. The positioning bracket provided by the present invention is similar to a spectacle frame, and is fixed above the human eye, similar to a spectacle frame. The point is set at the predetermined position of the positioning device, so that the position of the human eye can be determined simply and conveniently according to the position of the marked point. The positioning device is simple in structure and convenient in design and use.

Implementation Mode Two

Please refer to FIG. 11 to FIG. 14 , FIG. 11 is a schematic flowchart of a soft mirror-based stereoscopic display method according to Embodiment 2 of the present invention, FIG. 12 is a specific flowchart of S1 in FIG. 11 , and FIG. 13 is a specific flowchart of S12 in FIG. 12 Schematic diagram, FIG. 14 is a specific flowchart of S3 in FIG. 11 . As shown in Fig. 11 to Fig. 14, the stereoscopic display method based on the soft mirror according to the second embodiment of the present invention mainly includes the following steps:

S0 image capturing step, capturing a stereoscopic image of the second target object, and sending information of the captured stereoscopic image of the second target object in real time, the information including left view information and right view information.

S1 acquires location information of a first target object; uses a tracking device to track the location of the first target object, such as location information of a viewer.

S2 acquires the grating parameters of the light-splitting unit of the stereoscopic display device and the display parameters of the display unit; the grating parameters of the light-splitting unit mainly include the grating pitch (pitch) of the grating, the inclination angle of the grating relative to the display panel, and the placement distance of the grating relative to the display panel. .

S3 Perform real-time processing on the received stereoscopic image captured by the image capturing unit according to the position information, the grating parameter and the display parameter. Before the stereoscopic image is to be played, it is necessary to pre-process the image in combination with the position information of human eyes, the grating parameters and the display parameters of the display unit, so as to provide the best stereoscopic display effect to the viewer.

S4 displays the image to be played.

The three-dimensional display method based on the soft mirror of the present invention obtains the position information and grating parameters of the first target object in time, and directly performs image processing accordingly, improves the speed of image playback, can meet the requirements of real-time three-dimensional display, and is convenient The advantages of the doctor's surgical operation and assisting the doctor to improve the success rate of the operation.

In addition, the second target object here mainly refers to various scenes captured by the camera, which may be actual people, or a live broadcast of a football game, or an image of a patient's body captured by some equipment. By shooting stereoscopic images in real time and displaying the captured stereoscopic images on the display unit in real time, without additional image processing, various scenes captured can be displayed in a timely and true manner, which meets the needs of users for real-time display and improves user experience.

In a specific variant embodiment, the above step S0 further includes: an image acquisition step, acquiring a stereoscopic image of the second target object, and extracting left view information and right view information from the stereoscopic image. By extracting the left view information and the right view information of the stereoscopic image while shooting the stereoscopic image, the image processing speed is improved, and the display effect of real-time stereoscopic display is guaranteed.

Example 5

Referring to FIG. 12 , Embodiment 5 of the present invention mainly describes in detail how S1 obtains the location information of the first target object. These first target objects are, for example, parts related to human viewing, such as human eyes, human head, human face, or upper body of human body. The above-mentioned "S1 obtaining the location information of the first target object" mainly includes the following steps:

S11 setting a mark point corresponding to the spatial position of the first target object; the mark point here can be set on the first target object, or not set on the first target object, but set in a relative position relationship with the first target object, And also on an object that moves synchronously with the first target object. For example, if the target object is the human eye, marking points can be set around the orbit of the human eye; or a positioning bracket is arranged around the human eye, and the marking point is set on the frame of the positioning bracket; On the ear of a person whose positional relationship is relatively fixed. The marking point can be various components such as infrared emission sensors that send signals, LED lights, GPS sensors, laser positioning sensors, etc., or other physical signs that can be captured by the camera, such as objects with shape characteristics and/or color characteristics . Preferably, in order to avoid the interference of external stray light and improve the robustness of marker point tracking, it is preferable to use infrared LED lights with a relatively narrow spectrum as marker points, and use a corresponding infrared camera that can only pass through the spectrum used by infrared LEDs to Mark points to snap to. Considering that the external stray light is mostly irregular in shape and the brightness distribution is uneven, the marking points can be set to emit regular-shaped light spots, with high luminous intensity and uniform brightness. In addition, multiple marking points can be set, each marking point corresponds to a light spot, and each marking point forms a regular geometric shape, such as a triangle, a quadrilateral, etc., so that it is easy to track the marking point, obtain the spatial position information of the marking point, and improve the light spot. Extraction accuracy.

S12 Obtain the position information of the mark point; this may be to determine the position information of the mark point by receiving the signal sent by the mark point, or it may be to use a camera to capture an image containing the mark point, and extract the mark point in the image. The location information of the marked points is obtained through image processing algorithms.

S13 Reconstruct the spatial position of the first target object according to the position information of the marked point. After obtaining the position information of the mark point, the spatial position of the mark point is reconstructed, and then according to the relative positional relationship between the mark point and the first target object, the spatial position of the mark point is converted to the spatial position of the first target object ( For example, the spatial position of the left and right eyes of a person).

Embodiment 2 of the present invention obtains the position information of the marked point corresponding to the first target object, and reconstructs the spatial position of the first target object according to the position information. Compared with the existing technology that uses a camera as a human eye capture device and needs to perform feature analysis on a two-dimensional image to obtain the position of the human eye or use other human eye capture devices that use the reflection effect of the iris of the human eye to obtain the position of the human eye, it has good stability. , the accuracy of capturing the position information of human eyes is high, the cost is low, and there is no requirement for the distance between the tracking device and the first target object.

Please refer to Figure 13, the above step S12 further includes:

S121 preset a standard image, the reference mark point is set in the standard image, and the spatial coordinates and plane coordinates of the reference mark point are obtained; the standard image can be, for example, a standard image collected by an image acquisition device, and the reference mark is obtained The image coordinates of the points, and use other accurate stereo measurement equipment such as laser scanners, structured light scanners (such as Kinect, etc.) to obtain the spatial coordinates and plane coordinates of the reference marker points in the standard image.

S122 Acquire the current image including the target object and the marker point, and the plane coordinates of the marker point in the current image;

S123 Match the marker points in the current image with the reference marker points in the standard image. Here, it is first necessary to establish a corresponding relationship between the plane coordinates of the marker points in the current image and the plane coordinates of the reference marker points in the standard image, and then match the marker points with the reference marker points.

By setting the standard image and the reference mark point, it is convenient to have a reference object when acquiring the spatial position of the current image, which further ensures the stability and accuracy of the target tracking method in the embodiment of the present invention.

Further, before the above step S11, the method further includes: S10 performing calibration on the camera used to obtain the position information of the marking point.

The above calibration has the following situations:

(1) When the camera of the S10 is a monocular camera, the common Zhang’s checkerboard calibration algorithm can be used, for example, the following formula is used for calibration:

sm'=A[R|t]M'(1)

In formula (1), A is an internal parameter, R is an external parameter, t is a translation vector, m' is the coordinate of the image point in the image, and M' is the space coordinate of the object point (that is, the three-dimensional coordinate in space); where A, R and t are respectively determined by the following formulas:

$A=\left[\begin{array}{ccc}\mathrm{fx}& 0& \mathrm{cx}\\ 0& \mathrm{fy}& \mathrm{cy}\\ 0& 0& 1\end{array}\right]$ and $R=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{\mathrm{twenty\; one}}& r_{\mathrm{twenty\; two}}& r_{\mathrm{twenty\; three}}\\ r_{31}& r_{32}& r_{33}\end{array}\right],$ translation vector $t=\left[\begin{array}{c}{t}_1\\ t_2\\ t_3\end{array}\right].$

Of course, there are many kinds of camera calibration algorithms, and other commonly used calibration algorithms in the industry can also be used. The present invention does not limit it. The calibration algorithm is mainly used to improve the accuracy of the first object tracking method of the present invention.

(2) When the camera of the S10 is a binocular camera or a multi-eye camera, the following steps are used for calibration:

S101 first calibrates any one of the binocular cameras or multi-cameras, using the common Zhang checkerboard calibration algorithm, for example, using the following formula:

sm'=A[R|t]M'(1)

In formula (1), A is an internal parameter, R is an external parameter, t is a translation vector, m' is the coordinate of the image point in the image, and M' is the space coordinate of the object point; where A, R and t are determined by the following formula Sure:

$A=\left[\begin{array}{ccc}\mathrm{fx}& 0& \mathrm{cx}\\ 0& \mathrm{fy}& \mathrm{cy}\\ 0& 0& 1\end{array}\right]$ and $R=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{\mathrm{twenty\; one}}& r_{\mathrm{twenty\; two}}& r_{\mathrm{twenty\; three}}\\ r_{31}& r_{32}& r_{33}\end{array}\right],$ translation vector $t=\left[\begin{array}{c}{t}_1\\ t_2\\ t_3\end{array}\right];$

S102 Calculate the relative rotation matrix and relative translation between the binocular camera or the multi-camera, using the following formula:

relative rotation matrix $R^{\&prime;}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{\mathrm{twenty\; one}}& r_{\mathrm{twenty\; two}}& r_{\mathrm{twenty\; three}}\\ r_{31}& r_{32}& r_{33}\end{array}\right]$ and the relative translation $T=\left[\begin{array}{c}{T}_1\\ T_2\\ T_3\end{array}\right].$

Of course, the above-mentioned calibration algorithm for binocular cameras or multi-eye cameras is only one of the more common ones, and other commonly used calibration algorithms in the industry can also be used. The present invention is not limited, mainly using calibration algorithms to improve the first goal of the present invention Accuracy of object tracking methods.

Further, between the above S11 and S12, it also includes:

S14 collects the mark points;

S15 Screen target markers from the markers.

Specifically, when there are multiple marker points, a camera is used to collect all marker points corresponding to the first target object, and the marker point most relevant to the first target object is selected from all marker points, and then corresponding image processing is used to The algorithm extracts the marked points on the image, and the extraction needs to be carried out according to the characteristics of the marked points. Generally speaking, the method of extracting the features of the marked points is to use the feature extraction function H on the image I to obtain the feature score of each point in the image, and to filter out the marked points with sufficiently high feature values. Here it can be summed up by the following formula:

s(x,y)=H(I(x,y))

F={arg _{(x, y)} (S(x, y)>s0)}

In the above formula, H is the feature extraction function, I(x, y) is the image value corresponding to each pixel (x, y), which can be the gray value or the color energy value of the three channels, etc., S(x, y ) is the feature score of each pixel (x, y) after feature extraction, s0 is a feature score threshold, S(x, y) greater than s0 can be considered as a marker point, and F is a set of marker points. Preferably, the embodiment of the present invention uses the infrared marker points and the energy characteristics of the image formed by the infrared camera to be relatively obvious. Due to the use of narrow-band LED infrared lamps and corresponding infrared cameras, most of the pixels of the image formed by the camera have low energy, and only the pixels corresponding to the marking points have high energy. Therefore, the corresponding function H(x, y) can perform region growth on the image B(x, y) after using the threshold segmentation operator to obtain several sub-images, and extract the center of gravity of the obtained sub-images. The feature extraction function H(x, y) may be a feature point function such as Harris, SIFT, or FAST, or an image processing function such as circular spot extraction. At the same time, according to the stray light that can be imaged in the infrared camera in the ambient light, we can add constraints such as the area of the spot formed by the marker point and the positional relationship of the marker point in the two-dimensional image to the extraction process of the infrared marker point. mark points for filtering.

When the number of cameras is greater than 1, it is necessary to perform marker matching on images acquired by different cameras at or near the same time, so as to provide conditions for subsequent 3D reconstruction of markers. The method of mark point matching needs to be determined according to the feature extraction function H. We can use some classic feature point extraction operators based on image grayscale gradient maps and matching matching methods such as Harris, SIFT, FAST and other methods to obtain and match marker points. You can also use limit constraints, prior conditions for markers, etc. to match markers. The method of matching and screening using limit constraints here is: according to the principle that the projections of the same point on two different camera images are on the same plane, for a certain marker point p0 in a certain camera c0, we can use it in other cameras An epipolar line equation is calculated in c1, and the marker point p0 corresponds to the marker point p1 on the other camera c1 in accordance with the following relationship:

[p1;1] ^TF [p0;1]=0

In the above formula, F is the fundamental matrix of camera c0 to camera c1. By using the above relationship, we can greatly reduce the number of candidates for the marker point p1 and improve the matching accuracy.

In addition, we can use the prior conditions of the markers to be the spatial order of the markers, the dimensions of the markers, etc. For example, according to the mutual positional relationship of the two cameras, each pair of two pixels corresponding to the same spatial point on the captured image is equal in a certain dimension such as the y-axis. This process is also called image rectification. At this time, the matching of the marked points can also be performed according to the x-axis order of the marked points, that is, the smallest x corresponds to the smallest x, and so on, and the largest x corresponds to the largest x.

The object tracking method of the present invention will be introduced in detail below according to the number of cameras used for tracking.

Please refer to FIG. 14 , which is a schematic flowchart of a first modified example of S13 in FIG. 11 . As shown in Figure 13, in this embodiment, the first target object tracked by the first target object tracking method corresponds to no more than four marker points, and when a monocular camera is used to obtain the position information of the marker points, the preceding steps S13 further includes:

S131 Calculate the difference between the current image and the standard image according to the plane coordinates of the marker points in the current image, the plane coordinates of the reference marker points in the standard image, and the assumed conditions of the scene where the first target object is located. The homography transformation relationship between the current image and the reference marker point in the standard image is matched, and the homography transformation relationship between the current image and the standard image is calculated according to their respective plane coordinates. The so-called homography transformation refers to the homography in corresponding geometry, and it is a transformation method commonly used in the field of computer vision.

S132 Calculate the rigid transformation from the spatial position of the marker point at the moment when the standard image is captured to the spatial position at the current moment according to the homography transformation relationship, and then calculate the spatial position of the marker point at the current moment, and according to the The current spatial position of the first target object is calculated based on the spatial position of the marker point at the current moment.

Specifically, for the assumed conditions of the scene, we can assume that the value of a certain dimension does not change during the rigid transformation of the marked points in the scene. For example, in a three-dimensional space scene, the spatial coordinates are x, y, z, and x and y are respectively It is parallel to the x-axis and y-axis in the image coordinates (plane coordinates) of the camera, and the z-axis is perpendicular to the image of the camera. The assumption can be that the coordinates of the marked point on the z-axis are unchanged, or that the marked point is on the x-axis and /or the coordinates on the y-axis are unchanged. Different scenario assumptions require different inference methods. For another example, under another assumption, assuming that the rotation angle between the orientation of the first target object and the orientation of the camera remains constant during use, the distances and The ratio of the distances between the marker points on the standard image estimates the current spatial position of the first target object.

Through the above calculation method, the spatial position of the first target object can be reconstructed when the number of marker points of the monocular camera is not more than four, the operation is simple, and the tracking result is also relatively accurate. reduces the cost of tracking the first target object.

In the above-mentioned method of recovering the three-dimensional coordinates of an object by collecting images with a single camera, since the acquired image information is less, it is necessary to increase the number of marker points to provide more image information to calculate the three-dimensional coordinates of the object. According to machine vision theory, to deduce the stereoscopic information of the scene from a single image, at least five calibration points in the image need to be determined. Therefore, the monocular solution increases the number of marking points and the complexity of the design, but at the same time, only one camera is needed, which reduces the complexity of image acquisition and reduces the cost.

Please refer to FIG. 15 , which is a schematic flowchart of a second modified example of S13 in FIG. 11 . As shown in FIG. 14, in this embodiment, when the number of the marked points is more than five, and the position information of the marked points is obtained by using a monocular camera, the S13 further includes:

S133 Calculate a homography transformation relationship between the current image and the standard image according to the plane coordinates of the marker points in the current image and the plane coordinates of the reference marker points in the standard image;

S134 Calculate the rigid transformation from the spatial position of the marker point at the moment when the standard image is captured to the spatial position at the current moment according to the homography transformation relationship, and then calculate the spatial position of the marker point at the current moment, and according to the The current spatial position of the first target object is calculated based on the spatial position of the marker point at the current moment.

Specifically, a standard image is collected first, the spatial position of the reference mark point is measured using an accurate depth camera or laser scanner, and the two-dimensional image coordinates (ie plane coordinates) of the reference mark point at this time are obtained.

During use, the camera continuously captures the two-dimensional image coordinates of all marker points in the current image containing the first target object, and calculates the current state based on the two-dimensional image coordinates and the two-dimensional coordinates of the standard image reference marker points at this time. The rigid transformation between the marked point and the marked point when shooting the standard image, assuming that the relative position between the marked points remains unchanged, and then calculate the spatial position transformation of the marked point relative to the standard image at this time, so that Calculate the spatial position of the current marker.

Here, the space position rigid transformation [R|T] between the current marker point and the marker point when shooting the standard image can be calculated by using more than five points. Preferably, the five or more points are not on the same plane, and the camera’s The projection matrix P is calibrated in advance. The specific way to calculate [R|T] is as follows:

The homogeneous coordinates of each marked point in the standard image and the current image are X0, Xi respectively. Both satisfy the limit constraint, that is, X ₀ P ^-1 [R|T]P=X _i . All marked points form a system of equations whose unknown parameter is [R|T]. When the number of marked points is greater than 5, [R|T] can be solved; when the number of marked points is greater than 6, the optimal solution can be found for [R|T], and the method can use singular value decomposition SVD, and/or use The iterative method computes the nonlinear optimal solution. After calculating the spatial position of the marker point, we can infer the spatial position of the first target object (eg human eye) according to the mutual positional relationship between the pre-marked marker point and the first target object (eg human eye).

In this embodiment, only one camera and five or more marking points can be used to accurately construct the spatial position of the first target object, which is not only easy to operate, but also low in cost.

Please refer to FIG. 16 , which is a schematic flowchart of a third modified example of S13 in FIG. 11 . As shown in FIG. 16, Embodiment 3 uses two or more cameras and one or more marking points. When using a binocular camera or a multi-eye camera to obtain the position information of the marker point, the S3 further includes:

S35 adopts binocular or multi-eye three-dimensional reconstruction principle to calculate the spatial position of each marker point at the current moment; the so-called binocular or trinocular reconstruction principle can use the following methods, such as using the parallax between the marker points matched by the left and right cameras , to calculate the spatial position of each marker point at the current moment. Or use other existing common methods to achieve.

S36 Calculate the current spatial position of the first target object according to the current spatial position of the marker point.

Specifically, firstly, a multi-camera calibration method is used to calibrate the mutual positional relationship between each camera. Then in the process of use, extract the coordinates of the marker points from the images acquired by each camera, and match each marker point, that is, obtain its corresponding marker point in each camera, and then use the matched marker points and between cameras The mutual positional relationship calculates the spatial position of the marker points.

In a specific example, multi-cameras (that is, the number of cameras is greater than or equal to 2) are used to shoot marked points to achieve stereoscopic reconstruction. Knowing the coordinate u of a marker point on the image captured by a certain camera and the parameter matrix M of the camera, we can calculate a ray, and the marker point is located on this ray in space.

α ^j u ^j = M ^j Xj = 1...n (where n is a natural number greater than or equal to 2)

Similarly, according to the above formula, the marking point can also calculate the ray corresponding to the other camera on other cameras. Theoretically, these two rays converge on a point, which is the spatial position of this marked point. In fact, due to the digitization error of the camera, the calibration error of the internal and external parameters of the camera, etc., these rays cannot converge at one point, so it is necessary to use the method of triangulation to approximate the spatial position of the marker point. For example, the least square criterion can be used to determine the point closest to all rays as the object point.

${\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}}\mathrm{<span\; class="notranslate">\; min</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>$

After calculating the spatial position of the marker point, we can infer the spatial position of the first target object (human eye) according to the mutual positional relationship between the pre-marked marker point and the first target object (such as the human eye).

Among the above methods for realizing stereo reconstruction using multi-cameras, a better method is to use binocular cameras for calculation. The principle is the same as the aforementioned multi-camera reconstruction principle, which is to calculate the spatial position of the marker point based on the mutual positional relationship between the two cameras and the two-dimensional coordinates of the marker point imaged by the two cameras. The slight difference is that the binocular cameras are placed in parallel, and the images of the two cameras are calibrated as described above after simple calibration, so that the two matching two-dimensional marker points are equal on the y (or x) axis , then the depth from the marker point to the camera can be calculated from the distance between the calibrated two-dimensional marker points on the x (or y) axis. This method can be regarded as a special method for multi-eye stereo reconstruction in the case of binoculars, which simplifies the steps of stereo reconstruction and is easier to implement on the device hardware.

Example 6

Please refer to FIG. 17 , which is a schematic flowchart of S3 in FIG. 11 . As shown in FIG. 17 , based on the aforementioned second embodiment and the aforementioned examples, the step S3 of the stereoscopic display method based on the soft mirror of the present invention further includes:

S301 Step of determining the layout parameters, calculating the layout parameters on the display unit according to the acquired position information of the first target object, the grating parameters of the light splitting unit and the display parameters of the display unit;

S302 Parallax image arranging step, arranging the parallax images on the display unit according to the layout parameters;

S303 Parallax image playing step, playing the parallax image.

Through the above steps, the stereoscopic images to be played are rearranged, and the stereoscopic display effect is improved.

Further, before the step S301, the method further includes: S304 a stereoscopic image acquisition step, acquiring the information of the stereoscopic image captured in real time. While playing parallax images, acquire real-time stereoscopic image information, which improves the efficiency of image processing, not only ensures real-time playback, but also reduces the need for large memory due to the large amount of data storage occupied by stereoscopic display images requirements, reducing costs.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (25)

1. null1. the three-dimensional display system based on soft lens，Including: display unit、Spectrophotometric unit、Tracking equipment and image capturing unit，Described spectrophotometric unit is positioned at the display side of described display unit，Left view and right view it is divided on the image space shown by described display unit，Described tracking equipment is for obtaining the positional information of first object object，Described image capturing unit is used for shooting the second destination object，It is characterized in that，The described three-dimensional display system based on soft lens also includes image player and processes equipment，Respectively with described tracking equipment、Described display unit and described image capturing unit connect，Described image player processes the equipment positional information according to described first object object、The grating parameter of described spectrophotometric unit and the display parameters of described display unit，The stereo-picture that the described image capturing unit received photographs is processed in real time，Send described display unit after process to show in real time.
2. 2. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described image capturing unit is located on described soft lens.
3. 3. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described image capturing unit includes monocular-camera, shoots and obtain the stereo-picture of described second destination object with a photographic head.
4. 4. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described image capturing unit includes binocular camera, shoots and obtain the stereo-picture of described second destination object with two photographic head.
5. 5. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described image capturing unit includes multi-lens camera, by the photographic head of more than three stereo-picture shooting and obtaining described second destination object arranged in arrays.
6. 6. the three-dimensional display system based on soft lens as described in any one of claim 2 to 5, it is characterized in that, described image capturing unit farther includes collecting unit, described collecting unit is for gathering the stereo-picture of described second destination object, and extracts left view information and right view information from described stereo-picture.
7. 7. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described tracking equipment includes video camera, the change in location of first object object described in described Camera location.
8. 8. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterized in that, described tracking equipment includes infrared remote receiver, and described infrared remote receiver receives and comes from the infrared framing signal that the infrared transmitter set by corresponding described first object object sends.
9. 9. the three-dimensional display system based on soft lens as claimed in claim 1, it is characterised in that described tracking equipment includes:

   Labelling point arranges unit, and the locus of corresponding described first object object arranges labelling point；

   Acquiring unit, obtains the positional information of described labelling point；

   Rebuild unit, according to the positional information of described labelling point, rebuild the locus of described first object object.
10. 10. the three-dimensional display system based on soft lens as claimed in claim 9, it is characterised in that described acquiring unit farther includes:

    Presetting module, presets a standard picture, is provided with reference marker point, and obtains described reference marker point space coordinates in described standard picture and plane coordinates in described standard picture；

    Acquisition module, obtains the present image comprising described first object object with described labelling point, and described labelling point is at the plane coordinates of described present image；

    Matching module, mates the labelling point in described present image with the described reference marker point of described standard picture.
11. 11. the three-dimensional display system based on soft lens as claimed in claim 10, it is characterised in that when the quantity of described labelling point is less than four, and when adopting positional information that monocular-camera obtains described labelling point, described in rebuild unit and also include:

    First computing module, for calculating the homograph relation between described present image and described standard picture according to the plane coordinates of the described reference marker point of the plane coordinates of the labelling point in described present image and described standard picture and the assumed conditions of described first object object place scene；

    First reconstructed module, for calculating the described labelling point rigid transformation in the locus to the locus of current time that shoot the described standard picture moment according to described homograph relation, then calculate described labelling point in the locus of current time, and calculate the current locus of described first object object in the locus of current time according to described labelling point.
12. 12. the three-dimensional display system based on soft lens as claimed in claim 10, it is characterised in that when the quantity of described labelling point is more than five, and when adopting positional information that monocular-camera obtains described labelling point, described in rebuild unit and also include:

    Second computing module, is used for the plane coordinates of the plane coordinates according to the labelling point in described present image and the described reference marker point of described standard picture, calculates the homograph relation between described present image and described standard picture；

    Second reconstructed module, for calculating the described labelling point rigid transformation in the locus to the locus of current time that shoot the described standard picture moment according to described homograph relation, then calculate described labelling point in the locus of current time, and the locus according to described labelling point current time calculates the locus that first object object is current.
13. 13. the three-dimensional display system based on soft lens as claimed in claim 10, it is characterised in that when the positional information adopting binocular camera or multi-lens camera to obtain described labelling point, described in rebuild unit and also include:

    3rd computing module, is used for adopting binocular or many orders three-dimensional reconstruction principle, calculates each labelling point in the locus of current time；

    Third reconstruction module, calculates, for the locus according to described labelling point current time, the locus that first object object is current.
14. 14. the three-dimensional display system based on soft lens as described in any one of claim 2 to 13, it is characterized in that, described image player processes equipment and includes image player processing unit and memory element, described image player processing unit is for the display parameters according to the positional information of described first object object, the grating parameter of described spectrophotometric unit and described display unit, the stereo-picture received is carried out real-time row's figure process, sends described display unit after process and show in real time；Described memory element is for storing the image that described image capturing unit transmits, and wherein, described image player processing unit is connected with described memory element.
15. 15. the three-dimensional display system based on soft lens as claimed in claim 14, it is characterised in that described image player processing unit includes:

    Stereo-picture acquisition module, obtains the information of the described stereo-picture of described image capturing unit shooting；

    Row's graph parameter determines module, according to row's graph parameter that the positional information of described first object object got and the grating parameter of described spectrophotometric unit calculate on described display unit；

    Parallax image arrangement module, for arranging the anaglyph of the described stereo-picture on described display unit according to described row's graph parameter；

    Anaglyph playing module, plays described anaglyph.
16. 16. the three-dimensional display system based on soft lens as described in any one of claim 1 to 15, it is characterised in that be provided with laminating unit between described spectrophotometric unit and described display unit, by described laminating unit, described spectrophotometric unit is fitted on described display unit.
17. 17. the three-dimensional display system based on soft lens as claimed in claim 16, it is characterised in that described laminating unit includes first substrate, second substrate and the air layer between described first substrate and described second substrate.
18. 18. the stereo display method based on soft lens, it is characterised in that said method comprising the steps of:

    S0 shoots the stereo-picture of the second destination object, and sends the information of described stereo-picture in real time；

    S1 obtains the positional information of first object object；

    S2 obtains the grating parameter of the spectrophotometric unit of display device and the display parameters of the display unit of described display device；

    The stereo-picture that the described image capturing unit received photographs is processed in real time by S3 according to described positional information and described grating parameter and described display parameters；

    S4 shows described image to be played.
19. 19. the stereo display method based on soft lens as claimed in claim 18, it is characterised in that described S0 also includes:

    Image acquisition step, gathers the stereo-picture of described second destination object, and extracts left view information and right view information from described stereo-picture.
20. 20. the stereo display method based on soft lens as claimed in claim 19, it is characterised in that described S1 also includes:

    The locus of the corresponding described first object object of S11 arranges labelling point；

    S12 obtains the positional information of described labelling point；

    S13, according to the positional information of described labelling point, rebuilds the locus of described first object object.
21. 21. the stereo display method based on soft lens as claimed in claim 20, it is characterized in that, described S12 farther includes: S121 presets a standard picture, is provided with reference marker point, and obtains described reference marker point space coordinates in described standard picture and plane coordinates in described standard picture；

    S122 obtains the present image comprising described first object object, described labelling point, and described labelling point is at the plane coordinates of described present image；

    Labelling point in described present image is mated by S123 with the described reference marker point of described standard picture.
22. 22. the stereo display method based on soft lens as claimed in claim 21, it is characterised in that when the quantity of described labelling point is less than four, and when adopting the positional information of the monocular-camera described labelling point of acquisition, described S13 farther includes:

    S131 calculates the homograph relation between described present image and described standard picture according to the plane coordinates of the labelling point in described present image and the plane coordinates of described reference marker point of described standard picture and the assumed conditions of described first object object place scene；

    S132 calculates the described labelling point rigid transformation in the locus to the locus of current time that shoot the described standard picture moment according to described homograph relation, then calculate described labelling point in the locus of current time, and calculate the current locus of described first object object in the locus of current time according to described labelling point.
23. 23. the stereo display method based on soft lens as claimed in claim 21, it is characterised in that when the quantity of described labelling point is more than five, and when adopting the positional information of the monocular-camera described labelling point of acquisition, described S13 farther includes:

    S133, according to the plane coordinates of the described reference marker point of plane coordinates and the described standard picture of the labelling point in described present image, calculates the homograph relation between described present image and described standard picture；

    S134 calculates the described labelling point rigid transformation in the locus to the locus of current time that shoot the described standard picture moment according to described homograph relation, then calculate described labelling point in the locus of current time, and the locus according to described labelling point current time calculates the locus that first object object is current.
24. 24. the stereo display method based on soft lens as claimed in claim 21, it is characterised in that when the positional information adopting binocular camera or multi-lens camera to obtain described labelling point, described S13 farther includes:

    S135 adopts binocular or many orders three-dimensional reconstruction principle, calculates each labelling point in the locus of current time；

    S136 calculates, according to the locus of described labelling point current time, the locus that destination object is current.
25. 25. the stereo display method based on soft lens as described in any one of claim 18 to 24, it is characterised in that described S3 farther includes:

    S304 stereo-picture obtaining step, obtains the information of the described stereo-picture that captured in real-time arrives；

    S301 arranges graph parameter and determines step, calculates row's graph parameter on the display unit according to the display parameters of the positional information of described first object object got and the grating parameter of described spectrophotometric unit and described display unit；

    S302 parallax image arrangement step, according to the anaglyph of the described stereo-picture on described row's graph parameter described display unit of arrangement；

    S303 anaglyph plays step, plays described anaglyph.