CN117319631A - Light field display system and image rendering method based on human eye tracking - Google Patents

Abstract

The application discloses a light field display system and an image rendering method based on human eye tracking, wherein the positions of eyes of a viewer are positioned and tracked, and basic information such as real-time viewing angles, directions and the like required by light field reconstruction is calculated, so that the real-time rendering of light field images is completed through a nerve radiation field body rendering algorithm. By combining the human eye tracking technology and the nerve radiation field rendering technology, the spatial bandwidth product of the light field display is greatly improved, and simultaneous watching of multiple persons is supported, so that the design dependence of the light field display on the high pixel density of the liquid crystal panel is reduced, the stereoscopic display effect of the light field display is improved, and a viewer can obtain better stereoscopic immersion feeling at different watching positions. In addition, the proposed nerve radiation field is adopted to reconstruct the light field image, so that the problem of depth inversion is effectively avoided, and meanwhile, the data volume and the computational complexity of a light field image rendering algorithm can be reduced.

Description

A light field display system and image rendering method based on human eye tracking

Technical field

This application relates to the fields of image processing, stereoscopic display and artificial intelligence, and specifically relates to a light field display system and image rendering method based on human eye tracking.

Background technique

Current stereoscopic display technology generally uses Parallax Barrier (PB) technology to generate parallax so that viewers can gain a sense of stereoscopic immersion. Stereoscopic display technology can be roughly divided into two categories: assisted and unassisted. Assisted stereoscopic displays can be divided into switch glasses, polarized glasses, VR helmets, chromatic glasses, etc. Unassisted can be divided into parallax barrier type, and true three-dimensional display. The parallax barrier type mainly includes lenticular grating, slit grating, directional backlight technology, etc. True three-dimensional display is divided into integrated imaging, holographic display, light field display, volume display and other technologies. Among them, parallax barrier stereoscopic display technology is the most mature type of display technology. However, if the process control of this type of stereoscopic display technology is not good, it will cause greater stereoscopic crosstalk. The light field display system is a geometric optical system that cleverly uses crosstalk. The initial way to obtain light field images is through direct imaging through a lens array, but direct display will cause depth inversion problems.

In the existing auxiliary stereoscopic display mode, the driver of the stereoscopic display will alternately render images for the left and right eyes. If the first frame is a left-eye image, then the next frame corresponds to a right-eye image. The viewer wears LCD shutter glasses, which are synchronized with the graphics card and monitor through wired or wireless means. When the left eye image is displayed on the monitor, the glasses open the shutter of the left lens and close the shutter of the right lens. When the right eye image is displayed on the monitor, When taking an image, the glasses open the shutter of the right lens and close the shutter of the left lens. Due to the fast refresh rate, the view of the temporarily invisible eye will be synthesized by the brain based on the visual temporary effect. Therefore, as long as the shutter-type stereoscopic glasses interface is worn within the coverage of the synchronization signal, the stereoscopic image can be viewed on the display.

Auxiliary stereoscopic display requires that the left and right lenses of the stereoscopic shutter glasses must be synchronized with the corresponding projected image signal, and must have a high refresh rate of the stereoscopic display to achieve a good stereoscopic calming effect. The advantages of this stereoscopic display technology are low cost, good stereoscopic immersion, and seamless switching between flat and stereoscopic images. However, the disadvantage is that the stereoscopic brightness is low and it is prone to visual fatigue.

Unassisted autostereoscopic display technology is generally called "naked-eye 3D display" technology, which allows the left and right eyes to see two images with parallax from the display screen without using any tools, and reflects them to the brain. The viewer will have a sense of three-dimensional immersion. The unassisted stereoscopic display also takes advantage of the parallax principle of the human eye to achieve a three-dimensional visual viewing effect by providing different display images to the viewer's left and right eyes. Currently, the unassisted display technology is mainly based on multi-viewpoint stereoscopic display technology. Since the observer does not need to wear glasses, this technology is commonly used in public places for display and promotion. However, the shortcomings of multi-viewpoint free stereoscopic display technology are reduced resolution, limited viewing angles, low brightness, large crosstalk, and relatively poor display effects such as stereoscopic vertigo.

Directional backlight technology is one of the more practical stereoscopic display technologies among current unassisted stereoscopic display technologies. The directional backlight stereoscopic display technology is paired with two sets of LEDs and a fast-response LCD panel and driving method to allow 3D content to be viewed in a sequential manner. The user's left and right eyes exchange images to produce parallax, which in turn allows the human eye to experience a 3D stereoscopic effect, greatly enhancing communication and interaction based on mobile devices. Its advantage is that the resolution and light transmittance are guaranteed, it will not affect the existing display design structure, and the 3D display effect is excellent. The disadvantage is that the technology is still under development and the products are immature.

Compared with traditional parallax barrier stereoscopic display technology, light field display technology has the advantages of good stereoscopic immersion, multiple dimensions of displayed information, no dizziness, high resolution and fast refresh, etc. It is regarded as the most promising next-generation technology. One of the generation reserve display technologies, it is a key technology to realize large-format three-dimensional display.

To sum up, the existing technology has the following shortcomings:

1. Parallax barrier type free stereoscopic display, due to the limitation of the resolution of the liquid crystal display panel, it is generally difficult to increase the spatial bandwidth product, usually only about 40%.

2. Due to its optical structural characteristics, the parallax barrier autostereoscopic display will produce stereoscopic crosstalk during the stereoscopic display process. Due to its optical structural characteristics, the parallax barrier autostereoscopic display needs to modulate the light to the position of the user's left and right eyes, resulting in The brightness of the autostereoscopic display is low, which aggravates visual fatigue.

3. Dense-viewpoint stereoscopic display technology requires high display density of the LCD panel, making it difficult to increase the spatial bandwidth product. The amount of data required for image rendering is large, resulting in slower stereoscopic image rendering speed.

4. For light field displays, due to the traditional ray tracing pipeline rendering scheme, the algorithmic rendering process is complex, slow, low-precision, and may cause depth inversion. The sampling and reconstruction process of the target model to be displayed is complex. As a result, the display efficiency of the display is lowered and the stereoscopic performance of the stereoscopic display is deteriorated.

Contents of the invention

In order to solve the above problems, this application provides a light field display system and image rendering method based on human eye tracking.

According to the first aspect of this application, this application provides a light field display system based on human eye tracking, including:

A display control unit is used to provide stereoscopic display data and human-computer interaction related interaction information for the light field display;

A light field rendering unit, connected to the display control unit, used for real-time rendering of light field images based on the light field model and human eye position information output by the video input interface in the display control unit;

The driving unit is connected to the light field rendering unit. The driving unit adopts a modular design. The display resolution of each driving module is 2K×1K or 4K×2K. It is used for partition refresh and high-density liquid crystal display panels. Synchronous control;

The display unit is used to provide a display carrier for the presentation of light field information and build a high-density display physical model with a resolution of 16K×8K;

A high-speed eyeball three-dimensional tracking unit, connected to the display unit, is used to locate and track the human eye position information of the viewer in real time, and output the human eye position information from the serial bus according to a predetermined communication protocol;

A visual drive unit, connected to the light field rendering unit and the high-speed eyeball three-dimensional tracking unit, is used to bridge the viewer's perspective information and the light field rendering model to create a light field rendering unit based on the viewer's three-dimensional position information of the human eye. Outputs the desired perspective rendering parameters.

The light field display system based on human eye tracking is characterized in that the display control unit is a 2D interactive unit. The user can interact with the operating system by equipping a 2D display screen, and can also set the light field display system. The working mode of the field display is a 2D working mode, and the 2D and 3D switching control can be switched through the mouse or gestures.

The light field display system based on human eye tracking is characterized in that the light field rendering unit is a parallel rendering core with high stability and high real-time performance, and uses a neural radiation field deep learning algorithm for rendering.

The light field display system based on human eye tracking is characterized in that the driving unit is a Tcon driving module with independent driving and array synchronization functions, and the resolution of the driving board is 1920×1080 or 3840×2160.

The light field display system based on human eye tracking is characterized in that the visual driving unit provides the light field rendering unit with perspective information for image rendering, wherein the image rendering adopts reverse ray integration rendering technology.

According to the second aspect of this application, this application provides a light field image rendering method based on human eye tracking, which is characterized in that it includes two stages: model training and model inference, and includes the following steps:

Obtain the target scene step: the target scene that needs to be displayed is used to provide the original training data set for the light field image rendering unit;

Adaptive scene acquisition step: According to the target scene, use a multi-scale adaptive method to sample the target scene to generate a digital image signal that needs to be output;

Camera pose estimation step: estimate the pose of the shooting camera based on the sampled scene information;

Neural radiation field model training step: preprocess the digital waveform signal output in the data acquisition step and output the processing result;

Obtain implicit expression steps: According to the neural radiation field algorithm, use a fully connected network for model training to obtain the node information and weight parameters of the model;

Light field reproduction step: Using the parameters in the implicit expression of the model as network weights, convert the human eye position signal in the neural radiation field rendering step into perspective information to complete model inference. The data output by the rendering step according to the human eye position is rasterized and then converted to the coordinate system of the display screen to achieve the purpose of light field display.

The light field image rendering method based on human eye tracking is characterized in that in the target scene acquisition step, the data can be a real scene that needs to be displayed, or it can be a virtual scene in a game or in the field of animation. 3D model.

The described light field image rendering method based on human eye tracking is characterized in that the data acquisition step can use an aircraft to sample the scene in real time, sample data from a light field camera, or use a virtual camera to virtualize a three-dimensional model. sampling.

The light field image rendering method based on human eye tracking is characterized in that, in the step of obtaining model parameters, the rendering process adopts a reverse rendering algorithm to obtain an orthographic observation image.

The light field image rendering method based on human eye tracking is characterized in that the light field rendering algorithm is a real-time rendering algorithm

The beneficial effects of this application are:

According to the above embodiment, a light field display system and image rendering method based on human eye tracking are used. Since the position of the viewer's eyes is tracked and the corresponding image rendering viewpoint information is calculated, the neural radiation field reconstruction algorithm is used to perform Rendering of light field images. Combined with human eye tracking, the spatial bandwidth product of the light field stereoscopic display is greatly improved, and it supports multiple people to watch at the same time, thus reducing the design requirements of the light field display for the high pixel density of the liquid crystal panel, improving the display effect of the light field display, and This allows viewers to obtain better three-dimensional immersion at different viewing positions, while also reducing the amount of data and computational complexity of the light field image rendering algorithm.

Description of drawings

Figure 1 is a schematic diagram of a light field display system based on human eye tracking in this application;

Figure 2 is a schematic structural diagram of a light field display system based on human eye tracking in this application;

Figure 3 is a schematic diagram of a light field display system driving circuit based on human eye tracking in this application;

Figure 4 is a schematic flow chart of a light field image rendering method based on human eye tracking in this application;

Figure 5 is a schematic diagram of a light field acquisition and reproduction method based on human eye tracking in this application;

Figure 6 is a schematic diagram of a light field image rendering method based on human eye tracking in this application;

Detailed ways

The present application will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

Please refer to Figure 1. This application discloses a light field display system based on human eye tracking (hereinafter referred to as light field display), which includes a display control unit 10, a light field rendering unit 11, a driving unit 12, a display unit 13, a high-speed The three-dimensional eye tracking unit 14 and the visual driving unit 15 are described separately below.

The display control unit 10 is used to provide stereoscopic display data and related interactive information for the light field display. In a specific embodiment, the display control unit 10 is a computer with high stability and high computing power. For example, the display control unit 10 can be a server. Of course, in some other embodiments, the display control unit 10 can also be a conventional computer or a computer with an HDMI video interface.

The light field rendering unit 11 is connected to the display control unit 10 and is used to render the light field image according to the light field model and human eye position information output by the video input interface in the display control unit.

The driving unit 12 is connected to the light field rendering unit 11. The driving unit adopts a modular design. The size of each module driven is 1920×1080, and is used for partition refresh and control of high-density liquid crystal display panels.

The driving unit 12 has a resolution of 1920×1080 and is used to display, distribute and refresh the light field digital signal output by the light field rendering unit. In a specific embodiment, the driver may support HDMI, DP or Type-C input.

The display unit 13 is used as a presentation carrier for light field display information and provides display information with a physical resolution of 16K×8K; in a specific embodiment, the display unit 13 is composed of a high-resolution liquid crystal display panel and an optical microstructure. composite display carrier. For example, the display unit 13 may be a display device composed of a backlight structure, a liquid crystal display panel and a grating. Of course, in some other embodiments, the display unit 13 may also be a non-liquid crystal display panel. In another specific embodiment, as shown in Figure 2, the backlight module 24 provides a light source for the liquid crystal display panel, the human eye tracking module 26 can track the user's viewing position in real time, and the input terminal of the high-density liquid crystal display module 27 is connected to the display driver. The modules 23 are connected, and the front panel of the high-density liquid crystal display module is bonded to the aspherical microlens array module 28 .

The high-speed eyeball three-dimensional tracking unit 14 is connected to the display unit 13 and is used to locate and track the human eye position information of the viewer in real time, and output the human eye position information from the serial port according to a predetermined communication protocol. In a specific embodiment, the high-speed eyeball three-dimensional tracking unit 14 is a visual tracking device with human eye three-dimensional position detection and tracking functions.

The visual driving unit 15 is connected to the light field rendering unit 11 and the high-speed eyeball three-dimensional tracking unit 14, and is used to bridge the viewer's perspective information and the light field rendering model to form a light field based on the viewer's three-dimensional human eye information. The rendering unit outputs the desired perspective rendering parameters.

Accordingly, this application proposes a light field display image rendering method based on human eye tracking, which is characterized by including two stages of model training and model inference, as shown in Figure 4, including the following steps:

S41. Step of obtaining the target scene: The target scene that needs to be displayed can provide the original training data set for the light field image rendering unit, as shown in Figure 4. In a specific embodiment, the target scene can be a virtual three-dimensional scene in computer software or a direct collection of real objects in the real environment;

S42. Adaptive scene collection step: According to the target scene, use a multi-scale adaptive method to perform scene sampling on the target scene to generate digital image signals that need to be output. In a specific embodiment, sampling at different scales is performed according to the complexity of the model to reduce information redundancy and obtain images of key areas;

S43. Camera pose estimation step: estimate the pose of the captured camera based on the sampled scene information. In a specific embodiment, the camera pose is estimated by performing feature extraction and feature matching on the collected images;

S44. Neural radiation field model training steps: Based on the position and direction of the viewing angle, color is used as a supervision item, and a fully connected network is used for model training;

S45. Obtain implicit expression step: According to the neural radiation field algorithm, in a specific embodiment, obtain the node information and weight parameters of the model from the model that has been trained by the fully connected network;

S46. Light field reproduction step: Use the parameters in the implicit expression of the model as network weights to convert the human eye position signal in the neural radiation field rendering step into perspective information to complete model inference. In a specific embodiment, the data output from the rendering step based on the human eye position is rasterized and then converted to the coordinate system of the display screen to achieve the purpose of light field display.

This application proposes a light field display system and image rendering method based on human eye tracking. Since the position of the viewer's eyes is tracked and the corresponding image rendering viewpoint information is calculated, the light field is reconstructed through a neural radiation field reconstruction algorithm. Rendering of images. Combined with human eye tracking, the spatial bandwidth product of the light field stereoscopic display is greatly improved, and it supports multiple people to watch at the same time, thus reducing the design requirements of the light field display for the high pixel density of the liquid crystal panel, improving the display effect of the light field display, and This allows viewers to obtain better three-dimensional immersion at different viewing positions, and at the same time, it can also reduce the amount of data and computational complexity of the light field image rendering algorithm, which will be analyzed in detail below.

As shown in Figure 1, the light field display technical solution proposed in this application uses a display control unit to control the content that needs to be displayed. The light field model output by the display control unit can, in addition to outputting a standard three-dimensional point cloud model (. ply, .pcl, .txt and other formats), it can also be an implicit expression of the three-dimensional model, including the weight file of the model and the connection relationship between each fully connected layer. By encoding non-standard three-dimensional model and other data and then transmitting it through the standard HDMI output interface, it not only has better compatibility, but can also be used for compressed transmission and encrypted transmission of light field data, etc., with high timeliness and For security, the output resolution of light field data is 1920×1080.

As shown in Figure 1, for a large light field model, the rendering unit 11 needs to perform data segmentation. The purpose is to distribute the segmented data to the arrayed parallel acceleration unit to accelerate the rendering algorithm. The segmented data blocks are respectively The rendering of the light field image is completed in the arrayed rendering unit. In Figure 1, the data segmentation module, light field image rendering module, and video interface module output are collectively referred to as the light field rendering unit. The light field data rendered by the light field rendering unit needs to be output to the drive unit through the video output interface module.

As shown in FIG. 1 , after receiving the light field data from the light field rendering unit 11 , the driving unit 12 further drives the high-density liquid crystal display unit 13 to display. In order to increase the spatial bandwidth product of the light field display, on the one hand, it is necessary to increase the information amount of the light field display, and on the other hand, a high-density liquid crystal panel is required as a physical support. However, with the increase in the amount of displayed information, there is no doubt that the light field image rendering unit 11 This will cause huge computing power overhead, causing the display system to be unable to achieve smooth light field image playback and interaction.

Inspired by the peripheral vision mechanism, this application uses human eye tracking technology to accelerate the rendering algorithm and improve the display bandwidth product. As shown in Figure 1, this application uses a high-speed eyeball three-dimensional tracking unit 14 to determine the viewing position of the viewer. This allows the display information to be accurately generated for users, and the light field image will not be rendered in areas that are not viewed by viewers, thus reducing the amount of data for light field image rendering and saving a large number of effective pixels, allowing more LCD Display resources contribute to the valid viewer area. Therefore, due to the existence of the high-speed eyeball three-dimensional tracking unit 14, it is possible to increase the spatial bandwidth product of the light field display. The usual calculation method of the spatial bandwidth product (Space Bandwidth Product, SBP) is SBP=horizontal resolution*vertical resolution *Field frequency*1.344. In order to achieve good stereoscopic viewing effects, the spatial bandwidth product SBP of the light field display is generally required to be ≥ 10 ⁸ . Therefore, the core value of the visual drive unit 15 lies in establishing the physical and logical connection between the light field rendering unit and the high-speed eyeball three-dimensional tracking unit. .

As shown in Figure 2, it is a system structure diagram of a light field display system based on human eye tracking in this application. The system includes a video interface module 21, a unit module 22, a driver module 23, a backlight module 24, a light field rendering module 25, Eye tracking module 26, high-density liquid crystal display module 27, aspherical lens array module 28, it can be seen that in this application, the human eye tracking module and the light field image rendering module work together.

As shown in Figure 3, the video interface unit consists of a unit module 31, a PC interface module 32, an ESD protection module 33, a signal conversion module 34, a display panel 35, an audio decoding module 36, a video synchronization interface module 37, and a core processor module 38 It adopts modular design ideas, thereby reducing the difficulty of designing the array display driver.

Please refer to Figure 4, a light field display image rendering method based on human eye tracking. The algorithm includes the step of obtaining the target scene S41, the adaptive scene acquisition step S42, the camera attitude estimation step S43, the neural radiation field model training step S44, and obtaining the hidden scene. The formula expresses step S45 and light field reproduction step S46. After multi-scale adaptive sampling of the target scene, the multi-view information of the scene can be obtained. The pose estimation is performed based on the sampled scene information. After the pose estimation is completed, the network input of the neural radiation field can be obtained based on the pose parameters and multi-view images. , the NeRF function represents a continuous scene as a function whose input is a 5D vector, including the 3D coordinate position x = (x, y, z) and direction (θ, φ) of a spatial point, and the output is perspective-related. The color of the 3D point c = (r, g, b) and the volume density σ of the position. The volume density σ(x) can be understood as, at the x position in space, a differential of infinitesimal particles (the differential of the opacity at the position) has nothing to do with the radiation direction. Camera imaging radiation r(t)=o+td; camera position o and imaging point d constitute the camera imaging ray, accompanied by a near point and far point range t. The color C(r) of this ray is calculated by integrating The way is expressed as:

Among them, t _f and t _n represent the far point and near point of the ray respectively, σ represents the volume density of the current point, c represents the RGB value of the current point, and T(t) represents the accumulation of the ray from t _n to t. transparency:

Among them, t _n represents the near point of the ray, t represents any point on the ray, and σ represents the volume density of the current point.

In a continuous radiation field, to render for any viewing angle, it is necessary to calculate the above color integral for the ray passing through each pixel of the target virtual camera to obtain the color of each pixel and render the imaging image from that viewing angle. . The difference between the color and the actual color obtained by calculating volume rendering through L2 loss is used as the loss function to supervise the learning of the neural network.

Complete the training of the neural network and convert the scene into an implicit expression. If you want to obtain new perspective information, you only need to input the viewer's perspective information to render the perspective information of the current viewing position. After rasterization, the perspective image can be matched with the physical coordinate system of the light field display to achieve stereoscopic display effect.

As shown in Figure 5, it is a schematic diagram of a light field acquisition and reproduction method based on human eye tracking in this application. The acquisition plane 53 captures the three-dimensional scene 51 through the acquisition lens array 52. This process is the acquisition process of the light field display, because the acquisition process Each lens of the lens array 52 can collect specific information of the corresponding scene from different angles, so each collection lens can collect a specific unit image at a specific position of the collection plane 53, and all unit images form a sub-image array. 54. Load the sub-image array 54 onto the display plane 55. There is a reconstruction lens array 56 behind the display plane. The reconstruction lens array 56 should accurately correspond to the sub-image array 54. According to the principle of reversible optical path, the reconstructed three-dimensional scene 57 can be obtained. The process is the reproduction process of light field display.

As shown in Figure 6, it is a schematic diagram of a light field image rendering method based on human eye tracking in this application. The display plane 61 mentioned above can reconstruct a three-dimensional scene by loading the sub-image array and the reconstruction lens array 62 above the display plane. The light field information 63 is not required for actual viewing. It is only necessary to generate the image required for human eyes to view. The light that the observer can see at the current position is called effective light, and the light that the observer cannot see at the current position is called invalid light 66. Therefore, the viewer's eyes 64 and 65 are tracked through the human eye tracking module. , the human eye position is sent to the light field rendering unit, and only the sub-image array effective for the observer needs to be rendered through the neural radiation field, which can effectively reduce the data volume and computational complexity of the light field image rendering algorithm.

In summary, since the position of the viewer's eyes is tracked and the corresponding image rendering viewpoint information is calculated, the light field image is rendered through the neural radiation field reconstruction algorithm. Combined with human eye tracking, the spatial bandwidth product of the light field stereoscopic display is greatly improved, and it supports multiple people to watch at the same time, thus reducing the design requirements of the light field display for the high pixel density of the liquid crystal panel, improving the display effect of the light field display, and This allows viewers to obtain better three-dimensional immersion at different viewing positions, while also reducing the amount of data and computational complexity of the light field image rendering algorithm. This application combines human eye tracking technology to reduce the amount of data for data rendering, thereby reducing the high-performance requirements of light field image rendering on the processor, reducing the design difficulty of light field displays, and enabling real-time rendering of light field images and high-density liquid crystal display panels. The design is no longer the bottleneck to improve the spatial bandwidth product of the light field display, allowing the output signal bandwidth of the light field display to break through the spatial bandwidth product (10 ⁸ ) limit of the existing technology. The spatial bandwidth product refers to the information of the optical imaging of the light field display. Flux, spatial bandwidth product is abbreviated as SPB, SPB = horizontal resolution * vertical resolution * field frequency * 1.344. The spatial bandwidth product SPB of the light field display needs to be greater than 10 ⁸ , and the neural radiation field rendering technology can be used at any viewing angle You can get better viewing effects.

In addition, the visual fatigue phenomenon caused by traditional parallax free stereoscopic displays due to crosstalk, etc., such as the convergence point and focusing are not in the same plane, will also be affected by the increase in the number of light rays and the increase in spatial bandwidth product in the light field display. Alleviation, thereby improving the stereoscopic immersion and stereoscopic viewing comfort of light field displays.

The above content is a further detailed description of the present application in combination with specific implementation modes, and it cannot be concluded that the specific implementation of the present application is limited to these descriptions. For those of ordinary skill in the technical field to which this application belongs, several simple deductions or substitutions can be made without departing from the inventive concept of this application.

Claims (10)

1. A light field display system based on eye tracking, comprising:

the display control unit is used for providing three-dimensional display data and man-machine interaction related interaction information for the light field display;

the light field rendering unit is connected with the display control unit and is used for performing light field image real-time rendering according to the light field model and human eye position information output by the video input interface in the display control unit;

the driving units are connected with the light field rendering units, the driving units are in a modularized design, and the display resolution of each driving module is 2K multiplied by 1K or 4K multiplied by 2K and is used for carrying out partition refreshing and synchronous control on the high-density liquid crystal display panel;

the display unit is used for providing a display carrier for the presentation of the light field information and constructing a high-density display physical model with the resolution of 16K multiplied by 8K;

the high-speed eyeball three-dimensional tracking unit is connected with the display unit and is used for positioning and tracking the human eye position information of a viewer in real time and outputting the human eye position information from the serial bus according to a preset communication protocol;

the visual driving unit is connected with the light field rendering unit and the high-speed eyeball three-dimensional tracking unit and is used for bridging the visual angle information of the viewer with the light field rendering model so as to output expected visual angle rendering parameters for the light field rendering unit according to the three-dimensional position information of the human eye of the viewer.

2. The light field display system based on eye tracking as claimed in claim 1, wherein the display control unit is a 2D interaction unit, a user can interact with the operating system by providing a 2D display screen on one hand, and can also set the operation mode of the light field display to be a 2D operation mode, and the 2D and 3D switching control can be switched by a mouse or a gesture.

3. The eye-tracking based light field display system of claim 1, wherein the light field rendering unit is a parallel rendering core with high stability and high real-time, and performs rendering using a neural radiation field deep learning algorithm.

4. The eye-tracking based light field display system of claim 1, wherein the driving unit is a Tcon driving module having independent driving and array synchronization functions, and the resolution of the driving board is 1920 x 1080 or 3840 x 2160.

5. The eye-tracking based light field display system of any one of claims 1-4, wherein the visual drive unit provides visual angle information for image rendering to the light field rendering unit, wherein the image rendering employs an inverse ray-integral rendering technique.

6. The light field image rendering method based on human eye tracking is characterized by comprising two stages of model training and model reasoning, and comprises the following steps:

a step of acquiring a target scene: the target scene to be displayed is used for providing an original training data set for the light field image rendering unit;

and (3) self-adaptive scene acquisition: sampling the target scene by adopting a multi-scale self-adaptive method according to the target scene, and generating a digital image signal to be output;

and a camera pose estimation step: estimating the pose of a shooting camera according to the sampled scene information;

training a nerve radiation field model: preprocessing the digital waveform signals output in the data acquisition step and outputting processing results;

obtaining an implicit expression step: according to a neural radiation field algorithm, training a model by adopting a fully connected network to obtain node information and weight parameters of the model;

a light field reproduction step: and converting the human eye position signal in the neural radiation field rendering step into visual angle information by taking parameters in the model implicit expression as network weights, so as to complete model reasoning. And rasterizing the data output according to the human eye position rendering step and converting the data into a display screen coordinate system so as to achieve the aim of light field display.

7. The method for rendering a light field image based on eye tracking as claimed in claim 6, wherein in said target scene obtaining step, the data may be a real scene to be displayed, or may be a virtual scene in a game or a three-dimensional model in an animation field.

8. The method of claim 6, wherein the data sampling step is performed by sampling the scene in real time by an aircraft, sampling data by a light field camera, or virtually sampling a three-dimensional model by a virtual camera.

9. The method of light field image rendering based on eye tracking as claimed in claim 6, wherein in the step of obtaining model parameters, a reverse rendering algorithm is adopted in the rendering process, so as to obtain a front-looking observation image.

10. A light field image rendering method based on human eye tracking as claimed in any one of claims 6 to 9, wherein the light field rendering algorithm is a real-time rendering algorithm.