CN118537468A - Multi-level parallel rendering method and system for light field image - Google Patents

Abstract

The present invention discloses a multi-level parallel rendering method and system for light field images, and belongs to the field of image rendering technology. The multi-level parallel rendering method for light field images of the present invention obtains light field images by constructing a first-level rendering model, a second-level rendering model, and a third-level rendering model, and can realize three-level parallel rendering, thereby effectively reducing redundant calculations and improving rendering efficiency, thereby realizing fast rendering of large data volumes and high-quality light field images. Furthermore, the present invention utilizes a multi-level parallel rendering method to effectively overcome the rendering bottlenecks of high-dimensional data and graphics encountered by traditional methods in the problem of parallel rendering of high-dimensional parallax images, thereby greatly improving rendering efficiency. At the same time, the present invention utilizes a multi-level parallel rendering method to give full play to the parallel computing capabilities of the GPU, greatly reducing the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving communication and computing efficiency.

Description

A multi-level parallel rendering method and system for light field images

Technical Field

The invention relates to a multi-level parallel rendering method and system for a light field image, belonging to the technical field of image rendering.

Background Art

Light field display technology requires a large amount of light field images as display materials. How to quickly and effectively generate high-resolution dynamic light field image information to achieve high-quality three-dimensional display is still an important issue in the field. Among various light field information acquisition methods, the method of using computers to generate light field image information has the advantages of simple equipment, flexible parameter control, and the ability to record light field image data of virtual objects. Moreover, this method can be effectively compatible with various computer graphics technologies, so it has received more and more attention.

The technology of computer-generated light field images can be divided into image-based rendering technology and model-based rendering technology. Image-based rendering technology uses image data of known viewpoints to generate light field images of new viewpoints through reprojection, interpolation, deformation and other methods. Model-based rendering technology requires input of three-dimensional model data and then generates images of new viewpoints through a virtual camera array and a corresponding graphics rendering pipeline. This method can obtain higher quality light field images.

However, traditional model-based computer-generated light field image technology usually requires the use of a pinhole camera model to render each camera's image one by one. The process is shown in Figure 1. The program first reads the 3D model data and then enters the camera-by-camera loop rendering step. Each loop requires the 3D data to be transformed into a model view, perspective projection, viewport, and rasterized according to the camera parameters. This rendering method has a lot of redundant calculations and is inefficient until all cameras are rendered.

Therefore, the current rendering technology, whether it is image-based or model-based, generally has low rendering efficiency, and there are still great challenges in achieving fast rendering of large amounts of data and high-quality light field images.

The information disclosed in this Background Art is only for understanding the background of the inventive concept and therefore it may include information that does not constitute the prior art.

Summary of the invention

In view of the above problem or one of the above problems, an object of the present invention is to provide a multi-level parallel rendering method for a light field image, by constructing a first-level rendering model, a second-level rendering model, and a third-level rendering model to obtain a light field image, which can realize three-level parallel rendering, thereby effectively reducing redundant calculations and improving rendering efficiency, thereby realizing rapid rendering of large data volume and high-quality light field images.

In view of the above problem or one of the above problems, the second object of the present invention is to provide a multi-level parallel rendering method and system for light field images, which can effectively overcome the rendering bottleneck of high-dimensional data and graphics encountered in the traditional method in the parallel rendering problem of high-dimensional parallax images, and greatly improve the rendering efficiency; and can give full play to the parallel computing capability of the GPU, and greatly reduce the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving the communication and computing efficiency.

In response to the above problem or one of the above problems, the third object of the present invention is to provide a multi-level parallel rendering method and system for light field images, which can quickly calculate light field images with complex lighting, shadows, textures and other effects, and greatly improve the realism of light field images.

To achieve one of the above purposes, the first technical solution of the present invention is:

A multi-level parallel rendering method for light field images, comprising the following contents:

Obtaining three-dimensional model data and camera array parameters of light field images;

Using the pre-built first-level rendering model, according to the camera array parameters, the 3D model data is copied, and the 3D model data is transformed according to the camera parameters of each row or each column to obtain the transformed 3D model data, thereby realizing the first-level parallel rendering;

Through the pre-built second-level rendering model, based on perspective correlation, the transformed three-dimensional model data is used to render the views at multiple viewpoints in parallel to obtain the polar plane image, thereby realizing the second-level parallel rendering;

Using a pre-built third-level rendering model, the polar plane image is parallelly calculated at the data processing level based on parallel pixel remapping to obtain a light field image, thereby realizing third-level parallel rendering and completing multi-level parallel rendering of the light field image.

After continuous exploration and experiments, the present invention obtains light field images by constructing a first-level rendering model, a second-level rendering model, and a third-level rendering model, and can realize three-level parallel rendering, thereby effectively reducing redundant calculations and improving rendering efficiency, thereby realizing rapid rendering of large data volumes and high-quality light field images.

Furthermore, the present invention utilizes a multi-level parallel rendering method to effectively overcome the high-dimensional data and graphics rendering bottlenecks encountered by traditional methods in the high-dimensional parallax image parallel rendering problem, thereby greatly improving the rendering efficiency. At the same time, the present invention utilizes a multi-level parallel rendering method to fully utilize the parallel computing capabilities of the GPU, greatly reducing the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving the communication and computing efficiency.

As the preferred technical measures:

The method to obtain 3D model data and camera array parameters is as follows:

Using a pre-built rendering pipeline model, read the three-dimensional model data stored in a hard disk or a memory, wherein the three-dimensional model data includes position coordinates and/or index coordinates and/or texture coordinates and/or normal vectors of model vertices;

Using the rendering pipeline model, the viewpoint parameters of the camera array are set according to the light field acquisition requirements;

The viewpoint parameters include the number of viewpoints and/or the viewpoint interval and/or the viewpoint arrangement and/or the single viewpoint resolution and/or the camera row and column information and/or the zero parallax plane; and the models, view parameters and projection matrices of the cameras at the left and right ends of each row or the cameras at the top and bottom ends of each column are calculated according to the viewpoint parameters to obtain the camera array parameters.

As the preferred technical measures:

The method of implementing the first-level parallel rendering using the first-level rendering model is as follows:

Step 11, obtaining three-dimensional model data, which includes position coordinates and/or index coordinates and/or texture coordinates and/or normal vectors of model vertices;

Step 12, using a pre-built vertex shader to process the three-dimensional model data to obtain triangle primitive data;

Step 13, according to the triangle primitive data and the camera array parameters, using the pre-built instantiation model to copy a set of triangle primitive data for each row or column of cameras;

Step 14, based on the different vertical or horizontal parallaxes corresponding to cameras in different rows or columns, and according to the transformation matrices of the leftmost and rightmost cameras in each row of cameras or the topmost and bottommost cameras in each column of cameras, the triangle primitive data are transformed into the screen coordinate system to obtain the transformed three-dimensional model data, and the three-dimensional model data corresponding to multiple rows or multiple columns are sent to the second-level rendering in parallel to realize the parallel rendering of the first level.

As the preferred technical measures:

The method of implementing the second-level parallel rendering through the second-level rendering model is as follows:

Step 21, performing parallel triangle vertex sorting processing, parallel prefix sum calculation processing and parallel triangle slicing processing on the three-dimensional model data to obtain triangle slices; Step 22, based on the correlation, using a pre-built parallel processing unit to convert the triangle slices into polygonal primitives, and rendering each triangle slice in parallel at all viewpoint positions in the same row or/and column camera in a polar plane coordinate system to obtain polygonal primitives of the triangle slices; the correlation includes geometric correlation, perspective correlation, texture correlation and illumination correlation;

Step 23, rasterizing the views of the polygonal primitives of the triangle slice at all viewpoint positions in the same row or/and column of cameras through a pre-built parallel polygon rasterization unit to obtain an extreme plane image of the same row or column of cameras;

Alternatively or alternatively, the method for implementing the third-level parallel rendering using the third-level rendering model is as follows:

Step 31, obtaining an extreme plane image;

Step 32, according to the correlation between illumination and texture, the polar plane image is shaded using a pre-built fragment shader to obtain a shaded image;

Step 33, outputting pixel data of the polar plane image according to the colored image;

Step 34 , performing parallel pixel reorganization on the pixel data to form a final light field image.

As the preferred technical measures:

The method of parallel triangle slicing processing includes the following:

Step 221, obtaining a triangle and its three vertex coordinates in the three-dimensional model data, wherein the three vertices of the triangle are v ₁ (x ₀ , y ₀ ), v ₂ (x ₁ , y ₁ ), and v ₃ (x ₂ , y ₂ );

Step 222, slicing the triangle along the direction of the screen scan line, where the triangle intersects with each screen scan line to obtain two intersection points _va and _vb of a slice;

Step 223, the coordinates _va ( _xa , _ya ) and _vb ( _xb , _yb ) of the two intersection points _va and _vb of the screen scan line and the triangle are obtained by linear interpolation between the three known vertex coordinates. The corresponding x-direction coordinates are as follows:

x _b =(1-b)*x ₀ +b*x ₂ y ₀ ≤y≤y ₂

Among them, a and b are interpolation parameters, and different scan lines have different interpolation parameters:

b＝(yy ₀ )/(y ₂ -y ₀ ) y ₀ ≤y≤y ₂

Step 224, interpolating the texture coordinates and normal vector data of the intersection point to obtain the vertex data of the slice, and storing it for subsequent data processing;

Or/and, a method for parallel triangle slicing processing includes two levels of parallel processing; the first level uses different thread blocks to parallelly process the slicing process of different triangles; the second level uses different threads in the thread block to parallelly process the slicing calculation in a triangle, and finally obtains all the slicing data of all the triangles;

Before executing the parallel scanning and slicing algorithm, the position data of the three vertices of the triangle satisfy the condition y ₀ ≤y ₁ ≤y ₂ ;

The triangle vertex sorting processing method uses threads of a kernel function to arrange the vertex data of a triangle in parallel to satisfy the condition of y ₀ ≤y ₁ ≤y ₂ .

As the preferred technical measures:

The method for parallel prefix sum calculation processing is as follows:

Get the slice data of each triangle;

Use the prefix sum algorithm to calculate the prefix sum of the slice data, obtain the memory space required to store the slice data and the position of each slice in the global memory, so as to save the storage space of the slice data;

The parallel prefix sum algorithm uses each thread to perform a sum operation on every two slice quantity elements, which includes the following: the first step is to perform a reduction sum to obtain the partial sum of the slice data; the second step is to use an inverted tree to distribute the partial sum to obtain the prefix sum of the slice quantity;

Alternatively, the parallel prefix sum algorithm is a parallel prefix sum algorithm with a multi-level scanning structure, and its implementation steps are as follows:

(1) First, the entire slice quantity data is divided into multiple data blocks and scanned in parallel. After scanning, only the results of scanning in each block are retained in the result array;

(2) The last number of each data block is saved in an auxiliary array S of length m, and the auxiliary array S is scanned;

(3) The first m-1 elements of the auxiliary array S are added to each element of the last m-1 scan blocks of the result array. This algorithm further reduces the time proportion of data processing in the rendering process, thereby improving the overall processing efficiency.

As the preferred technical measures:

Parallel processing units for converting triangle slices into polygon primitives, including the following:

Get the view of a triangle slice taken at the leftmost and rightmost viewpoints of a row of cameras;

For a triangle slice v ₀ v ₁ , the line segment S _L is recorded in the view of the left viewpoint, and the line segment S _R is recorded in the view of the right viewpoint;

According to the correlation, all the images formed by the scan lines on this row of cameras can be expressed in the EPI coordinate system as a quadrilateral v _L0 v _L1 v _R1 v _R0 consisting of two different slopes and two parallel lines; the two parallel edge segments are line segment _SL and line segment _SR ; the slopes of the two straight line segments are determined by the depth of the object point _v0 and _v1 and the focal length of the camera, respectively, and the two endpoints of the two straight line segments are determined by the coordinates of the images of the points recorded by the left and right endpoint cameras; this quadrilateral is a polygonal primitive of a triangular slice; each triangular slice is generated by the corresponding thread block in parallel to generate multiple polygonal primitives of triangular slices;

or/and, the parallel polygon rasterization unit performs rasterization operation on the sliced polygon primitives using the rendering pipeline, thereby realizing parallel rendering of one-dimensional multi-viewpoint image information of scan lines v ₀ v ₁ in the EPI coordinate system;

Or/and, it also includes lighting processing for the sliced polygonal primitives, the specific contents are as follows:

Based on the pre-built fragment shader, the lighting effects of the sliced polygon primitives are rendered using the correlation of lighting; and the Phone lighting model is used to achieve isotropic and anisotropic lighting effects;

Taking specular light as an example, the specular light values of the four endpoints of the polygon primitive are S _L0 , S _L1 , S _R0 , and S _R1 , which are obtained by the following formula:

specularResult＝lightColor*specularMaterial*pow(max(0,V·R),shiness)

Where specularResult refers to the specular lighting value of the sliced polygonal primitive surface, lightColor refers to the color of the lighting, specularMaterial refers to the specular material of the sliced polygonal primitive surface, V refers to the view vector of the viewpoint, R refers to the reflection vector of the incident light vector, and shines refers to the specular reflectivity. The pow() function is used to calculate the power of a number. Its basic syntax is pow(base, exponent), where base refers to the base and exponent refers to the exponent.

Finally, the result of the specular lighting of the polygonal primitive is obtained by interpolating the barycentric coordinates of the fragment shader.

To achieve one of the above purposes, the second technical solution of the present invention is:

A multi-level parallel rendering system for light field images, comprising a rendering pipeline and rendering hardware;

The rendering pipeline is used to implement multi-level parallel rendering, which includes an application program, an application program interface, a general parallel computing architecture and an operating system;

The rendering hardware is used to provide the hardware resources required to run the rendering pipeline, including memory, CPU, GPU and video memory;

The operating system is used to control applications, call application program interfaces and general parallel computing frameworks;

The application, application program interface and general parallel computing architecture drive the CPU, GPU, memory and video memory, and when executed, implement the multi-level parallel rendering method of a light field image as described above, completing the multi-level parallel rendering from three-dimensional model data to light field images.

By setting up a rendering pipeline and rendering hardware, the present invention can effectively overcome the rendering bottleneck of high-dimensional data and graphics encountered by traditional methods in the problem of parallel rendering of high-dimensional parallax images, and greatly improve the rendering efficiency. At the same time, the present invention can give full play to the parallel computing capability of the GPU, greatly reduce the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving the communication and computing efficiency. Furthermore, the present invention can quickly calculate light field images with complex lighting, shadows, textures and other effects, and greatly improve the realism of the light field images.

To achieve one of the above purposes, the third technical solution of the present invention is:

A multi-level parallel rendering method for light field images, using the above-mentioned multi-level parallel rendering system for light field images, comprises the following steps:

The first step is to set the world coordinates XYZ, where the camera array is located in the XY plane; the 3D model is located in front of the camera array; when all cameras complete shooting the 3D model, the light field image is recorded;

In the second step, the rendering pipeline reads the 3D model data from the memory through the CPU and transmits it to the vertex shader on the GPU side;

The third step is to use the vertex shader to assemble the vertex data in the 3D model data into triangle primitives and pass them into the geometry shader. At the same time, the parameters of the camera array are set from the CPU side and passed into the geometry shader on the GPU side.

The fourth step is to use the instantiation function to copy a set of triangle primitives for each column or row of cameras;

Step 5: transform the triangle primitives according to the transformation matrices of the top and bottom cameras in each column or the left and right cameras in each row to obtain transformed three-dimensional model data;

Step 6: Map the transformed image metadata to the CUDA space using the interoperability between the CUDA and OpenGL.

Step 7: Perform vertex parallel sorting calculation, parallel prefix sum calculation and parallel triangle slicing calculation in the general parallel computing framework CUDA space to obtain triangle slicing data;

Step 8: Map the triangle slice data to the vertex shader by using the interoperation between the general parallel computing framework CUDA and the application program interface OpenGL; assemble the vertex data in the triangle slice data into the polygon primitives of the slice by using the vertex shader and correlation;

In the ninth step, the sliced polygonal primitives are rasterized and colored in the fragment shader according to the correlation of lighting and texture to obtain a polar plane image;

In the tenth step, a light field image is obtained based on the pixel data of the polar plane image through parallel pixel remapping.

The present invention utilizes a multi-level parallel rendering method to effectively overcome the high-dimensional data and graphics rendering bottlenecks encountered by traditional methods in the parallel rendering of high-dimensional parallax images, thereby greatly improving rendering efficiency. At the same time, the present invention utilizes a multi-level parallel rendering method to fully utilize the parallel computing capabilities of the GPU, greatly reducing the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving communication and computing efficiency. Furthermore, the present invention utilizes a multi-level parallel rendering method to quickly calculate light field images with complex lighting, shadows, textures and other effects, greatly improving the realism of the light field images.

To achieve one of the above purposes, the fourth technical solution of the present invention is:

A multi-level parallel rendering method for light field images comprises the following steps:

Using the pre-built first-level rendering model, parallax images of different dimensions are rendered in parallel;

Through the pre-built second-level rendering model, different viewpoint images of the same dimension are rendered in parallel;

A pre-built third-level rendering model is used to perform parallel computing at the data processing level.

To achieve one of the above purposes, the fifth technical solution of the present invention is:

An electronic device comprising:

one or more processors;

A storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-level parallel rendering method of light field images.

To achieve one of the above purposes, the sixth technical solution of the present invention is:

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned multi-level parallel rendering method for light field images.

Compared with the prior art solutions, the present invention has the following beneficial effects:

After continuous exploration and experiments, the present invention obtains light field images by constructing a first-level rendering model, a second-level rendering model, and a third-level rendering model, and can realize three-level parallel rendering, thereby effectively reducing redundant calculations and improving rendering efficiency, thereby realizing rapid rendering of large data volumes and high-quality light field images.

Furthermore, the present invention utilizes a multi-level parallel rendering method to effectively overcome the high-dimensional data and graphics rendering bottlenecks encountered by traditional methods in the high-dimensional parallax image parallel rendering problem, thereby greatly improving the rendering efficiency. At the same time, the present invention utilizes a multi-level parallel rendering method to fully utilize the parallel computing capabilities of the GPU, greatly reducing the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving the communication and computing efficiency.

Furthermore, the present invention utilizes a multi-level parallel rendering method to quickly calculate light field images with complex lighting, shadow, texture and other effects, thereby greatly improving the realism of the light field images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a single-view cyclic rendering method;

FIG2 is a flow chart of a multi-level parallel rendering method according to the present invention;

FIG. 3 is a schematic diagram of a light field image acquisition system of the present invention.

FIG4 is a processing flow chart of the rendering pipeline of the present invention;

FIG5 is a structural diagram of a multi-level parallel rendering system of the present invention;

FIG6 is a flow chart of the parallel triangle slicing algorithm and the related vertex sorting algorithm of the present invention;

FIG7 is a flow chart of a parallel prefix sum algorithm of the present invention;

FIG8 is a flow chart of the slice polygon primitive generation algorithm of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or/and" as used herein includes any and all combinations of one or more of the associated listed items.

A first specific embodiment of the multi-level parallel rendering method of light field images of the present invention:

A multi-level parallel rendering method for light field images, comprising the following contents:

Obtaining three-dimensional model data and camera array parameters of light field images;

Using the pre-built first-level rendering model, according to the camera array parameters, the 3D model data is copied, and the 3D model data is transformed according to the camera parameters of each row or each column to obtain the transformed 3D model data, thereby realizing the first-level parallel rendering;

Through the pre-built second-level rendering model, based on perspective correlation, the transformed three-dimensional model data is used to render the views at multiple viewpoints in parallel to obtain the polar plane image, thereby realizing the second-level parallel rendering;

Using a pre-built third-level rendering model, the polar plane image is parallelly calculated at the data processing level based on parallel pixel remapping to obtain a light field image, thereby realizing third-level parallel rendering and completing multi-level parallel rendering of the light field image.

A second specific embodiment of the multi-level parallel rendering method of light field images of the present invention:

A multi-level parallel rendering method for light field images includes the following levels of parallel rendering process:

First level: parallel rendering of parallax images of different dimensions;

Second level: parallel rendering of images from different viewpoints in the same dimension;

The third level: parallel computing at the data processing level.

The first level of parallel rendering includes parallel rendering in two dimensions: horizontal parallax and vertical parallax. The second level of parallel rendering includes parallel rendering of different viewpoint images in the horizontal parallax dimension or parallel rendering of different viewpoint images in the vertical parallax dimension. The third level of parallel rendering includes parallel processing of 3D model data and parallel processing of pixel data.

The first level of parallel rendering includes instantiation operations performed row by row or column by column on the model; and parallel rendering of instantiation data of different rows or columns.

The instantiation operation includes an instantiation operation performed by combining an observation matrix and a projection matrix corresponding to the vertical parallax of a row of cameras or the horizontal parallax of a column of cameras.

The second level of parallel rendering includes utilizing the correlation of the three-dimensional model to implement multi-viewpoint parallel rendering including disparity information, illumination information and texture information in the same dimension.

The correlation includes perspective projection correlation or orthogonal projection correlation of information such as geometry, parallax, illumination and texture.

The parallel processing of the model data of the third level includes parallel triangle vertex sorting calculation, parallel prefix sum calculation, parallel triangle scan slicing calculation, rasterization, etc.

The parallel processing of the pixel data at the third level includes parallel pixel remapping. The parallel pixel remapping completes the conversion of the epipolar plane image (EPI) to the light field image.

Therefore, the present invention can achieve fast rendering of large data volumes and high-quality light field images. The present invention uses a multi-level parallel rendering method to effectively overcome the rendering bottleneck of high-dimensional data and graphics encountered by traditional methods in the problem of parallel rendering of high-dimensional parallax images, thereby greatly improving the rendering efficiency. Furthermore, the present invention uses a multi-level parallel rendering method to give full play to the parallel computing capabilities of the GPU, greatly reducing the communication between the CPU and the GPU required in the traditional light field rendering method, thereby improving the communication and computing efficiency. At the same time, the present invention uses a multi-level parallel rendering method to quickly calculate light field images with complex lighting, shadows, textures and other effects, greatly improving the realism of the light field images.

As shown in FIG. 2 , a third specific embodiment of the multi-level parallel rendering method of a light field image of the present invention is as follows:

A multi-level parallel rendering method for light field images includes file reading, three-level parallel rendering and light field image output.

The method first reads the three-dimensional model data, and then sets the parameters of the camera array, which includes n rows and m columns of cameras.

Then use the instantiation technology to copy a set of model data for each row of cameras.

Then, the 3D model is transformed using the parameters corresponding to the cameras at the left and right ends of each row, and the 3D model is converted to the screen coordinate system.

Cameras in different rows correspond to different vertical parallaxes. The program inputs the transformed data into parallel horizontal parallax processing pipelines to achieve the first level of parallel rendering.

The horizontal disparity processing pipeline uses the perspective correlation of geometry, disparity, texture, lighting and other information to render the views of the model at all viewpoints in the same row of cameras in parallel, thereby achieving the second level of parallel rendering.

The parallel rendering of the model in the second level is achieved by parallel processing of each triangle in the model, wherein the model includes k triangles. The operations include parallel triangle vertex sorting, parallel prefix sum calculation, parallel triangle slicing, parallel slice polygon primitive generation, parallel polygon rasterization, etc. The parallel slicing polygon primitive generation process will render the views of each triangle slice at all viewpoints in the same row of cameras in parallel in the polar plane coordinate system. The parallel polygon rasterization will rasterize the views of all slices at all viewpoints in the same row of cameras.

Finally, the rasterized pixel data is subjected to parallel pixel reorganization to form a final light field image.

The above-mentioned parallel data processing constitutes the third level of parallel rendering, including parallel processing of model data and parallel processing of pixel data.

The first specific embodiment of the multi-level parallel rendering system of the light field image of the present invention:

A multi-level parallel rendering system for light field images includes a rendering pipeline and rendering hardware.

The rendering pipeline includes the following processes:

1. Read 3D model data and assemble graphics elements.

2. Set the parameters of the camera array.

3. Instantiate 3D model data for each row or column of cameras according to camera parameters.

4. Use the instantiated model data to render light field images of cameras in different rows or columns in parallel.

5. Use correlation to render the polar plane images of the cameras in the same row or column of the model in parallel.

6. Convert the epipolar plane image to a light field image using parallel pixel remapping.

The rendering pipeline also includes software such as applications, application programming interfaces, general parallel computing architectures, operating systems, etc. required to execute the above processes.

In the process 1, the rendering pipeline reads the 3D model data from the hard disk or memory and passes it to the vertex shader. The vertex shader assembles the vertex data into triangles and then passes them to the geometry shader. The 3D model data includes the position coordinates, index coordinates, texture coordinates, normal vectors and other data of the model vertices.

In the process 2, the rendering pipeline sets the number of viewpoints, viewpoint interval, viewpoint arrangement, single viewpoint resolution, zero parallax plane and other parameters of the camera array according to the light field acquisition requirements, and calculates the model, observation, and projection matrices of the cameras at the left and right ends of each row or the upper and lower ends of each column based on these parameters, and passes them into the geometry shader.

In the process 3, a set of 3D model data is copied for each row or column of cameras using the instantiation function, and the vertices are transformed using the model matrix, observation matrix, projection matrix and viewport matrix in process 2. The transformed vertex data is used for subsequent program calls.

The process 4 inputs the transformed instantiation data of different rows or columns into a plurality of sub-pipelines composed of the processes 5 and 6 in parallel.

The operation object of the process 5 includes the triangle data in the instantiation data, and the process 5 uses the correlation to render the images of the triangles in parallel at all viewpoints in a row or a column of cameras in the polar plane coordinate system.

The process 5 specifically includes sorting calculation of triangle vertices, parallel prefix sum calculation, parallel triangle scanning slice calculation, polygon primitive generation and rasterization calculation of triangle slices in the polar plane coordinate system.

The process 5 processes multiple triangle data in the model in parallel. The process 5 processes multiple scan slice data in a triangle in parallel. The process 5 generates and rasterizes multiple polygonal primitives in parallel in the polar plane coordinate system, and finally forms a polar plane image of the three-dimensional model.

The process 6 processes a plurality of polar plane images in parallel, and the process processes a plurality of pixels in one polar plane image in parallel.

The rendering hardware includes the hard disk, memory, CPU, GPU, video memory, etc. required to run the above rendering pipeline.

A second specific embodiment of the multi-level parallel rendering system of light field images of the present invention:

First, the world coordinates XYZ are defined as shown in Figure 3, where the camera array is located in the XY plane. The 3D model is located in front of the camera array. When all cameras complete shooting the 3D model, the light field image is recorded.

As shown in Figure 4, the rendering pipeline reads 3D model data from the hard disk through the CPU and passes it to the vertex shader on the GPU side, and then uses the vertex shader to assemble the vertex data into triangle primitives and pass them to the geometry shader. Then, the parameters of the camera array are set from the CPU side and passed to the geometry shader on the GPU side.

Use instancing to replicate a set of primitives for each column of cameras.

The metadata is then transformed according to the transformation matrices of the top and bottom cameras in each column of cameras.

Then, the transformed image metadata is mapped to the CUDA space by using the interoperability between the general parallel computing framework CUDA and the application program interface OpenGL.

Parallel vertex sorting calculations, parallel prefix sum calculations, and parallel triangle slicing calculations are performed in CUDA space.

Then, the triangle slice data is mapped to the vertex shader by using the interoperation between the general parallel computing framework CUDA and the application program interface OpenGL. Then, the slice vertex data is assembled into slice polygon primitives by using the vertex shader and correlation.

The tile polygon primitives are then rasterized and shaded in the fragment shader based on lighting and texture dependencies.

The pixel data of the output epipolar plane image (EPI) is obtained by shading and then parallel pixel remapping.

The rendering pipeline also includes software such as applications, application programming interfaces, general parallel computing architectures, operating systems, etc. required to execute the above processes.

The rendering hardware in this embodiment includes a computer system composed of hardware such as a hard disk, a CPU, a memory, a GPU, a video memory, and a display.

As shown in FIG5 , the rendering system of this embodiment is embedded in a computer system, and the computer system includes an operating system to provide a running environment for the rendering system. The rendering system includes an application program, and the application program calls the multi-level parallel rendering pipeline, and the rendering pipeline calls an application program interface and a general parallel computing framework. The application program, the application program interface and the general parallel computing framework drive computing hardware such as a CPU, a GPU, a memory and a video memory to complete the rendering from 3D model data to a light field image.

As shown in FIG6 , a specific embodiment of the parallel triangle slicing algorithm and the related vertex sorting algorithm of the present invention is as follows:

The parallel triangle slicing algorithm and the related vertex sorting algorithm of the present invention include the following contents:

Take the triangle on the left side of Figure 6 as an example. The three vertices of the triangle are v ₁ , v ₂ , and v ₃ . In order to render the triangle, it needs to be sliced along the direction of the screen scan line. Given the coordinates of the three vertices, the coordinates of the two intersection points of the scan line and the triangle, v _a and v _b , can be obtained by linear interpolation between the known coordinates of the three vertices. Taking the x direction as an example, its coordinates are:

x _b =(1-b)*x ₀ +b*x ₂ y ₀ ≤y≤y ₂

Among them, a and b are interpolation parameters, and different scan lines have different interpolation parameters:

b＝(yy ₀ )/(y ₂ -y ₀ ) y ₀ ≤y≤y ₂

When a triangle intersects with each scan line, two vertex coordinates _va and _vb of a slice are obtained. Similarly, the texture coordinates, normal vector and other data of the vertex are also interpolated in the same way, and then the vertex data of these slices are stored for subsequent data processing.

As shown on the right side of Figure 6, the parallel triangle slicing algorithm includes two levels of parallel processing. The first level uses different CUDA thread blocks to parallelly process the slicing process of different triangles. The second level uses different threads in the thread block to parallelly process the slicing calculation in a triangle, and finally obtains all the slicing data of all triangles.

Before executing the parallel scan slicing algorithm, it is necessary to ensure that the three vertex position data of the triangle meet the condition y ₀ ≤y ₁ ≤y _2. The parallel triangle vertex sorting algorithm uses threads of the CUDA kernel function to arrange the vertex data of the triangle in parallel to meet the condition y ₀ ≤y ₁ ≤y ₂ .

As shown in FIG. 7 , a specific embodiment of the parallel prefix sum algorithm of the present invention is as follows:

After obtaining the slice data of each triangle, the prefix sum algorithm is used to calculate the prefix sum of the number of slices, thereby obtaining the memory space required to store the slice data and the position of each slice in the global memory, which can save the storage space of the slice data. As shown on the left side of Figure 7. The parallel prefix sum algorithm uses each thread to perform a sum operation on every two slice number elements. In the first step of the reduction and summation stage, the partial sum of the slice data is obtained. In the second step, the inverted tree is used to distribute the partial sum to obtain the prefix sum of the number of slices.

As shown on the right side of Figure 7, this embodiment also provides a parallel prefix sum algorithm with a multi-level scanning structure, and its implementation steps are as follows: (1) First, the entire slice quantity data is divided into multiple data blocks, and parallel scanning processing is performed. After scanning, only the results of scanning in each block are retained in the result array. (2) The last number of each block is retained in an auxiliary array S of length m, and S is scanned. (3) The first m-1 elements of S are added to each element of the last m-1 scan blocks of the result array. This algorithm further reduces the time proportion of data processing in the rendering process, thereby improving the overall processing efficiency.

As shown in FIG8 , a specific embodiment of the slice polygon primitive generation algorithm of the present invention is as follows:

The slice polygon primitive generation algorithm of the present invention includes the following contents:

(a) in FIG8 shows the views of a triangle in the model taken at the leftmost and rightmost viewpoints of a row of cameras. For a scan line slice v ₀ v ₁ , the line segment _SL is recorded on the view of the left end viewpoint, and the line segment _SR is recorded on the view of the right end viewpoint. As shown in (b) in FIG8 , based on the correlation, all the images formed by the scan line on this row of cameras can be represented in the EPI coordinate system as a quadrilateral v _L0 v _L1 v _R1 v _R0 consisting of two different slopes and two parallel lines. The two parallel edge segments are _SL and _SR . The slopes of the two straight line segments are determined by the depth of the object point v ₀ and v ₁ and the focal length of the camera, respectively, and the two endpoints of the two straight line segments are determined by the coordinates of the images of the point recorded by the left and right endpoint cameras. This quadrilateral is the slice polygon primitive. Each slice is generated in parallel by the corresponding thread block. The rendering pipeline can realize parallel rendering of one-dimensional multi-viewpoint image information of scan line v ₀ v ₁ in the EPI coordinate system by rasterizing sliced polygon primitives.

In order to improve the realism of the image, it is necessary to render the lighting effect of the model in the fragment shader using the correlation of lighting. In this embodiment, the Phone lighting model is adopted, and the anisotropic specular lighting implementation process is shown in (c) of FIG8 . is the lighting direction of the two end points of the triangle slice, is the normal direction of the two end points, is the direction of the specular light reflection at the two end points. The position information of the middle camera can be obtained by rasterization interpolation of the slice polygon primitive constructed in (b) of Figure 8. At the same time, the line of sight directions of the left and right camera viewpoints relative to the two end points can be obtained respectively. The specular illumination values of the four endpoints in (d) of FIG8 are S _L0 , S _L1 , S _R0 , and S _R1 , which can be obtained by the following formula:

specularResult＝lightColor*specularMaterial*pow(max(0,V·R),shiness)

Among them, lightColor refers to the color of the light, specularMaterial refers to the specular material of the object surface, V refers to the vector observed by the viewpoint, and R refers to the reflection vector of the incident light vector. Finally, the specular lighting result of the polygon primitive is obtained by interpolating the barycentric coordinates of the fragment shader.

An embodiment of a device applying the method of the present invention:

An electronic device comprising:

one or more processors;

A storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-level parallel rendering method of light field images.

A computer medium embodiment using the method of the present invention:

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned multi-level parallel rendering method for light field images.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, and computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present application is described by flowcharts or/and block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process or/and box in the flowchart or/and block diagram and the combination of the processes or/and boxes in the flowchart or/and block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart or/and one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims (10)

1. A multi-level parallel rendering method for light field images, characterized in that: Includes the following: Obtaining three-dimensional model data and camera array parameters of light field images; Using the pre-built first-level rendering model, according to the camera array parameters, the 3D model data is copied, and the 3D model data is transformed according to the camera parameters of each row or each column to obtain the transformed 3D model data, thereby realizing the first-level parallel rendering; Through the pre-built second-level rendering model, based on perspective correlation, the transformed three-dimensional model data is used to render the views at multiple viewpoints in parallel to obtain the polar plane image, thereby realizing the second-level parallel rendering; Using a pre-built third-level rendering model, the polar plane image is parallelly calculated at the data processing level based on parallel pixel remapping to obtain a light field image, thereby realizing third-level parallel rendering and completing multi-level parallel rendering of the light field image. 2. The multi-level parallel rendering method of a light field image according to claim 1, characterized in that: The method to obtain 3D model data and camera array parameters is as follows: Using a pre-built rendering pipeline model, read the three-dimensional model data stored in a hard disk or a memory, wherein the three-dimensional model data includes position coordinates and/or index coordinates and/or texture coordinates and/or normal vectors of model vertices; Using the rendering pipeline model, the viewpoint parameters of the camera array are set according to the light field acquisition requirements; The viewpoint parameters include the number of viewpoints and/or the viewpoint interval and/or the viewpoint arrangement and/or the single viewpoint resolution and/or the camera row and column information and/or the zero parallax plane; and the models, view parameters and projection matrices of the cameras at the left and right ends of each row or the cameras at the top and bottom ends of each column are calculated according to the viewpoint parameters to obtain the camera array parameters. 3. The multi-level parallel rendering method of a light field image according to claim 2, characterized in that: The method of implementing the first-level parallel rendering using the first-level rendering model is as follows: Step 11, obtaining three-dimensional model data, which includes position coordinates and/or index coordinates and/or texture coordinates and/or normal vectors of model vertices; Step 12, using a pre-built vertex shader to process the three-dimensional model data to obtain triangle primitive data; Step 13, according to the triangle primitive data and the camera array parameters, using the pre-built instantiation model to copy a set of triangle primitive data for each row or column of cameras; Step 14, based on the different vertical or horizontal parallaxes corresponding to cameras in different rows or columns, and according to the transformation matrices of the leftmost and rightmost cameras in each row of cameras or the topmost and bottommost cameras in each column of cameras, the triangle primitive data are transformed into the screen coordinate system to obtain the transformed three-dimensional model data, and the three-dimensional model data corresponding to multiple rows or multiple columns are sent to the second-level rendering in parallel to realize the parallel rendering of the first level. 4. The multi-level parallel rendering method of a light field image according to claim 1, characterized in that: The method of implementing the second-level parallel rendering through the second-level rendering model is as follows: Step 21, performing parallel triangle vertex sorting processing, parallel prefix sum calculation processing and parallel triangle slicing processing on the three-dimensional model data to obtain triangle slices; Step 22, based on the correlation, using a pre-built parallel processing unit to convert the triangle slices into polygonal primitives, and rendering each triangle slice in parallel at all viewpoint positions in the same row or/and column camera in a polar plane coordinate system to obtain polygonal primitives of the triangle slices; the correlation includes geometric correlation, perspective correlation, texture correlation and illumination correlation; Step 23, rasterizing the views of the polygonal primitives of the triangle slice at all viewpoint positions in the same row or/and column of cameras through a pre-built parallel polygon rasterization unit to obtain an extreme plane image of the same row or column of cameras; Alternatively or alternatively, the method for implementing the third-level parallel rendering using the third-level rendering model is as follows: Step 31, obtaining an extreme plane image; Step 32, according to the correlation between illumination and texture, the polar plane image is shaded using a pre-built fragment shader to obtain a shaded image; Step 33, outputting pixel data of the polar plane image according to the colored image; Step 34 , performing parallel pixel reorganization on the pixel data to form a final light field image. 5. The multi-level parallel rendering method of light field images according to claim 4, characterized in that: The method of parallel triangle slicing processing includes the following: Step 221, obtaining a triangle and its three vertex coordinates in the three-dimensional model data, wherein the three vertices of the triangle are v ₁ (x ₀ , y ₀ ), v ₂ (x ₁ , y ₁ ), and v ₃ (x ₂ , y ₂ ); Step 222, slicing the triangle along the direction of the screen scan line, where the triangle intersects with each screen scan line to obtain two intersection points _va and _vb of a slice; Step 223, the coordinates _va ( _xa , _ya ) and _vb ( _xb , _yb ) of the two intersection points _va and _vb of the screen scan line and the triangle are obtained by linear interpolation between the three known vertex coordinates. The corresponding x-direction coordinates are as follows: x _b =(1-b)*x ₀ +b*x ₂ y ₀ ≤y≤y ₂ Among them, a and b are interpolation parameters, and different scan lines have different interpolation parameters: b＝(yy ₀ )/(y ₂ -y ₀ ) y ₀ ≤y≤y ₂ Step 224, interpolating the texture coordinates and normal vector data of the intersection point to obtain the vertex data of the slice, and storing it for subsequent data processing; Or/and, a method for parallel triangle slicing processing includes two levels of parallel processing; the first level uses different thread blocks to parallelly process the slicing process of different triangles; the second level uses different threads in the thread block to parallelly process the slicing calculation in a triangle, and finally obtains all the slicing data of all the triangles; Before executing the parallel scanning and slicing algorithm, the position data of the three vertices of the triangle satisfy the condition y ₀ ≤y ₁ ≤y ₂ ; The triangle vertex sorting processing method uses threads of a kernel function to arrange the vertex data of a triangle in parallel to satisfy the condition of y ₀ ≤y ₁ ≤y ₂ . 6. The multi-level parallel rendering method of light field images according to claim 5, characterized in that: The method for parallel prefix sum calculation processing is as follows: Get the slice data of each triangle; Use the prefix sum algorithm to calculate the prefix sum of the slice data, and obtain the memory space required to store the slice data and the location of each slice in the global memory; The parallel prefix sum algorithm uses each thread to perform a sum operation on every two slice quantity elements, which includes the following: the first step is to perform a reduction sum to obtain the partial sum of the slice data; the second step is to use an inverted tree to distribute the partial sum to obtain the prefix sum of the slice quantity; Alternatively, the parallel prefix sum algorithm is a parallel prefix sum algorithm with a multi-level scanning structure, and its implementation steps are as follows: (1) First, the entire slice quantity data is divided into multiple data blocks and scanned in parallel. After scanning, only the results of scanning in each block are retained in the result array; (2) The last number of each data block is saved in an auxiliary array S of length m, and the auxiliary array S is scanned; (3) Add the first m-1 elements of the auxiliary array S to each element of the last m-1 scan blocks of the result array. 7. The multi-level parallel rendering method of light field images according to claim 6, characterized in that: Parallel processing units for converting triangle slices into polygon primitives, including the following: Get the view of a triangle slice taken at the leftmost and rightmost viewpoints of a row of cameras; For a triangle slice v ₀ v ₁ , the line segment S _L is recorded in the view of the left viewpoint, and the line segment S _R is recorded in the view of the right viewpoint; According to the correlation, all the images formed by the scan lines on this row of cameras can be expressed in the EPI coordinate system as a quadrilateral v _L0 v _L1 v _R1 v _R0 consisting of two different slopes and two parallel lines; the two parallel edge segments are line segment _SL and line segment _SR ; the slopes of the two straight line segments are determined by the depth of the object point _v0 and _v1 and the focal length of the camera, respectively, and the two endpoints of the two straight line segments are determined by the coordinates of the images of the points recorded by the left and right endpoint cameras; this quadrilateral is a polygonal primitive of a triangular slice; each triangular slice is generated by the corresponding thread block in parallel to generate multiple polygonal primitives of triangular slices; or/and, the parallel polygon rasterization unit performs rasterization operation on the sliced polygon primitives using the rendering pipeline, thereby realizing parallel rendering of one-dimensional multi-viewpoint image information of scan lines v ₀ v ₁ in the EPI coordinate system; Or/and, it also includes lighting processing for the sliced polygonal primitives, the specific contents are as follows: Based on the pre-built fragment shader, the lighting effects of the sliced polygon primitives are rendered using the correlation of lighting; and the Phone lighting model is used to achieve isotropic and anisotropic lighting effects; Taking specular light as an example, the specular light values of the four endpoints of the polygon primitive are S _L0 , S _L1 , S _R0 , and S _R1 , which are obtained by the following formula: specularResult＝lightColor*specularMaterial*pow(max(0,V·R),shiness) Where specularResult refers to the specular lighting value of the sliced polygonal primitive surface, lightColor refers to the color of the lighting, specularMaterial refers to the specular material of the sliced polygonal primitive surface, V refers to the view vector of the viewpoint, R refers to the reflection vector of the lighting incident vector, shines refers to the specular reflectivity, and the pow() function is used to calculate the power of a number; Finally, the result of the specular lighting of the polygonal primitive is obtained by interpolating the barycentric coordinates of the fragment shader. 8. A multi-level parallel rendering system for light field images, characterized in that: It has a rendering pipeline and rendering hardware; The rendering pipeline is used to implement multi-level parallel rendering, which includes an application program, an application program interface, a general parallel computing architecture and an operating system; The rendering hardware is used to provide the hardware resources required to run the rendering pipeline, including memory, CPU, GPU and video memory; The operating system is used to control applications, call application program interfaces and general parallel computing frameworks; The application, application program interface and general parallel computing architecture drive the CPU, GPU, memory and video memory, and when executed, implement a multi-level parallel rendering method for light field images as described in any one of claims 1 to 7, and complete multi-level parallel rendering from three-dimensional model data to light field images. 9. A multi-level parallel rendering method for light field images, characterized in that: A multi-level parallel rendering system for light field images as claimed in claim 8 comprises the following steps: The first step is to set the world coordinates XYZ, where the camera array is located in the XY plane; the 3D model is located in front of the camera array; when all cameras complete shooting the 3D model, the light field image is recorded; In the second step, the rendering pipeline reads the 3D model data from the memory through the CPU and transmits it to the vertex shader on the GPU side; The third step is to use the vertex shader to assemble the vertex data in the 3D model data into triangle primitives and pass them into the geometry shader. At the same time, the parameters of the camera array are set from the CPU side and passed into the geometry shader on the GPU side. The fourth step is to use the instantiation function to copy a set of triangle primitives for each column or row of cameras; Step 5: transform the triangle primitives according to the transformation matrices of the top and bottom cameras in each column or the left and right cameras in each row to obtain transformed three-dimensional model data; Step 6: Map the transformed image metadata to the CUDA space using the interoperability between the CUDA and OpenGL. Step 7: Perform vertex parallel sorting calculation, parallel prefix sum calculation and parallel triangle slicing calculation in the general parallel computing framework CUDA space to obtain triangle slicing data; Step 8: Map the triangle slice data to the vertex shader by using the interoperation between the general parallel computing framework CUDA and the application program interface OpenGL; assemble the vertex data in the triangle slice data into the polygon primitives of the slice by using the vertex shader and correlation; In the ninth step, the sliced polygonal primitives are rasterized and colored in the fragment shader according to the correlation of lighting and texture to obtain a polar plane image; In the tenth step, a light field image is obtained based on the pixel data of the polar plane image through parallel pixel remapping. 10. A multi-level parallel rendering method for light field images, characterized in that: The following steps are involved: Using the pre-built first-level rendering model, parallax images of different dimensions are rendered in parallel; Through the pre-built second-level rendering model, different viewpoint images of the same dimension are rendered in parallel; A pre-built third-level rendering model is used to perform parallel computing at the data processing level.