CN112738010B - Data interaction method and system, interaction terminal and readable storage medium - Google Patents

Abstract

The data interaction method and system, the interaction terminal and the readable storage medium, wherein the data interaction method comprises the following steps: acquiring a data stream to be played from a play control device in real time and playing and displaying the data stream in real time, wherein the data stream to be played comprises video data and interactive identifications, and each interactive identification is associated with a designated frame time of the data stream to be played; responding to triggering operation of an interaction identifier, and acquiring interaction data corresponding to the interaction identifier at a designated frame time, wherein the interaction data comprises multi-angle free view angle data; and based on the interaction data, performing image display of the multi-angle free view angles at the appointed frame time. By adopting the scheme, the interactive data can be acquired according to the triggering operation of the interactive identification in the playing process, and then multi-angle free view display is performed, so that the user interaction experience is improved.

Description

Data interaction method and system, interactive terminal, and readable storage medium

technical field

The embodiments of the present invention relate to the technical field of data processing, and in particular to a data interaction method and system, an interaction terminal, and a readable storage medium.

Background technique

With the continuous development of Internet technology, data transmission capability has been greatly improved, and users can watch videos through various terminal devices.

In various playback scenarios, users usually watch videos based on a fixed viewing angle, and users cannot freely switch viewpoints for viewing, and user experience needs to be improved.

Contents of the invention

The problem to be solved by the embodiments of the present invention is to provide a data interaction method and system, an interactive terminal, and a readable storage medium, which can freely switch viewpoint positions for interaction when watching a video.

The embodiment of the present invention discloses a data interaction method, which includes: acquiring data streams to be played from a playback control device in real time and performing real-time playback and display, the data streams to be played include video data and interaction identifiers, and each interaction identifier is related to the to-be-played data streams. The specified frame time of the playback data stream is associated; in response to a trigger operation on an interactive mark, the interactive data corresponding to the specified frame time of the interactive mark is obtained, and the interactive data includes multi-angle free view data; based on the interactive data , performing multi-angle free-view image display at the specified frame time.

Optionally, the multi-angle free viewing angle data is generated based on multiple received frame images corresponding to the specified frame time, and the multiple frame images are synchronously collected by the data processing device for multiple acquisition devices in the acquisition array. The video data stream is obtained by intercepting the specified frame time, and the multi-angle free view data includes pixel data, depth data, and parameter data of the multiple frame images, wherein the pixel data and depth data of each frame image There is a relationship between them.

Optionally, the data processing device intercepts the multiple video data streams synchronously collected by multiple collection devices in the collection array at the specified frame time, including: the data processing device intercepts the instruction based on the received video frame, Intercepting frame-level synchronized video frames at the specified frame time in the multiple video data streams.

Optionally, multiple acquisition devices in the acquisition array are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and the data processing equipment is placed in the on-site non-acquisition area or in the cloud.

Optionally, the interaction identifier is generated by the playback control device, and the playback control device generates a video frame associated with the corresponding time in the data stream to be played based on the frame time information of the intercepted video frame from the data processing device. interaction logo.

Optionally, the interaction data further includes at least one of the following: on-site analysis data, information data of the collection object, information data of equipment associated with the collection object, information data of items deployed on-site, and information data of logos displayed on-site .

Optionally, the data interaction method further includes: when an interaction end signal is detected, switching to the data stream to be played obtained in real time from the playback control device and performing real-time playback and presentation.

Optionally, when an interaction end signal is detected, switching to the data stream to be played obtained in real time from the playback control device and performing real-time playback display, including at least one of the following: when receiving an interaction end operation instruction, switching to The data stream to be played is obtained in real time from the playback control device and displayed in real time; when the image of the multi-angle free viewing angle at the specified frame time is detected to be displayed to the last image, switch to from the playback control device The data stream to be played is obtained in real time and displayed in real time.

Optionally, the multi-angle free viewing angle image display based on the interaction data includes: determining a virtual viewpoint according to the interactive operation, the virtual viewpoint being selected from a multi-angle free viewing range, and the multi-angle free viewing range In order to support the switching viewing range of the virtual viewpoint of the region to be viewed; displaying an image of viewing the region to be viewed based on the virtual viewpoint, the image is generated based on the interaction data and the virtual viewpoint.

The embodiment of the present invention also provides a data processing system, including: an acquisition array, a data processing device, a server, a playback control device, and an interactive terminal; wherein:

The acquisition array includes a plurality of acquisition devices, and the plurality of acquisition devices are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and are suitable for real-time synchronous acquisition of multiple video data streams, and real-time upload of video data streams To the data processing device; the data processing device, for the uploaded multiple video data streams, is adapted to intercept the multiple video data streams at a specified frame time according to the received video frame interception instruction to obtain the corresponding Multiple frame images at the specified frame time and frame time information corresponding to the specified frame time, and uploading the multiple frame images at the specified frame time and frame time information corresponding to the specified frame time to the server , sending the frame time information of the specified frame time to the playback control device; the server is adapted to receive the plurality of frame images and the frame time information uploaded by the data processing device, and based on the A plurality of frame images is used to generate interaction data for interaction, the interaction data includes multi-angle free view data, the interaction data is associated with the frame time information; the playback control device is adapted to determine the data stream to be played In the specified frame time corresponding to the frame time information uploaded by the data processing device, an interaction identifier associated with the specified frame time is generated, and the data stream to be played including the interaction identifier is transmitted to the interactive terminal; The interactive terminal is adapted to play and display the video containing the interactive logo in real time based on the received data stream to be played, and based on a trigger operation on the interactive logo, obtain the video stored in the server and corresponding to the specified frame Momentary interactive data for multi-angle free-view image display.

Optionally, the data processing device is adapted to intercept multiple video data streams synchronously collected by multiple collection devices in the collection array at the specified frame moment;

The server is adapted to generate the multi-angle free viewing angle data based on the received multiple frame images corresponding to the specified frame time, the multi-angle free viewing angle data including pixel data and depth data of the multiple frame images, And parameter data, wherein there is an association relationship between the pixel data and the depth data of each frame image.

Optionally, multiple acquisition devices in the acquisition array are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, the data processing equipment is placed in the on-site non-acquisition area or in the cloud, and the server is placed on-site Non-acquisition area, cloud or terminal.

Optionally, the playback control device is adapted to generate an interaction identifier associated with a video frame at a corresponding time in the data stream to be played based on the frame information time of the video frame intercepted by the data processing device.

Optionally, the interactive terminal is further adapted to switch to the data stream to be played acquired in real time from the playback control device and perform real-time playback and presentation when the interaction end signal is detected.

The embodiment of the present invention also provides a data processing system, including: an acquisition array, a data processing device, a playback control device, and an interactive terminal; wherein:

The acquisition array includes a plurality of acquisition devices, and the plurality of acquisition devices are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and are suitable for real-time synchronous acquisition of multiple video data streams, and real-time upload of video data streams To the data processing device; the data processing device, for the uploaded multiple video data streams, is adapted to intercept the multiple video data streams at a specified frame time according to the received video frame interception instruction to obtain the corresponding A plurality of frame images at the specified frame time and frame time information corresponding to the specified frame time, and sending the frame time information at the specified frame time to the playback control device; the playback control device is adapted to determine In the data stream to be played, at the specified frame time corresponding to the frame time information uploaded by the data processing device, an interaction identifier associated with the specified frame time is generated, and the data stream to be played containing the interaction identifier is transmitted to the The interactive terminal; the interactive terminal is adapted to play and display the video containing the interactive logo in real time based on the received data stream to be played, and obtain the corresponding video from the data processing device based on a trigger operation on the interactive logo. Based on the multiple frame images at the specified frame time of the interaction mark, and based on the multiple frame images, interactive data for interaction is generated, and then multi-angle free-view image display is performed, wherein the interactive data includes multiple Angle free view data.

The embodiment of the present invention also provides an interactive terminal, including:

The data stream acquisition unit is adapted to obtain the data stream to be played in real time from the playback control device, the data stream to be played includes video data and an interactive identifier, and the interactive identifier is associated with a specified frame time of the data stream to be played; The unit is adapted to play in real time the video displaying the data stream to be played and the interactive sign; the interactive data acquisition unit is adapted to respond to the trigger operation on the interactive sign and acquire the interactive data corresponding to the specified frame moment, so The interactive data includes multi-angle free-view data; the interactive display unit is adapted to perform multi-angle free-view image display at the specified frame time based on the interactive data.

Optionally, the interaction terminal further includes: a switching unit, adapted to trigger switching to the data stream to be played acquired by the data stream acquisition unit from the playback control device in real time when an interaction end signal is detected, and the playback The display unit performs real-time playback and display.

The embodiment of the present invention also provides another interactive terminal, including: a processor, a network component, a memory, and a display component; wherein:

The processor is adapted to obtain the data stream to be played in real time through the network component, and in response to a trigger operation on an interaction mark, obtain the interaction data corresponding to the specified frame time of the interaction mark, wherein the data to be played The stream includes video data and an interactive identifier, the interactive identifier is associated with the specified frame time of the data stream to be played, the interactive data includes multi-angle free view data; the memory is suitable for storing the data stream to be played acquired in real time The display component is adapted to play in real time the video and interactive logo showing the data stream to be played based on the data stream to be played acquired in real time, and perform multi-angle free viewing angles at the specified frame time based on the interactive data image display. The embodiment of the present invention also provides another interactive terminal, including a memory and a processor, the memory stores computer instructions that can be run on the processor, and it is characterized in that the processor runs the computer instructions When performing the steps of the data interaction method described in any embodiment of the present invention.

The embodiment of the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and the steps of the data interaction method described in any embodiment of the present invention are executed when the computer instructions are run.

Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

In the embodiment of the present invention, the data stream to be played can be acquired in real time from the playback control device and displayed in real time, and each interaction identifier in the data stream to be played is associated with the specified frame time of the data stream to be played, and then, can respond to For the trigger operation of an interactive sign, the interaction data corresponding to the specified frame moment of the interactive sign is acquired. Since the interactive data may include multi-angle free view data, the specified frame moment can be calculated based on the interaction data. Perform multi-angle free viewing angle display. It can be seen from the above that, by adopting the embodiment of the present invention, during the playback process, the interaction data can be obtained according to the trigger operation of the interaction sign, and then multi-angle free-view display can be performed, so as to improve the user interaction experience.

Further, by including one or more interactive data including on-site analysis data, information data of the collection object, information data of equipment associated with the collection object, information data of items deployed on-site, information data of logos displayed on-site, etc. , performing multi-angle free viewing angle display, can display richer interactive information to the user through multi-angle free viewing angles, thereby further enhancing the user interaction experience.

Description of drawings

Fig. 1 is a schematic structural diagram of a data processing system in an embodiment of the present invention;

Fig. 2 is a flow chart of a data processing method in an embodiment of the present invention;

3 is a schematic structural diagram of a data processing system in an application scenario in an embodiment of the present invention;

FIG. 4 is a schematic diagram of an interactive interface of an interactive terminal in an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a server in an embodiment of the present invention;

FIG. 6 is a flowchart of a data interaction method in an embodiment of the present invention;

7 is a schematic structural diagram of another data processing system in an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a data processing system in another application scenario according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of an interactive terminal in an embodiment of the present invention;

FIG. 10 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention;

Fig. 11 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention;

FIG. 12 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention;

Fig. 13 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention;

FIG. 14 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention;

Fig. 15 is a flow chart of another data processing method in the embodiment of the present invention;

16 is a flow chart of a method for intercepting synchronized video frames in compressed video data in an embodiment of the present invention;

Fig. 17 is a flowchart of another data processing method in the embodiment of the present invention;

Fig. 18 is a schematic structural diagram of a data processing device in an embodiment of the present invention;

Fig. 19 is a schematic structural diagram of another data processing system in an embodiment of the present invention;

FIG. 20 is a flowchart of a data synchronization method in an embodiment of the present invention;

FIG. 21 is a timing diagram of pull-stream synchronization in an embodiment of the present invention;

22 is a flow chart of another method for intercepting synchronized video frames in compressed video data in an embodiment of the present invention;

Fig. 23 is a schematic structural diagram of another data processing device in an embodiment of the present invention;

Fig. 24 is a schematic structural diagram of a data synchronization system in an embodiment of the present invention;

Fig. 25 is a schematic structural diagram of a data synchronization system in an application scenario in an embodiment of the present invention;

Fig. 26 is a flowchart of a method for generating a depth map in an embodiment of the present invention;

Fig. 27 is a schematic structural diagram of a server in an embodiment of the present invention;

Fig. 28 is a schematic diagram of a server cluster performing depth map processing in an embodiment of the present invention;

Fig. 29 is a flow chart of a method for generating a virtual viewpoint image in an embodiment of the present invention;

FIG. 30 is a flow chart of a method for combined rendering by GPU in an embodiment of the present invention;

Fig. 31 is a schematic diagram of a void filling method in an embodiment of the present invention;

Fig. 32 is a schematic structural diagram of a virtual viewpoint image generation system in an embodiment of the present invention;

Fig. 33 is a schematic structural diagram of an electronic device in an embodiment of the present invention.

Fig. 34 is a schematic structural diagram of another data synchronization system in an embodiment of the present invention;

Fig. 35 is a schematic structural diagram of another data synchronization system in an embodiment of the present invention;

Fig. 36 is a schematic structural diagram of a collection device in an embodiment of the present invention.

Fig. 37 is a schematic diagram of an acquisition array in an application scenario according to an embodiment of the present invention.

Fig. 38 is a schematic structural diagram of another data processing system in an embodiment of the present invention.

Fig. 39 is a schematic structural diagram of another interactive terminal in an embodiment of the present invention.

Fig. 40 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 41 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 42 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 43 is a schematic diagram of connection of an interactive terminal in an embodiment of the present invention.

Fig. 44 is a schematic diagram of an interactive operation of an interactive terminal in an embodiment of the present invention.

Fig. 45 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 46 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 47 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Fig. 48 is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention.

Detailed ways

In traditional broadcasting scenarios such as live broadcasting, rebroadcasting, and recording, as mentioned above, users often can only watch the game through one viewpoint during the viewing process, and cannot freely switch viewpoints to watch the game at different viewpoints. Or the game process, it is impossible to experience the feeling of watching the game while moving the viewpoint on the spot.

The 6 Degree of Freedom (6DoF) technology can provide a high-degree-of-freedom viewing experience. Users can adjust the viewing angle of the video through interactive means during the viewing process, and watch from the free viewpoint they want to watch. to enhance the viewing experience.

In order to realize 6DoF scenes, there are Free-D playback technology, light field rendering technology and 6DoF video generation technology based on depth map, etc. Among them, the Free-D playback technology is to obtain the point cloud data of the scene through multi-angle shooting to express the 6DoF image, and the light field rendering technology is to obtain the depth of field information and three-dimensional position information of the pixels through the focal length and spatial position changes of the dense light field. Then express the 6DoF image. The 6DoF video generation method based on the depth map is based on the parameter data corresponding to the virtual viewpoint position and the texture map and the depth map of the corresponding group, and performs the texture map and the depth map of the corresponding group in the image combination of the video frame at the moment of user interaction. Combined rendering for 6DoF image or video reconstruction.

For example, when the Free-D playback solution is used on site, a large number of cameras are required to collect raw data, which is collected to the on-site computer room through a digital component serial interface (Serial Digital Interface, SDI) capture card, and then through the on-site computer room. The computing server processes the raw data, obtains point cloud data expressing the three-dimensional positions and pixel information of all points in the space, and reconstructs the 6DoF scene. This solution makes the amount of data collected, transmitted and calculated on-site extremely large, especially for broadcast scenarios such as live broadcast and rebroadcast that have high requirements on transmission networks and computing servers, the implementation cost of 6DoF reconstruction scenarios is too high, and there are too many restrictions. Moreover, there are currently no good technical standards and industrial-grade software and hardware to support point cloud data. Therefore, it takes a long time to process data from the original data collection on site to the final 6DoF reconstruction scene, which cannot meet the multi-angle requirements. Requirements for low-latency playback and real-time interaction of free-view videos.

For another example, when using the light field rendering scheme on site, it is necessary to obtain the depth information and three-dimensional position information of pixels through the focal length and spatial position changes of the dense light field. Since the resolution of the light field image obtained by the dense light field is too large, it often needs to be decomposed There are hundreds of conventional two-dimensional pictures. Therefore, this solution also makes the amount of data collected, transmitted and calculated on site extremely large. It has high requirements for the transmission network and calculation server on site, and the implementation cost is too high. Too many to process data quickly. Moreover, the technical means of reconstructing 6DoF scenes from light field images is still under experimental exploration, and currently cannot effectively meet the needs of low-latency playback and real-time interaction of multi-angle free-view videos.

To sum up, both the Free-D playback technology and the light field rendering technology require a large amount of storage and calculation, so a large number of servers need to be deployed on site for processing, resulting in high implementation costs and too many restrictions , cannot quickly process data, thus cannot meet the needs of viewing and interaction, and is not conducive to popularization.

Although the 6DoF video reconstruction method based on the depth map can reduce the amount of data calculation in the video reconstruction process, it is difficult to meet the low-latency playback of multi-angle free-view video due to the constraints of various factors such as network transmission bandwidth and device decoding capability. and real-time interactive needs.

In response to the above problems, some embodiments of the present invention propose a multi-angle free-view image generation scheme, using a distributed system architecture, wherein an acquisition array composed of multiple acquisition devices is set in the on-site acquisition area to perform synchronous acquisition of frame images from multiple angles , the frame image collected by the acquisition device is intercepted by the data processing device according to the frame interception instruction, and the frame image of a plurality of synchronous video frames uploaded by the data processing device is used as an image combination by the server, and the image combination can be determined The corresponding parameter data and the depth data of each frame image in the image combination, based on the corresponding parameter data of the image combination, the pixel data and depth data of the preset frame image in the image combination, the preset virtual viewpoint path Perform frame image reconstruction to obtain corresponding multi-angle free-view video data, and insert the multi-angle free-view video data into the data stream to be played of the playback control device for transmission to the playback terminal for playback.

With reference to the schematic structural diagram of a data processing system in an application scenario in the embodiment of the present invention, the data processing system 10 includes: a data processing device 11, a server 12, a playback control device 13 and a playback terminal 14, wherein the data processing device 11 can be on-site The frame image collected by the collection array in the collection area is used to intercept the video frame. By intercepting the video frame to generate the multi-angle free view image, a large amount of data transmission and data processing can be avoided. After that, the server 12 performs multi-angle free view The generation of images can make full use of the powerful computing power of the server to quickly generate multi-angle free-view video data, so that it can be inserted into the data stream to be played by the playback control device in time, and realize multi-angle free-view at low cost. Playback to meet users' needs for low-latency playback and real-time interaction of multi-angle free-view videos.

In order for those skilled in the art to understand and implement the embodiments of this specification more clearly, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described implementation Examples are some of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

Referring to the flow chart of the data processing method shown in Figure 2, in the embodiment of the present invention, the following steps may be specifically included:

S21. Receive frame images of multiple synchronous video frames uploaded by the data processing device as an image combination.

Wherein, the plurality of synchronous video frames are obtained by the data processing device based on video frame interception instructions, and are obtained by intercepting video frames at specified frame times from multiple video data streams that are synchronously collected and uploaded in real time from different locations in the on-site collection area, The shooting angles of the multiple synchronous video frames are different.

In a specific implementation, the video frame interception instruction may include information of a designated frame time, and the data processing device intercepts the corresponding frame time from multiple video data streams according to the information of the designated frame time in the video frame interception instruction video frames. Wherein, the specified frame time may be in units of frames, and the Nth to M frames are used as the specified frame time, N and M are both integers not less than 1, and N≤M; or, the specified frame time may also be in The unit of time is X to Y seconds as the specified frame time, both X and Y are positive numbers, and X≤Y. Therefore, the plurality of synchronized video frames may include all frame-level synchronized video frames corresponding to a specified frame moment, and the pixel data of each video frame forms a corresponding frame image.

For example, the data processing device can obtain the second frame in the multi-channel video data stream at the specified frame time according to the received video frame interception instruction, then the data processing device intercepts the video frame of the second frame in each video data stream respectively, And the frame-level synchronization between the video frames of the second frame of the intercepted video data streams of each channel is used as a plurality of synchronized video frames obtained.

For another example, assume that the acquisition frame rate is set to 25fps, that is, 25 frames are acquired in 1 second, and the data processing device can obtain the video frame within the first second of the multi-channel video data stream at the specified frame time according to the received video frame interception instruction , the data processing device can respectively intercept 25 video frames in the first second of each video data stream, and the frame-level synchronization between the first video frames in the first second of the intercepted video data streams in each channel, the interception The frame-level synchronization between the second video frame in the first second in each video data stream of the obtained video data stream, until the frame-level synchronization between the 25th video frame in the first second in each video data stream obtained, as the acquisition The resulting multiple synchronized video frames.

Also for example, the data processing device can obtain the second frame and the third frame in the multi-channel video data stream at the specified frame time according to the received video frame interception instruction, then the data processing device can intercept the first frame in each video data stream respectively. The video frames of the 2nd frame and the video frame of the 3rd frame, and the video frames of the 2nd frame and the video frames of the 3rd frame of the intercepted video data streams of each channel are respectively frame-level synchronized, as multiple synchronized video frames .

S22. Determine parameter data corresponding to the image combination.

In a specific implementation, parameter data corresponding to the image combination can be obtained through a parameter matrix, and the parameter matrix can include an internal parameter matrix, an external parameter matrix, a rotation matrix, and a translation matrix. Thus, the relationship between the three-dimensional geometric position of the specified point on the surface of the space object and its corresponding point in the image combination can be determined.

In the embodiment of the present invention, the structure from motion (SFM) algorithm can be used to perform feature extraction, feature matching and global optimization on the obtained image combination based on the parameter matrix, and the obtained parameter estimation value is used as the image combination corresponding parameter data. Among them, the algorithm used for feature extraction can include any of the following: Scale-Invariant Feature Transform (SIFT) algorithm, Speeded-Up Robust Features (SURF) algorithm, accelerated section test features ( Features from AcceleratedSegment Test, FAST) algorithm. Algorithms adopted for feature matching may include: a Euclidean distance calculation method, a random sample consensus (Random Sample Consensus, RANSC) algorithm, and the like. Algorithms for global optimization may include: bundle adjustment (Bundle Adjustment, BA) and the like.

S23. Determine depth data of each frame image in the image combination.

In a specific implementation, the depth data of each frame image may be determined based on multiple frame images in the image combination. Wherein, the depth data may include depth values corresponding to pixels of each frame image in the image combination. The distance from the collection point to each point in the scene can be used as the above-mentioned depth value, and the depth value can directly reflect the geometric shape of the visible surface in the area to be viewed. For example, taking the origin of the shooting coordinate system as the optical center, the depth value can be the distance from each point in the scene to the optical center along the shooting optical axis. Those skilled in the art can understand that the above distance may be a relative value, and multiple frame images may use the same reference.

In an embodiment of the present invention, a binocular stereo vision algorithm may be used to calculate the depth data of each frame of image. In addition, the depth data can also be indirectly estimated by analyzing the photometric features, light and dark features and other features of the frame image.

In another embodiment of the present invention, a multi-view three-dimensional reconstruction (Mult-View Stereo, MVS) algorithm may be used for frame image reconstruction. In the reconstruction process, all pixels can be used for reconstruction, or pixels can be down-sampled and only some pixels can be used for reconstruction. Specifically, the pixel points of each frame image can be matched, the three-dimensional coordinates of each pixel point can be reconstructed, and the points with image consistency can be obtained, and then the depth data of each frame image can be calculated. Alternatively, the pixel points of the selected frame images can be matched, and the three-dimensional coordinates of the pixel points of each selected frame image can be reconstructed to obtain points with image consistency, and then the depth data of the corresponding frame images can be calculated. Among them, the pixel data of the frame image corresponds to the calculated depth data, the method of selecting the frame image can be set according to the specific situation, for example, the distance between the frame image of the depth data and other frame images can be calculated according to the needs, and the selected part frame image.

S24. Based on the corresponding parameter data of the image combination, the pixel data and depth data of the preset frame image in the image combination, perform frame image reconstruction on the preset virtual viewpoint path, and obtain corresponding multi-angle free-view video data.

The multi-angle free-view video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time.

Wherein, the pixel data of the frame image may be any one of YUV data or RGB data, or other data capable of expressing the frame image; the depth data may include a depth value corresponding to the pixel data of the frame image, Or, it may be some values selected from the depth value set corresponding to the pixel data of the frame image one-to-one. The specific selection method depends on the specific scene; The angle-free viewing angle range is a range that supports switching viewing of the viewing area.

In a specific implementation, the preset frame images may be all frame images in the image combination, or may be selected partial frame images. Among them, the selection method can be set according to the specific situation. For example, according to the positional relationship between the collection points, some frame images of the corresponding positions in the image combination can be selected; , select a partial frame image at the corresponding frame moment in the image combination.

Since the preset frame images can correspond to different frame moments, each virtual viewpoint in the virtual viewpoint path can be associated with each frame moment, and the corresponding frame image can be obtained according to the frame moment corresponding to each virtual viewpoint, and then based on The image combines corresponding parameter data, depth data and pixel data of the frame image corresponding to the frame time of each virtual viewpoint, performs frame image reconstruction on each virtual viewpoint, and obtains corresponding multi-angle free-view video data. At this time, multi-angle The free-view video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time. In other words, in a specific implementation, in addition to realizing a multi-angle free-view image at a certain moment, time-series continuous or discontinuous multi-angle free-view videos may also be realized.

In an embodiment of the present invention, the image combination includes A synchronous video frames, wherein, a1 synchronous video frames correspond to the first frame moment, a2 synchronous video frames correspond to the second frame moment, a1+a2=A; and , a virtual viewpoint path consisting of B virtual viewpoints is preset, wherein b1 virtual viewpoints correspond to the first frame moment, b2 virtual viewpoints correspond to the second frame moment, and b1+b2≤2B, then based on the image Combining the corresponding parameter data, the pixel data and depth data of the frame images of a1 synchronous video frames at the first frame time, reconstructing the first frame image on the path composed of b1 virtual viewpoints, and combining the corresponding parameter data based on the image , the pixel data and depth data of the frame images of a21 synchronous video frames at the moment of the first frame, the second frame image reconstruction is performed on the path composed of b2 virtual viewpoints, and finally the corresponding multi-angle free-view video data is obtained, wherein, The multi-angle free-view video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time.

It can be understood that the specified frame time and virtual viewpoint can be further divided, thereby obtaining more synchronous video frames and virtual viewpoints corresponding to different frame times. limits.

In the embodiment of the present invention, a Depth Image Based Rendering (DIBR) algorithm can be used to combine the corresponding parameter data and the preset virtual viewpoint path according to the image, and perform pixel data of the preset frame image Combined rendering with depth data to realize frame image reconstruction based on the preset virtual viewpoint path, and obtain corresponding multi-angle free-view video data.

S25. Insert the multi-angle free-view video data into the data stream to be played of the playback control device and play it through the playback terminal.

The playback control device can take multiple video data streams as input, wherein the video data streams can come from each collection device in the collection array, or from other collection devices. The playback control device can select one of the input video data streams as the data stream to be played according to needs, wherein the multi-angle free-view video data obtained in the aforementioned step S24 can be selected to be inserted into the data stream to be played, or switched by video data streams of other input interfaces To the input interface of the multi-angle free viewing angle video data, the playback control device outputs the selected data stream to be played to the playback terminal, which can be played through the playback terminal, so the user can watch the video of the multi-angle free viewing angle through the playback terminal image. The playback terminal may be a video playback device such as a TV, a mobile phone, a tablet, or a computer, or an electronic device including a display screen.

In a specific implementation, the multi-angle free viewing angle video data of the data stream to be played inserted into the playback control device can be reserved in the playback terminal, so that the user can watch with time shift, wherein the time shift can be the pause and rewind when the user watches , fast forward to the current moment and other operations.

Using the above data processing method, the data processing equipment in the distributed system architecture can be used to process the interception of the specified video frame and the reconstruction of the multi-angle free-view video after the server intercepts the preset frame image, which can avoid arranging a large number of servers on site. It can also avoid directly uploading the video data stream collected by the collection equipment of the collection array, so it can save a lot of transmission resources and server processing resources, and in the case of limited network transmission bandwidth, it makes the multi-angle of the specified video frame free Perspective video can be reconstructed in real time to realize low-latency playback of multi-angle free-view video and reduce the limitation of network transmission bandwidth, so it can reduce implementation costs and restrictions, and is easy to implement, meeting the requirements of low-latency playback of multi-angle free-view video and The need for real-time interaction.

In a specific implementation, according to the relationship between the virtual parameter data of each virtual viewpoint in the preset virtual viewpoint path and the corresponding parameter data of the image combination, the depth of the frame image preset in the image combination The data are respectively mapped to the corresponding virtual viewpoints; according to the pixel data and depth data of the preset frame images respectively mapped to the corresponding virtual viewpoints, as well as the preset virtual viewpoint path, the frame image reconstruction is performed to obtain the corresponding multi-angle free-view video data.

Wherein, the virtual parameter data of the virtual view point may include: virtual viewing position data and virtual viewing angle data; the corresponding parameter data of the image combination may include: collection position data, shooting angle data, and the like. The reconstructed image can be obtained by using forward mapping first and then performing reverse mapping.

In a specific implementation, the collected position data and shooting angle data may be referred to as external parameter data, and the parameter data may also include internal parameter data, and the internal parameter data may include attribute data of the collection device, so that the mapping relationship can be determined more accurately. For example, the internal parameter data may include distortion data, and since the distortion factor is taken into account, the mapping relationship can be further accurately determined spatially.

In a specific embodiment, in order to facilitate subsequent data acquisition, a mosaic image corresponding to the image combination may be generated based on the pixel data and depth data of the image combination, and the mosaic image may include a first field and a second field , wherein the first field includes the pixel data of the image combination, the second field includes the depth data of the image combination, and then stores the corresponding spliced image and corresponding parameter data of the image combination.

In another specific embodiment, in order to save storage space, the spliced image corresponding to the preset frame image in the image combination may be generated based on the pixel data and depth data of the preset frame image in the image combination, and the preset The spliced image corresponding to the frame image may include a first field and a second field, wherein the first field includes pixel data of the preset frame image, and the second field includes depth data of the preset frame image, Then, only the spliced image corresponding to the preset frame image and the corresponding parameter data are stored.

Wherein, the first field corresponds to the second field, the spliced image can be divided into an image area and a depth map area, the pixel field of the image area stores the pixel data of the plurality of frame images, and the pixel data of the depth map area The pixel field stores the depth data of the plurality of frame images; the pixel field storing the pixel data of the frame image in the image area is used as the first field, and the pixel field storing the depth data of the frame image in the depth map area is used as The second field: the spliced image of the obtained image combination and the corresponding parameter data of the image combination can be stored in the data file. When it is necessary to obtain the spliced image or corresponding parameter data, according to the header file of the data file The contained memory address is read from the corresponding memory space.

In addition, the storage format of the image combination may be a video format, and the number of image combinations may be multiple, and each image combination may be an image combination corresponding to a different frame time after the video is decapsulated and decoded.

In a specific implementation, based on the image reconstruction instruction received from the interactive terminal, the interactive frame time information at the interactive time can be determined, and the corresponding stored image at the corresponding interactive frame time can be combined with the spliced image of the preset frame image and the corresponding image combination The corresponding parameter data is sent to the interactive terminal, so that the interactive terminal selects the corresponding pixel data, depth data and corresponding parameter data in the mosaic image according to preset rules based on the virtual viewpoint position information determined by the interactive operation, Combining and rendering the selected pixel data and depth data with the parameter data, reconstructing and playing multi-angle free-view video data corresponding to the virtual viewpoint position to be interacted.

Wherein, the preset rule can be set according to specific scenarios, for example, based on the virtual viewpoint location information determined by the interactive operation, the location information of W adjacent virtual viewpoints sorted by distance closest to the virtual viewpoint at the moment of interaction can be selected, And in the spliced image, the pixel data and depth data corresponding to the W+1 virtual viewpoints including the virtual viewpoints at the interaction time that satisfy the time information of the interaction frame are acquired.

Wherein, the interaction frame time information is determined based on a trigger operation from the interactive terminal. The trigger operation may be a trigger operation input by the user, or may be a trigger operation automatically generated by the interactive terminal. For example, when the interactive terminal detects that there is a multi-angle The trigger operation can be initiated automatically when the free viewpoint data frame is identified. When the user manually triggers, it may be the time information when the user chooses to trigger the interaction after the interactive terminal displays the interactive prompt information, or it may be the historical time information that the interactive terminal receives the user operation to trigger the interaction, and the historical time information may be at the current playback time previous time information.

In a specific implementation, the interactive terminal may be based on the spliced image of the preset frame image and the corresponding parameter data in the acquired image combination at the interaction frame time, the information of the interaction frame time and the virtual viewpoint position information at the interaction frame time, using the same method as above In step S24, the same method is used to combine and render the pixel data and depth data of the spliced image of the preset frame image in the acquired image combination at the time of the interactive frame, to obtain the multi-angle free-view video data corresponding to the virtual viewpoint position of the interaction, and The multi-angle free-view video starts to be played at the interactive virtual viewpoint position.

By adopting the above solution, multi-angle free-view video data corresponding to the interactive virtual view point position can be generated at any time based on the image reconstruction instruction from the interactive terminal, which can further improve user interaction experience.

Referring to the schematic structural diagram of the data processing system shown in FIG. 1, in an embodiment of the present invention, as shown in FIG. in:

The data processing device 11 is adapted to obtain a plurality of synchronous video frames by intercepting video frames at specified frame times from multiple video data streams synchronously collected in real time at different positions in the on-site collection area based on a video frame interception instruction, and will obtain A plurality of synchronous video frames at the specified frame moment are uploaded to the server, wherein the multiple video data streams may be video data streams in a compressed format, or video data streams in an uncompressed format;

The server 12 is adapted to use the received frame images of a plurality of synchronous video frames uploaded by the data processing device 11 at a specified frame time as an image combination, and determine the corresponding parameter data of the image combination and the corresponding parameter data in the image combination. The depth data of each frame image, and based on the corresponding parameter data of the image combination, the pixel data and depth data of the preset frame image in the image combination, perform frame image reconstruction on the preset virtual viewpoint path, and obtain the corresponding multiple Angle free viewing angle video data, the multi-angle free viewing angle video data includes: multi-angle free viewing angle space data and multi-angle free viewing angle time data of frame images sorted according to frame time;

The playback control device 13 is adapted to insert the multi-angle free-view video data into the data stream to be played;

The playback terminal 14 is adapted to receive the data stream to be played from the playback control device 13 and play it in real time.

In a specific implementation, the playback control terminal 13 may output the data stream to be played based on the control instruction. As an optional example, the playback control device 13 may select one of the multiple data streams as the data stream to be played, or continuously switch the selection among the multiple data streams to continuously output the data stream to be played. The broadcast control device can be used as a playback control device in the embodiment of the present invention. The broadcast control device can be a manual or semi-manual broadcast control device based on external input control instructions, or a virtual guide control device that can automatically perform broadcast control based on artificial intelligence or big data learning or preset algorithms.

Using the above data processing system, the data processing equipment in the distributed system architecture can be used to process the interception of the specified video frame and the reconstruction of the multi-angle free-view video after the server intercepts the preset frame image, which can avoid a large number of servers on site. It can also avoid directly uploading the video data stream collected by the collection equipment of the collection array, so it can save a lot of transmission resources and server processing resources, and in the case of limited network transmission bandwidth, it makes the multi-angle of the specified video frame free Perspective video can be reconstructed in real time to realize low-latency playback of multi-angle free-view video and reduce the limitation of network transmission bandwidth, so it can reduce implementation costs and restrictions, and is easy to implement, meeting the requirements of low-latency playback of multi-angle free-view video and The need for real-time interaction.

In a specific implementation, the server 12 is further adapted to generate a mosaic image at a moment of a preset frame in the image combination based on the pixel data and depth data of the preset frame image in the image combination, and the mosaic image includes the first field and a second field, wherein the first field includes the pixel data of the preset frame image in the image combination, and the second field includes the second field of the depth data of the preset frame image in the image combination, And store the spliced image of the image combination and the corresponding parameter data of the image combination.

In a specific implementation, the data processing system 10 may also include an interactive terminal 15, which is adapted to determine the time information of the interactive frame based on the trigger operation, send an image reconstruction instruction containing the time information of the interactive frame to the server, and receive the information returned from the server. The mosaic image of the preset frame image and the corresponding parameter data in the image combination corresponding to the interactive frame moment, and determine the virtual viewpoint position information based on the interactive operation, and select the corresponding pixel data and depth data in the mosaic image according to the preset rules, based on The selected pixel data and depth data are combined with the parameter data for rendering, and the multi-angle free-view video data corresponding to the virtual viewpoint position at the moment of the interactive frame is reconstructed and played.

Wherein, the number of the playback terminal 14 may be one or more, the number of the interaction terminal 15 may be one or more, and the playback terminal 14 and the interaction terminal 15 may be the same terminal device. In addition, at least one of a server, a playback control device, or an interactive terminal can be used as the transmitting end of the video frame interception instruction, and other devices capable of transmitting the video frame interception instruction can also be used.

It should be noted that, in a specific implementation, the locations of the data processing device and the server may be flexibly deployed according to user requirements. For example, the data processing device may be placed in an on-site non-acquisition area or in the cloud. As another example, the server can be placed in the on-site non-collection area, on the cloud or on the terminal access side. For example, on the terminal access side, edge node devices such as base stations, set-top boxes, routers, home data center servers, and hotspot devices can be used as all The above-mentioned server is used to obtain multi-angle free-view data. Alternatively, the data processing device and the server can also be centrally set together to work together as a server cluster to realize rapid generation of multi-angle free-view data, so as to realize low-latency playback and real-time multi-angle free-view video. interactive.

By adopting the above solution, multi-angle free-view video data corresponding to the virtual viewpoint position to be interacted can be generated at any time based on the image reconstruction instruction from the interactive terminal, which can further improve user interaction experience.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the data processing system will be described in detail below through specific application scenarios.

As shown in FIG. 3 , it is a schematic structural diagram of a data processing system in an application scenario, in which it shows a layout scene of a data processing system for a basketball game, and the data processing system includes a collection system composed of multiple collection devices. An array 31 , a data processing device 32 , a cloud server cluster 33 , a playback control device 34 , a playback terminal 35 and an interactive terminal 36 .

Referring to Figure 3, take the basketball hoop on the left as the core point of view, take the core point of view as the center of the circle, and the fan-shaped area on the same plane as the core point of view as the preset multi-angle free viewing angle range. Each acquisition device in the acquisition array 31 can be fan-shaped and placed at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and can collect video data streams from corresponding angles in real time and synchronously.

In a specific implementation, the collection device can also be set on the ceiling area of the basketball court, on the basketball hoop, and the like. Each collection device can be arranged and distributed along a straight line, sector, arc, circle or irregular shape. The specific arrangement can be set according to one or more factors such as the specific on-site environment, the number of acquisition devices, the characteristics of the acquisition devices, and the requirements for imaging effects. The collection device may be any device with a camera function, for example, an ordinary camera, a mobile phone, a professional camera, and the like.

In order not to affect the work of the acquisition equipment, the data processing equipment 32 can be placed in a non-acquisition area on site, which can be regarded as an on-site server. The data processing device 32 can send a streaming instruction to each collection device in the collection array 31 through a wireless local area network, and each collection device in the collection array 31 will obtain a stream based on the streaming instruction sent by the data processing device 32. The video data stream is transmitted to the data processing device 32 in real time. Wherein, each collection device in the collection array 31 can transmit the obtained video data stream to the data processing device 32 in real time through the switch 37 .

When the data processing device 32 receives a video frame interception instruction, it intercepts the video frame at the specified frame time from the received multi-channel video data stream to obtain frame images of a plurality of synchronous video frames, and uses the obtained specified A plurality of synchronous video frames at frame times are uploaded to the server cluster 33 in the cloud.

Correspondingly, the server cluster 33 in the cloud regards the received frame images of a plurality of synchronous video frames as an image combination, determines the corresponding parameter data of the image combination and the depth data of each frame image in the image combination, and based on the image Combining the corresponding parameter data, the pixel data and depth data of the preset frame image in the image combination, performing frame image reconstruction on the preset virtual viewpoint path, and obtaining corresponding multi-angle free-view video data, the multi-angle free-view The video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time.

The server can be placed in the cloud, and in order to process data in parallel more quickly, a server cluster 33 in the cloud can be composed of multiple different servers or server groups according to different data to be processed.

For example, the cloud server cluster 33 may include: a first cloud server 331 , a second cloud server 332 , a third cloud server 333 , and a fourth cloud server 334 . Wherein, the first cloud server 331 can be used to determine the corresponding parameter data of the image combination; the second cloud server 332 can be used to determine the depth data of each frame image in the image combination; the third cloud server 333 can be based on the The parameter data corresponding to the image combination, the pixel data and the depth data of the image combination, using a depth map-based virtual viewpoint reconstruction (Depth Image Based Rendering, DIBR) algorithm, perform frame image reconstruction on the preset virtual viewpoint path; The fourth cloud server 334 may be used to generate multi-angle free-view video, wherein the multi-angle free-view video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time.

It can be understood that, the first cloud server 831, the second cloud server 832, the third cloud server 833, and the fourth cloud server 834 may also be server groups composed of server arrays or server sub-clusters, which are not described in the embodiment of the present invention. limit.

In specific implementation, the server cluster 33 in the cloud can store the pixel data and depth data of the image combination in the following manner:

Based on the pixel data and depth data of the image combination, a spliced image corresponding to the frame moment is generated, and the spliced image includes a first field and a second field, wherein the first field includes a preset frame image in the image combination pixel data, the second field includes the second field of the depth data of the preset frame image in the image combination. The obtained mosaic image and corresponding parameter data can be stored in the data file, and when the mosaic image or parameter data needs to be obtained, it can be read from the corresponding storage space according to the corresponding storage address in the header file of the data file.

Then, the playback control device 34 may insert the received multi-angle free-view video data into the data stream to be played, and the playback terminal 35 receives the data stream to be played from the playback control device 34 and plays it in real time. Wherein, the playback control device 34 may be a manual playback control device, or a virtual playback control device. In a specific implementation, a special server that can automatically switch video streams can be set as a virtual playback control device to control the data source. A broadcast control device such as a broadcast control station can be used as a playback control device in the embodiment of the present invention.

It can be understood that the data processing device 32 can be placed in the on-site non-acquisition area or in the cloud according to the specific scenario, and the server (cluster) and playback control device can be placed in the on-site non-acquisition area, cloud or terminal access according to the specific scenario. On the other hand, this embodiment is not intended to limit the specific implementation and protection scope of the present invention.

The schematic diagram of the interactive interface of the interactive terminal as shown in FIG. The progress bar is associated, and several interaction signs can be generated on the progress bar 41 , such as interaction signs 42 and 43 . Wherein, the black segment of the progress bar 41 is the played part 41a, and the blank segment of the progress bar 41 is the unplayed part 41b.

When the system of the interactive terminal reads the corresponding interactive identification 43 on the progress bar 41, the interface of the interactive terminal 40 can display interactive prompt information. For example, when the user selects an operation to trigger the current interaction logo 43, the interactive terminal 40 generates an image reconstruction instruction at the interaction frame time corresponding to the interaction logo 43 after receiving the feedback, and sends the image reconstruction instruction including the interaction frame time information to the A server cluster 33 in the cloud. When the user does not select a trigger, the interactive terminal 40 may continue to read subsequent video data, and the played part 41a on the progress bar continues to advance. The user can also choose to trigger the historical interaction logo when watching, for example, trigger the interaction logo 42 displayed in the played part 41a on the progress bar, and the interactive terminal 40 will generate an image reconstruction instruction corresponding to the interaction frame moment of the interaction logo 42 after receiving the feedback.

When the server cluster 33 in the cloud receives the image reconstruction instruction from the interactive terminal 40, it can extract the mosaic image of the preset frame image in the corresponding image combination and the corresponding parameter data of the corresponding image combination and transmit it to the interactive terminal 40 .

The interactive terminal 40 determines the time information of the interactive frame based on the trigger operation, sends an image reconstruction instruction including the time information of the interactive frame to the server, and receives the mosaic image of the preset frame image in the image combination corresponding to the time of the interactive frame returned from the server cluster 33 in the cloud and the corresponding parameter data, and determine the position information of the virtual viewpoint based on the interactive operation, select the corresponding pixel data, depth data and corresponding parameter data in the mosaic image according to the preset rules, and perform combined rendering of the selected pixel data and depth data The multi-angle free-view video data corresponding to the virtual viewpoint position at the time of the interaction frame is reconstructed and played.

It can be understood that each acquisition device in the acquisition array can be connected to the data processing device through a switch and/or a local area network, and the number of playback terminals and interactive terminals can be one or more. The interactive terminal can be the same terminal device, the data processing device can be placed in the on-site non-collection area or the cloud according to the specific situation, and the server can be placed in the on-site non-collection area, the cloud or the terminal access side according to the specific situation. The embodiments are not intended to limit the specific implementation and protection scope of the present invention.

The embodiment of the present invention also provides a server corresponding to the above data processing method. To enable those skilled in the art to better understand and implement the embodiment of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings.

Referring to the schematic structural diagram of the server shown in FIG. 5, in the embodiment of the present invention, as shown in FIG. 5, the server 50 may include:

The data receiving unit 51 is adapted to receive frame images of a plurality of synchronous video frames uploaded by the data processing device as an image combination;

A parameter data calculation unit 52, adapted to determine the corresponding parameter data of the image combination;

Depth data calculation unit 53, adapted to determine the depth data of each frame image in the image combination;

The video data acquisition unit 54 is adapted to perform frame image reconstruction on the preset virtual viewpoint path based on the corresponding parameter data of the image combination, the pixel data and depth data of the preset frame image in the image combination, and obtain corresponding multiple Angle free-view video data, wherein the multi-angle free-view video data includes: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time.

The first data transmission unit 55 is adapted to insert the multi-angle free-view video data into the data stream to be played of the playback control device and play it through the playback terminal.

Wherein, the plurality of synchronous video frames can be obtained by the data processing device intercepting the video frames at the specified frame time from the multi-channel video data streams synchronously collected and uploaded in real time from different positions in the on-site collection area based on the video frame interception instruction. , the shooting angles of the multiple synchronous video frames are different.

The server can be placed in the on-site non-acquisition area, on the cloud or on the terminal access side according to specific scenarios.

In a specific implementation, the multi-angle free viewing angle video data of the data stream to be played inserted into the playback control device can be reserved in the playback terminal, so that the user can watch with time shift, wherein the time shift can be the pause and rewind when the user watches , fast forward to the current moment and other operations.

In specific implementation, as shown in Figure 5, the video data acquisition unit 54 may include:

The data mapping subunit 541 is adapted to combine the preset frame images in the image combination according to the relationship between the virtual parameter data of each virtual viewpoint in the preset virtual viewpoint path and the corresponding parameter data of the image combination The depth data of are respectively mapped to the corresponding virtual viewpoints;

The data reconstruction subunit 542 is adapted to perform frame image reconstruction according to the pixel data and depth data of the preset frame images respectively mapped to the corresponding virtual viewpoints, and the preset virtual viewpoint path, to obtain corresponding multi-angle free-view video data .

In specific implementation, as shown in Figure 5, the server 50 may also include:

The spliced image generating unit 56 is adapted to generate a spliced image corresponding to the image combination based on the pixel data and depth data of the preset frame image in the image combination, and the spliced image may include a first field and a second field, wherein , the first field includes pixel data of a preset frame image in the image combination, and the second field includes depth data of a preset frame image in the image combination;

The data storage unit 57 is adapted to store the spliced image of the image combination and the corresponding parameter data of the image combination.

In specific implementation, as shown in Figure 5, the server 50 may also include:

The data extraction unit 58 is adapted to determine the interaction frame time information at the interaction time based on the image reconstruction instruction received from the interaction terminal, and extract the mosaic image of the preset frame image and the corresponding image combination corresponding to the corresponding image combination at the corresponding interaction frame time. parameter data;

The second data transmission unit 59 is adapted to transmit the mosaic image and corresponding parameter data extracted by the data extraction unit 58 to the interactive terminal, so that the interactive terminal determines the virtual viewpoint position information based on the interactive operation, Select the corresponding pixel data, depth data and corresponding parameter data in the mosaic image according to preset rules, combine and render the selected pixel data and depth data, and reconstruct the multi-angle free viewing angle corresponding to the virtual viewpoint position at the time of the interactive frame video data and play it.

By adopting the above solution, multi-angle free-view video data corresponding to the virtual viewpoint position to be interacted can be generated at any time based on the image reconstruction instruction from the interactive terminal, which can further improve user interaction experience.

The embodiment of the present invention also provides a data interaction method and a data processing system, which can obtain the data stream to be played in real time from the playback control device and perform real-time playback display, and the interaction identifiers in the data stream to be played and the specified video data Frame moment correlation, then, in response to a trigger operation on an interaction identifier, the interaction data corresponding to the specified frame moment of the interaction identifier can be obtained, since the interaction data can include multi-angle free view data, it can be based on the Interactive data, multi-angle free-view display for the specified frame moment.

Using the data interaction scheme in the embodiment of the present invention, during the playback process, interactive data can be obtained according to the trigger operation of the interactive logo, and then multi-angle free-view display can be performed to improve user interaction experience. Referring to the accompanying drawings, the data interaction method and the data processing system will be described in detail below through specific embodiments.

Referring to the flow chart of the data interaction method shown in FIG. 6 , the following describes the data interaction method adopted in the embodiment of the present invention through specific steps.

S61. Obtain a data stream to be played from the playback control device in real time and perform real-time playback and display. The data stream to be played includes video data and an interaction identifier, and each interaction identifier is associated with a specified frame time of the data stream to be played.

Wherein, the specified frame time may be in units of frames, and frames N to M are used as the specified frame time, N and M are integers not less than 1, and N≤M; or, the specified frame time may also be time The unit is X to Y seconds as the specified frame time, X and Y are positive numbers, and X≤Y.

In a specific implementation, the data stream to be played may be associated with several designated frame times, and the playback control device may generate an interactive identifier corresponding to each designated frame time based on the information of each designated frame time, so as to play and display in real time When the data stream is to be played, a corresponding interactive logo can be displayed at a specified frame time. Wherein, each interaction identifier and video data may be associated in different ways according to actual conditions.

In an embodiment of the present invention, the data stream to be played may include several frame times corresponding to the video data, since each interaction logo also has a corresponding designated frame time, therefore, the corresponding designated frame of each interaction logo can be matched The time information and the information of each frame time in the data stream to be played can associate the frame time of the same information with the interaction identifier, so that when the data stream to be played is displayed in real time and the corresponding frame time is reached, it can be Display the corresponding interaction logo.

For example, the data stream to be played includes N frame times, and the playback control device generates corresponding M interaction identifiers based on information of M specified frame times. If the information at the i-th frame time is the same as the information at the j-th specified frame time, you can associate the i-th frame time with the j-th interaction identifier, and when the real-time playback display reaches the i-th frame time, The jth interactive logo can be displayed, wherein i is a natural number not greater than N, and j is a natural number not greater than M.

S62. In response to a trigger operation on an interaction sign, acquire interaction data corresponding to a specified frame moment of the interaction sign, where the interaction data includes multi-angle free view data.

In a specific implementation, each interaction data respectively corresponding to each designated frame time can be stored in a preset storage device. Since the interaction logo has a corresponding relationship with the designated frame time, by performing a trigger operation on the interactive terminal, it can be triggered The interactive logo displayed by the interactive terminal can obtain the specified frame time corresponding to the triggered interactive logo according to the trigger operation on the interactive logo. In this way, the interaction data at the specified frame time corresponding to the triggered interaction identifier can be acquired.

For example, the preset storage device may store M pieces of interaction data, wherein the M pieces of interaction data respectively correspond to M specified frame times, and the M specified frame times correspond to M interaction identifiers. Assuming that the triggered interaction identifier is Pi, the specified frame time Ti corresponding to the interaction identifier Pi can be obtained according to the triggered interaction identifier Pi. Thus, the interaction data of the specified frame time Ti corresponding to the interaction identifier Pi is acquired. Among them, i is a natural number.

Wherein, the trigger operation may be a trigger operation input by the user, or may be a trigger operation automatically generated by the interactive terminal.

Moreover, the preset storage device can be placed in the on-site non-acquisition area, the cloud or the terminal access side. Specifically, the preset storage device may be a data processing device, a server, or an interactive terminal in the embodiment of the present invention, or an edge node device located on the interactive terminal side, such as a base station, a set-top box, a router, a home data center server, a hotspot equipment etc.

S63. Based on the interaction data, perform multi-angle free-view image presentation at the specified frame time.

In a specific implementation, an image reconstruction algorithm may be used to perform image reconstruction on the multi-angle free-view data of the interaction data, and then perform multi-angle free-view image display at the specified frame moment.

Moreover, if the specified frame time is one frame time, then a static image of multi-angle free viewing angles can be displayed; if the specified frame time corresponds to multiple frame times, then a dynamic image of multi-angle free viewing angles can be displayed.

By adopting the above scheme, during the video playing process, the interactive data can be obtained according to the trigger operation on the interactive logo, and then multi-angle free-view display can be performed, so as to improve the user interactive experience.

In a specific implementation, the multi-angle free viewing angle data may be generated based on multiple received frame images corresponding to the specified frame time, and the multiple frame images are synchronously collected by the data processing device for multiple acquisition devices in the acquisition array Multiple video data streams are obtained by intercepting at the specified frame time, and the multi-angle free view data may include pixel data, depth data, and parameter data of the multiple frame images, wherein the pixel data of each frame image and There is an association relationship between the depth data.

Wherein, the pixel data of the frame image may be any one of YUV data or RGB data, or may be other data capable of expressing the frame image. The depth data may include depth values corresponding one-to-one to the pixel data of the frame image, or may be partial values selected from a set of depth values corresponding to the pixel data of the frame image one-to-one. The specific selection method of depth data depends on the specific scenario.

In a specific implementation, parameter data corresponding to the plurality of frame images may be obtained through a parameter matrix, and the parameter matrix may include an internal parameter matrix, an external parameter matrix, a rotation matrix, and a translation matrix. Thus, the relationship between the three-dimensional geometric position of the specified point on the surface of the space object and its corresponding point in multiple frame images can be determined.

In the embodiment of the present invention, the SFM algorithm can be used to perform feature extraction, feature matching and global optimization on the acquired multiple frame images based on the parameter matrix, and the obtained parameter estimates are used as the corresponding parameter data of the multiple frame images. The specific algorithms used in the process of feature extraction, feature matching and global optimization can be found in the previous introduction.

In a specific implementation, the depth data of each frame image may be determined based on the plurality of frame images. Wherein, the depth data may include depth values corresponding to pixels of each frame image. The distance from the collection point to each point in the scene can be used as the above-mentioned depth value, and the depth value can directly reflect the geometric shape of the visible surface in the area to be viewed. For example, taking the origin of the shooting coordinate system as the optical center, the depth value can be the distance from each point in the scene to the optical center along the shooting optical axis. Those skilled in the art can understand that the above distance may be a relative value, and multiple frame images may use the same reference.

In an embodiment of the present invention, a binocular stereo vision algorithm may be used to calculate depth data from each frame of image. In addition, the depth data can also be indirectly estimated by analyzing the photometric features, light and dark features and other features of the frame image.

In another embodiment of the present invention, the MVS algorithm can be used for frame image reconstruction, and the pixels of each frame image can be matched, and the three-dimensional coordinates of each pixel can be reconstructed to obtain points with image consistency, and then calculate Depth data of each frame image. Alternatively, the pixel points of the selected frame images can be matched, and the three-dimensional coordinates of the pixel points of each selected frame image can be reconstructed to obtain points with image consistency, and then the depth data of the corresponding frame images can be calculated. Among them, the pixel data of the frame image corresponds to the calculated depth data, the method of selecting the frame image can be set according to the specific situation, for example, the distance between the frame image of the depth data and other frame images can be calculated according to the needs, and the selected part frame image.

In a specific implementation, the data processing device may, based on the received video frame interception instruction, intercept the frame-level synchronized video frame at the specified frame moment in the multiple video data streams.

In a specific implementation, the video frame interception instruction may include frame time information for intercepting video frames, and the data processing device intercepts corresponding video frames from multiple video data streams according to the frame time information in the video frame interception instruction. The video frame at frame time. Moreover, the data processing device sends the frame time information in the video frame interception instruction to the playback control device, and the playback control device can obtain the corresponding specified frame time according to the received frame time information, and according to the received frame time information The information generates a corresponding interactive logo.

In a specific implementation, multiple acquisition devices in the acquisition array are placed in different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and the data processing equipment can be placed in the on-site non-acquisition area or in the cloud.

In a specific implementation, the multi-angle free viewing angle may refer to the spatial position and viewing angle of the virtual viewing point enabling the scene to be switched freely. For example, the multi-angle free viewing angle can be a viewing angle of 6 degrees of freedom (6DoF), where the spatial position of the virtual viewing point can be expressed as (x, y, z), and the viewing angle can be expressed as three rotation directions A total of 6 degrees of freedom are used as the viewing angle of 6 degrees of freedom (6DoF).

Moreover, the range of multi-angle free viewing angles can be determined according to the needs of application scenarios.

In a specific implementation, the playback control device may generate an interaction identifier associated with a video frame at a corresponding time in the data stream to be played based on the frame time information of the intercepted video frame from the data processing device. For example, after the data processing device receives the video frame clipping instruction, it sends the frame time information in the video frame clipping instruction to the playback control device. Then, the playback control device may generate a corresponding interaction identifier based on the time information of each frame.

In a specific implementation, corresponding interaction data may be generated according to the objects displayed on site and the associated information of the displayed objects. For example, the interaction data may also include at least one of the following: on-site analysis data, information data of the collection object, information data of equipment associated with the collection object, information data of items deployed on-site, and information data of logos displayed on-site. Then, based on the interaction data, a multi-angle free-view display is performed, and richer interactive information can be displayed to the user through the multi-angle free view, thereby further enhancing the user interaction experience.

For example, when playing a basketball game, the interactive data may not only include multi-angle free view data, but also include analysis data of the game, information data of a certain player, information data of shoes worn by the player, information data of basketball, One or more of information data such as logos of on-site sponsors.

In a specific implementation, in order to return to the data stream to be played conveniently after the image presentation is over, continue referring to FIG. 6 , after the step 63, it may also include:

S64. When the interaction end signal is detected, switch to acquiring the data stream to be played in real time from the playback control device and perform real-time playback and presentation.

For example, when receiving an interaction end operation indication, switch to the data stream to be played obtained in real time from the playback control device and perform real-time playback and presentation.

For another example, when it is detected that the last image displayed by the multi-angle free-view image at the specified frame time is displayed, switch to the data stream to be played obtained in real time from the playback control device and perform real-time playback and display.

In a specific embodiment, the display of multi-angle free-view images based on the interaction data described in step 63 may specifically include the following steps:

Determine the virtual viewpoint according to the interactive operation, the virtual viewpoint is selected from the multi-angle free viewing angle range, and the multi-angle free viewing angle range is a range that supports switching viewing of the virtual viewpoint in the area to be viewed, and then, the display based on the virtual viewpoint An image viewed on the area to be viewed, where the image is generated based on the interaction data and the virtual viewpoint.

During specific implementation, a virtual viewpoint path may be preset, and the virtual viewpoint path may include several virtual viewpoints. Since the virtual viewpoint is selected from the multi-angle free viewing angle range, the corresponding first virtual viewpoint can be determined according to the image viewing angle displayed during interactive operation, and then the preset virtual viewpoint can be started from the first virtual viewpoint In the order of , the corresponding images of each virtual viewpoint are displayed in turn.

In the embodiment of the present invention, the DIBR algorithm can be used to combine the pixel data and depth data corresponding to the specified frame time of the triggered interactive logo according to the parameter data in the multi-angle free viewing angle data and the preset virtual viewpoint path Rendering, so as to realize image reconstruction based on a preset virtual viewpoint path, obtain corresponding multi-angle free-view video data, and then display corresponding images sequentially from the first virtual viewpoint in the order of preset virtual viewpoints.

And, if the specified frame time corresponds to the same frame time, the obtained multi-angle free-view video data may include multi-angle free-view space data of images sorted according to the frame time, and may display a static image of multi-angle free view; if said The specified frame time corresponds to different frame times, and the obtained multi-angle free-view video data can include multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time, which can display dynamic images of multi-angle free view , which shows the frame image of the video frame of the multi-angle free view.

The embodiment of the present invention also provides a system corresponding to the above-mentioned data interaction method. In order to enable those skilled in the art to better understand and implement the embodiment of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings.

Referring to the structural representation of the data processing system shown in Figure 7, the data processing system 70 may include: an acquisition array 71, a data processing device 72, a server 73, a playback control device 74, and an interactive terminal 75, wherein:

The acquisition array 71 may include a plurality of acquisition devices, which are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and are suitable for real-time synchronous acquisition of multiple video data streams, and real-time upload of video data flow to said data processing device 72;

The data processing device 72 is suitable for intercepting the multi-channel video data stream at a specified frame time according to the received video frame interception instruction for the uploaded multiple video data streams, and obtains the video data corresponding to the specified frame time. A plurality of frame images and frame time information corresponding to the specified frame time, uploading the multiple frame images of the specified frame time and the frame time information corresponding to the specified frame time to the server 73, and uploading the specified The frame time information of the frame time is sent to the playback control device 74;

The server 73 is adapted to receive the multiple frame images and the frame time information uploaded by the data processing device 72, and generate interaction data for interaction based on the multiple frame images, the interaction The data includes multi-angle free view data, and the interaction data is associated with the frame time information;

The playback control device 74 is adapted to determine the specified frame time corresponding to the frame time information uploaded by the data processing device 72 in the data stream to be played, generate an interaction identifier associated with the specified frame time, and include all The data stream to be played of the interactive logo is transmitted to the interactive terminal 75;

The interactive terminal 75 is adapted to play and display the video containing the interactive logo in real time based on the received data stream to be played, and based on the trigger operation on the interactive logo, obtain and store in the server 73 and corresponding to the video. Specify interactive data at frame time for multi-angle free-view image display.

It should be noted that, in a specific implementation, the locations of the data processing device and the server may be flexibly deployed according to user requirements. For example, the data processing device may be placed in an on-site non-acquisition area or in the cloud. As another example, the server can be placed in the on-site non-collection area, on the cloud or on the terminal access side. For example, on the terminal access side, edge node devices such as base stations, set-top boxes, routers, home data center servers, and hotspot devices can be used as all The above-mentioned server is used to obtain multi-angle free-view data. Alternatively, the data processing device and the server can also be centrally set together to work together as a server cluster to realize rapid generation of multi-angle free-view data, so as to realize low-latency playback and real-time multi-angle free-view video. interactive.

By adopting the above solution, during the playback process, the interactive data can be obtained according to the trigger operation of the interactive logo, and then multi-angle free-view display can be performed to improve the user interactive experience.

In a specific implementation, the multi-angle free viewing angle may refer to the spatial position and viewing angle of the virtual viewing point enabling the scene to be switched freely. Moreover, the range of multi-angle free viewing angles can be determined according to the needs of application scenarios. The multi-angle free view may be a view of 6 degrees of freedom (6DoF).

In a specific implementation, the acquisition device itself may have the functions of encoding and encapsulation, so that the original video data collected synchronously in real time from corresponding angles may be encoded and encapsulated. In addition, the collection device may have a compression function.

In a specific implementation, the server 73 is adapted to generate the multi-angle free viewing angle data based on the received multiple frame images corresponding to the specified frame time, and the multi-angle free viewing angle data includes the multiple frame images Pixel data, depth data, and parameter data, wherein there is an association relationship between the pixel data and the depth data of each frame image.

In a specific implementation, multiple acquisition devices in the acquisition array 71 can be placed in different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and the data processing device 72 can be placed in the on-site non-acquisition area or in the cloud, so The above-mentioned server 73 can be placed in the on-site non-acquisition area, the cloud or the terminal access side.

In a specific implementation, the playback control device 74 is adapted to generate an interaction identifier associated with a corresponding video frame in the to-be-played data stream based on the frame information moment of the video frame intercepted by the data processing device 72 .

In a specific implementation, the interaction terminal 75 is further adapted to switch to the data stream to be played acquired in real time from the playback control device 74 and perform real-time playback and display when detecting the interaction end signal.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the data processing system is described in detail below through specific application scenarios, as shown in FIG. 8 , which is a data processing system in another application scenario in the embodiments of the present invention. is a schematic structural diagram of a basketball game playing application scenario, wherein the scene is the basketball court area on the left, and the data processing system 80 may include: an acquisition array 81 composed of various acquisition devices, a data processing device 82, a cloud server cluster 83, play control device 84 and interactive terminal 85.

With the basketball hoop as the core viewing point, the core viewing point as the center, and the fan-shaped area on the same plane as the core viewing point can be used as a preset multi-angle free viewing range. Correspondingly, each acquisition device in the acquisition array 81 can be fan-shaped and placed at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and can collect video data streams from corresponding angles in real time and synchronously.

In a specific implementation, the collection device can also be set on the ceiling area of the basketball court, on the basketball hoop, and the like. Each collection device can be arranged and distributed along a straight line, sector, arc, circle or irregular shape. The specific arrangement can be set according to one or more factors such as the specific on-site environment, the number of acquisition devices, the characteristics of the acquisition devices, and the requirements for imaging effects. The collection device may be any device with a camera function, for example, an ordinary camera, a mobile phone, a professional camera, and the like.

In order not to affect the work of the collection equipment, the data processing equipment 82 can be placed in a non-collection area on site. The data processing device 82 may send streaming instructions to each collection device in the collection array 81 through a wireless local area network. Each acquisition device in the acquisition array 81 transmits the obtained video data stream to the data processing device 82 in real time based on the streaming instruction sent by the data processing device 82 . Wherein, each collection device in the collection array 81 can transmit the obtained video data stream to the data processing device 82 in real time through the switch 87 . Each collection device can compress and transmit the collected raw video data to the data processing device in real time, so as to further save transmission resources of the local area network.

When the data processing device 82 receives a video frame interception instruction, it intercepts the video frame at the specified frame time from the received multiple video data streams to obtain the frame images corresponding to the multiple video frames and the frame images corresponding to the specified frame time. frame time information, and upload a plurality of frame images at the designated frame time and the frame time information corresponding to the designated frame time to the server cluster 83 in the cloud, and send the frame time information at the designated frame time to the The playback control device 84 described above. Wherein, the video frame interception instruction may be issued manually by the user, or may be automatically generated by the data processing device.

The server can be placed in the cloud, and in order to process data in parallel more quickly, the server 83 in the cloud can be composed of multiple different servers or server groups according to different data to be processed.

For example, the cloud server cluster 83 may include: a first cloud server 831 , a second cloud server 832 , a third cloud server 833 and a fourth cloud server 834 . Wherein, the first cloud server 831 can be used to determine the parameter data corresponding to the plurality of frame images; the second cloud server 832 can be used to determine the depth data of each frame image in the plurality of frame images; the third cloud server 833 Based on the parameter data corresponding to the multiple frame images, the depth data and pixel data of the preset frame images in the multiple frame images, the DIBR algorithm can be used to perform frame image reconstruction on the preset virtual viewpoint path; the first The four-cloud server 834 can be used to generate multi-angle free-view video.

It can be understood that, the first cloud server 831, the second cloud server 832, the third cloud server 833, and the fourth cloud server 834 may also be server groups composed of server arrays or server sub-clusters, which are not described in the embodiment of the present invention. limit.

In a specific implementation, the multi-angle free-view video data may include: multi-angle free-view space data and multi-angle free-view time data of frame images sorted by frame time. The interaction data may include multi-angle free viewing angle data, and the multi-angle free viewing angle data may include pixel data, depth data and parameter data of multiple frame images, and there is an association between the pixel data and depth data of each frame image .

The server cluster 83 in the cloud can store the interaction data according to the specified frame time information.

The playback control device 84 may generate an interaction identifier associated with the specified frame moment according to the frame time information uploaded by the data processing device, and transmit the to-be-played data stream including the interaction identifier to the interaction terminal 85 .

Based on the received data stream to be played, the interactive terminal 85 can play the presentation video in real time and display the interaction logo at the corresponding video frame time. When an interactive sign is triggered, the interactive terminal 85 can obtain the interactive data stored in the server cluster 83 in the cloud and corresponding to the specified frame time, so as to display multi-angle free-view images. When the interaction terminal 85 detects the interaction end signal, it may switch to obtain the data stream to be played from the playback control device 84 in real time and perform real-time playback and presentation.

With reference to the schematic structural diagram of another data processing system shown in FIG. 38, the data processing system 380 may include: an acquisition array 381, a data processing device 382, a playback control device 383, and an interactive terminal 384; wherein:

The acquisition array 381 includes a plurality of acquisition devices, and the plurality of acquisition devices are placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and are suitable for real-time synchronous acquisition of multiple video data streams, and real-time upload of video data flow to said data processing facility;

The data processing device 382 is adapted to intercept the multiple video data streams at a specified frame time according to the received video frame interception instruction for the uploaded multiple video data streams, and obtain the video data corresponding to the specified frame time. A plurality of frame images and frame time information corresponding to the designated frame time, and sending the frame time information of the designated frame time to the playback control device 383;

The playback control device 383 is adapted to determine the specified frame time corresponding to the frame time information uploaded by the data processing device 382 in the data stream to be played, generate an interaction identifier associated with the specified frame time, and include all The data stream to be played of the interactive logo is transmitted to the interactive terminal 384;

The interactive terminal 384 is adapted to play and display the video containing the interactive logo in real time based on the received data stream to be played, and based on the trigger operation on the interactive logo, obtain from the data processing device 382 the corresponding Multiple frame images at the specified frame time of the interactive logo, and based on the multiple frame images, generate interactive data for interaction, and then perform multi-angle free-view image display, wherein the interactive data includes multi-angle free Perspective data.

In a specific implementation, the data processing device can be flexibly deployed according to user requirements, for example, the data processing device can be placed in an on-site non-acquisition area or in the cloud.

Using the above data processing system, during the playback process, interactive data can be obtained according to the trigger operation of the interactive logo, and then multi-angle free-view display can be performed to improve user interactive experience.

The embodiment of the present invention also provides a terminal corresponding to the above data interaction method. In order to enable those skilled in the art to better understand and implement the embodiment of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings.

Referring to the schematic structural diagram of the interactive terminal shown in FIG. 9, the interactive terminal 90 may include:

The data stream obtaining unit 91 is adapted to obtain the data stream to be played in real time from the playback control device, the data stream to be played includes video data and an interaction identifier, and the interaction identifier is associated with a specified frame time of the data stream to be played;

Play display unit 92, adapted to play and display the video and interactive logo of the data stream to be played in real time;

The interaction data acquisition unit 93 is adapted to acquire the interaction data corresponding to the specified frame moment in response to the trigger operation on the interaction identifier, the interaction data including multi-angle free view data;

The interactive display unit 94 is adapted to perform multi-angle free-view image display at the specified frame time based on the interactive data;

The switching unit 95 is adapted to trigger switching to the data stream to be played acquired by the data stream acquisition unit 91 from the playback control device in real time when an interaction end signal is detected, and the playback display unit 92 performs real-time playback display .

Wherein, the interaction data may be generated by the server and transmitted to the interaction terminal, or may be generated by the interaction terminal.

During the process of playing the video, the interactive terminal can obtain the data stream to be played from the playback control device in real time, and can display the corresponding interactive logo at the corresponding frame time. In a specific implementation, as shown in FIG. 4 , it is a schematic diagram of an interactive interface of an interactive terminal in an embodiment of the present invention.

The interactive terminal 40 obtains the data stream to be played in real time from the playback control device. When the real-time playback display reaches the first frame time T1, the first interactive logo 42 can be displayed on the progress bar 41. When the real-time playback display reaches the first frame time T1 At the time T2 of the second frame, the second interaction logo 43 may be displayed on the progress bar. Among them, the black part of the progress bar is the played part, and the white part is the unplayed part.

The trigger operation may be a trigger operation input by the user, or may be a trigger operation automatically generated by the interactive terminal. For example, the interactive terminal may automatically initiate the trigger operation when it detects that there is an identifier of a multi-angle free viewpoint data frame. When the user manually triggers, it may be the time information when the user chooses to trigger the interaction after the interactive terminal displays the interactive prompt information, or it may be the historical time information that the interactive terminal receives the user operation to trigger the interaction, and the historical time information may be at the current playback time previous time information.

With reference to Fig. 4, Fig. 7 and Fig. 9, when the system of the interactive terminal reads the corresponding interactive logo 43 on the progress bar 41, it can display interactive prompt information. When the user does not select the trigger, the interactive terminal 40 can continue to read the follow-up video Data, the played part of the progress bar 41 continues to advance. When the user selects a trigger, the interactive terminal 40 generates an image reconstruction instruction at a specified frame time of the corresponding interactive logo after receiving the feedback, and sends it to the server 73 .

For example, when the user chooses to trigger the current interaction logo 43 , the interaction terminal 40 generates an image reconstruction instruction corresponding to the specified frame time T2 of the interaction logo 43 after receiving the feedback, and sends it to the server 73 . The server may send interactive data corresponding to the specified frame time T2 according to the image reconstruction instruction.

The user can also choose to trigger the historical interactive logo when watching, for example, trigger the interactive logo 42 shown in the played part 41a on the progress bar. After receiving the feedback, the interactive terminal 40 generates an image reconstruction instruction corresponding to the specified frame time T1 of the interactive logo 42, and sends to the server 73. The server may send interactive data corresponding to the specified frame time T1 according to the image reconstruction instruction. The interactive terminal 40 may use an image reconstruction algorithm to perform image processing on the multi-angle free-view data of the interaction data, and then display the multi-angle free-view images at the specified frame time. If the specified frame time is one frame time, then a static image of multi-angle free viewing angles is displayed; if the specified frame time corresponds to multiple frame times, then a dynamic image of multi-angle free viewing angles is displayed.

With reference to Figure 4, Figure 38 and Figure 9, when the system of the interactive terminal reads the corresponding interactive logo 43 on the progress bar 41, it can display interactive prompt information, and when the user does not select the trigger, the interactive terminal 40 can continue to read the follow-up video Data, the played part of the progress bar 41 continues to advance. When the user selects a trigger, the interactive terminal 40 generates an image reconstruction instruction at a specified frame time corresponding to the interactive logo after receiving the feedback, and sends it to the data processing device 382 .

For example, when the user chooses to trigger the current interaction logo 43, the interaction terminal 40 generates an image reconstruction instruction corresponding to the specified frame time T2 of the interaction logo 43 after receiving the feedback, and sends it to the data processing device. The data processing device 382 can send a plurality of frame images corresponding to the specified frame time T2 according to the image reconstruction instruction.

The user can also choose to trigger the historical interactive logo when watching, for example, trigger the interactive logo 42 shown in the played part 41a on the progress bar. After receiving the feedback, the interactive terminal 40 generates an image reconstruction instruction corresponding to the specified frame time T1 of the interactive logo 42, and sends to the data processing device. According to the image reconstruction instruction, the data processing device can send a plurality of frame images corresponding to the specified frame time T1.

The interactive terminal 40 can generate interactive data for interaction based on the plurality of frame images, and can use an image reconstruction algorithm to perform image processing on the multi-angle free view data of the interactive data, and then perform image processing at the specified frame time. Image display from multiple angles and free viewing angles. If the specified frame time is one frame time, then a static image of multi-angle free viewing angles is displayed; if the specified frame time corresponds to multiple frame times, then a dynamic image of multi-angle free viewing angles is displayed.

In a specific implementation, the interactive terminal in the embodiment of the present invention may be an electronic device with a touch screen function, a head-mounted virtual reality (Virtual Reality, VR) terminal, an edge node device connected to a display, or an IoT (The Internet of Things, Internet of Things) devices.

As shown in FIG. 40 , it is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. The interactive terminal is an electronic device 400 with a touch screen function. When the corresponding interactive logo 402 on the progress bar 401 is read, the electronic The interface of the device 400 may display an interactive prompt message box 403 . The user can make a choice according to the content of the interactive prompt information box 403. When the user makes a trigger operation of selecting "Yes", the electronic device 400 can generate an image reconstruction instruction corresponding to the interactive frame moment of the interactive logo 402 after receiving the feedback. When the user selects "No" to not trigger the operation, the electronic device 400 may continue to read subsequent video data.

As shown in Figure 41, it is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. The interactive terminal is a head-mounted VR terminal 410. When the corresponding interactive logo 412 on the progress bar 411 is read, the head-mounted VR terminal The interface of the VR terminal 410 may display an interactive prompt message box 413 . The user can make a choice according to the content of the interactive prompt information box 413. When the user makes a trigger operation (such as nodding) to select "Yes", the head-mounted VR terminal 410 can generate an interactive frame corresponding to the interactive logo 412 after receiving the feedback. The image reconstruction instruction at the moment, when the user selects "No" and does not trigger the operation (such as shaking the head), the head-mounted VR terminal 410 can continue to read the subsequent video data.

As shown in FIG. 42 , it is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. The interactive terminal is an edge node device 421 connected to a display 420. When the edge node device 421 reads the corresponding interaction on the progress bar 422 When 423 is identified, the display 420 may display an interactive prompt message box 424 . The user can make a selection according to the content of the interactive prompt information box 424. When the user makes a trigger operation of selecting "Yes", the edge node device 421 can generate an image reconstruction instruction corresponding to the interactive frame time corresponding to the interactive logo 423 after receiving the feedback. When the user selects "No" as an untriggered operation, the edge node device 421 may continue to read subsequent video data.

In a specific implementation, the interactive terminal may establish a communication connection with at least one of the above-mentioned data processing device and server, and a wired connection or a wireless connection may be used.

As shown in FIG. 43 , it is a schematic diagram of connection of an interactive terminal in the embodiment of the present invention. The edge node device 430 establishes wireless connections with the interaction devices 431 , 432 and 433 through the Internet of Things.

In a specific implementation, after the interactive terminal triggers the interactive logo, it can display the multi-angle free-view image at the specified frame time corresponding to the triggered interactive logo, and determine the position information of the virtual viewpoint based on the interactive operation, as shown in Figure 44. A schematic diagram of an interactive operation of an interactive terminal in an embodiment of the present invention, the user can operate horizontally or vertically on the interactive operation interface, and the operation track can be a straight line or a curve.

In a specific implementation, as shown in FIG. 45 , it is a schematic diagram of an interactive interface of another interactive terminal in an embodiment of the present invention. After the user clicks on the interaction logo, the interaction terminal acquires the interaction data at the specified frame time of the interaction logo.

If the user does not take a new operation, the triggering operation is an interactive operation, and the corresponding first virtual viewpoint can be determined according to the angle of view of the image played and displayed during the interactive operation. If the user takes a new operation, the new operation is an interactive operation, and the corresponding first virtual viewpoint can be determined according to the angle of view of the image played and displayed during the interactive operation.

Then, starting from the first virtual viewpoint, images corresponding to each virtual viewpoint may be displayed sequentially in accordance with a preset sequence of virtual viewpoints. If the specified frame time corresponds to the same frame time, the obtained multi-angle free-view video data may include multi-angle free-view space data of images sorted according to the frame time, and may display a static image of multi-angle free view; if the specified frame Time corresponds to different frame times, and the obtained multi-angle free-view video data can include multi-angle free-view space data and multi-angle free-view time data of frame images sorted according to frame time, and can display multi-angle free-view dynamic images, namely What is shown is the frame image of the video frame of the multi-angle free view.

In one embodiment of the invention, refer to FIGS. 45 and 46 . The multi-angle free-view video data obtained by the interactive terminal may include multi-angle free-view space data and multi-angle free-view time data of frame images sorted according to frame time, and the user slides horizontally to the right to generate an interactive operation to determine the corresponding first Virtual viewpoint, and because different virtual viewpoints can correspond to different multi-angle free-view space data and multi-angle free-view time data, as shown in Figure 46, the frame images displayed in the interactive interface occur in time and space with interactive operations The content displayed by the frame image changes from the athlete running towards the finish line in Figure 45 to the athlete who is about to cross the finish line in Figure 46, and with the athlete as the target object, the perspective displayed by the frame image changes from the left view to the front view .

In the same way, in Figures 45 and 47, the content displayed by the frame image changes from the athlete running towards the finish line in Figure 45 to the athlete who has crossed the finish line in Figure 47, and with the athlete as the target object, the viewing angle displayed by the frame image is from the left The view becomes the right view.

Similarly, in Figures 45 and 48, the user slides vertically upwards to generate an interactive operation, and the content displayed in the frame image changes from the athlete running towards the finish line in Figure 45 to the athlete who has crossed the finish line in Figure 48, and the athlete is the target object , the perspective displayed by the frame image changes from the left view to the top view.

It can be understood that different interactive operations can be obtained according to the user's operation, and the corresponding first virtual viewpoint can be determined according to the image angle of view displayed during the interactive operation; according to the obtained multi-angle free-view video data, multi-angle free view can be displayed. The static image or the dynamic image of the viewing angle is not limited in this embodiment of the present invention.

In a specific implementation, the interaction data may also include at least one of the following: on-site analysis data, information data of the collection object, information data of equipment associated with the collection object, information data of items deployed on-site, and logo information displayed on-site. information data.

In an embodiment of the present invention, FIG. 10 is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. After the interactive terminal 100 triggers the interactive logo, it can perform multi-angle free-view image display at the specified frame time corresponding to the triggered interactive logo, and can superimpose on-site analysis data on the image (not shown), as shown in FIG. 10 Field analysis data 101 is shown.

In an embodiment of the present invention, FIG. 11 is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. After the user triggers the interactive logo, the interactive terminal 110 can display the multi-angle free-view image at the specified frame time corresponding to the triggered interactive logo, and can superimpose the information data of the collected object on the image (not shown), as shown in Fig. The information data 111 of the acquisition object in 11 is shown.

In an embodiment of the present invention, FIG. 12 is a schematic diagram of an interactive interface of another interactive terminal in the embodiment of the present invention. After the user triggers the interactive logo, the interactive terminal 120 can display the multi-angle free-view image at the specified frame time corresponding to the triggered interactive logo, and can superimpose the information data of the collected object on the image (not shown), as shown in Fig. The information data 121-123 of the acquisition object in 12 is shown.

In an embodiment of the present invention, FIG. 13 is a schematic diagram of an interactive interface of another terminal in the embodiment of the present invention. After the user triggers the interactive logo, the interactive terminal 130 can display the multi-angle free-view image at the specified frame time corresponding to the triggered interactive logo, and can superimpose the information data of the items deployed on the scene on the image (not shown), As shown in the information data 131 of the file package in FIG. 13 .

In an embodiment of the present invention, FIG. 14 is a schematic diagram of an interactive interface of another terminal in the embodiment of the present invention. After the interactive terminal 140 triggers the interactive logo, it can display the multi-angle free-view image at the specified frame time corresponding to the triggered interactive logo, and can superimpose the information data of the logo displayed on the spot on the image (not shown), such as The logo information data 141 in FIG. 14 is shown.

Thus, the user can obtain more associated interactive information through the interactive data, and have a more in-depth, comprehensive and professional understanding of the watched content, thereby further enhancing the user interactive experience.

Referring to the schematic structural diagram of another interactive terminal shown in FIG. 39, the interactive terminal 390 may include: a processor 391, a network component 392, a memory 393 and a display component 394; wherein:

The processor 391 is adapted to acquire the data stream to be played in real time through the network component 392, and in response to a trigger operation on an interaction identifier, acquire the interaction data corresponding to the specified frame time of the interaction identifier, wherein the to-be-played The play data stream includes video data and an interaction identifier, the interaction identifier is associated with the specified frame time of the data stream to be played, and the interaction data includes multi-angle free view data;

The memory 393 is suitable for storing the data stream to be played acquired in real time;

The display unit 394 is adapted to play in real time the video showing the data stream to be played and the interactive logo based on the data stream to be played acquired in real time, and perform multi-angle free viewing at the specified frame time based on the interactive data image display.

Wherein, the interaction terminal 390 can obtain the interaction data at the specified frame time from the server storing the interaction data, or can obtain multiple frame images corresponding to the specified frame time from the data processing device storing the frame images, and then generate corresponding interactive data.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the processing scheme of the multi-angle free-view video image at the scene side is further described in detail below.

Referring to the flowchart of the data processing method shown in FIG. 15, in the embodiment of the present invention, the following steps may be specifically included:

S151. When it is determined that the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than the preset bandwidth threshold, respectively send a streaming instruction to each acquisition device in the acquisition array, wherein the Each acquisition device in the acquisition array is placed at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range.

The multi-angle free viewing angle may refer to the spatial position and viewing angle of the virtual viewing point enabling the scene to be switched freely. Moreover, the range of multi-angle free viewing angles can be determined according to the needs of application scenarios.

In a specific implementation, the preset bandwidth threshold may be determined according to the transmission capacity of the transmission network where each collection device in the collection array is located. For example, if the uplink bandwidth of the transmission network is 1000 Mbps, the preset bandwidth value may be 1000 Mbps.

S152. Receive the compressed video data stream transmitted in real time by each acquisition device in the acquisition array based on the streaming command, where the compressed video data stream is the real-time synchronous acquisition and data compression of each acquisition device in the acquisition array from corresponding angles respectively. get.

In specific implementation, the acquisition device itself can have the function of encoding and encapsulation, so that the original video data collected synchronously from the corresponding angle can be encoded and encapsulated in real time, wherein the encapsulation format adopted by the acquisition equipment can be AVI, QuickTime File Format , MPEG, WMV, Real Video, Flash Video, Matroska and other formats, or other encapsulation formats, the encoding format used by the acquisition device can be H.261, H.263, H.264, H. 265, MPEG, AVS and other encoding formats, or other encoding formats. In addition, the acquisition device can have a compression function. The higher the compression rate, the smaller the amount of compressed data when the amount of data before compression is the same, which can relieve the bandwidth pressure of real-time synchronous transmission. Therefore, the acquisition device can use predictive coding , transform coding and entropy coding technologies to improve video compression rate.

Using the above data processing method, it is determined whether the transmission bandwidth matches before pulling the stream, which can avoid data transmission congestion during the pulling process, enable the data collected by each acquisition device and data compression to be transmitted synchronously in real time, and speed up multi-angle free-view video data With high processing speed, low-latency playback of multi-angle free-view video can be realized in the case of limited bandwidth resources and data processing resources, reducing implementation costs.

In a specific implementation, the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array can be calculated to be no greater than the preset bandwidth threshold by acquiring the values of the parameters of each acquisition device. For example, 40 acquisition devices can be included in the acquisition array, and the code rate of the compressed video data stream of each acquisition device is 15 Mbps, then the overall code rate of the acquisition array is 15*40=600 Mbps, if the preset bandwidth threshold is 1000 Mbps, Then it is determined that the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than the preset bandwidth threshold. Then, according to the IP addresses of the 40 collection devices in the collection array, streaming commands can be sent to each collection device respectively.

In the specific implementation, in order to ensure that the values of the parameters of each acquisition device in the acquisition array are unified, so that each acquisition device can collect and compress data synchronously in real time, before sending the streaming command to each acquisition device in the acquisition array, you can set The value of the parameter of each acquisition device in the acquisition array. Wherein, the parameters of the collection device may include: collection parameters and compression parameters, and each collection device in the collection array acquires the values obtained by real-time synchronous collection and data compression from corresponding angles according to the set values of the parameters of each collection device. The sum of the code rates of the compressed video data streams is not greater than the preset bandwidth threshold.

Since the acquisition parameters and compression parameters complement each other, when the value of the compression parameter remains unchanged, the data size of the original video data can be reduced by setting the value of the acquisition parameter, so that the time for data compression processing is shortened; Under the same condition, setting the value of the compression parameter can reduce the amount of compressed data correspondingly, so that the time of data transmission is shortened. As another example, setting a higher compression rate can save transmission bandwidth, and setting a lower sampling rate can also save transmission bandwidth. Therefore, acquisition parameters and/or compression parameters may be set according to actual conditions.

Therefore, before starting to pull the stream, the values of the parameters of each acquisition device in the acquisition array can be set to ensure that the values of the parameters of each acquisition device in the acquisition array are unified, and each acquisition device can synchronize acquisition and data compression in real time from corresponding angles , and the sum of the code rates of the obtained compressed video data streams is not greater than the preset bandwidth threshold, so that network congestion can be avoided, and low-latency playback of multi-angle free-view videos can also be realized in the case of limited bandwidth resources.

In a specific embodiment, the acquisition parameters may include focal length parameters, exposure parameters, resolution parameters, encoding bit rate parameters, and encoding format parameters, etc., and the compression parameters may include compression rate parameters, compression format parameters, etc., by setting the values of different parameters , to obtain the most suitable value for the transmission network where each acquisition device is located.

In order to simplify the setting process and save setting time, before setting the value of the parameters of each acquisition device in the acquisition array, it can be determined that each acquisition device in the acquisition array performs acquisition and data compression according to the value of the parameter that has been set. Whether the sum of the bit rates of the compressed video data streams is greater than the preset bandwidth threshold, when the sum of the bit rates of the obtained compressed video data streams is greater than the preset bandwidth threshold, each acquisition device in the acquisition array sends a pull Before the streaming instruction, the value of the parameter of each acquisition device in the acquisition array can be set. It can be understood that, in a specific implementation, the value of the acquisition parameter and the value of the compression parameter can also be set according to imaging quality requirements such as the resolution of the multi-angle free-view image to be displayed.

In a specific implementation, the process from transmission to writing of the compressed video data streams obtained by each acquisition device occurs continuously. Therefore, before sending a streaming command to each acquisition device in the acquisition array, it can also be determined that the Whether the sum of the bit rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is greater than the preset write speed threshold, and whether the sum of the bit rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array When it is greater than the preset write speed threshold, the value of the parameter of each acquisition device in the acquisition array can be set, so that each acquisition device in the acquisition array can read from a corresponding angle according to the set value of the parameter of each acquisition device. The sum of the code rates of the compressed video data stream obtained by real-time synchronous acquisition and data compression is not greater than the preset writing speed threshold.

In a specific implementation, the preset writing speed threshold may be determined according to the data storage writing speed of the storage medium. For example, the data storage writing speed upper limit of a solid state disk (Solid State Disk or Solid State Drive, SSD) of the data processing device is 100 Mbps, and the preset writing speed threshold may be 100 Mbps.

Using the above solution, before starting to pull the stream, it can be ensured that the sum of the code rates of the compressed video data streams obtained by each acquisition device from the corresponding angle in real-time synchronous acquisition and data compression is not greater than the preset write speed threshold, thereby avoiding Data writing congestion ensures that the compressed video data stream is unblocked during the process of collection, transmission and writing, so that the compressed video stream uploaded by each collection device can be processed in real time, and then realize the playback of multi-angle free-view video.

In a specific implementation, the compressed video data streams obtained by each collection device may be stored. When a video frame interception instruction is received, frame-level synchronized video frames in each compressed video data stream may be intercepted according to the received video frame interception instruction, and the intercepted video frames may be synchronously uploaded to the designated target end.

Wherein, the specified target end may be a preset target end, or a target end specified by a video frame interception instruction. In addition, the intercepted video frames may be encapsulated first, and uploaded to the designated target through a network transmission protocol, and then analyzed to obtain frame-level synchronized video frames in the corresponding compressed video data stream.

Therefore, the subsequent processing of the video frame intercepted by the compressed video data stream is handed over to the designated target end, which can save network transmission resources, reduce the pressure and difficulty of deploying a large number of server resources on site, and can also greatly reduce the data processing load. , to shorten the transmission delay of multi-angle free-view video frames.

In a specific implementation, in order to ensure that the frame-level synchronized video frames in each compressed video data stream are intercepted, as shown in Figure 16, the following steps may be included:

S161. Determine one of the compressed video data streams received in real time from the compressed video data streams of each acquisition device in the acquisition array as a reference data stream;

S162. Based on the received video frame interception instruction, determine the video frame to be intercepted in the reference data stream, and select one of the remaining compressed video data streams that is synchronized with the video frame to be intercepted in the reference data stream The video frame is used as the video frame to be intercepted of the remaining compressed video data streams;

S163. Intercept video frames to be intercepted in each compressed video data stream.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, how to determine video frames to be intercepted in each compressed video data stream will be described in detail below through a specific application scenario.

In an embodiment of the present invention, 40 acquisition devices can be included in the acquisition array, therefore, 40 compressed video data streams can be received in real time, assuming that in the compressed video data streams of each acquisition device in the acquisition array received in real time, Determine the compressed video data stream A1 corresponding to the acquisition device A1' as the reference data stream, and then, based on the received video frame interception instruction indicating the feature information X of the object in the intercepted video frame, determine the The video frame a1 with the same characteristic information X of the object is used as the video frame to be intercepted, and then according to the characteristic information x1 of the object in the video frame a1 to be intercepted in the reference data stream, select the remaining compressed video data streams A2-A40 The video frames a2-a40 consistent with the feature information x1 of the object are used as video frames to be intercepted for the remaining compressed video data streams.

Wherein, the feature information of the object may include at least one of shape feature information, color feature information, position feature information, and the like. The feature information X of the object in the clipped video frame indicated in the video frame clipping instruction and the feature information x1 of the object in the video frame a1 to be clipped in the reference data stream may be identical to the feature information of the same object. Representation, for example, the feature information X and x1 of the object are both two-dimensional feature information; the feature information X of the object and the feature information x1 of the object can also be different representations of the feature information of the same object, for example, the feature of the object The information X may be two-dimensional characteristic information, and the characteristic information x1 of the object may be three-dimensional characteristic information. Moreover, a similarity threshold can be preset, and when the similarity threshold is met, it can be considered that the characteristic information X of the object is consistent with x1, or the characteristic information x1 of the object is the same as the characteristic information x2-x40 of the object in the other compressed video data streams A2-A40 unanimous.

The specific representation manner and the similarity threshold of the feature information of the object may be determined according to the preset multi-angle free viewing angle range and the on-site scene, which are not limited in this embodiment of the present invention.

In another embodiment of the present invention, 40 acquisition devices can be included in the acquisition array, therefore, 40 compressed video data streams can be received in real time, assuming that in the compressed video data streams of each acquisition device in the acquisition array received in real time Determining the compressed video data stream B1 corresponding to the acquisition device B1' as the reference data stream, and then, based on the time stamp information Y indicating the intercepted video frame in the received video frame interception instruction, determining the difference between the reference data stream and the The video frame b1 corresponding to the timestamp information Y is used as the video frame to be intercepted, and then according to the timestamp information y1 in the video frame b1 to be intercepted in the reference data stream, select the remaining compressed video data streams B2-B40 and The video frames b2-b40 whose timestamp information is consistent with y1 are used as video frames to be intercepted for the remaining compressed video data streams.

Wherein, the time stamp information Y indicating the video frame to be intercepted in the video frame interception instruction may have a certain error from the time stamp information y1 in the video frame b1 to be intercepted in the reference data stream, for example, the The timestamp information corresponding to the video frame in the reference data stream is inconsistent with the timestamp information Y. If there is an error of 0.1ms, an error range can be preset. For example, if the error range is ±1ms, then the error of 0.1ms is within the error range Therefore, the video frame b1 corresponding to the time stamp information y1 with a difference of 0.1 ms from the time stamp information Y may be selected as the video frame to be intercepted in the reference data stream. The specific error range and the selection rule of the time stamp information y1 in the reference data stream can be determined according to the on-site collection equipment and the transmission network, which is not limited in this embodiment.

It can be understood that the method for determining the video frame to be intercepted in each compressed video data stream in the above embodiment can be used alone or at the same time, which is not limited in this embodiment of the present invention.

Using the above data processing method, the data processing device can smoothly and smoothly pull the data collected and compressed by each collection device.

The following will clearly and completely describe the technical solution for data processing by the acquisition array in the embodiment of the present specification with reference to the accompanying drawings in the embodiment of the present specification.

Referring to the flow chart of the data processing method shown in FIG. 17, in the embodiment of the present invention, the following steps may be specifically included:

S171. Each acquisition device in the acquisition array placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range collects raw video data synchronously in real time from corresponding angles, and performs real-time data compression on the collected raw video data respectively, A corresponding compressed video data stream is obtained.

S172. When the data processing device connected to the acquisition array link determines that the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than the preset bandwidth threshold, send Each acquisition device in the array sends a streaming command.

In a specific implementation, the preset bandwidth threshold may be determined according to the transmission capability of the transmission network where each collection device in the collection array is located. For example, if the uplink bandwidth of the transmission network is 1000 Mbps, the preset bandwidth value may be 1000 Mbps.

S173. Each acquisition device in the acquisition array transmits the obtained compressed video data stream to the data processing device in real time based on the streaming instruction.

In a specific implementation, the data processing device may be set according to an actual scenario. For example, when there is suitable space on site, the data processing equipment can be placed in the non-collection area of the site as an on-site server; when there is no suitable space on site, the data processing equipment can be placed in the cloud as a cloud server.

Using the above scheme, when the data processing device connected to the acquisition array link determines that the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than the preset bandwidth threshold, the Each acquisition device in the acquisition array sends a streaming command, so that the data collected by each acquisition device and data compression can be transmitted synchronously in real time, so that real-time streaming can be performed through the transmission network where it is located, and data transmission congestion during the streaming process can be avoided; Then, each acquisition device in the acquisition array transmits the obtained compressed video data stream to the data processing device in real time based on the streaming instruction. Since the data transmitted by each acquisition device is compressed, the bandwidth pressure of real-time synchronous transmission can be alleviated. , speeding up the processing speed of multi-angle free-view video data.

As a result, it is possible to avoid arranging a large number of servers for data processing on site, and it is not necessary to summarize the collected raw data through the SDI capture card, and then process the raw data through the computing server in the on-site computer room, which can avoid the use of expensive SDI video transmission lines Instead of cable and SDI interfaces, data transmission is carried out through ordinary transmission networks. In the case of limited bandwidth resources and data processing resources, low-latency playback of multi-angle free-view videos can be achieved, reducing implementation costs.

In a specific implementation, in order to simplify the setting process and save setting time, before setting the values of the parameters of each acquisition device in the acquisition array, the data processing device can first determine that each acquisition device in the acquisition array is set according to the set parameters. Whether the sum of the code rates of the compressed video data streams obtained by collecting and compressing the data is greater than the preset bandwidth threshold, when the sum of the code rates of the compressed video data streams obtained is greater than the preset bandwidth threshold, the data processing device can set The value of the parameter of each acquisition device in the acquisition array is collected, and then a streaming instruction is sent to each acquisition device in the acquisition array respectively.

In the specific implementation, the process from transmission to writing of the compressed video data stream obtained by each acquisition device occurs continuously, and it is also necessary to ensure that the data processing device writes the compressed video data stream obtained by each acquisition device smoothly. Before sending the streaming command to each acquisition device in the acquisition array, the data processing device may also determine whether the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is greater than a preset write speed threshold , and when the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is greater than the preset write speed threshold, the data processing device can set the value of the parameter of each acquisition device in the acquisition array , so that the sum of the code rates of the compressed video data streams obtained by real-time synchronous acquisition and data compression from corresponding angles for each acquisition device in the acquisition array according to the set parameter values of each acquisition device is not greater than the preset Write speed threshold.

In a specific implementation, the preset writing speed threshold may be determined according to the data storage writing speed of the data processing device.

In a specific implementation, data transmission may be performed in at least one of the following ways between each collection device in the collection array and the data processing device:

1. Data transmission through the switch;

Each acquisition device in the acquisition array is connected to the data processing device through a switch, and the switch can summarize and uniformly transmit the compressed video data streams of more acquisition devices to the data processing device, which can reduce the number of ports supported by the data processing device quantity. For example, the switch supports 40 inputs, so the data processing device can simultaneously receive video streams from a collection array composed of 40 collection devices at most through the switch, thereby reducing the number of data processing devices.

2. Data transmission through LAN.

Each acquisition device in the acquisition array is connected to the data processing device through a local area network, and the local area network can transmit the compressed video data stream of the acquisition device to the data processing device in real time, reducing the number of ports supported by the data processing device, thereby reducing data The number of processing devices.

In a specific implementation, the data processing device can store (can be buffered) the compressed video data streams obtained by each acquisition device, and when receiving a video frame interception instruction, the data processing device can according to the received The video frame interception instruction intercepts frame-level synchronized video frames in each compressed video data stream, and synchronously uploads the intercepted video frames to the designated target end.

Wherein, the data processing device may establish a connection with a target end through a port or an IP address in advance, and may also synchronously upload the intercepted video frame to the port or IP address specified by the video frame interception instruction. In addition, the data processing device may first encapsulate the intercepted video frames, upload them to the designated target through a network transmission protocol, and then analyze them to obtain frame-level synchronized video frames in the corresponding compressed video data stream.

By adopting the above scheme, the compressed video data stream obtained by real-time synchronous acquisition and data compression of each acquisition device in the acquisition array can be uniformly transmitted to the data processing device, and the data processing device will cut the frame after receiving the video frame interception instruction The initial processing of the compressed video data streams can upload the frame-level synchronized video frames in each compressed video data stream to the designated target end synchronously, and hand over the subsequent processing of the video frames intercepted by the compressed video data streams to the designated target end, In this way, network transmission resources can be saved, the pressure and difficulty of on-site deployment can be reduced, the data processing load can also be greatly reduced, and the transmission delay of multi-angle free-view video frames can be shortened.

In a specific implementation, in order to intercept frame-level synchronized video frames in each compressed video data stream, the data processing device may first determine one of the compressed video data streams received in real time in the compressed video data streams of each acquisition device in the acquisition array. stream as a reference data stream, and then, the data processing device can determine the video frame to be intercepted in the reference data stream based on the received video frame interception instruction, and select the video frame to be intercepted in the reference data stream. The video frames in the remaining compressed video data streams synchronized with the video frames are used as video frames to be intercepted in the remaining compressed video data streams, and finally, the data processing device intercepts the video frames to be intercepted in each compressed video data stream. For a specific frame truncating method, reference may be made to the examples in the foregoing embodiments, and details are not repeated here.

The embodiment of the present invention also provides a data processing device corresponding to the data processing method in the above embodiment. In order to enable those skilled in the art to better understand and realize the embodiment of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings .

Referring to the schematic structural diagram of the data processing device shown in FIG. 18, in the embodiment of the present invention, as shown in FIG. 18, the data processing device 180 may include:

The first transmission matching unit 181 is adapted to determine whether the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than a preset bandwidth threshold, wherein each acquisition device in the acquisition array according to the preset The multi-angle free viewing angle range is placed in different positions of the field collection area.

The instruction sending unit 182 is adapted to send a pull request to each acquisition device in the acquisition array when it is determined that the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is not greater than a preset bandwidth threshold. stream instructions.

The data stream receiving unit 183 is adapted to receive the compressed video data stream transmitted in real time by each acquisition device in the acquisition array based on the stream pulling instruction, and the compressed video data stream is obtained from each acquisition device in the acquisition array from corresponding angles. Real-time synchronous acquisition and data compression are obtained.

Using the above-mentioned data processing device, before sending the streaming command to each collection device in the collection array, it is determined whether the transmission bandwidth matches, which can avoid data transmission congestion during the streaming process, so that the data collected by each collection device and data compression can be obtained. Real-time synchronous transmission speeds up the processing speed of multi-angle free-view video data, realizes multi-angle free-view video under the condition of limited bandwidth resources and data processing resources, and reduces implementation costs.

In an embodiment of the present invention, as shown in FIG. 18, the data processing device 180 may further include:

The first parameter setting unit 184 is adapted to set the value of the parameter of each acquisition device in the acquisition array before sending the streaming instruction to each acquisition device in the acquisition array respectively;

Wherein, the parameters of the collection device may include: collection parameters and compression parameters, and each collection device in the collection array acquires the values obtained by real-time synchronous collection and data compression from corresponding angles according to the set values of the parameters of each collection device. The sum of the code rates of the compressed video data streams is not greater than the preset bandwidth threshold.

In an embodiment of the present invention, in order to simplify the setup process and save setup time, as shown in FIG. 18, the data processing device 180 may further include:

The second transmission matching unit 185 is adapted to determine the compressed video obtained by each acquisition device in the acquisition array by collecting and compressing data according to the set parameter value before setting the value of the parameter of each acquisition device in the acquisition array Whether the sum of the code rates of the data streams is not greater than the preset bandwidth threshold.

In an embodiment of the present invention, as shown in FIG. 18, the data processing device 180 may further include:

Write matching unit 186, adapted to determine whether the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is greater than a preset write speed threshold;

The second parameter setting unit 187 is adapted to set the data rate of each acquisition device in the acquisition array when the sum of the code rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array is greater than the preset write speed threshold The value of the parameter, so that each acquisition device in the acquisition array according to the numerical value of the parameter of each acquisition device set, the sum of the code rates of the compressed video data stream obtained by synchronous acquisition and data compression in real time from the corresponding angle is not greater than the said Preset write speed threshold.

Therefore, before starting to pull the stream, it can be ensured that the sum of the code rates of the compressed video data streams obtained by each acquisition device from the corresponding angle in real-time synchronous acquisition and data compression is not greater than the preset write speed threshold, thereby avoiding data writing. Incoming congestion, to ensure that the link of the compressed video data stream is smooth during the process of collection, transmission and writing, so that the compressed video stream uploaded by each collection device can be processed in real time, and then realize the playback of multi-angle free-view video.

In an embodiment of the present invention, as shown in FIG. 18, the data processing device 180 may further include:

The frame interception processing unit 188 is adapted to intercept the frame-level synchronized video frames in each compressed video data stream according to the received video frame interception instruction;

The uploading unit 189 is adapted to synchronously upload the intercepted video frames to the specified target.

Wherein, the specified target end may be a preset target end, or a target end specified by a video frame interception instruction.

Thus, the subsequent processing of the video frames intercepted by the compressed video data stream is handed over to the specified target end, thereby saving network transmission resources, reducing the pressure and difficulty of on-site deployment, and also greatly reducing the data processing load and shortening the length of time. The transmission delay of an angle-free perspective video frame.

In an embodiment of the present invention, as shown in FIG. 18, the frame truncation processing unit 188 may include:

The reference data stream selection subunit 1881 is adapted to determine one of the compressed video data streams of each acquisition device in the acquisition array received in real time as the reference data stream;

The video frame selection subunit 1882 is adapted to determine the video frame to be intercepted in the reference data stream based on the received video frame interception instruction, and select the remaining video frames synchronized with the video frame to be intercepted in the reference data stream The video frames in each compressed video data stream are used as video frames to be intercepted for the remaining compressed video data streams;

The video frame interception subunit 1883 is adapted to intercept video frames to be intercepted in each compressed video data stream.

In an embodiment of the present invention, as shown in FIG. 18, the video frame selection subunit 1882 may include at least one of the following:

The first video frame selection module 18821 is adapted to select video frames consistent with the feature information of the object in the remaining compressed video data streams according to the feature information of the object in the video frame to be intercepted in the reference data stream, as The video frames to be intercepted of the remaining compressed video data streams;

The second video frame selection module 18822 is adapted to select, according to the time stamp information of the video frame to be intercepted in the reference data stream, video frames consistent with the time stamp information in the remaining compressed video data streams, as the remaining video frames The video frame to be intercepted in the compressed video data stream.

The embodiment of the present invention also provides a data processing system corresponding to the above-mentioned data processing method, and adopts the above-mentioned data processing device to realize real-time reception of multiple compressed video data streams. In order to enable those skilled in the art to better understand and implement the embodiment of the present invention, The following describes in detail through specific embodiments with reference to the accompanying drawings.

With reference to the schematic structural diagram of the data processing system shown in Figure 19, in the embodiment of the present invention, the data processing system 190 may include: an acquisition array 191 and a data processing device 192, the acquisition array 191 includes a preset multi-angle free viewing angle A range of multiple collection devices placed at different locations in the on-site collection area, of which:

Each acquisition device in the acquisition array 191 is suitable for synchronously acquiring original video data from corresponding angles in real time, and performing real-time data compression on the acquired original image data respectively, to obtain compressed video data streams acquired synchronously in real time from corresponding angles, And based on the streaming instruction sent by the data processing device 192, the obtained compressed video data stream is transmitted to the data processing device 192 in real time;

The data processing device 192 is adapted to, when it is determined that the sum of the code rates of the compressed video data streams pre-transmitted by each collection device in the collection array is not greater than the preset bandwidth threshold, send data to each of the collection arrays 191. The device sends a streaming instruction, and receives the compressed video data stream transmitted in real time by each collection device in the collection array 191 .

Adopting the above-mentioned scheme can avoid arranging a large number of servers on site for data processing, and it is not necessary to summarize the collected raw data through the SDI capture card, and then process the raw data through the computing server in the on-site computer room, which can avoid the use of expensive SDI video transmission Instead of cables and SDI interfaces, data transmission and streaming are carried out through ordinary transmission networks. In the case of limited bandwidth resources and data processing resources, low-latency playback of multi-angle free-view videos can be realized, and implementation costs can be reduced.

In an embodiment of the present invention, the data processing device 192 is further adapted to set the value of the parameter of each collection device in the collection array 191 before sending the streaming instruction to each collection device in the collection array 191;

Wherein, the parameters of the acquisition equipment include: acquisition parameters and compression parameters, and each acquisition equipment in the acquisition array according to the set values of the parameters of each acquisition equipment, real-time synchronous acquisition and data compression obtained from the corresponding angle The sum of the code rates of the video data streams is not greater than the preset bandwidth threshold.

Therefore, before starting to pull the stream, the data processing device can set the value of the parameters of each acquisition device in the acquisition array to ensure that the values of the parameters of each acquisition device in the acquisition array are unified, and each acquisition device can collect synchronously in real time from the corresponding angle and data compression, and the sum of the bit rates of the obtained compressed video data streams is not greater than the preset bandwidth threshold, so as to avoid network congestion and realize low-latency playback of multi-angle free-view videos in the case of limited bandwidth resources .

In an embodiment of the present invention, before the data processing device 192 sends the streaming command to each collection device in the collection array 191, it determines the compressed video data stream pre-transmitted by each collection device in the collection array 191. Whether the sum of bit rates is greater than the preset write speed threshold, and when the sum of the bit rates of the compressed video data streams pre-transmitted by each acquisition device in the acquisition array 191 is greater than the preset write speed threshold, set the The numerical value of the parameter of each acquisition equipment in acquisition array 191, makes each acquisition equipment in described acquisition array 192 according to the numerical value of the parameter of described each acquisition equipment set, the compressed video data flow that obtains from corresponding angle real-time synchronous acquisition and data compression The sum of the code rates is not greater than the preset write speed threshold.

Therefore, before starting to pull the stream, it can be ensured that the sum of the code rates of the compressed video data streams obtained by each acquisition device from corresponding angles in real-time synchronous acquisition and data compression is not greater than the preset write speed threshold, thereby avoiding data processing The data writing congestion of the device ensures that the link of the compressed video data stream is unblocked during the process of collection, transmission and writing, so that the compressed video stream uploaded by each collection device can be processed in real time, thereby realizing the playback of multi-angle free-view video .

In a specific implementation, each collection device in the collection array is adapted to be connected to the data processing device through a switch and/or a local area network.

In an embodiment of the present invention, the data processing system 190 may further include a specified target end 193 .

The data processing device 192 is adapted to intercept frame-level synchronized video frames in each compressed video stream according to the received video frame interception instruction, and upload the intercepted video frames to the designated target terminal 193 synchronously;

The specified target end 193 is adapted to receive the video frame intercepted by the data processing device 192 based on the video frame interception instruction.

Wherein, the data processing device may establish a connection with a target end through a port or an IP address in advance, and may also synchronously upload the intercepted video frame to the port or IP address specified by the video frame interception instruction.

By adopting the above scheme, the compressed video data stream obtained by real-time synchronous acquisition and data compression of each acquisition device in the acquisition array can be uniformly transmitted to the data processing device, and the data processing device will cut the frame after receiving the video frame interception instruction The preliminary processing of the compressed video data streams can upload the frame-level synchronized video frames in each compressed video data stream to the designated target end synchronously, and the subsequent processing of the video frames intercepted by the compressed video data streams is carried out by the designated target end, thereby It can save network transmission resources, reduce the pressure and difficulty of on-site deployment, and can also greatly reduce the data processing load and shorten the transmission delay of multi-angle free-view video frames.

In an embodiment of the present invention, the data processing device 192 is adapted to determine one of the compressed video data streams of each acquisition device in the acquisition array 191 received in real time as a reference data stream; and based on the received The video frame interception instruction, determine the video frame to be intercepted in the described reference data stream, and select the video frame in the other compressed video data streams synchronous with the video frame to be intercepted in the described reference data stream, as the rest The video frames to be intercepted in each compressed video data stream; finally, the video frames to be intercepted in each compressed video data stream are intercepted.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the frame synchronization solution between the data processing device and the acquisition device will be described in detail below through specific embodiments.

Referring to the flow chart of the data synchronization method shown in FIG. 20, in the embodiment of the present invention, the following steps may be specifically included:

S201. Send streaming instructions to each acquisition device in the acquisition array, wherein each acquisition device in the acquisition array is placed at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and each acquisition device in the acquisition array The equipment collects video data streams in real time and synchronously from corresponding angles respectively.

In a specific implementation, in order to realize synchronization of streaming, there may be multiple implementation manners. For example, the streaming command can be issued to all the acquisition devices in the acquisition array at the same time; or, the streaming command can be sent only to the main acquisition device in the acquisition array to trigger the main acquisition device to pull the current, and then the main acquisition device will The above streaming commands are synchronized to all slave acquisition devices, triggering all slave acquisition devices to pull streams.

S202. Receive in real time the video data streams respectively transmitted by each acquisition device in the acquisition array based on the streaming instruction, and determine whether the video data streams respectively transmitted by each acquisition device in the acquisition array are frame-level synchronized.

In a specific implementation, the acquisition device itself may have the functions of encoding and encapsulation, so that the original video data collected synchronously in real time from corresponding angles may be encoded and encapsulated. In addition, each acquisition device can also have a compression function. The higher the compression rate, the smaller the amount of compressed data when the amount of data before compression is the same, which can relieve the bandwidth pressure of real-time synchronous transmission. Therefore, the acquisition device can use Technologies such as predictive coding, transform coding, and entropy coding improve the compression rate of video.

S203. When there is no frame-level synchronization between the video data streams respectively transmitted by the acquisition devices in the acquisition array, re-send streaming instructions to each acquisition device in the acquisition array until each acquisition device in the acquisition array Frame-level synchronization between separately transmitted video data streams.

Using the above data synchronization method, by determining whether the video data streams transmitted by each acquisition device in the acquisition array are frame-level synchronous, it can ensure the synchronous transmission of multi-channel data, thereby avoiding the problem of frame leakage and multi-frame transmission, and improving data processing. Speed, and thus meet the needs of low-latency playback of multi-angle free-view video.

In a specific implementation, when each acquisition device in the acquisition array is started manually, there is a start-up time error, and it is possible that the acquisition of the video data stream may not start at the same time. Therefore, at least one of the following methods can be adopted to ensure that each collection device in the collection array collects video data streams in real time and synchronously from corresponding angles:

1. When at least one collection device obtains the collection start instruction, the collection device that has obtained the collection start instruction synchronizes the collection start instruction to other collection devices, so that each collection device in the collection array is based on The collection start instruction starts synchronously collecting video data streams from corresponding angles in real time.

For example, the collection array may contain 40 collection devices. When the collection device A1 obtains the collection start instruction, the collection device A1 synchronously sends the obtained collection start instruction to other collection devices A2-A40. After all the collection devices receive the collection start instruction, each collection device starts synchronously collecting video data streams from corresponding angles in real time based on the collection start instruction. Since the data transmission speed between the acquisition devices is far faster than the speed of manual startup, the startup time error caused by manual startup can be reduced.

2. Each acquisition device in the acquisition array synchronously acquires video data streams in real time from corresponding angles based on a preset clock synchronization signal.

For example, a clock signal synchronizing device can be set, and each acquisition device can be connected to the clock signal synchronizing device respectively, and the clock signal synchronizing device can A clock synchronization signal is transmitted to each acquisition device, and each acquisition device starts synchronously collecting video data streams from corresponding angles in real time based on the clock synchronization signal. Since the clock signal transmitting device can transmit a clock synchronization signal to each acquisition device based on a preset trigger signal, so that each acquisition device can collect synchronously and is not easily disturbed by external conditions and manual operations, therefore, the synchronization accuracy of each acquisition device can be improved and synchronization efficiency.

In the specific implementation, due to the impact of the network transmission environment, each acquisition device in the acquisition array may not be able to receive the streaming command at the same time, and there may be a time difference of a few milliseconds or less between each acquisition device, resulting in each acquisition The video data streams transmitted by the devices in real time are not synchronized. As shown in Figure 21, the acquisition array includes acquisition devices 1 and 2. The acquisition parameters of acquisition devices 1 and 2 are set the same, where the acquisition frame rate is X fps, and acquisition device 1 And 2 captured video frames frame-level synchronous capture.

The acquisition interval T of each frame in acquisition devices 1 and 2 is Assume that the data processing device sends the streaming command r at time t0, the collection device 1 receives the streaming command r at time t1, and the collection device 2 receives the streaming command r at time t2. If both collection devices 1 and 2 are at the same collection interval Received within T, it can be considered that acquisition devices 1 and 2 receive streaming instructions at the same time, and acquisition devices 1 and 2 can transmit frame-level synchronous video data streams respectively; if acquisition devices 1 and 2 do not receive in the same acquisition interval , it can be considered that the acquisition devices 1 and 2 did not receive the streaming command at the same time, and the acquisition devices 1 and 2 cannot realize the synchronous transmission of frame-level video data streams. Frame-level synchronization of video data stream transmission may also be referred to as pull-stream synchronization. Once streaming synchronization is achieved, it will continue automatically until the streaming is stopped.

Reasons for not being able to transmit a frame-level synchronized video stream can be:

1) It is necessary to send streaming commands to each collection device separately;

2) There is a delay when the LAN transmits streaming commands.

Therefore, at least one of the following methods can be used to determine whether the video data streams respectively transmitted by the acquisition devices in the acquisition array are frame-level synchronized:

1. When obtaining the Nth frame of the video data stream transmitted by each acquisition device in the acquisition array, match the feature information of the object in the Nth frame of each video data stream, when the Nth frame of each video data stream When the characteristic information of the object satisfies the preset similarity threshold, it is determined that the characteristic information of the object in the Nth frame of the video data stream transmitted by each acquisition device in the acquisition array is consistent, and then the video data streams respectively transmitted by each acquisition device are consistent. Frame-level synchronization.

Wherein, N is an integer not less than 1, and the feature information of the object in the Nth frame of each video data stream may include at least one of shape feature information, color feature information, and position feature information.

2. When acquiring the Nth frame of the video data stream respectively transmitted by each acquisition device in the acquisition array, match the time stamp information of the Nth frame of each video data stream, where N is an integer not less than 1. When the time stamp information of the Nth frame of each video data stream is consistent, it is determined that the frame-level synchronization between the video data streams respectively transmitted by each acquisition device is determined.

When the video data streams transmitted by each acquisition device in the acquisition array are not synchronized at the frame level, re-send streaming instructions to each acquisition device in the acquisition array, and at least one of the above methods can be used to determine whether frame-level synchronization until the frame-level synchronization between the video data streams respectively transmitted by the acquisition devices in the acquisition array.

In a specific implementation, the video frames in the video data streams of each acquisition device can also be intercepted and transmitted to the designated target end. In order to ensure the frame-level synchronization of the intercepted video frames, as shown in Figure 22, the following steps can be included:

S221. Determine one of the video data streams among the video data streams of each acquisition device in the acquisition array received in real time as a reference data stream.

S222. Based on the received video frame interception instruction, determine the video frame to be intercepted in the reference data stream, and select videos in the remaining video data streams that are synchronized with the video frame to be intercepted in the reference data stream frame, as the video frame to be intercepted for the remaining video data streams.

S223. Intercept video frames to be intercepted in each video data stream.

S224. Synchronously upload the intercepted video frames to the designated target end.

Wherein, the specified target end may be a preset target end, or a target end specified by a video frame interception instruction.

By adopting the above solution, the synchronization of frame cutting can be realized, the efficiency of frame cutting can be improved, the display effect of the generated multi-angle free-view video can be further improved, and user experience can be enhanced. Moreover, it can reduce the coupling between the process of selecting and intercepting video frames and the process of generating multi-angle free-view video, enhance the independence between each process, facilitate later maintenance, and upload the captured video frames to the specified On the target side, it can save network transmission resources and reduce data processing load, and increase the speed of data processing to generate multi-angle free-view video.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, how to determine video frames to be intercepted in each video data stream will be described in detail below through specific application examples.

One way is to select, according to the feature information of the object in the video frame to be intercepted in the reference data stream, the video frame consistent with the feature information of the object in the remaining video data streams, as the waiting frame of the remaining video data streams. Captured video frame.

For example, the acquisition array contains 40 acquisition devices, therefore, 40 video data streams can be received in real time, assuming that in the video data streams of each acquisition device in the acquisition array received in real time, the video data corresponding to the acquisition device A1' is determined Stream A1 is used as the reference data stream, and then, based on the received video frame clipping instruction indicating the feature information X of the object in the clipped video frame, determine the video frame a1 in the reference data stream that is consistent with the feature information X of the object As the video frame to be intercepted, then according to the characteristic information x1 of the object in the video frame a1 to be intercepted in the reference data stream, select the video frame a2 consistent with the characteristic information x1 of the object in the remaining video data streams A2-A40 -a40, as the video frames to be intercepted for the rest of the video data streams.

Wherein, the feature information of the object may include shape feature information, color feature information, and position feature information, etc.; the feature information X of the object in the video frame indicated in the video frame clipping instruction is the same as that to be clipped in the reference data stream The feature information x1 of the object in the video frame a1 can be the same representation of the feature information of the same object, for example, the feature information X and x1 of the object are two-dimensional feature information; the feature information X of the object and the feature information of the object x1 may also be different representations of the feature information of the same object, for example, the feature information X of the object may be two-dimensional feature information, and the feature information x1 of the object may be three-dimensional feature information. Moreover, a similarity threshold can be preset, and when the similarity threshold is met, it can be considered that the characteristic information X of the object is consistent with x1, or the characteristic information x1 of the object is consistent with the characteristic information x2-x40 of the object in the remaining video data streams A2-A40 .

The specific representation manner of the feature information of the object and the similarity threshold may be determined according to the preset multi-angle free viewing angle range and the on-site scene, which are not limited in this embodiment.

Another way is, according to the timestamp information of the video frame in the reference data stream, select the video frame consistent with the timestamp information in the remaining video data streams as the video to be intercepted in the remaining video data streams frame.

For example, the acquisition array may include 40 acquisition devices, therefore, 40 video data streams may be received in real time, assuming that the video data corresponding to acquisition device B1 is determined in the video data streams of each acquisition device in the acquisition array received in real time stream B1 as the reference data stream, and then, based on the time stamp information Y indicating the video frame to be intercepted in the received video frame interception instruction, determine the video frame b1 corresponding to the time stamp information Y in the reference data stream as the to-be The intercepted video frame, then according to the timestamp information y1 in the video frame b1 to be intercepted in the reference data stream, select the video frame b2-b40 consistent with the timestamp information y1 in the remaining video data streams B2-B40, As the video frames to be intercepted for the remaining video data streams.

Wherein, the time stamp information Y indicating the video frame to be intercepted in the video frame interception instruction may have a certain error from the time stamp information y1 in the video frame b1 to be intercepted in the reference data stream, for example, the The timestamp information corresponding to the video frame in the reference data stream is inconsistent with the timestamp information Y. If there is an error of 0.1ms, an error range can be preset. For example, if the error range is ±1ms, then the error of 0.1ms is within the error range Therefore, the video frame b1 corresponding to the time stamp information y1 with a difference of 0.1 ms from the time stamp information Y may be selected as the video frame to be intercepted in the reference data stream. The specific error range and the selection rule of the time stamp information y1 in the reference data stream can be determined according to the on-site collection equipment and the transmission network, which is not limited in this embodiment.

It can be understood that, the method for determining the video frame to be intercepted in each video data stream in the foregoing embodiment may be used alone or at the same time, which is not limited in this embodiment of the present invention.

By adopting the above solution, the efficiency and result accuracy of synchronous selection and synchronous interception of video frames can be improved, thereby improving the integrity and synchronization of transmitted data.

The embodiment of the present invention also provides a data processing device corresponding to the above data processing method. In order to enable those skilled in the art to better understand and implement the embodiment of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings.

Referring to the schematic structural diagram of the data processing device shown in FIG. 23, in the embodiment of the present invention, as shown in FIG. 23, the data processing device 230 may include:

The instruction sending unit 231 is adapted to send streaming instructions to each acquisition device in the acquisition array, wherein each acquisition device in the acquisition array is placed at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, and the Each acquisition device in the acquisition array collects video data streams in real time and synchronously from corresponding angles;

The data stream receiving unit 232 is adapted to receive in real time the video data streams respectively transmitted by each acquisition device in the acquisition array based on the stream pulling instruction;

The first synchronization judging unit 233 is adapted to determine whether the video data streams transmitted by the acquisition devices in the acquisition array are frame-level synchronous, and whether the video data streams transmitted by the acquisition devices in the acquisition array are not synchronized at the frame level. During frame-level synchronization, the instruction sending unit 231 is re-triggered until the frame-level synchronization between the video data streams respectively transmitted by the acquisition devices in the acquisition array.

Wherein, the data processing device may be set according to actual scenarios. For example, when there is free space on site, the data processing device can be placed in a non-acquisition area on site as an on-site server; when there is no free space on site, the data processing device can be placed in the cloud as a cloud server.

Using the above data processing equipment, by determining whether the video data streams transmitted by each acquisition equipment in the acquisition array are frame-level synchronous, it can ensure the synchronous transmission of multi-channel data, thereby avoiding the problem of frame leakage and multi-frame transmission, and improving data processing Speed, and thus meet the needs of low-latency playback of multi-angle free-view video.

In an embodiment of the present invention, as shown in FIG. 23, the data processing device 230 may further include:

The reference video stream determination unit 234 is adapted to determine one of the video data streams among the video data streams of each acquisition device in the acquisition array received in real time as the reference data stream;

The video frame selection unit 235 is adapted to determine the video frame to be intercepted in the reference data stream based on the received video frame interception instruction, and select the remaining video frames synchronized with the video frame to be intercepted in the reference data stream. The video frame in the video data stream is used as the video frame to be intercepted of the remaining video data streams;

A video frame intercepting unit 236, adapted to intercept video frames to be intercepted in each video data stream;

The uploading unit 237 is adapted to synchronously upload the intercepted video frames to the designated target.

Wherein, the data processing device 230 may establish a connection with a target end through a port or an IP address in advance, and may also synchronously upload the intercepted video frame to the port or IP address specified by the video frame interception instruction.

By adopting the above solution, the synchronization of frame cutting can be realized, the efficiency of frame cutting can be improved, the display effect of the generated multi-angle free-view video can be further improved, and user experience can be enhanced. Moreover, the coupling between the process of selecting and intercepting video frames and the process of generating multi-angle free-view video is reduced, the independence between each process is enhanced, and it is convenient for later maintenance, and the captured video frames are uploaded to the specified target synchronously The terminal can save network transmission resources and reduce data processing load, and improve the speed of data processing to generate multi-angle free-view video.

In an embodiment of the present invention, as shown in FIG. 23, the video frame selection unit 235 includes at least one of the following:

The first video frame selection module 2351 is adapted to select video frames consistent with the feature information of the object in the remaining video data streams according to the feature information of the object in the video frame to be intercepted in the reference data stream, as the rest Video frames to be intercepted for each video data stream;

The second video frame selection module 2352 is adapted to select, according to the time stamp information of the video frames in the reference data stream, video frames consistent with the time stamp information in the remaining video data streams, as the time stamp information of the remaining video data streams The video frame to be captured.

By adopting the above solution, the efficiency and result accuracy of synchronous selection and synchronous interception of video frames can be improved, thereby improving the integrity and synchronization of transmitted data.

The embodiment of the present invention also provides a data synchronization system corresponding to the above-mentioned data processing method. The above-mentioned data processing device is used to realize real-time reception of multiple video data streams. In order to enable those skilled in the art to better understand and implement the embodiment of the present invention, the following With reference to the accompanying drawings, detailed description will be given through specific embodiments.

With reference to the structural diagram of the data synchronization system shown in Figure 24, in the embodiment of the present invention, the data synchronization system 240 may include: a collection array 241 placed in the on-site collection area and a collection array 241 placed in a link connection with the collection array Data processing device 242, the acquisition array 241 includes a plurality of acquisition devices, each acquisition device in the acquisition array 241 is located at different positions in the on-site acquisition area according to the preset multi-angle free viewing angle range, wherein:

Each acquisition device in the acquisition array 241 is suitable for synchronously acquiring video data streams from corresponding angles in real time, and based on the streaming command sent by the data processing device 242, the obtained video data streams are transmitted to the data processing unit in real time. device 242;

The data processing device 242 is adapted to send streaming instructions to each collection device in the collection array 241, and receive in real time the video data streams respectively transmitted by each collection device in the collection array 241 based on the streaming instructions, And when there is no frame-level synchronization between the video data streams transmitted by each acquisition device in the acquisition array 241, re-send the streaming instruction to each acquisition device in the acquisition array 241, until each acquisition device in the acquisition array 241 Frame-level synchronization between the video data streams transmitted by the capture device.

By adopting the data synchronization system in the embodiment of the present invention, by determining whether the video data streams transmitted by each acquisition device in the acquisition array are frame-level synchronized, it can ensure the synchronous transmission of multi-channel data, thereby avoiding frame-missing and multi-frame transmission problem, improve the data processing speed, and then meet the needs of low-latency playback of multi-angle free-view video.

In a specific implementation, the data processing device 242 is also adapted to determine one of the video data streams in the video data streams of each acquisition device in the acquisition array 241 received in real time as a reference data stream; based on the received video frame interception Instructions, determine the video frame to be intercepted in the reference data stream, and select video frames in the remaining video data streams synchronized with the video frame to be intercepted in the reference data stream, as the video frames of the remaining video data streams The video frame to be intercepted: intercepting the video frame to be intercepted in each video data stream and synchronously uploading the intercepted video frame to the specified target end.

Wherein, the data processing device 240 may establish a connection with a target end through a port or an IP address in advance, and may also synchronously upload the intercepted video frame to the port or IP address specified by the video frame interception instruction.

In an embodiment of the present invention, the data synchronization system 240 may also include a cloud server, which is suitable as a designated target.

In another embodiment of the present invention, as shown in FIG. 34 , the data synchronization system 240 may further include a playback control device 341, which is suitable as a designated target end.

In yet another embodiment of the present invention, as shown in FIG. 35 , the data synchronization system 240 may further include an interactive terminal 351, which is suitable as a specified target terminal.

In an embodiment of the present invention, at least one of the following methods can be adopted to ensure that each acquisition device in the acquisition array 241 collects video data streams in real time and synchronously from corresponding angles:

1. The collection devices in the collection array are connected through a synchronization line, wherein, when at least one collection device obtains a collection start instruction, the collection device that has obtained the collection start instruction sends the The acquisition start instruction is synchronized to other acquisition devices, so that each acquisition device in the acquisition array starts synchronously collecting video data streams from corresponding angles in real time based on the acquisition start instruction;

2. Each acquisition device in the acquisition array synchronously acquires video data streams in real time from corresponding angles based on a preset clock synchronization signal.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the data synchronization system is described in detail below through specific application scenarios. FIG. 25 shows a schematic structural diagram of the data synchronization system in the application scenario, wherein the data The synchronization system includes a collection array 251 composed of various collection devices, a data processing device 252, and a server cluster 253 in the cloud.

At least one acquisition device in each acquisition device in the acquisition array 251 acquires the acquisition start command, and synchronizes the acquired acquisition start command to other acquisition devices through the synchronization line 254, so that each acquisition device in the acquisition array The acquisition device respectively starts synchronously acquiring video data streams from corresponding angles in real time based on the acquisition start instruction.

The data processing device 252 may send streaming instructions to each collection device in the collection array 251 through a wireless local area network. Each acquisition device in the acquisition array 251 transmits the obtained video data stream to the data processing device 252 in real time through the switch 255 based on the streaming instruction sent by the data processing device 252 .

The data processing device 252 determines whether the video data streams transmitted by the acquisition devices in the acquisition array 251 are frame-level synchronous, and whether the video data streams transmitted by the acquisition devices in the acquisition array 251 are synchronized at the frame level. During frame-level synchronization, re-send streaming instructions to each acquisition device in the acquisition array 251 until the frame-level synchronization between the video data streams transmitted by each acquisition device in the acquisition array 251 .

After the data processing device 252 determines the frame-level synchronization between the video data streams transmitted by the acquisition devices in the acquisition array 251, it determines that one of the video data streams in the video data streams of each acquisition device in the acquisition array 251 received in real time stream as the reference data stream, and, after receiving the video frame clipping instruction, determine the video frame to be clipped in the reference data stream according to the video frame clipping instruction, and then, the data processing device 252 selects the The video frames in the remaining video data streams synchronized with the video frames to be intercepted in the reference data stream are used as the video frames to be intercepted in the remaining video data streams, and then the video frames to be intercepted in each video data stream are intercepted and The captured video frames are uploaded to the cloud synchronously.

The server cluster 253 in the cloud will perform subsequent processing on the intercepted video frames to obtain a multi-angle free-view video for playback.

In a specific implementation, the cloud server cluster 253 may include: a first cloud server 2531 , a second cloud server 2532 , a third cloud server 2533 , and a fourth cloud server 2534 . Wherein, the first cloud server 2531 can be used for parameter calculation; the second cloud server 2532 can be used for depth calculation to generate a depth map; the third cloud server 2533 can be used for DIBR to preset virtual viewpoint path Perform frame image reconstruction; the fourth cloud server 2534 can be used to generate multi-angle free-view video.

It can be understood that the data processing equipment can be placed in the non-acquisition area of the site according to the actual situation, or placed in the cloud. In practical applications, the data synchronization system can use at least one of a cloud server, a playback control device or an interactive terminal. The transmitting end of the video frame interception instruction may also use other devices capable of transmitting the video frame interception instruction, which is not limited in this embodiment of the present invention.

It should be noted that the data synchronization system in the embodiment of the present invention can be applied to the data processing system and the like in the foregoing embodiments.

The embodiment of the present invention also provides a collection device corresponding to the above-mentioned data processing method. The video data stream is collected synchronously in real time, and when receiving the streaming instruction sent by the data processing device, the obtained video data stream is transmitted to the data processing device in real time. In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the following describes in detail through specific embodiments with reference to the accompanying drawings.

Referring to the schematic structural diagram of the acquisition device shown in Figure 36, in the embodiment of the present invention, the acquisition device 360 includes: a photoelectric conversion camera assembly 361, a processor 362, an encoder 363, and a transmission component 365, wherein:

A photoelectric conversion camera assembly 361, suitable for collecting images;

The processor 362 is adapted to synchronize the acquisition initiation instruction to other acquisition devices through the transmission component 365 when the acquisition initiation instruction is obtained, and start to perform real-time processing of the images collected by the photoelectric conversion camera assembly 361. Processing to obtain the image data sequence, and when the streaming instruction is obtained, the video data stream obtained is transmitted to the data processing device in real time through the transmission component 365;

The encoder 363 is adapted to encode the image data sequence to obtain a corresponding video data stream.

As an optional solution, as shown in FIG. 36 , the collection device 360 may further include a recording component 364 adapted to collect sound signals and obtain audio data.

The collected image data sequence and audio data can be processed by the processor 362 , and then the collected image data sequence and audio data can be encoded by the encoder 363 to obtain a corresponding video data stream. And when the processor 362 obtains the collection start instruction, it can synchronize the collection start instruction to other collection devices through the transmission component 365; The stream is transmitted in real time to the data processing device.

In a specific implementation, the acquisition device can be placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and the acquisition device can be fixed at a certain point in the on-site acquisition area, or can be moved within the on-site acquisition area To form an acquisition array. Therefore, the acquisition device may be a fixed device or a mobile device, so that video data streams may be flexibly collected from multiple angles.

As shown in Figure 37, it is a schematic diagram of an acquisition array in an application scenario in an embodiment of the present invention, with the center of the stage as the core point of view, the core point of view as the center of the circle, and the fan-shaped area where the core point of view is located on the same plane as the preset Multi-angle free viewing angle range. The collection devices 371-375 in the collection array are fan-shaped and placed at different positions in the on-site collection area according to the preset multi-angle free viewing angle range. The collection device 376 is a mobile device that can be moved to a designated location according to instructions for flexible collection. Moreover, the collection device can be a handheld device, which is used to supplement the collection data when the collection device breaks down or in a small space. For example, the handheld device 377 located in the stage audience area in FIG. 37 can be added to the collection array to provide stage audiences Region's video data stream.

As mentioned above, in order to generate multi-angle free-view data, depth map calculation is required, but currently the depth map calculation takes a long time, how to reduce the time for depth map generation and improve the depth map generation rate has become an urgent problem to be solved.

In view of the above problems, embodiments of the present invention provide computing node clusters, and multiple computing nodes can simultaneously generate depth maps in parallel and batch-processed texture data synchronously collected by the same collection array. Specifically, the depth map calculation process can be divided into multiple steps: obtaining a rough depth map through the first depth calculation, determining the unstable area in the rough depth map, and then calculating the second depth. In each step, the calculation Multiple computing nodes in the node cluster can perform the first depth calculation on the texture data collected by multiple acquisition devices in parallel to obtain a rough depth map, and verify and perform the second depth calculation on the obtained rough depth map in parallel, so that It can save the time for calculating the depth map and improve the generation rate of the depth map. Hereinafter, with reference to the accompanying drawings, it will be further described in detail through specific embodiments.

Referring to the flowchart of a method for generating a depth map shown in FIG. 26 , in the embodiment of the present invention, multiple computing nodes in the computing node cluster are used to generate the depth map respectively. For the convenience of description, any of the computing node clusters A computing node is referred to as a first computing node. The following is a detailed description of the method for generating the depth map of the computing node cluster through specific steps:

S261. Receive texture data, where the texture data is synchronously collected by multiple collection devices in the same collection array.

In a specific implementation, the plurality of acquisition devices can be placed in different positions of the on-site acquisition area according to the preset multi-angle free viewing angle range, and the acquisition devices can be fixed at a certain point in the on-site acquisition area, or can be placed in the on-site acquisition area Move within to form an acquisition array. Wherein, the multi-angle free viewing angle may refer to the spatial position and viewing angle of the virtual viewpoint enabling the scene to be switched freely. For example, the multi-angle free viewing angle can be a viewing angle of 6 degrees of freedom (6DoF), and the acquisition equipment used in the acquisition array can be a general-purpose camera, camera, video recorder, handheld device such as a mobile phone, etc. For specific implementation, please refer to other embodiments of the present invention , which will not be repeated here.

The texture data is the pixel data of the two-dimensional image frame collected by the aforementioned acquisition device, which may be an image at a frame time, or may be the pixel data of a frame image corresponding to a video stream formed by continuous or discontinuous frame images.

S262. The first calculation node performs a first depth calculation according to the first texture data and the second texture data to obtain a first rough depth map.

Here, in order to describe more clearly and concisely, among the texture data, the texture data that satisfies the preset first mapping relationship with the first computing node is referred to as the first texture data; The texture data collected by the collection device of the set first spatial position relationship is called the second texture data.

In a specific implementation, the first mapping relationship may be obtained based on a preset first mapping relationship table or through random mapping. For example, the texture data processed by each computing node may be pre-allocated according to the number of computing nodes in the computing node cluster and the number of acquisition devices in the acquisition array corresponding to the texture data. A dedicated allocation node may be set up to allocate computing tasks of each computing node in the computing node cluster, and the allocation node may obtain the first mapping relationship based on a preset first mapping relationship table or through random mapping. For example, if there are 40 collection devices in the collection array, in order to achieve the highest concurrent processing efficiency, 40 computing nodes can be configured, and each collection device corresponds to a computing node. However, if there are only 20 computing nodes, and the processing capabilities of each computing node are the same or roughly equivalent, in order to achieve the highest concurrent processing efficiency and balance load requirements, you can set each computing node to correspond to two collection devices to collect texture data. Specifically, the mapping relationship between the acquisition device identifier corresponding to the texture data and the identifier of each computing node can be set as the first mapping relationship, and the corresponding acquisition device in the acquisition array is directly acquired based on the first mapping relationship The texture data of is distributed to the corresponding computing nodes. In the specific implementation, computing tasks can also be assigned randomly, and the texture data collected by each acquisition device in the acquisition array can be randomly assigned to each computing node in the computing node cluster. Therefore, in order to improve processing efficiency, the acquisition array can be allocated in advance All the texture data collected are copied on each computing node in the computing node cluster.

As an example, any server in the server cluster may perform the first depth calculation according to the first texture data and the second texture data.

Wherein, for the preset first spatial position relationship between the first texture data and the second texture data, for example, the second texture data may meet the preset first texture data acquisition device. The texture data collected by a collection device with a distance relationship, or the texture data collected by a collection device that satisfies a preset first quantitative relationship with the collection device of the first texture data, may also be the texture data that is related to the first texture data The texture data collected by the data collection device that meets the preset first distance relationship and the preset first quantitative relationship.

Wherein, the first preset number can take any integer value from 1 to N-1, and N is the total number of acquisition devices in the acquisition array. In an embodiment of the present invention, the first preset number is 2, so that the highest possible image quality can be obtained with as little computation as possible. For example, assuming that the computing node 9 corresponds to the camera 9 in the preset first mapping relationship, the texture data of the camera 9 and the texture data of the cameras 5, 6, 7, 10, 11, and 12 adjacent to the camera 9 can be used , to obtain a rough depth map of the camera 9 through calculation.

It can be understood that, in a specific implementation, the second texture data may also be data collected by a collection device that satisfies another type of first spatial position relationship with the collection device of the first texture data, for example, the first texture data The spatial position relationship may also satisfy a preset angle, satisfy a preset relative position, and the like.

S263. The first computing node synchronizes the first rough depth map to other computing nodes in the computing node cluster to obtain a rough depth map set.

The rough depth map obtained after the rough calculation of the depth map needs to be cross-validated to determine the unstable area in each rough depth map, so as to perform a refined solution in the next step. Wherein, for any rough depth map in the rough depth map set, cross-validation needs to be performed through the rough depth maps corresponding to multiple collection devices around the collection device corresponding to the rough depth map. (Typically, the rough depth map to be verified and the rough depth map corresponding to all other acquisition devices are cross-validated together), so the rough depth map calculated by each computing node needs to be synchronized to the rest of the computing node cluster After the computing nodes are synchronized in step S263, each computing node in the computing node cluster obtains a rough depth map calculated by other computing nodes in the computing node cluster, and each server obtains the same rough depth map set.

S264. The first calculation node uses a third rough depth map to verify the second rough depth map in the rough depth map set, and obtains an unstable region in the second rough depth map.

Wherein, the second rough depth map and the first computing node may satisfy a preset second mapping relationship; the third rough depth map may be that the acquisition device corresponding to the second rough depth map satisfies a preset A rough depth map corresponding to the acquisition device of the second spatial position relationship.

The second mapping relationship may be obtained based on a preset second mapping relationship table or through random mapping. For example, the texture data processed by each computing node may be pre-allocated according to the number of computing nodes in the computing node cluster and the number of acquisition devices in the acquisition array corresponding to the texture data. In a specific implementation, a dedicated allocation node can be set to allocate computing tasks of each computing node in the computing node cluster, and the allocation node can obtain the second mapping relationship based on a preset second mapping relationship table or through random mapping. For a specific example of setting the second mapping relationship, reference may be made to the aforementioned implementation example of the first mapping relationship.

It can be understood that, in a specific implementation, the second mapping relationship may completely correspond to the first mapping relationship, or may not correspond to the first mapping relationship. For example, when the number of cameras is equal to the number of computing nodes, a one-to-one second mapping relationship can be established between the acquisition device corresponding to the data (including texture data and a rough depth map) and the identification of the computing node processing the data according to the hardware identifier .

It can be understood that the descriptions of the first rough depth map, the second rough depth map and the third rough depth map here are only for clarity and brevity. In a specific implementation, the first rough depth map may be the same as or different from the second rough depth map; the acquisition device corresponding to the third rough depth map is the same as the acquisition device corresponding to the second rough depth map It only needs to satisfy the preset second spatial position relationship.

For the second spatial position relationship, as a specific example, the texture data corresponding to the third rough depth map may be collected by a collection device corresponding to the second rough depth map that satisfies the preset second distance relationship. The obtained texture data, or the texture data corresponding to the third texture depth map may be the texture data collected by the collection device corresponding to the second rough depth map that satisfies the preset second quantitative relationship, or The texture data corresponding to the third rough depth map is texture data collected by a collection device corresponding to the second rough depth map that satisfies a preset second distance relationship and a second quantity relationship.

Wherein, the second preset number can take any integer value from 1 to N-1, and N is the total number of acquisition devices in the acquisition array. In a specific implementation, the second preset number may be equal to or different from the first preset number. In an embodiment of the present invention, the second preset number is 2, so that the highest possible image quality can be obtained with as little computation as possible.

In a specific implementation, the second spatial position relationship may also be another type of spatial position relationship, such as satisfying a preset angle, satisfying a preset relative position, and the like.

S265. The first calculation node performs a second depth calculation according to the unstable region in the second rough depth map, the texture data corresponding to the second rough depth map, and the texture data corresponding to the third rough depth map. Calculate to obtain the corresponding fine depth map.

It should be noted here that the difference between the second depth calculation and the first depth calculation is that the depth map candidate values in the second rough depth map selected by the second depth calculation do not include the depth value of the unstable region, Therefore, unstable regions in the generated depth map can be eliminated, so that the generated depth map is more accurate, and the quality of the generated multi-angle free-view image can be improved.

To illustrate with an example application scenario:

The first round of depth calculation (first depth calculation) can be performed by the server S based on the texture data of the allocated camera M and the texture data of the camera that meets the preset first spatial position relationship with the camera M to obtain a rough depth map .

After the cross-validation in step S264, the refined solution of the depth map can be continuously performed on the same server. Specifically, the server S can cross-validate the assigned rough depth map corresponding to the camera M with the results of all other rough depth maps, and can obtain the unstable region in the rough depth map corresponding to the camera M. After that, the server S The unstable area in the rough depth map corresponding to the assigned camera M, the texture data collected by the camera M, and the texture information of the N cameras around the camera M can be used for another round of depth map calculation (the second depth calculation), namely A refined depth map corresponding to the first texture data (texture data collected by the camera M) can be obtained.

Here, the rough depth map corresponding to the camera M is a rough depth map calculated based on the texture data collected by the camera M and the texture data collected by a collection device satisfying the preset first spatial position relationship with the camera M.

S266. Use the fine depth atlas set of the fine depth maps obtained by the computing nodes as the finally generated depth map.

By adopting the above-mentioned embodiments, multiple computing nodes can simultaneously generate depth maps in parallel and batch-processed texture data synchronously collected by the same collection array, thus greatly improving the efficiency of depth map generation.

In addition, by using the above scheme to calculate the depth twice, the unstable regions in the generated depth map are eliminated, so the obtained fine depth map is more accurate, and the quality of the generated multi-angle free-view image can be improved.

In specific implementation, according to the size of the data volume of the texture data to be processed and the demand for the generation speed of the depth map, an appropriate configuration of computing nodes and the number of computing nodes in the computing node cluster can be selected. For example, the computing node cluster may be a server cluster composed of multiple servers, and the multiple servers in the server cluster may be deployed in a centralized manner or in a distributed manner. In some embodiments of the present invention, part or all of the computing node devices in the computing node cluster may serve as local servers, or as edge node devices, or as cloud computing devices.

As another example, the computing node cluster may also be a computing device formed by multiple CPUs or GPUs. An embodiment of the present invention also provides a computing node, which is suitable for forming a computing node cluster with at least another computing node to generate a depth map. Referring to the schematic structural diagram of a computing node shown in FIG. 27 , the computing node 270 may include:

The input unit 271 is adapted to receive texture data, where the texture data is acquired synchronously from multiple acquisition devices in the same acquisition array;

The first depth calculation unit 272 is adapted to perform first depth calculation according to the first texture data and the second texture data to obtain a first rough depth map, wherein: the first texture data and the calculation nodes meet the preset requirements The first mapping relationship; the second texture data is texture data collected by a collection device that satisfies a preset first spatial position relationship with the collection device of the first texture data;

The synchronization unit 273 is adapted to synchronize the first rough depth map to other computing nodes in the computing node cluster to obtain a rough depth map set;

The verification unit 274, for the second rough depth map in the rough depth atlas, is adapted to use the third rough depth map for verification, and obtain the unstable area in the second rough depth map, wherein: the second rough The depth map and the computing node satisfy a preset second mapping relationship; the third rough depth map corresponds to a collection device corresponding to the second rough depth map that satisfies a preset second spatial position relationship rough depth map;

The second depth calculation unit 275 is adapted to perform the second calculation according to the unstable region in the second rough depth map, the texture data corresponding to the second rough depth map, and the texture data corresponding to the third rough depth map. Depth calculation to obtain a corresponding fine depth map, wherein: the depth map candidate values in the second rough depth map selected by the second depth calculation do not include the depth value of the unstable region;

The output unit 276 is adapted to output the fine depth map, so that the computing node cluster obtains a fine depth map set as a finally generated depth map.

Using the above calculation node, the depth map calculation process can include multiple steps such as obtaining a rough depth map through the first depth calculation, determining the unstable area in the rough depth map and subsequent second depth calculation, etc., and performing depth map calculation through the above steps , which is beneficial for multiple computing nodes to calculate separately, so that the generation efficiency of the depth map can be improved.

An embodiment of the present invention also provides a computing node cluster, the computing node cluster may include a plurality of computing nodes, and the multiple computing nodes in the computing node cluster may concurrently collect texture data synchronously acquired by the same acquisition array, Batch-based depth map generation. For convenience of description, any computing node in the computing node cluster is referred to as a first computing node.

In some embodiments of the present invention, the first calculation node is adapted to perform a first depth calculation according to the first texture data and the second texture data in the received texture data to obtain a first rough depth map; The first rough depth map is synchronized to the remaining computing nodes in the computing node cluster to obtain a rough depth map set; for the second rough depth map in the rough depth map set, the third rough depth map is used for verification, and the obtained An unstable area in the second rough depth map; and according to the unstable area in the second rough depth map, the texture data corresponding to the second rough depth map, and the texture data corresponding to the third rough depth map , performing a second depth calculation, obtaining a corresponding fine depth map, and outputting the obtained fine depth map so that the computing node cluster uses the obtained fine depth map set as a final generated depth map;

Wherein, the first texture data and the first computing node satisfy a preset first mapping relationship; the second texture data satisfy a preset first spatial position relationship with the acquisition device of the first texture data The texture data collected by the collection device; the second rough depth map and the first computing node satisfy a preset second mapping relationship; the third rough depth map is the collection corresponding to the second rough depth map The rough depth map corresponding to the acquisition device whose device satisfies the preset second spatial position relationship; and the depth map candidate values in the second rough depth map selected by the second depth calculation do not include the depth value of the unstable region.

Referring to the schematic diagram of depth image processing by a server cluster shown in FIG. 28 , the texture data collected by N cameras in the camera array are respectively input into N servers in the server cluster, and the first depth calculation is performed respectively to obtain rough depth images 1- N. After that, each server copies the rough depth map calculated by itself to other servers in the server cluster and realizes time synchronization. After that, each server verifies the rough depth map allocated by itself, and performs the second depth calculation. The depth map after fine calculation is obtained as the depth map generated by the server cluster. It can be seen from the above calculation process that each server in the server cluster can perform the first depth calculation on the texture data collected by multiple cameras in parallel, and verify and perform the second depth calculation on each rough depth map in the rough depth atlas. The entire depth map generation process is performed in parallel by multiple servers, which can greatly save the time of depth map calculation and improve the efficiency of depth map generation.

For the specific implementation manners and beneficial effects of the computing nodes and computing node clusters in the embodiments of the present invention, reference may be made to the methods for generating depth maps in the foregoing embodiments of the present invention, which will not be repeated here.

The server cluster can further store the generated depth map, or output it to the terminal device according to the request, so as to further generate and display the virtual view point image, which will not be repeated here.

An embodiment of the present invention also provides a computer-readable storage medium, on which computer instructions are stored. When the computer instructions are run, the steps of the method for generating a depth map described in any of the foregoing embodiments can be executed. For details, please refer to the foregoing depth map The steps of the generation method will not be repeated here.

In addition, currently known methods for generating virtual viewpoint images based on Depth-Image-Based Rendering (DIBR) are difficult to meet the requirements of multi-angle free-viewing applications in playback.

The inventor found through research that the current DIBR virtual view point image generation method has low concurrency and is usually processed by the CPU. However, for each virtual view point image, since the generation method involves many steps, each step is also relatively complicated. So it is more difficult to achieve through parallel processing methods.

In order to solve the above problems, the embodiment of the present invention provides a method for generating virtual view point images through parallel processing, which can greatly accelerate the timeliness performance of virtual view point image generation with multi-angle free view angles, thereby meeting the requirements of low time requirements for multi-angle free view point images. To meet the needs of delayed playback and real-time interaction, and improve user experience.

In order to make the purpose, features and advantages of the embodiments of the present invention more obvious to those skilled in the art, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Referring to the flow chart of the method for generating a virtual viewpoint image shown in FIG. 29 , in a specific implementation, the virtual viewpoint image can be generated through the following steps:

S291. Acquire an image combination of multi-angle free viewing angles, parameter data of the image combination, and preset virtual viewpoint path data, wherein the image combination includes multiple sets of corresponding texture maps and depth maps that are synchronized at multiple angles .

Wherein, the multi-angle free viewing angle may refer to the spatial position and viewing angle of the virtual viewing point enabling the scene to be switched freely. The range of multi-angle free viewing angles can be determined according to the needs of the application scenario.

In a specific implementation, by arranging a collection array composed of multiple collection devices on the site, each collection device in the collection array can be placed in different positions of the on-site collection area according to the preset multi-angle free viewing angle range, each collection device On-site images can be collected synchronously to obtain texture maps synchronized from multiple angles. For example, multiple cameras, video cameras, etc. may be used to collect images of a certain scene at multiple angles synchronously.

The images in the multi-angle free viewing angle image combination may be completely free viewing angle images. In a specific implementation, it may be a viewing angle of 6 degrees of freedom (Degree of Freedom, DoF), that is, the spatial position and viewing angle of the viewing point may be freely switched. As mentioned earlier, the spatial position of the viewpoint can be expressed as coordinates (x, y, z), and the viewing angle can be expressed as three rotation directions Therefore, it can be called 6DoF.

In the process of generating the virtual view point image, the image combination of multi-angle free viewing angles and the parameter data of the image combination can be obtained first.

In a specific implementation, there is a one-to-one correspondence between the texture map and the depth map in the image combination. Wherein, the texture map may adopt any type of two-dimensional image format, such as any one of BMP, PNG, JPEG, and webp formats. The depth map can represent the distance of each point in the scene relative to the shooting device, that is, each pixel value in the depth map represents the distance between a point in the scene and the shooting device.

A texture map in an image combination is a plurality of synchronized two-dimensional images. Depth data for each two-dimensional image may be determined based on the plurality of two-dimensional images.

Wherein, the depth data may include depth values corresponding to pixels of the two-dimensional image. The distance from the acquisition device to each point in the area to be viewed can be used as the above-mentioned depth value, and the depth value can directly reflect the geometric shape of the visible surface in the area to be viewed. For example, the depth value can be the distance from each point in the area to be viewed along the optical axis of the camera to the optical center, and the origin of the camera coordinate system can be used as the optical center. Those skilled in the art can understand that the distance can be a relative value, and the same reference can be used for multiple images.

The depth data may include depth values that correspond one-to-one to pixels of the two-dimensional image, or may be partial values selected from a set of depth values that correspond one-to-one to pixels of the two-dimensional image. Those skilled in the art can understand that the depth value set can be stored in the form of a depth map. In a specific implementation, the depth data can be the data obtained after downsampling the original depth map, and the two-dimensional image (texture map) The pixel-to-one depth value set is stored according to the pixel arrangement of the two-dimensional image (texture map), and the image form is the original depth map.

In the embodiment of the present invention, the image combination of multi-angle free viewing angles and the parameter data of the image combination can be obtained through the following steps, which will be described below through specific application scenarios.

As a specific embodiment of the present invention, it may include the following steps: the first step is to collect and calculate the depth map, including three main steps, which are respectively: multi-camera video capture (Multi-camera Video Capturing), camera internal and external parameter calculation (Camera Parameter Estimation), and depth map calculation (Depth MapCalculation). For multi-camera capture, it is required that the videos captured by each camera can be aligned at the frame level.

The texture image (Texture Image), that is, multiple synchronized images can be obtained through multi-camera video acquisition; the camera parameter (Camera Parameter), that is, the parameter data of image combination, including internal parameter data, can be obtained through the calculation of internal and external parameters of the camera and external parameter data; through the calculation of the depth map, the depth map (DepthMap) can be obtained.

Multiple sets of corresponding texture maps and depth maps synchronized in the image combination can be spliced together to form a spliced image. The stitched image can have multiple stitching structures. Each frame of spliced images can be combined as an image. Multiple sets of texture maps and depth maps in the image combination can be spliced and combined according to the preset relationship. Specifically, the texture map and depth map of the image combination can be divided into a texture map area and a depth map area according to the positional relationship. The texture map area stores the pixel values of each texture map respectively, and the depth map area stores each The depth value corresponding to the texture map. The texture map area and the depth map area can be continuous or distributed at intervals. In the embodiment of the present invention, there is no restriction on the positional relationship between the texture map and the depth map in the image combination.

In a specific implementation, the parameter data of each image in the image combination can be obtained from the attribute information of the image. Wherein, the parameter data may include external parameter data, and may also include internal parameter data. The external parameter data is used to describe the spatial coordinates and attitude of the shooting device, and the internal parameter data is used to describe the attribute information of the shooting device such as the optical center and focal length of the shooting device. Internal parameter data may also include distortion parameter data. The distortion parameter data includes radial distortion parameter data and tangential distortion parameter data. Radial distortion occurs in the process of converting the coordinate system of the shooting device to the physical coordinate system of the image. The tangential distortion occurs in the production process of the shooting equipment, which is because the plane of the photosensitive element is not parallel to the lens. Information such as the shooting position and shooting angle of the image can be determined based on the external parameter data. In the generation of virtual viewpoint images, combining internal parameter data including distortion parameter data can make the determined spatial mapping relationship more accurate.

In a specific implementation, the virtual viewpoint path may be preset. For example, for a sports game, such as a basketball game or a football match, an arc path can be planned in advance, for example, whenever a wonderful shot occurs, a corresponding virtual viewpoint image is generated according to the arc path.

In the specific application process, it can be based on a specific position or perspective on the scene (such as the basket, the sideline, the referee's perspective, the coach's perspective, etc.), or based on a specific object (such as players on the court, the host on the spot, the audience, and actors in film and television images, etc.) to set the virtual viewpoint path.

The route data corresponding to the virtual viewpoint route may include position data of a series of virtual viewpoints in the route.

S292. According to the preset virtual viewpoint path data and the parameter data of the image combination, select a corresponding group of texture maps and depth maps for each virtual viewpoint in the virtual viewpoint path from the image combination.

In a specific implementation, according to the position data of each virtual viewpoint in the virtual viewpoint path data and the parameter data of the image combination, select from the image combination that satisfies the preset positional relationship and/or quantity with each virtual viewpoint position The texture map and depth map of the corresponding set of relations. For example, for a virtual viewpoint position area with a high camera density, only the texture maps and corresponding depth maps captured by the two cameras closest to the virtual viewpoint can be selected; Select texture maps and corresponding depth maps captured by three or four cameras closest to the virtual viewpoint.

In an embodiment of the present invention, texture maps and depth maps corresponding to 2 to N acquisition devices nearest to each virtual viewpoint position in the virtual viewpoint path may be respectively selected, where N is the number of all acquisition devices in the acquisition array. For example, the texture map and the depth map corresponding to the two capture devices nearest to each virtual viewpoint position may be selected by default. In a specific implementation, the user can set the number of the selected collection devices closest to the virtual viewpoint position by himself, and the maximum number does not exceed the number of collection devices corresponding to the image combination.

In this way, there is no special requirement for the spatial position distribution of the acquisition devices in the acquisition array (for example, it can be linear distribution, arc array arrangement, or any irregular arrangement form), according to the acquired The virtual view point position data and the parameter data corresponding to the image combination determine the actual distribution of the acquisition equipment, and then adopt an adaptive strategy to select the texture map and depth map of the corresponding group in the image combination, so as to reduce the amount of data calculation, While ensuring the quality of the generated virtual viewpoint image, it provides a high degree of freedom of choice and flexibility, and also reduces the installation requirements for the acquisition equipment in the acquisition array, which is convenient for adapting to different site requirements and easy installation and operability.

In an embodiment of the present invention, according to the virtual viewpoint position data and the parameter data of the image combination, select a preset number of corresponding sets of texture maps and depth maps closest to the virtual viewpoint position from the image combination.

It can be understood that, in a specific implementation, other preset rules may also be used to select a corresponding set of texture maps and depth maps from the image combination. For example, it can also be selected from the image combination according to the processing capability of the virtual viewpoint image generation device, or according to the user's requirements for the generation speed and the definition requirements of the generated image (such as normal definition, high definition or ultra-high definition, etc.). Corresponding sets of texture maps and depth maps.

S293, input the corresponding texture map and depth map of each virtual view point into the graphics processor, and for each virtual view point in the virtual view point path, take the pixel as the processing unit, and use multiple threads to combine the selected images correspondingly The pixel points in the texture map and the depth map of the group are combined and rendered to obtain an image corresponding to the virtual viewpoint.

Graphics Processing Unit (GPU), also known as display core, visual processor, display chip, etc., is a microprocessor that specializes in image and graphics related operations, and can be configured in personal computers, workstations, video games, etc. Computers and some mobile terminals (such as tablet computers, smart phones, etc.) have image-related computing requirements in electronic devices.

In order to enable those skilled in the art to better understand and implement the embodiments of the present invention, the architecture of a GPU used in some embodiments of the present invention is briefly introduced below. It should be noted that the GPU architecture is only a specific example, and does not constitute a limitation on the applicable GPU in this embodiment of the present invention.

In some embodiments of the present invention, the GPU may use a unified device architecture (Compute Unified Device Architecture, CUDA) parallel programming framework to perform combined rendering on pixels in a corresponding group of texture maps and depth maps in the selected image combination. CUDA is a new hardware and software architecture for distributing and managing computations on GPUs as data-parallel computing devices without mapping them to a graphics Application Programming Interface (API).

When programmed through CUDA, a GPU can be thought of as a computing device capable of executing large numbers of threads in parallel. It runs as the main CPU or as a coprocessor to the host, in other words, the data-parallel, computationally intensive parts of the application running on the host are offloaded to the GPU.

Rather, parts of an application that execute multiple times independently of different data can be isolated into a single function that runs on the GPU device as if it were many different threads. To this end, such functions can be compiled into an instruction set of a GPU device, and the generated program (called a kernel (Kernel)) is downloaded to the GPU. Batches of threads executing a kernel are organized into thread blocks.

A thread block is a batch of threads that can be coordinated by sharing data efficiently and synchronizing their execution with some fast shared memory to coordinate memory access. In a specific implementation, a synchronization point can be specified in the kernel, and the threads in its thread block will be suspended until they all reach the synchronization point.

In a specific implementation, the maximum number of threads that a thread block can contain is limited. However, blocks of the same dimension and size executing the same kernel can be batched into a Grid of Thread Blocks so that the total number of threads that can be launched in a single kernel call is much larger.

It can be seen from the above that, with the CUDA structure, a large number of threads can process data in parallel on the GPU at the same time, so the generation speed of virtual viewpoint images can be greatly improved.

In order to enable those skilled in the art to better understand and implement, the process of processing each step of combined rendering in units of pixels will be introduced in detail below.

In specific implementation, referring to the flow chart of the method for combined rendering by GPU shown in FIG. 30 , step S293 can be implemented through the following steps:

S2931. Perform forward mapping on the corresponding group of depth maps in parallel to the virtual viewpoint.

The forward mapping of the depth map is to map the depth map of the original camera (acquisition device) to the position of the virtual camera through the transformation of the coordinate space position, so as to obtain the depth map of the virtual camera position. Specifically, the forward mapping of the depth map is an operation of mapping each pixel of the depth map of the original camera (acquisition device) to a virtual viewpoint according to a preset coordinate mapping relationship.

In a specific implementation, the first kernel (Kernel) function may be run on the GPU, and the pixels in the depth map of the corresponding group are forward-mapped in parallel to the corresponding virtual viewpoint positions.

The inventor found in the process of research and practice that in the process of forward mapping, there may be the problem of occlusion of the foreground and the background, as well as the effect of the mapping gap, which affects the quality of the generated image. First, to solve the problem of foreground and background occlusion, in the embodiment of the present invention, for multiple depth values mapped to the same pixel of the virtual viewpoint, an atomic operation can be used to obtain the maximum value of the pixel value to obtain the first value of the corresponding virtual viewpoint position. depth map. Afterwards, in order to improve the impact of the mapping gap effect, a second depth map of the virtual view point position may be created based on the first depth map of the virtual view point position, and each pixel in the second depth map is parallelized The processing is to obtain the maximum value of the pixel points in the preset area around the corresponding pixel position in the first depth map.

In the forward mapping process, since each pixel can be processed in parallel, the processing speed of the forward mapping can be greatly accelerated, and the time-effective performance of the forward mapping can be improved.

S2932. Perform post-processing on the forward-mapped depth map in parallel.

After the forward mapping is completed, the virtual viewpoint depth map can be post-processed, specifically, the preset second kernel function can be run on the GPU, and for each pixel in the second depth map obtained by the forward mapping, A median filtering process is performed in a preset area around the pixel position. Since the median filtering process can be performed on each pixel in the second depth map in parallel, the post-processing speed can be greatly accelerated, and the time-effective performance of the post-processing can be improved.

S2933. Perform reverse mapping on the corresponding group of texture maps in parallel.

This step is to calculate the coordinates of the virtual viewpoint position in the original camera texture map according to the value of the depth map, and calculate the corresponding value through sub-pixel interpolation calculation. In the GPU, the sub-pixel value can be directly interpolated according to bilinearity, so in this step, it is only necessary to directly obtain the value in the original camera texture according to the coordinates calculated by each pixel. In a specific implementation, the preset third core function may be run on the GPU, and the pixels in the corresponding selected texture map may be interpolated in parallel to generate a corresponding virtual texture map.

By running the third core function on the GPU, the pixels in the texture map of the selected corresponding group are interpolated in parallel to generate the corresponding virtual texture map, which can greatly speed up the processing speed of reverse mapping and improve the timeliness of reverse mapping performance.

S2934, merging the pixels in each virtual texture map generated after the reverse mapping in parallel.

In a specific implementation, the fourth kernel function may be run on the GPU, and the pixels at the same position in each virtual texture map generated after reverse mapping are weighted and fused in parallel.

Running the fourth core function on the GPU, the pixels in the same position in each virtual texture map generated after reverse mapping are weighted and fused in parallel, which can greatly speed up the fusion of virtual texture maps and improve the efficiency of image fusion. aging performance.

A specific example is used below to describe in detail.

In step S2931, for the forward mapping of the depth map, first, the projection mapping relationship of each pixel can be calculated through the first Kernel function of the GPU.

Assuming a certain pixel point (u, v) in the image of the real camera, first change the image coordinates (u, v) to the coordinates [X, Y, Z] ^T in the camera coordinate system through the perspective projection model of the corresponding camera. It is understandable that there are different conversion methods for perspective projection models of different cameras.

For example, for a perspective projection model:

Among them, [u, v, 1] ^T is the homogeneous coordinates of the pixel (u, v), [X, YZ] ^T is the coordinates of (u, v) corresponding to the real object in the camera coordinate system, f _x , f _y are the focal lengths in the x and y directions respectively, and c _x and _cy are the optical center coordinates in the x and y directions respectively.

Therefore, for a certain pixel point (u, v) in the image, the depth value Z of the known pixel and the physical parameters of the corresponding camera lens (f _x , f _y , c _x , _cy can be obtained from the parameter data of the aforementioned image combination obtained), the coordinates [X, YZ] ^T of the corresponding point in the camera coordinate system can be obtained through the above formula (1).

After the transformation from the image coordinate system to the camera coordinate system, the coordinates of the object in the current camera coordinate system can be transformed into the coordinate system of the camera where the virtual viewpoint is located according to the coordinate transformation in the three-dimensional space. Specifically, the following conversion formula can be used:

Wherein, R ₁₂ is a 3x3 rotation matrix, and T ₁₂ is a translation vector.

Assuming that the transformed three-dimensional coordinates are [X ₁ , Y ₁ , Z ₁ ] ^T , through the previous description from the image coordinate system to the camera coordinate system, and applying its inverse transformation, the transformed virtual camera three-dimensional coordinates and virtual camera can be obtained The corresponding relationship position of the image coordinates. Thus, the projection relationship of the points from the real viewpoint image to the virtual viewpoint image is established. By transforming each pixel in the real viewpoint and rounding the coordinate points, the projected depth map in the virtual viewpoint image can be obtained.

After establishing the point-to-point mapping relationship between the original camera depth map and the virtual camera depth map, during the projection process of the depth map, there may be multiple positions in the original camera depth map mapped to the same position in the virtual camera depth map, As a result, there is a foreground and background occlusion relationship in the forward mapping process of the depth map. To solve this problem, in the embodiment of the present invention, an atomic operation can be used to take the smallest depth map as the final result of the mapping position. As shown in formula (3):

Depth(u,v)=min[Depth _1-N (u,v)] (3)

It should be noted that the smallest depth value is also the largest pixel value in the depth map. Therefore, the first depth map corresponding to the virtual viewpoint position can be obtained by taking the largest pixel value on the mapped depth map.

In a specific implementation, the operation of obtaining the maximum or minimum value of multiple point maps can be provided in a CUDA parallel environment, specifically by calling the atomic operation function atomicMin or atomicMax provided by CUDA.

In the above process of obtaining the first depth map, a gap effect may occur, that is, some pixels may not be covered due to the problem of mapping accuracy. To solve this problem, the embodiment of the present invention may perform seam masking processing on the obtained first depth map. In an embodiment of the present invention, a 3*3 seam masking process is performed on the first depth map. The specific masking process is as follows:

First create a second depth map of the virtual view point position, and then, for each pixel D(x, y) in the second depth map, take the surrounding 3* in the first depth map of the virtual view point position 3 existing pixel points D_old(x, y), take the maximum value of the pixel points within the surrounding 3*3 range in the first depth map, which can be realized by the following kernel function operation:

D(x,y)=Max[D_old(X,y)] (4)

It can be understood that the size range of the surrounding area during the seam masking process may also take other values, such as 5*5. In order to obtain a better processing effect, it can be set according to experience.

For step S2932, in specific implementation, 3*3 or 5*5 median filtering may be performed on the second depth map of the virtual viewpoint position. For example, for a median filter of 3*3, the second kernel mapping function of the GPU may operate according to the following formula:

In step S2933, the third kernel function running on the GPU calculates the coordinates of the virtual viewpoint position in the original camera texture map according to the value of the depth map, and the third kernel function can be realized by executing the reverse process of step S2391.

In step S2934, for the pixel point f(x, y) of the virtual viewpoint position (x, y), the pixel values of the corresponding positions of the texture map obtained by all original camera mappings can be calculated according to the confidence degree conf(x, y). weighted. The fourth core function can be calculated using the following formula:

f(x,y)=∑conf(x,y)*f(x,y) (6)

Through the above steps S2931 to S2934, a virtual viewpoint image can be obtained. In a specific implementation, further processing and optimization can be performed on the virtual texture map obtained after weighted fusion. For example, hole filling may be performed on each pixel in the weighted and fused texture map in parallel to obtain an image corresponding to the virtual viewpoint.

For the hole filling of the virtual texture map, in a specific implementation, for each pixel, a separate windowing method may be used to perform parallel operations. For example, for each hole pixel, a window of size N*M may be opened, and then the value of the hole pixel is calculated by weighting according to the values of non-hole pixels in the window. Through the above method, the generation of the virtual viewpoint image can be completely processed in parallel on the GPU, so that the generation process can be greatly accelerated.

As shown in the schematic diagram of the hole filling method in FIG. 31 , for the generated virtual viewpoint view G, there is a hole area F, and rectangular windows a and b are respectively opened for the pixel f1 and the pixel f2 in the hole area F. After that, for pixel f1, all pixels are obtained from the existing non-hole pixels in the rectangular window (some pixels can also be obtained by down-sampling), and the value of pixel f1 in the hole area F is obtained according to distance weighting (or average weighting). Similarly, for the pixel f2, the value of the pixel f2 can be obtained by using the same operation. In a specific implementation, the fifth kernel function can be run on the GPU to parallelize the processing and speed up the hole filling time.

The fifth core function can be calculated using the following formula:

P(x,y)=Average[Window(x,y)] (7)

Among them, P(x,y) is the value of a certain point in the hole, Window(x,y) is the value (or downsampled value) of all existing pixels in the hole area, and Average is the average (or weighted average) of these pixels value.

In the embodiment of the present invention, in addition to generating the virtual viewpoint images of each virtual viewpoint position in parallel in units of pixels, in order to further speed up the generation efficiency of virtual viewpoint path images, the corresponding groups of virtual viewpoints in the virtual viewpoint path can be The texture map and depth map are input to multiple GPUs respectively, and multiple virtual viewpoint images are generated in parallel.

In a specific implementation, in order to further improve processing efficiency, each of the above steps may be performed by different block grids respectively.

Referring to the schematic structural diagram of the virtual viewpoint image generation system shown in FIG. 32, in the embodiment of the present invention, the virtual viewpoint image generation system 320 may include a CPU321 and a GPU322, wherein:

CPU 321, adapted to acquire multi-angle free-view image combination, parameter data of the image combination, and preset virtual viewpoint path data, wherein the image combination includes multiple sets of corresponding texture maps synchronized at multiple angles and a depth map; according to the preset virtual viewpoint path data and the parameter data of the image combination, select a texture map and a depth map corresponding to each virtual viewpoint in the virtual viewpoint path from the image combination;

The GPU322 is adapted to call corresponding kernel functions for each virtual viewpoint in the virtual viewpoint path, and combine and render corresponding groups of texture maps and pixels in the depth map in the selected image combination in parallel, so as to obtain an image corresponding to the virtual viewpoint. image.

Specifically, the GPU322 is adapted to perform forward mapping on the corresponding group of depth maps in parallel to the virtual viewpoint; perform post-processing on the forward-mapped depth maps in parallel; parallelize the corresponding group of texture maps performing reverse mapping; the pixels in each virtual texture map generated after reverse mapping are fused in parallel.

Wherein, the GPU322 can generate the virtual viewpoint images of each virtual viewpoint by using steps S2931 to S2934 and the hole filling step in the aforementioned method for generating virtual viewpoint images. For details, please refer to the introduction of the foregoing embodiments, which will not be repeated here.

In a specific implementation, there may be one GPU or multiple GPUs, as shown in FIG. 32 .

In a specific application, the GPU can be an independent GPU chip, or a GPU core in a GPU chip, or a GPU server, or a GPU chip packaged by multiple GPU chips or multiple GPU cores. , or a GPU cluster composed of multiple GPU servers.

Correspondingly, the texture maps and depth maps of the respective groups of virtual viewpoints in the virtual viewpoint path can be respectively input into multiple GPU chips, multiple GPU cores, or multiple GPU servers to generate multiple virtual viewpoints in parallel image. For example, the virtual view point path data corresponding to a certain virtual view point path contains 20 virtual view point position coordinates, and the data corresponding to the 20 virtual view point position coordinates can be input into multiple GPU chips in parallel, for example, there are 10 GPU chips in total. Then the data corresponding to the 20 virtual viewpoint position coordinates can be divided into two batches for parallel processing, and each GPU chip can generate the virtual viewpoint images corresponding to the virtual viewpoint positions in parallel in units of pixels, so the virtual viewpoint images can be greatly accelerated. The generation speed improves the time-sensitive performance of virtual viewpoint image generation.

The embodiment of the present invention also provides an electronic device. Referring to the schematic structural diagram of the electronic device shown in FIG. The computer instructions running on the CPU332 and the GPU333, when the CPU332 and the GPU333 cooperate to run the computer instructions, are suitable for executing the steps of the virtual viewpoint image generation method described in any of the foregoing embodiments of the present invention, for details, please refer to the foregoing embodiments A detailed introduction will not be repeated here.

In a specific implementation, the electronic device may be a server, or a server cluster composed of multiple servers.

Each of the above embodiments can be applied to a live broadcast scene, and two or more embodiments can be used in combination as required during the application process. Those skilled in the art can understand that the solutions in the above embodiments are not limited to live broadcast scenarios, and the solutions in the embodiments of the present invention for video or image acquisition, data processing of video data streams, and image generation by servers can also be applied to The playback requirements of non-live scenarios, such as recording, rebroadcasting, and other scenarios with low latency requirements.

For the specific implementation, working principles, and specific functions and effects of each device or system in the embodiments of the present invention, refer to the specific introduction in the corresponding method embodiments.

The embodiment of the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and the steps of the method in any one of the above-mentioned embodiments of the present invention can be executed when the computer instructions are run.

Wherein, the computer-readable storage medium may be any suitable readable storage medium such as an optical disk, a mechanical hard disk, or a solid-state hard disk. For the method executed by the instructions stored in the computer-readable storage medium, reference may be made to the embodiments of the foregoing methods for details, and details are not repeated here.

The embodiment of the present invention also provides a server, including a memory and a processor, the memory stores computer instructions that can be run on the processor, and the processor can execute the above-mentioned server of the present invention when running the computer instructions. The steps of the method described in any embodiment. For the specific implementation of the method executed when the computer instructions are running, reference may be made to the steps of the method in the foregoing embodiments, and details are not repeated here.

Although the present invention is disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, so the protection scope of the present invention should be based on the scope defined in the claims.

Claims (19)

1. A method of data interaction, comprising:

the method comprises the steps that an interactive terminal acquires a data stream to be played from a play control device in real time and plays and displays the data stream in real time, wherein the data stream to be played comprises video data and interactive identifications, and each interactive identification is associated with a designated frame time of the data stream to be played;

the method comprises the steps that an interactive terminal responds to triggering operation of an interactive identifier, interactive data corresponding to appointed frame moments of the interactive identifier are obtained, the interactive data comprise multi-angle free view angle data, the multi-angle free view angle data are generated by the interactive terminal or a server based on a plurality of received frame images corresponding to the appointed frame moments, the plurality of frame images are obtained by intercepting multi-channel video data streams synchronously collected by a plurality of collecting devices in a collecting array at the appointed frame moments by a data processing device, the data processing device provides the plurality of frame images for the interactive terminal or the server, and the data processing device is arranged in a non-collecting area on site;

The interactive terminal performs image display of the multi-angle free view angles at the appointed frame time based on the interactive data;

the data processing device intercepts multiple paths of video data streams synchronously acquired by a plurality of acquisition devices in an acquisition array at the appointed frame time, and the method comprises the following steps:

the data processing device intercepts frame-level synchronous video frames at the appointed frame time in the multi-path video data stream based on the received video frame interception instruction.

2. The data interaction method according to claim 1, wherein the multi-angle freeview data includes pixel data, depth data, and parameter data of the plurality of frame images, wherein an association exists between the pixel data and the depth data of each frame image.

3. The data interaction method according to claim 1, wherein a plurality of acquisition devices in the acquisition array are placed at different positions in the field acquisition area according to a preset multi-angle free view angle range.

4. The data interaction method according to claim 1, wherein the interaction identifier is generated by the play control device, and the play control device generates the interaction identifier associated with the video frame at the corresponding time in the data stream to be played based on the frame time information of the intercepted video frame from the data processing device.

5. The data interaction method of claim 1, wherein the interaction data further comprises at least one of: the method comprises the steps of on-site analysis data, information data of an acquisition object, information data of equipment associated with the acquisition object, information data of an on-site deployed object and information data of a logo displayed on site.

6. The data interaction method of claim 1, further comprising:

and when the interaction ending signal is detected, switching to a data stream to be played, which is acquired from the playing control equipment in real time, and playing and displaying in real time.

7. The data interaction method according to claim 6, wherein when the interaction end signal is detected, switching to a data stream to be played obtained from the play control device in real time and performing real-time play presentation, includes at least one of:

when receiving an interaction ending operation instruction, switching to a data stream to be played, which is obtained from the playing control equipment in real time, and playing and displaying the data stream in real time;

and when the image display of the multi-angle free view angle at the appointed frame moment is detected to the last image, switching to a data stream to be played, which is acquired from the playing control equipment in real time, and carrying out real-time playing display.

8. The data interaction method according to any one of claims 1 to 7, wherein the performing multi-angle freeview image presentation based on the interaction data includes:

determining a virtual viewpoint according to the interactive operation, wherein the virtual viewpoint is selected from a multi-angle free view angle range, and the multi-angle free view angle range is a range supporting switching view of the virtual viewpoint in a region to be watched;

and displaying an image for viewing the region to be viewed based on the virtual viewpoint, wherein the image is generated based on the interaction data and the virtual viewpoint.

9. A data processing system, comprising: the system comprises an acquisition array, data processing equipment, a server, play control equipment and an interactive terminal; wherein:

the acquisition array comprises a plurality of acquisition devices which are arranged at different positions of an on-site acquisition area according to a preset multi-angle free view angle range, and are suitable for synchronously acquiring multiple paths of video data streams in real time and uploading the video data streams to the data processing device in real time;

the data processing device is arranged in an on-site non-acquisition area, is suitable for intercepting the multipath video data stream at a designated frame time according to a received video frame intercepting instruction, obtains a plurality of frame images corresponding to the designated frame time and frame time information corresponding to the designated frame time, uploads the plurality of frame images at the designated frame time and the frame time information corresponding to the designated frame time to the server, and sends the frame time information at the designated frame time to the play control device;

The server is suitable for receiving the plurality of frame images and the frame time information uploaded by the data processing equipment, generating interaction data for interaction based on the plurality of frame images, wherein the interaction data comprises multi-angle free view angle data, the interaction data is associated with the frame time information, and the multi-angle free view angle data is generated based on the received plurality of frame images corresponding to the designated frame time;

the play control device is adapted to determine a designated frame time corresponding to the frame time information uploaded by the data processing device in a data stream to be played, generate an interactive identifier associated with the designated frame time, and transmit the data stream to be played containing the interactive identifier to the interactive terminal;

the interactive terminal is suitable for playing and displaying the video containing the interactive identifier in real time based on the received data stream to be played, and acquiring the interactive data which is stored in the server and corresponds to the appointed frame moment based on the triggering operation of the interactive identifier so as to display the multi-angle free view image.

10. The data processing system of claim 9, wherein the server is adapted to generate the multi-angle freeview data based on a plurality of received frame images corresponding to the specified frame time, the multi-angle freeview data including pixel data, depth data, and parameter data for the plurality of frame images, wherein an association exists between the pixel data and the depth data for each frame image.

11. The data processing system of claim 9, wherein the plurality of collection devices in the collection array are disposed at different locations in the field collection area according to a preset multi-angle free view range, and the server is disposed in the field non-collection area, the cloud or the terminal.

12. The data processing system according to claim 9, wherein the play control device is adapted to generate the interactive identifier associated with the video frame at the corresponding time in the data stream to be played based on the time of the frame information of the video frame captured by the data processing device.

13. The data processing system according to claim 9, wherein the interactive terminal is further adapted to switch to a data stream to be played acquired in real time from the play control device and to play the presentation in real time when the interactive end signal is detected.

14. A data processing system, comprising: the system comprises an acquisition array, data processing equipment, play control equipment and an interactive terminal; wherein:

the acquisition array comprises a plurality of acquisition devices which are arranged at different positions of an on-site acquisition area according to a preset multi-angle free view angle range, and are suitable for synchronously acquiring multiple paths of video data streams in real time and uploading the video data streams to the data processing device in real time;

The data processing device is arranged in an on-site non-acquisition area, is suitable for intercepting the multipath video data stream at a designated frame time according to a received video frame intercepting instruction, obtains a plurality of frame images corresponding to the designated frame time and frame time information corresponding to the designated frame time, and sends the frame time information of the designated frame time to the play control device;

the play control device is adapted to determine a designated frame time corresponding to the frame time information uploaded by the data processing device in a data stream to be played, generate an interactive identifier associated with the designated frame time, and transmit the data stream to be played containing the interactive identifier to the interactive terminal;

the interactive terminal is suitable for playing and displaying videos containing the interactive identification in real time based on the received data stream to be played, acquiring a plurality of frame images corresponding to the appointed frame moment of the interactive identification from the data processing equipment based on the triggering operation of the interactive identification, generating interactive data for interaction based on the plurality of frame images, and displaying multi-angle free view images, wherein the interactive data comprises multi-angle free view data, and the multi-angle free view data is generated based on the received plurality of frame images corresponding to the appointed frame moment.

15. An interactive terminal, comprising:

the data stream acquisition unit is suitable for acquiring a data stream to be played from the playing control equipment in real time, wherein the data stream to be played comprises video data and an interactive identifier, and the interactive identifier is associated with a designated frame time of the data stream to be played;

the playing display unit is suitable for playing and displaying the video and the interactive identification of the data stream to be played in real time;

the interactive data acquisition unit is adapted to respond to the triggering operation of the interactive identifier, acquire interactive data corresponding to the appointed frame time, wherein the interactive data comprises multi-angle free view angle data, the multi-angle free view angle data is generated based on a plurality of received frame images corresponding to the appointed frame time, the plurality of frame images are obtained by intercepting multi-channel video data streams synchronously acquired by a plurality of acquisition devices in an acquisition array at the appointed frame time by a data processing device, the data processing device is arranged in an on-site non-acquisition area, and the data processing device intercepts the multi-channel video data streams synchronously acquired by the plurality of acquisition devices in the acquisition array at the appointed frame time, and the interactive data acquisition unit comprises: the data processing equipment intercepts frame-level synchronous video frames at the appointed frame time in the multi-path video data stream based on the received video frame interception instruction;

And the interaction display unit is suitable for displaying the images of the multi-angle free viewing angles at the appointed frame moment based on the interaction data.

16. The interactive terminal of claim 15, further comprising: and the switching unit is suitable for triggering switching to the data stream to be played, which is acquired in real time by the data stream acquisition unit from the play control equipment, when the interaction ending signal is detected, and performing real-time play display by the play display unit.

17. An interactive terminal, comprising: a processor, a network component, a memory and a display component; wherein:

the processor is adapted to acquire a data stream to be played in real time through the network component, and acquire interactive data corresponding to a designated frame time of an interactive identifier in response to a triggering operation of the interactive identifier, wherein the data stream to be played comprises video data and the interactive identifier, the interactive identifier is associated with the designated frame time of the data stream to be played, the interactive data comprises multi-angle free view data, the multi-angle free view data is generated based on a plurality of received frame images corresponding to the designated frame time, the plurality of frame images are acquired by a data processing device by intercepting a plurality of paths of video data streams synchronously acquired by a plurality of acquisition devices in an acquisition array at the designated frame time, and the data processing device intercepts the plurality of paths of video data streams synchronously acquired by the plurality of acquisition devices in the acquisition array at the designated frame time, and the method comprises the following steps:

The data processing equipment is arranged in the on-site non-acquisition area and is used for intercepting the frame-level synchronous video frames at the appointed frame moment in the multi-path video data stream based on the received video frame interception instruction;

the memory is suitable for storing the data stream to be played, which is acquired in real time;

the display component is suitable for displaying the video and the interactive identification of the data stream to be played in real time based on the data stream to be played obtained in real time, and displaying the image of the multi-angle free view angle at the appointed frame moment based on the interactive data.

18. An interactive terminal comprising a memory and a processor, said memory having stored thereon computer instructions executable on said processor, characterized in that said processor executes the steps of the data interaction method according to any of claims 1 to 8 when said computer instructions are executed.

19. A computer readable storage medium having stored thereon computer instructions, which when run perform the steps of the data interaction method of any of claims 1 to 8.