WO2024109317A1 - Method and device for transmitting video frames and camera parameter information - Google Patents

Abstract

The objective of the present application is to provide a method and device for transmitting video frames and camera parameter information, comprising: acquiring a plurality of video frames captured by a camera apparatus and at least one piece of camera parameter information obtained when the plurality of video frames are captured; generating a coding sequence according to the plurality of video frames and the at least one piece of camera parameter information, wherein the coding sequence comprises a plurality of video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit comprises supplementary enhancement information used for indicating corresponding camera parameter information; and transmitting the coding sequence to a computer device. According to the present application, the camera parameter information of the camera apparatus and the collected video frames can be synchronized at an image sequence frame level, and a super-reality AR rendering tag in a mobile video scene can be realized, so that real-time synchronization of a stream pushing end and a stream pulling end is achieved, and real-time action command in application scenarios is facilitated.

Description

A method and device for transmitting video frames and camera parameter information

This application is based on the application with CN application number 202211481736.6 and application date 2022.11.24, and claims its priority. The disclosed content of the CN application is hereby introduced into this application as a whole.

Technical Field

The present application relates to the field of communications, and in particular to a technology for transmitting video frames and camera parameter information.

Background technique

In recent years, with the rapid development of science and technology, augmented reality (AR) technology has gradually come into people's vision. In particular, AR interaction based on real-time video streams has been rapidly applied to various industries. With the promotion of the metaverse, command systems based on AR real-time video streams, and expert guidance, mobile monitoring devices (drones, glasses, mobile phones, fixed cameras, high-point cameras) have also shown more and more application prospects. In particular, for example, drones fly high and see far, similar to the real-time video transmission of the God's perspective, AR glasses transmit the first-person perspective in real time, and Skynet fixed cameras transmit in real time, forming a real-time command system that integrates the sky and the earth. Due to the existence of network jitter, the monitoring device cannot transmit the video frames and the internal and external parameters of the camera to the monitoring platform in real time, resulting in the inability to achieve video frame-level rendering synchronization, affecting the command efficiency and accuracy.

Summary of the invention

One object of the present application is to provide a method and device for transmitting video frames and camera parameter information.

According to one aspect of the present application, a method for transmitting video frames and camera parameter information is provided, which is applied to a transmission device, wherein the transmission device is connected to a camera apparatus, wherein the method comprises:

Acquire a plurality of video frames captured by the camera device and at least one camera parameter information when the plurality of video frames are captured;

Generate a coding sequence according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information;

The coded sequence is transmitted to a computer device.

According to another aspect of the present application, a method for transmitting video frames and camera parameter information is applied to a computer device, wherein the computer device includes a display device, wherein the method includes:

Receive a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information;

The coded sequence is decoded, and a corresponding plurality of video frames are acquired and presented through the display device.

According to one aspect of the present application, a system method for transmitting video frames and camera parameter information is provided, the method comprising:

The transmission device obtains a plurality of video frames shot by a corresponding camera device and at least one camera parameter information when the plurality of video frames are shot; generates a coding sequence according to the plurality of video frames and the at least one camera parameter information, wherein the coding sequence includes a plurality of video frame coding units and at least one camera parameter coding unit. Each camera parameter encoding unit includes supplementary enhancement information for indicating corresponding camera parameter information; and transmitting the encoding sequence to a computer device;

The computer device receives the coding sequence of multiple video frames shot by the camera device and sent by the transmission device, decodes the coding sequence, obtains and presents the corresponding multiple video frames through the corresponding display device.

According to another aspect of the present application, a device for transmitting video frames and camera parameter information is provided, wherein the device is connected to a camera device, wherein the device comprises:

A module, used to obtain a plurality of video frames shot by the camera device and at least one camera parameter information when the plurality of video frames are shot;

A module 12 is used to generate a coding sequence according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information;

A module for transmitting the coding sequence to a computer device:

According to one aspect of the present application, a computer device for transmitting video frames and camera parameter information is provided, wherein the computer device includes a display device, wherein the computer device includes:

A module 21 is used to receive a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information;

Module 22 is used to decode the coding sequence, obtain and present the corresponding multiple video frames through the display device.

According to one aspect of the present application, a computer device is provided, wherein the device includes:

Processor; and

A memory arranged to store computer executable instructions, which when executed cause the processor to perform the steps of any of the methods described above.

According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program/instruction is stored, characterized in that when the computer program/instruction is executed, the system performs the steps of any of the methods described above.

According to one aspect of the present application, a computer program product is provided, including a computer program/instruction, characterized in that the computer program/instruction implements the steps of any of the above methods when executed by a processor.

Compared with the prior art, the present application writes the camera parameter information into the corresponding supplementary enhancement information while transmitting the video frame, and transmits it to the computer device together with the video frame, so as to display the video frame and perform real-time rendering of the video frame. The present application can synchronize the camera parameter information of the camera device with the collected video frame at the image sequence frame level, and can realize the hyper-realistic AR rendering label in the mobile video scene, so as to achieve real-time synchronization between the push stream end and the pull stream end, which is beneficial to the real-time action command in the application scene. Since SEI is not a necessary option in the decoding process, in other words, SEI has no direct impact on the decoding process. For scenes that do not need to achieve real-time synchronization of the camera parameter information of the camera device, there is no need to decode the camera parameter information and directly play the video normally. In addition, this solution can ignore network jitter. No matter how jittery the network is, this solution can ensure that the image data and the camera parameter information are synchronized at the image sequence frame level.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG1 shows a flow chart of a method for transmitting video frames and camera parameter information according to an embodiment of the present application;

FIG2 shows a flow chart of a method for transmitting video frames and camera parameter information according to another embodiment of the present application;

FIG3 shows a device structure diagram of a transmission device according to an embodiment of the present application;

FIG4 shows a device structure diagram of a computer device according to an embodiment of the present application;

FIG. 5 illustrates an exemplary system that may be used to implement various embodiments described herein.

The same or similar reference numerals in the drawings represent the same or similar components.

Detailed ways

The present application is described in further detail below in conjunction with the accompanying drawings.

In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), an input/output interface, a network interface and a memory.

Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase-Change Memory (PCM), Programmable Random Access Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The devices referred to in this application include but are not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic products that can interact with users (for example, interact with users through a touchpad), such as smart phones, tablet computers, etc. The mobile electronic products can use any operating system, such as Android operating system, iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field Programmable Gate Array, FPGA), digital signal processors (Digital Signal Processor, DSP), embedded devices, etc. The network devices include but are not limited to computers, network hosts, single network servers, multiple network server sets or multiple servers. The cloud is composed of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes but is not limited to the Internet, wide area networks, metropolitan area networks, local area networks, VPNs, etc. Network, wireless self-organizing network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, network device, or a device formed by integrating the user device and network device, network device, touch terminal, or network device and touch terminal through a network.

Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices that are applicable to the present application should also be included in the scope of protection of the present application and are included here by reference.

In the description of the present application, “plurality” means two or more, unless otherwise clearly and specifically defined.

In the traditional method of transmitting video frames and camera parameter information, the algorithm data set video frames and camera parameter information are stored separately. Since the video frame is the original image data that has not been compressed by video encoding, it occupies a large storage space. At the same time, it is necessary to synchronize the timestamps of the video frame and the camera parameter information. The process is relatively complicated and the transmission efficiency is low. In addition, the traditional method also includes using additional signaling channels (such as websocket, message queue telemetry transport (Message Queuing Telemetry Transport, MQTT), etc.), which cannot guarantee the synchronization of the image sequence frame level. Since video encoding and decoding takes time, network transmission also consumes time. The additional signaling channel can always reach the receiving end before the video frame, and due to network fluctuations and unstable video encoding and decoding time, the signaling data delay fluctuation is also uncontrollable and unstable. In addition, the traditional method also includes defining a new protocol based on H264/H.265 additional packaging data packets (Packet), but for scenes that do not require camera parameter information, the players on the market cannot normally pull the stream to play the video, and the compatibility of this method is zero.

The present application provides a system method for transmitting video frames and camera parameter information, which is applied to a system consisting of a transmission device and a computer device, and the method includes:

The transmission device obtains a plurality of video frames shot by the corresponding camera device and at least one camera parameter information when the plurality of video frames are shot; generates a coding sequence according to the plurality of video frames and the at least one camera parameter information, wherein the coding sequence includes a plurality of video frame coding units and at least one camera parameter coding unit, each camera parameter coding unit includes supplementary enhancement information for indicating the corresponding camera parameter information; and transmits the coding sequence to the computer device;

The computer device receives the coding sequence of multiple video frames shot by the camera device and sent by the transmission device, decodes the coding sequence, obtains and presents the corresponding multiple video frames through the corresponding display device.

The transmission device includes but is not limited to a streaming device that encodes and transmits video frames and camera parameter information. The transmission device may be provided with an internal camera device or connected to an external camera device, and may obtain multiple video frames shot by the camera device. Specific transmission devices include drone ground control equipment, augmented reality glasses, or other data processing equipment. The corresponding computer device includes but is not limited to a streaming device for decoding and presenting video frames. In some cases, the computer device may also superimpose virtual information in the video frame based on the camera parameter information, such as a mobile phone, a computer, a tablet computer, or other augmented reality glasses. The corresponding multiple video frames and at least one camera parameter information are encoded into a coding sequence at the transmission device end, and transmitted to the computer device through a communication line. After the computer device obtains the coding sequence, it decodes, obtains, and presents the corresponding video frame. Furthermore, the computer device may also decode, obtain the camera parameter information, and superimpose virtual information in the video frame based on the camera parameter information.

Referring to FIG1, a method for transmitting video frames and camera parameter information is shown, which is applied to a transmission device, wherein the transmission device is connected to a camera device, wherein the method comprises steps S101, S102 and S103. In step S101, a plurality of video frames captured by the camera device and the plurality of video frames are obtained. At least one camera parameter information when the frame is shot; in step S102, a coding sequence is generated according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information; in step S103, the coding sequence is transmitted to a computer device.

Specifically, in step S101, multiple video frames captured by the camera device and at least one camera parameter information when the multiple video frames are captured are obtained. For example, the camera device may be a built-in camera device of the transmission device, and the transmission device may issue a corresponding image acquisition instruction to the camera device to start the camera device to capture the corresponding multiple video frames. The corresponding camera device captures multiple video frames of the current field of view in real time based on the corresponding acquisition instruction. The transmission device may also obtain the camera parameter information when the camera device captures images, such as obtaining the internal and external parameters of the camera device, wherein the external parameters include the position and angle of the camera device. The camera device may also be an external camera device of the transmission device, such as an external camera or a camera device of other devices. The corresponding external camera device is used to capture images according to user instructions, and capture multiple video frames of the current field of view through the camera. At the same time, the corresponding camera device also obtains the camera parameter information when the image is captured, such as reading the internal and external parameters of the camera device. The camera device sends the multiple video frames and the corresponding camera parameter information to the transmission device through a communication connection with the transmission device. The multiple video frames include multiple video frames to be transmitted in the video stream captured by the camera device, which may be all the captured video frames or video frames filtered according to preset rules, wherein the preset rules include but are not limited to preset network quality, preset time interval, preset number of frames, and the change of the picture from the previous video frame exceeds a threshold, etc., which are not limited here. The camera parameter information includes indication information about the internal parameters and external parameters of the corresponding video frames (for example, one or more video frames, etc.) in the multiple video frames captured by the camera device.

For example, in some embodiments, the camera parameter information includes the internal parameter information and camera posture information of the camera device. For example, in order to simplify the amount of calculation and reduce the amount of data in the data transmission process, the transmission device does not directly transmit the coordinate transformation matrix corresponding to the video frame shot by the camera device, but transmits the internal parameters and camera posture information used to calculate the coordinate transformation matrix. The internal parameters include but are not limited to the long focal length, short focal length, long axis optical center offset, short axis optical center offset, relative zoom size of the camera device, video image sequence width, height, etc. when the camera device is not zoomed. The camera posture information includes the camera position information and camera angle information of the camera device, and the camera position information is usually expressed by geographic location information, such as represented by geographic coordinates (B, L, H), where L represents longitude information, B represents latitude information, and H represents height (such as altitude, etc.). Among them, the longitude and latitude information can be obtained through sensors such as GPS, and the corresponding altitude information can be obtained through a digital elevation model (DEM) or other methods. Here, the method of obtaining geographic coordinates (B, L, H) is only an example and is not limited here. The camera angle information includes yaw/pitch/roll three-axis angle information, yaw is the yaw angle, pitch is the pitch angle, roll is the roll angle, etc., which is obtained by the corresponding gimbal or attitude sensor (such as a three-axis gyroscope, etc.).

Here, the number of the at least one camera parameter information may be one or more. Accordingly, when the number of the camera parameter information is one, the camera parameter information when the multiple video frames are shot is the same, or the camera parameter information is the camera parameter selected from the multiple candidate camera parameter information obtained when the multiple video frames are shot, etc. When the number of the at least one camera parameter information is multiple, the multiple camera parameter information may correspond one-to-one to each of the multiple video frames, for example, the camera parameter information of the camera device is recorded once when each video frame is captured, etc.; in some cases, the multiple camera parameter information may also be some representative camera parameters of the multiple video frames, etc., and does not correspond one-to-one to the multiple video frames. Here, the plurality of camera parameter information may be selected from a plurality of candidate camera parameter information according to a preset condition, and the number of the plurality of candidate camera parameter information is not limited, and may be greater than or equal to the number of frames of the plurality of video frames, etc. In some embodiments, in step S101, a plurality of video frames shot by the camera device are obtained; a plurality of candidate camera parameter information when the plurality of video frames are shot is obtained, and at least one camera parameter information satisfying the preset condition is determined from the plurality of candidate camera parameter information. For example, the transmission device obtains a plurality of video frames shot by the camera device, and obtains a plurality of candidate camera parameter information when the plurality of video frames are shot, and the plurality of candidate camera parameter information may correspond to the plurality of video frames one by one, such as recording the candidate camera parameter information of the corresponding camera device once when each video frame is captured, or the plurality of candidate camera parameter information is a plurality of candidate camera parameter information collected based on a preset time interval, and the corresponding preset time interval may be greater than or equal to the acquisition interval of the video frame, or may be less than the acquisition interval of the video frame, etc. After the transmission device obtains multiple candidate camera parameter information, it selects at least one camera parameter information based on preset conditions. Specifically, in some embodiments, the preset conditions include but are not limited to: if the acquisition time of a certain candidate camera parameter information is the same as the shooting time of one of the video frames, then the certain candidate camera parameter information is determined as the camera parameter information; sampling is performed from the multiple candidate camera parameter information at a preset interval to determine the corresponding camera parameter information; if a certain candidate camera parameter information is different from the preceding candidate camera parameter information of the certain candidate camera parameter information, then the certain candidate camera parameter information is determined as the camera parameter information. For example, according to the shooting time of each video frame, determine the candidate camera parameter information whose acquisition time of a certain candidate camera parameter among the multiple candidate camera parameter information is the same as the shooting time of one of the multiple video frames, and determine it as the camera parameter information of the corresponding video frame. For example, when the number of candidate camera parameter information is not limited, in order to save transmission resources and improve transmission efficiency, it can be determined that the change in camera parameter information of video frames between multiple consecutive frames is small, and only intermittent camera parameter information is needed to describe the change trend of camera parameter information in the real-time video stream, etc. Then, the transmission device determines the camera parameter information that meets the conditions from multiple candidate camera parameter information according to a preset interval (for example, a preset time interval, a preset number interval, or a preset number of video frames, etc.), such as taking the 1st, 6th, 11th... candidate camera parameter information as the corresponding camera parameter information, etc. For another example, in order to improve the sensitivity of the change of the camera parameter information, and in order to save transmission resources and improve the transmission efficiency, the transmission of the same candidate camera parameter information can be ignored. When the candidate camera parameter information corresponding to multiple consecutive time points or multiple consecutive video frames is the same, the transmission device can determine one (such as the first, etc.) of the candidate camera parameter information corresponding to the multiple consecutive time points or multiple consecutive video frames as the camera parameter information, so that when a certain candidate camera parameter information is different from the previous candidate camera parameter information of the certain candidate camera parameter information, the certain candidate camera parameter information is determined as the camera parameter information. Among them, the same candidate camera parameter information means that the parameter difference between the corresponding candidate camera parameter information and the previous candidate camera parameter information of the candidate camera parameter information (for example, the candidate camera parameter information corresponding to the first N or the first N video frames, etc.) satisfies the following conditions: the corresponding internal parameter change is less than or equal to the preset change threshold, the position difference of the corresponding camera position information is less than or equal to the position difference threshold, and the angle difference of the camera angle information is less than or equal to the angle difference threshold, etc.

In step S102, a coding sequence is generated according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information. For example, after the transmission device obtains multiple video frames and at least one camera parameter information, it generates corresponding multiple video frame coding units and at least one camera parameter coding unit, and the corresponding video frame coding units and camera parameter coding units are NAL coding unit (such as NAL Unit, etc.) is composed of a corresponding unit header (such as NAL Header, etc.) and a unit body (such as NALU payload, etc.). The corresponding unit header is used to indicate the content type of the unit body of the NAL coding unit. Specifically, the unit header of the coding unit includes: forbidden_zero_bit (1 bit) + nal_ref_idc (2 bits) + nal_unit_type (5 bits), wherein forbidden_zero_bit is a forbidden bit, which is initially 0. When the network finds that the NAL coding unit has a bit error, the bit can be set to 1 so that the receiver can correct the error or discard the unit; wherein nal_ref_idc is the nal importance indication, which marks the importance of the NAL coding unit. The larger the value, the more important it is. When the decoder cannot process the decoding, it can discard the NALU with an importance of 0; nal_unit_type is used to indicate the content type of the unit body of the NAL coding unit. For example, when the type value is 6, the corresponding unit body is Supplemental Enhancement Information (Supplemental Enhancement Information). Information, SEI), when the type value is 5, the corresponding unit body is a slice of an IDR image, when the type value is other values, the corresponding unit body is the type corresponding to the nal_unit_type value, etc. Among them, the corresponding video frame coding unit is determined by one or more video frame codes in the corresponding multiple video frames, and the corresponding camera parameter coding unit is determined by one camera parameter information code in at least one camera parameter information. After the transmission device determines multiple video frame coding units and at least one camera parameter coding unit, it generates a corresponding coding sequence according to the multiple video frame coding units and at least one camera parameter coding unit. Optionally, in addition to the multiple video frame coding units and the at least one camera parameter coding unit, the coding sequence may also include other NAL coding units, wherein the content type of the unit body in the other NAL coding units is different from the content type of the unit body in the video frame coding unit and the camera parameter coding unit.

In some embodiments, in step S102, a corresponding plurality of video frame encoding units and at least one camera parameter encoding unit are generated according to the plurality of video frames and the at least one camera parameter information by using video encoding technology; a corresponding encoding sequence is generated according to the plurality of video frame encoding units and the at least one camera parameter encoding unit. For example, the video encoding technology includes but is not limited to video encoding technology corresponding to standards such as H.264 and H.265. The corresponding camera parameter encoding unit and the video frame encoding unit are data packets of the same level and both belong to the code stream category. For the camera parameter encoding unit, the value of nal_unit_type in its unit header is 6, and the content type of the unit body corresponding to the camera parameter encoding unit is SEI. The transmission device encapsulates the corresponding camera parameter information and fills it into SEI. For example, the camera parameter is stored in the supplementary enhancement information in any serialization or deserialization manner, wherein the serialization manner includes a JSON string manner, etc. For example, in some embodiments, the camera parameter information is stored in the supplementary enhancement information in the form of a JSON string. For example, the JSON data structure includes a collection of "key/value" pairs. In different languages, it is understood as an object, record, structure, dictionary, hash table, keyed list, or associative array. JSON data structure also includes an ordered list of values, which is understood as an array in most languages. The specific form is to enclose the string in double quotes, and arrays and objects can be nested in arrays, for example,

In some embodiments, generating a corresponding coding sequence according to the multiple video frame coding units and the at least one camera parameter coding unit includes: arranging the multiple video frame coding units and the at least one camera parameter coding unit into a coding sequence according to the acquisition order of the multiple video frames. For example, after the transmission device obtains the corresponding multiple video coding units and the at least one camera parameter coding unit, it transmits them according to the corresponding video frame acquisition order, thereby generating a corresponding coding sequence, wherein the arrangement order of each camera parameter coding unit is adjacent to the transmission position of the video frame coding unit corresponding to the camera parameter coding unit (for example, adjacent, etc.), and each coding unit is separated by a corresponding delimiter, such as Start code: 00 00 00 01/00 00 01, etc.

In step S103, the coding sequence is transmitted to the computer device. For example, after the transmission device obtains the corresponding coding sequence, it transmits the coding sequence to the computer device based on the communication connection with the computer device, so that the computer device can present the video frame in real time. In some cases, the computer device can also decode the coding sequence to obtain the corresponding camera parameter information, and superimpose virtual information in the video frame in real time, such as highlighting or superimposing other information (for example, text, image or video about a target in an image).

In some embodiments, the supplementary enhancement information includes corresponding byte length information and byte parameter information, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte length of the byte parameter information. For example, the corresponding camera parameter encoding unit includes a corresponding unit header and a unit body, and the unit header includes nal_unit_type, which is used to indicate the content type of the unit body in the NAL encoding unit. For example, when the type value is 6, the corresponding unit body is the supplementary enhancement information. The corresponding unit body of the supplementary enhancement information includes corresponding byte length information and byte parameter information, and the corresponding byte length information is used to indicate the byte length of the byte parameter information, such as SEI payload size, etc., and the corresponding byte parameter information is used to indicate the camera parameter information, such as SEI payload content, etc. The byte length information is used to count the byte length from the first byte to the end byte of the byte parameter information (wherein the end byte may or may not include the end mark), such as the byte length from the first byte to the end byte is the aforementioned byte length information, or the byte length information is used to count the byte length from the first byte to the end byte of the byte parameter information after anti-competition processing; in some cases, if the byte length information exceeds 255, a byte is added to represent the byte length information, and so on. In some cases, the main body of the supplementary enhancement information corresponding unit also includes the main body encoding specification type, such as SEI payload type, etc., byte identification information, such as SEI payload uuid, etc., and RBSP tail padding bytes, such as rbsp trailing bits with a value of 80, etc. In some embodiments, the supplementary enhancement information also includes corresponding byte identification information, and the byte length information is used to indicate the byte length of the byte identification information and the byte parameter information. For example, the main body coding specification type of the main body of the supplementary enhancement information corresponding unit can also be indicated by 05. If the SEI payload type takes a value of 05, the main body of the supplementary enhancement information corresponding unit also includes corresponding byte identification information, such as SEI payload uuid, etc. The corresponding byte length information is the total byte length corresponding to the first byte of the byte identification information to the end byte of the byte parameter information, etc., or the byte length information is used to count the total byte length corresponding to the first byte to the end byte of the byte identification information and the byte parameter information after anti-competition processing. Here, the specific content in the corresponding unit body is exemplified as follows: NALU payload = SEI payload type + SEI payload size + SEI payload uuid + SEI payload content + rbsp trailing bits. Among them, the byte length of the byte identification information before the anti-competition operation is a fixed value, usually 16 bytes, and its content is randomly generated.

In some embodiments, the method further comprises step S104 (not shown), in which: The byte identification information and the byte parameter information are processed for anti-competition, and preset characters are inserted into each byte of the byte identification information and the byte parameter information to obtain the byte identification information and the byte parameter information after the anti-competition processing; the corresponding byte length information is determined by counting the byte identification information and the byte parameter information after the anti-competition processing. For example, the corresponding coding unit is divided by the separator 0x000001 or 0x000000. If the corresponding sequence appears inside a coding unit, the coding unit will also be divided, which will cause the content of the corresponding coding unit to be incomplete. Therefore, it is necessary to perform anti-competition processing on the content in the coding unit, such as performing anti-competition processing on the byte identification information and the byte parameter information in the unit body. Specifically, according to the preset character (such as the system default preset character or the preset character input by the user, such as 03, etc.), the preset character is inserted in the middle of each byte of the byte identification information and the byte parameter information. The specific examples are as follows:
0x000000------------->0x00000300
0x000001-------------->0x00000301
0x000002-------------->0x00000302
0x000003-------------->0x00000303

The above-mentioned anti-competition processing is to add 03 after 0000 in each byte. After the transmission device determines the byte identification information and byte parameter information after the anti-competition processing, it counts the total byte length corresponding to the first byte to the end byte of the byte identification information and byte parameter information after the anti-competition processing to determine the corresponding byte length information.

Of course, those skilled in the art should understand that the above-mentioned anti-contention processing is only an example, and other existing or future anti-contention processing, if applicable to the present application, should also be included in the scope of protection of the present application and included here by reference.

In some embodiments, the method further includes step S105 (not shown), in which the color space conversion is performed on the multiple video frames to determine multiple converted video frames that conform to the preset color space; wherein, in step S102, a coding sequence is generated according to the multiple converted video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating the corresponding camera parameter information. For example, the color space of the multiple video frames acquired by the camera device is usually the RGB color space. For better transmission or when the streaming media does not support RGB images, it is necessary to perform color space conversion on the multiple video frames to determine whether they conform to the preset color space (for example, YUV, CMY, HSV, HSI color space, etc.), determine multiple converted video frames after the color space conversion, and perform subsequent encoding and transmission based on the multiple converted video frames. Of course, those skilled in the art should understand that the above color space is only an example, and other existing or future color spaces that may appear should also be included in the scope of protection of the present application if they are applicable to the present application, and are included here by reference.

Fig. 2 shows a method for transmitting video frames and camera parameter information according to another aspect of the present application, which is applied to a computer device, wherein the computer device includes a display device, wherein the method includes steps S201 and S202. In step S201, a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device is received, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information; in step S202, the coding sequence is decoded to obtain and present the corresponding multiple video frames through the display device.

Specifically, in step S201, a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device is received, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter. A number of encoding units, each of which contains supplementary enhancement information for indicating corresponding imaging parameter information. For example, a computer device receives an encoding sequence determined by a transmission device based on multiple video frames shot by an imaging device and at least one imaging parameter information, wherein the generation process of the encoding sequence is as described above and will not be repeated here.

In step S202, the coding sequence is decoded, and the corresponding multiple video frames are obtained and presented through the display device. For example, the computer device includes a corresponding display device for displaying the encoded multiple video frames, such as a display screen or a projector, and the display device can be a display device built into the computer device, or a display device external to the computer device. After the computer device obtains the coding sequence, it decodes the coding sequence based on the corresponding decoding technology, decodes the corresponding multiple video frame encoding units to obtain the corresponding multiple video frames, and the corresponding at least one camera parameter encoding unit determines whether decoding is required according to demand. For example, in some embodiments, in step S202, based on the pre-setting, the at least one camera parameter encoding unit is ignored, and only the multiple video frame encoding units are decoded to obtain and present the corresponding multiple video frames through the display device. For example, this solution can realize the real-time synchronous transmission of video frames and camera parameter information, and since SEI is a non-essential option in the decoding stage, we can determine whether to decode the current camera parameter coding unit according to the current video presentation requirements. For example, the default setting of the corresponding application is to present only the video, for example, there is no need to superimpose virtual information on the video, or the function of the corresponding application includes a selection function for decoding the camera parameter information, then based on the user operation, the camera parameter information can be not selected to be decoded in the application, and only the video frame coding unit is decoded and multiple video frames are presented, which saves computing resources and improves the efficiency of video frame decoding. The computer device can directly skip the coding unit whose content type of the unit body in the NAL coding unit is SEI based on the pre-settings (for example, the default setting of the application or the user's selection setting of the camera parameter information, etc.), ignore its decoding process, decode other coding units and obtain the frame data corresponding to multiple video frames, and present them through the display device.

In some embodiments, in step S202, the at least one camera parameter encoding unit is decoded to obtain corresponding camera parameter information; the multiple video frame encoding units are decoded to obtain and present corresponding multiple video frames through the display device; wherein the method further includes step S203 (not shown), in which coordinate transformation information from the corresponding world coordinate system to the pixel coordinate system of the camera device is determined according to the camera parameter information; and virtual information is superimposed and presented in the multiple video frames according to the coordinate transformation information. For example, the transmission device encapsulates the camera parameter information into a coding unit with a unit body content type of SEI and a video frame into a coding sequence corresponding to the video stream and transmits it to the computer device through a communication connection. After the computer device receives the video stream, it parses the coding sequence of the real-time video stream into image data of multiple video frames and SEI coding units through a video decoding module, and obtains the camera internal parameters and camera posture information from the SEI coding unit. Then, the geographic tag (for example, the geographic location information and/or map location information input by the user, an object determined by target recognition and reading the corresponding geographic location information and/or map location information, or directly calling the geographic tag information to determine the corresponding geographic location information and/or map location information) is calculated at the screen coordinate position through the camera internal parameters and the camera posture information. Then, all the geographic tags on the screen are rendered. In some embodiments, the computer device aligns the coordinates of the camera device in the physical world coordinate system in the virtual world coordinate system through the camera pose information in the SEI coding unit, the geotag transforms the object coordinates into virtual world coordinates through the model matrix, the view matrix transforms the virtual world coordinates into camera coordinates, and the camera device intrinsic parameters in the SEI coding unit are converted into a projection matrix to align the camera cropping coordinates (align the screen/video coordinates), etc.

In some implementations, the supplementary enhancement information includes the corresponding byte length information and the byte identification information and byte parameter information, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte length of the byte identification information and the byte parameter information after anti-competition processing; wherein, the decoding of the at least one camera parameter encoding unit to obtain the corresponding camera parameter information also includes: decoding the at least one camera parameter encoding unit to obtain the corresponding at least one supplementary enhancement information, reading the byte length information and the byte identification information and byte parameter information after anti-competition processing in the supplementary enhancement information; performing anti-competition removal processing on the byte identification information and byte parameter information after anti-competition processing to obtain the real byte identification information and real byte parameter information after anti-competition processing, and determining the real byte length information after anti-competition removal processing, wherein the real byte length information is used to indicate the byte length of the real byte identification information and the real byte parameter information; reading the corresponding camera parameter information from the real byte parameter information according to the real byte length information and the real byte identification information, thereby determining the corresponding at least one camera parameter information. For example, based on the aforementioned anti-competition processing, the corresponding byte length information in the supplementary enhancement information is used to indicate the total byte length of the byte identification information and byte parameter information after the anti-competition processing. According to the byte length information, the byte identification information and byte parameter information after the anti-competition processing can be determined from the video stream. Then, during decoding, the computer device needs to first perform anti-competition processing on the byte identification information and byte parameter information after the anti-competition processing to obtain the real byte identification information and real byte parameter information before the anti-competition processing is performed, such as removing the corresponding preset characters at specific positions of the byte identification information (such as SEI payload uuid, etc.) and byte parameter information (such as SEI payload content, etc.) in the supplementary enhancement information according to the preset characters. After the computer device performs anti-competition processing on the byte identification information and byte parameter information, the real byte identification information and the real byte parameter information are determined, and then the total byte length of the real byte identification information and byte parameter information is counted, thereby determining the corresponding real byte length information, and then determining the real byte parameters according to the real byte identification information, such as according to the byte length of the real byte identification information, thereby obtaining the camera parameter information. For example, assuming that the corresponding real byte length information is 47 bytes; the computer device determines the byte length of the corresponding real byte parameter information based on the real byte length information, minus the known byte length of 16 bytes of the real byte identification information, that is, 47-16=31 bytes, so that the real byte parameter information, that is, the camera parameter information, can be obtained from the real byte identification information and the real byte parameter information after the anti-competition processing is removed. In some embodiments, the byte length of the real byte parameter information includes the length of the terminator. For example, since the byte parameter information is a string, the string in the C language has an end mark, so there is an end mark 0x00, then the byte length corresponding to the real byte parameter information minus one corresponds to the byte length of the camera parameter information. Continuing with the above example, 30 bytes of camera parameter information can be read from the real byte parameter information.

In some embodiments, the method further includes step S204 (not shown), in which a packaged file for the multiple video frames is determined and stored according to the coding sequence; in step S202, if a presentation operation for the packaged file is obtained, the coding sequence in the packaged file is decoded, and the corresponding multiple video frames are obtained and presented through the display device. For example, after receiving the corresponding coding sequence, the computer device can directly decode and present the real-time video stream, or generate a corresponding packaged file. In some embodiments, the packaged file refers to a video file, such as a video file packaged into MP4, MOV, MPG, WMV, etc., and stored in a database for subsequent video playback calls, such as for plan exercises, review reviews, or as a data set to optimize coordinate transformation algorithms. If the computer device subsequently obtains a presentation operation for the packaged file, the packaged file is called from the database, the corresponding coding sequence is decoded, and the corresponding multiple video frames are presented, wherein the presentation operation can be based on user operations or intelligent recognition, etc. In some cases, when the computer device subsequently calls to present the packaged file, the camera in the packaged file is The encoding order of the parameter encoding unit is the same as that of the corresponding video frame encoding unit. Accordingly, when the corresponding video frame is presented, if the camera parameter encoding unit needs to be decoded, the camera parameter information obtained by decoding and the presented video frame can still be synchronized at the video frame level. On the one hand, it can restore the rendering of the virtual content in the scene, which can be used for scenarios such as plan exercises and review. On the other hand, it can restore the scene without the need to re-acquire the scene video and camera parameters each time the algorithm is executed/tested, thereby optimizing the coordinate transformation algorithm.

The above mainly introduces the various embodiments of a method for transmitting video frames and camera parameter information in one aspect of the present application. In addition, we also provide specific devices that can implement the above embodiments. Below we will give a specific introduction in conjunction with Figures 3 and 4.

Referring to FIG3 , a transmission device for transmitting video frames and camera parameter information is shown, the transmission device is connected to the camera device, wherein the transmission device includes a module-101, a module-202 and a module-303. The module-101 is used to obtain multiple video frames shot by the camera device and at least one camera parameter information when the multiple video frames are shot; the module-202 is used to generate a coding sequence according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information; the module-303 is used to transmit the coding sequence to a computer device.

In some implementations, the camera parameter information includes internal reference information and camera posture information of the camera device.

In some embodiments, a module 101 is used to obtain multiple video frames shot by the camera device; obtain multiple candidate camera parameter information when the multiple video frames are shot, and determine at least one camera parameter information that meets the preset conditions from the multiple candidate camera parameter information. In some embodiments, the preset conditions include but are not limited to: if the acquisition time of a certain candidate camera parameter information is the same as the shooting time of one of the video frames, then the certain candidate camera parameter information is determined as the camera parameter information; sampling from the multiple candidate camera parameter information at a preset interval to determine the corresponding camera parameter information; if a certain candidate camera parameter information is different from the previous candidate camera parameter information of the certain candidate camera parameter information, then the certain candidate camera parameter information is determined as the camera parameter information.

In some embodiments, the module 102 is used to generate corresponding multiple video frame encoding units and at least one camera parameter encoding unit according to the multiple video frames and the at least one camera parameter information using video encoding technology; and generate corresponding encoding sequences according to the multiple video frame encoding units and the at least one camera parameter encoding unit. For example, the high-definition video encoding technology includes but is not limited to video encoding technologies corresponding to standards such as H.264 and H.265. In some embodiments, the camera parameter information is stored in the supplemental enhancement information in the form of a JSON string.

In some embodiments, generating a corresponding coding sequence based on the multiple video frame coding units and the at least one camera parameter coding unit includes: arranging the multiple video frame coding units and the at least one camera parameter coding unit into a coding sequence according to the acquisition order of the multiple video frames.

In some embodiments, the supplemental enhancement information includes corresponding byte length information and byte parameter information, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte length of the byte parameter information. In some embodiments, the supplemental enhancement information also includes corresponding byte identification information, and the byte length information is used to indicate the byte length of the byte identification information and the byte parameter information.

In some embodiments, the transmission device also includes a module (not shown) for performing anti-contention processing on the byte identification information and the byte parameter information, inserting a preset character into each byte of the byte identification information and the byte parameter information to obtain the byte identification information and the byte parameter information after anti-contention processing; and determining the corresponding byte length information by counting the byte identification information and the byte parameter information after anti-contention processing.

In some embodiments, the transmission device also includes a module 15 (not shown) for performing color space conversion on the multiple video frames to determine multiple converted video frames that conform to a preset color space; wherein the module 12 102 is used to generate a coding sequence based on the multiple converted video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information.

Here, the specific implementations corresponding to the module 11 101, the module 12 102, the module 13 103, the module 14 and the module 15 are the same or similar to the aforementioned step S101, step S102, step S103, step S104 and step S105, and are therefore not repeated herein and are included herein by reference.

FIG4 shows a computer device for transmitting video frames and camera parameter information according to another aspect of the present application, the computer device includes a display device, wherein the computer device includes a 21 module 201 and a 22 module 202. The 21 module 201 is used to receive a coding sequence of multiple video frames sent by a corresponding transmission device and shot by a camera device, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information; the 22 module 202 is used to decode the coding sequence, obtain and present the corresponding multiple video frames through the display device.

In some embodiments, the module 202 is configured to ignore the at least one camera parameter encoding unit based on a preset setting, and only decode the multiple video frame encoding units to obtain and present the corresponding multiple video frames through the display device.

In some embodiments, the 22 module 202 is used to decode the at least one camera parameter encoding unit to obtain corresponding camera parameter information; decode the multiple video frame encoding units to obtain and present the corresponding multiple video frames through the display device; wherein the computer device also includes a 23 module (not shown) for determining the coordinate transformation information from the corresponding world coordinate system to the pixel coordinate system of the camera device according to the camera parameter information; and superimpose virtual information in the multiple video frames according to the coordinate transformation information. In some embodiments, the supplementary enhancement information includes corresponding byte length information and byte identification information and byte parameter information after anti-competition processing, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte length of the byte identification information and the byte parameter information after anti-competition processing; wherein, the decoding of the at least one camera parameter encoding unit to obtain the corresponding camera parameter information also includes: decoding the at least one camera parameter encoding unit to obtain the corresponding at least one supplementary enhancement information, reading the byte length information and the byte identification information and byte parameter information after anti-competition processing in the supplementary enhancement information; performing anti-competition processing on the byte identification information and byte parameter information after anti-competition processing to obtain the real byte identification information and real byte parameter information after anti-competition processing, and determining the real byte length information after anti-competition processing, wherein the real byte length information is used to indicate the byte length of the real byte identification information and the real byte parameter information; reading the corresponding camera parameter information from the real byte parameter information according to the real byte length information and the real byte identification information, thereby determining the corresponding at least one camera parameter information.

In some embodiments, the computer device also includes a 24 module (not shown), which is used to determine and store an encapsulation file regarding the multiple video frames based on the coding sequence; wherein the 22 module 202 is used to decode the coding sequence in the encapsulation file if a presentation operation regarding the encapsulation file is obtained, and obtain and present the corresponding multiple video frames through the display device.

Here, the specific implementations corresponding to the 21 module 201, 22 module 202, 23 module and 24 module are the same or similar to the embodiments of the aforementioned step S201, step S202, step S203 and step S204, and are therefore not repeated here and are included herein by reference.

In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the preceding items is executed.

The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any of the preceding items is executed.

The present application also provides a computer device, the computer device comprising:

one or more processors;

a memory for storing one or more computer programs;

When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any of the preceding items.

FIG5 illustrates an exemplary system that can be used to implement various embodiments described in this application;

As shown in FIG5 , in some embodiments, the system 300 can be used as any of the above-mentioned devices in each of the described embodiments. In some embodiments, the system 300 may include one or more computer-readable media (e.g., system memory or NVM/storage device 320) having instructions and one or more processors (e.g., (one or more) processors 305) coupled to the one or more computer-readable media and configured to execute instructions to implement modules to perform the actions described in the present application.

For one embodiment, system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 305 and/or any suitable device or component in communication with system control module 310 .

The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and/or a firmware module.

The system memory 315 may be used, for example, to load and store data and/or instructions for the system 300. For one embodiment, the system memory 315 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 315 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, system control module 310 may include one or more input/output (I/O) controllers to provide interfaces to NVM/storage device 320 and communication interface(s) 325 .

For example, NVM/storage device 320 may be used to store data and/or instructions. NVM/storage device 320 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and/or one or more digital versatile disk (DVD) drives).

NVM/storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or that are accessible to the device without being part of the device. Storage device 320 may be accessible over a network via communication interface(s) 325 .

Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and/or with any other suitable devices. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and/or protocols.

For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 to form a system on chip (SoC).

In various embodiments, the system 300 may be, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 300 may have more or fewer components and/or a different architecture. For example, in some embodiments, the system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

It should be noted that the present application can be implemented in software and/or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and/or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

Communication media include media by which communication signals containing, for example, computer readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media may include guided transmission media such as cables and wires (e.g., fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied as a modulated data signal in, for example, a wireless medium such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information/data for use with a computer system.

Here, according to an embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the methods and/or technical solutions based on the aforementioned multiple embodiments of the present application.

It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims (21)

A method for transmitting video frames and camera parameter information is applied to a transmission device, wherein the transmission device is connected to a camera device, wherein the method comprises: Acquire a plurality of video frames captured by the camera device and at least one camera parameter information when the plurality of video frames are captured; Generate a coding sequence according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information; The coded sequence is transmitted to a computer device. The method according to claim 1, wherein the acquiring of the plurality of video frames captured by the camera device and at least one camera parameter information when the plurality of video frames are captured comprises: Acquire multiple video frames captured by the camera device; A plurality of candidate camera parameter information when the plurality of video frames are shot is obtained, and at least one camera parameter information satisfying a preset condition is determined from the plurality of candidate camera parameter information. The method according to claim 2, wherein the preset condition includes at least any one of the following: If the acquisition time of a certain candidate camera parameter information is the same as the shooting time of one of the video frames, the certain candidate camera parameter information is determined as the camera parameter information; Sampling from the plurality of candidate camera parameter information at a preset interval to determine corresponding camera parameter information; If a certain candidate camera parameter information is different from the preceding candidate camera parameter information of the certain candidate camera parameter information, the certain candidate camera parameter information is determined as the camera parameter information. The method according to claim 1, wherein generating a coding sequence according to the plurality of video frames and the at least one camera parameter information comprises: Using video coding technology, generating corresponding multiple video frame coding units and at least one camera parameter coding unit according to the multiple video frames and the at least one camera parameter information; A corresponding coding sequence is generated according to the multiple video frame coding units and the at least one camera parameter coding unit. The method according to claim 4, wherein the generating a corresponding coding sequence according to the plurality of video frame coding units and the at least one camera parameter coding unit comprises: According to the acquisition order of the multiple video frames, the multiple video frame encoding units and the at least one camera parameter encoding unit are arranged to form a coding sequence. The method according to any one of claims 1 to 5, wherein the camera parameter information includes internal parameter information and camera posture information of the camera device. The method according to any one of claims 1 to 6, wherein the camera parameter information is stored in the supplemental enhancement information in the form of a JSON string. According to the method described in any one of claims 1 to 7, the supplementary enhancement information includes corresponding byte length information and byte parameter information, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte length of the byte parameter information. The method according to claim 8, wherein the supplemental enhancement information further includes a corresponding byte identifier Information, the byte length information is used to indicate the byte length of the byte identification information and the byte parameter information. The method according to claim 8 or 9, wherein the method further comprises: Performing anti-competition processing on the byte identification information and the byte parameter information, inserting a preset character into each byte of the byte identification information and the byte parameter information to obtain the byte identification information and the byte parameter information after the anti-competition processing; The corresponding byte length information is determined by counting the byte identification information and byte parameter information after anti-contention processing. The method according to claim 1, wherein the method further comprises: Performing color space conversion on the multiple video frames to determine multiple converted video frames that conform to a preset color space; The step of generating a coding sequence according to the plurality of video frames and the at least one camera parameter information includes: A coding sequence is generated according to the multiple converted video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information. A method for transmitting video frames and camera parameter information is applied to a computer device, wherein the computer device includes a display device, wherein the method includes: Receive a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information; The coded sequence is decoded, and a corresponding plurality of video frames are acquired and presented through the display device. The method according to claim 12, wherein decoding the coded sequence, obtaining and presenting the corresponding plurality of video frames through the display device comprises: Based on the pre-setting, the at least one camera parameter encoding unit is ignored, and only the multiple video frame encoding units are decoded to obtain and present the corresponding multiple video frames through the display device. The method according to claim 12, wherein the decoding of the coded sequence, obtaining and presenting the corresponding plurality of video frames through the display device comprises: Decoding the at least one camera parameter encoding unit to obtain corresponding camera parameter information; Decoding the plurality of video frame encoding units, acquiring and presenting the corresponding plurality of video frames through the display device; Wherein, the method further comprises: Determine coordinate transformation information corresponding to the world coordinate system to the pixel coordinate system of the camera device according to the camera parameter information; Virtual information is superimposed and presented in the multiple video frames according to the coordinate transformation information. The method according to claim 14, wherein the supplementary enhancement information includes corresponding byte length information and byte identification information after anti-competition processing, and byte parameter information, wherein the byte parameter information is used to fill in the corresponding camera parameter information, and the byte length information is used to indicate the byte identification information after anti-competition processing and the byte length of the byte parameter information; wherein the decoding of the at least one camera parameter encoding unit to obtain the corresponding camera parameter information further includes: Decoding the at least one camera parameter encoding unit to obtain corresponding at least one supplementary enhancement information, and reading the byte length information and the byte identification information and byte parameter information after the anti-competition processing in the supplementary enhancement information; Perform anti-competition processing on the byte identification information and byte parameter information after anti-competition processing to obtain the anti-competition The processed real byte identification information and real byte parameter information, and determine the real byte length information after the anti-competition processing is removed, wherein the real byte length information is used to indicate the byte length of the real byte identification information and the real byte parameter information; The corresponding camera parameter information is read from the real byte parameter information according to the real byte length information and the real byte identification information, thereby determining at least one corresponding camera parameter information. The method according to any one of claims 12 to 14, wherein the method further comprises: Determine and store a package file related to the plurality of video frames according to the coding sequence; The decoding of the coding sequence, obtaining and presenting the corresponding multiple video frames through the display device includes: If a presentation operation on the encapsulated file is obtained, the coding sequence in the encapsulated file is decoded, and the corresponding multiple video frames are obtained and presented through the display device. A transmission device for transmitting video frames and camera parameter information, wherein the transmission device is connected to a camera device, wherein the device comprises: A module, used to obtain a plurality of video frames shot by the camera device and at least one camera parameter information when the plurality of video frames are shot; A module 12 is used to generate a coding sequence according to the multiple video frames and the at least one camera parameter information, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit includes supplementary enhancement information for indicating corresponding camera parameter information; A module 3 is used to transmit the coding sequence to a computer device. A computer device for transmitting video frames and camera parameter information, the computer device comprising a display device, wherein the computer device comprises: A module 21 is used to receive a coding sequence of multiple video frames shot by a camera device and sent by a corresponding transmission device, wherein the coding sequence includes multiple video frame coding units and at least one camera parameter coding unit, and each camera parameter coding unit contains supplementary enhancement information for indicating corresponding camera parameter information; Module 22 is used to decode the coding sequence, obtain and present the corresponding multiple video frames through the display device. A computer device, wherein the device comprises: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of a method as claimed in any one of claims 1 to 16. A computer-readable storage medium having a computer program/instruction stored thereon, wherein the computer program/instruction, when executed, causes a system to perform the steps of the method as claimed in any one of claims 1 to 16. A computer program product, comprising a computer program/instruction, characterized in that when the computer program/instruction is executed by a processor, the steps of the method described in any one of claims 1 to 16 are implemented.