CN113810721A - Video stream error code masking method, device, terminal equipment and readable storage medium - Google Patents
Video stream error code masking method, device, terminal equipment and readable storage medium Download PDFInfo
- Publication number
- CN113810721A CN113810721A CN202111101332.5A CN202111101332A CN113810721A CN 113810721 A CN113810721 A CN 113810721A CN 202111101332 A CN202111101332 A CN 202111101332A CN 113810721 A CN113810721 A CN 113810721A
- Authority
- CN
- China
- Prior art keywords
- error code
- error
- video frame
- decoding
- current video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/164—Feedback from the receiver or from the transmission channel
- H04N19/166—Feedback from the receiver or from the transmission channel concerning the amount of transmission errors, e.g. bit error rate [BER]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/43—Hardware specially adapted for motion estimation or compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/89—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
- H04N19/895—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder in combination with error concealment
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The embodiment of the application provides a video stream error code concealment method, a device, a terminal device and a computer readable storage medium, wherein in the video stream error code concealment method, a hardware decoder is controlled to decode and detect an error code of a current video frame by receiving the current video frame to obtain a decoded frame and error code information; if the error code information indicates that the current video frame has an error code block, calculating an error code complexity value according to the error code information; based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, the corresponding error code compensation algorithm is selected according to the error code complexity value, and the selected error code compensation algorithm is adopted to perform error code compensation on the decoded frame, so that the chip area is saved, the equipment power consumption is reduced on the basis of ensuring the real-time error code covering, and the product cost is further saved.
Description
[ technical field ] A method for producing a semiconductor device
The embodiment of the application relates to the technical field of intelligent terminals, in particular to a video stream error code concealment method, a video stream error code concealment device, terminal equipment and a computer readable storage medium.
[ background of the invention ]
TCP and UDP are two representative network communication protocols. Since UDP communication does not require connection establishment and is superior to TCP protocol in real-time, UDP communication protocol is used in some scenarios with high real-time requirements, such as teleconferencing and video call services provided by mobile phone operators. However, UDP has a disadvantage that packet loss occurs under bad network conditions, and if a decoder at a video receiving end does not have an error concealment algorithm, a decoded video frame may have incomplete pictures, mosaics, and the like.
The video decoder can be divided into a software decoder and a hardware decoder at present, and the software decoder has the advantages of flexible realization, easy modification and scheme upgrade, CPU resource occupation and higher power consumption; the hardware decoder has the advantages of low power consumption, high decoding speed, no occupation of CPU resources, long design period and incapability of modifying all logic once a chip is streamed.
Therefore, in a scenario with high real-time requirement and strict power consumption requirement, how to implement the error concealment function of the video decoder while saving chip area and reducing device power consumption is a technical problem that needs to be solved at present.
[ summary of the invention ]
The embodiment of the application provides a video stream error code concealment method, a device, a terminal device and a computer readable storage medium, and the method of software and hardware cooperation is adopted to conceal the video stream error code, so that the chip area is saved, the device power consumption is reduced on the basis of ensuring the real-time error code concealment, and further the product cost is saved.
In a first aspect, an embodiment of the present application provides a method for concealing an error code in a video stream, including: receiving a current video frame, and controlling a hardware decoder to decode and detect an error code of the current video frame to obtain a decoded frame and error code information; if the error code information indicates that the current video frame has an error code block, calculating an error code complexity value according to the error code information; and selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm.
In the method for concealing the error codes of the video stream, after the terminal equipment receives the current video frame, a hardware decoder is controlled to decode and detect the error codes of the current video frame to obtain a decoded frame and error code information; if the error code information indicates that the current video frame has error code blocks, calculating an error code complexity value according to the error code information; and selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm. The method is characterized in that the hardware logic is used for taking charge of the video decoding and error code detecting part which is high in complexity and basically does not need to be upgraded and changed once the video decoding and error code detecting part is completed, and the error code compensating part is realized by a software program, so that the error code concealing function of the intelligent terminal video decoder is realized while the chip area is saved and the equipment power consumption is reduced.
In one possible implementation manner, the error code information includes video frame receiving time, an error code block occurrence starting position, and decoding completion time, the receiving of the current video frame, the controlling of the hardware decoder to decode the current video frame and perform error code detection, and obtaining the decoded frame and the error code information includes: receiving a current video frame and recording the receiving time of the video frame; controlling a hardware decoder to decode the current video frame, and if the hardware decoder decodes the error code block, terminating the decoding of the current video frame to generate a decoded frame and an initial position of the error code block; the decoding completion time is recorded.
In one possible implementation manner, if the error code information indicates that the current video frame has a wrong code block, calculating the error code complexity value according to the error code information includes: calculating a decoding time ratio according to the video frame receiving time, the decoding completion time and the total decoding time of each frame of image; calculating the occupied ratio of the error code blocks according to the initial position of the error code blocks and the total code block number of the current video frame; and calculating the error code complexity value according to the decoding time ratio, the error code block ratio, a prestored decoding time ratio coefficient and a prestored error code block ratio coefficient.
In one possible implementation manner, the greater the error code complexity value is, the greater the corresponding error code complexity is, and in the correspondence relationship between the pre-stored error code complexity value and the error code compensation algorithm: the N error code complexity values which are ordered from large to small sequentially correspond to N error code compensation algorithms which are ordered from small to large according to the complexity, so that the larger the error code complexity value is, the smaller the corresponding error code compensation algorithm complexity is, wherein N is a positive integer which is larger than 1.
In a second aspect, an embodiment of the present application provides an apparatus for concealing errors in a video stream, including: the decoding module is used for receiving the current video frame, controlling the hardware decoder to decode and detect the error code of the current video frame and obtaining the decoded frame and the error code information; the calculation module is used for calculating an error code complexity value according to the error code information if the error code information indicates that the current video frame has an error code block; and the compensation module is used for selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm.
In one possible implementation manner, the error code information includes a video frame receiving time, a start position of an error code block, and a decoding completion time, and the decoding module includes: the receiving submodule is used for receiving the current video frame and recording the receiving time of the video frame; the decoding submodule is used for controlling the hardware decoder to decode the current video frame, if the hardware decoder decodes the error code block, the decoding of the current video frame is stopped, and a decoding frame and an initial position of the error code block are generated; and the completion submodule is used for recording the decoding completion time.
In one possible implementation manner, the calculation module includes: the first calculation submodule is used for calculating a decoding time ratio according to the video frame receiving time, the decoding completion time and the total decoding time of each frame of image; the second calculation submodule is used for calculating the occupied ratio of the error code blocks according to the starting position of the error code blocks and the total code block number of the current video frame; and the third calculation submodule is used for calculating the error code complexity value according to the decoding time ratio, the error code block ratio, the prestored decoding time ratio coefficient and the prestored error code block ratio coefficient.
In a third aspect, an embodiment of the present application provides a terminal device, including: at least one processor, at least one hardware decoder communicatively coupled to the processor, and at least one memory communicatively coupled to the processor, wherein: the memory stores program instructions executable by the processor, the processor calling the program instructions to be able to perform the method provided by the first aspect.
In a fourth aspect, an embodiment of the present application provides a terminal device, including: at least one processor having at least one hardware decoder integrated therein, and at least one memory communicatively coupled to the processor, wherein: the memory stores program instructions executable by the processor, the processor calling the program instructions to be able to perform the method provided by the first aspect.
In a fifth aspect, embodiments of the present application provide a computer-readable storage medium storing computer instructions, which cause the computer to execute the method provided in the first aspect.
It should be understood that the second to fifth aspects of the embodiment of the present application are consistent with the technical solution of the first aspect of the embodiment of the present application, and beneficial effects obtained by the aspects and the corresponding possible implementation are similar, and are not described again.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present specification, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for concealing errors in a video stream according to an embodiment of the present application;
fig. 2 is a flowchart of a video stream error concealment method according to another embodiment of the present application;
fig. 3 is a flowchart of a method for concealing errors in a video stream according to yet another embodiment of the present application;
fig. 4 is a schematic diagram of a frame image error detection result according to an embodiment of the present application;
fig. 5 is a schematic diagram of a pre-stored error code complexity value and error code compensation algorithm correspondence table according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an error concealment apparatus for a video stream according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
[ detailed description ] embodiments
For better understanding of the technical solutions in the present specification, the following detailed description of the embodiments of the present application is provided with reference to the accompanying drawings.
It should be understood that the described embodiments are only a few embodiments of the present specification, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step are within the scope of the present specification.
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the specification. As used in the examples of this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
In the prior art, during a teleconference or a video call, when a terminal device encounters a situation with a poor network condition, packet loss occurs, and if a video receiving end does not implement an error concealment algorithm, a situation of incomplete picture and mosaic occurs in a decoded video frame. Due to the advantages and disadvantages of the hardware decoder and the software decoder, different problems exist when the hardware decoder or the software decoder is adopted at a video receiving end independently, and need to be solved.
For example, the hardware decoder of the current mobile phone end already supports a video decoding capability of 4K 30 frames/second, that is, the time required for decoding a frame of 3840x2160 video frame is less than 33ms, and the most important scenario requiring error concealment at the mobile phone end is the video call service based on the UDP protocol provided by the operator. The maximum resolution of the video call service supported by the current operator is 720p (1280x720), and if the network condition is poor, the video call can be performed with a smaller resolution, such as VGA (640x 480). With a decoding capability of 4k 30 frames/sec, the time for decoding a frame 720p video frame by the hardware decoder should be less than 5ms (1280 × 720/3840/2160 × 33), and a time margin of about 25ms is still provided without causing a jam.
Based on the above problems, embodiments of the present application provide a video stream error concealment method, apparatus, terminal device and computer readable storage medium, which integrate the advantages of a hardware and software decoder and the decoding capability of an existing hardware decoder, and use a method of combining hardware and software to conceal a video stream error, thereby saving chip area, reducing device power consumption and further saving product cost on the basis of ensuring the real-time performance of error concealment.
Fig. 1 is a flowchart of a video stream error concealment method according to an embodiment of the present application, and as shown in fig. 1, the video stream error concealment method may include:
And 102, if the error code information indicates that the current video frame has an error code block, calculating an error code complexity value according to the error code information.
And 103, selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm.
For example, the hardware decoder may be a hardware module having decoding and error detection functions, and the embodiment does not limit the specific structural form and the specific number of the hardware decoder.
In the above method for concealing error codes in video stream, the method starts to receive video stream after a teleconference of a terminal device or a video call program runs, and the same flow of the method for concealing error codes in video stream is adopted for each video frame in video stream, and may include: receiving a video frame, controlling a hardware decoder to decode and detect an error code of the current video frame to obtain a decoded frame and error code information, if the error code information indicates that the current video frame has an error code block, calculating an error code complexity value according to the error code information, selecting a corresponding error code compensation algorithm according to the error code complexity value based on a corresponding relation between a prestored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm. The method is characterized in that the hardware logic is used for taking charge of a video decoding and error code detecting part which is high in complexity and basically does not need to be upgraded and changed once the design is completed, and the error code compensating part is realized by a software program, so that the chip area is saved, the equipment power consumption is reduced on the basis of ensuring the real-time error code covering, and the product cost is further saved.
Fig. 2 is a flowchart of a video stream error concealment method according to another embodiment of the present application, as shown in fig. 2, in the embodiment shown in fig. 1 of the present application, step 101 may include:
Illustratively, the nth frame data in the video stream is received, and the current time point is recorded as t0. Here, n is to receive any one frame of video frame, and the same processing method is adopted, so that the video frame is not specifically limited to the number of frames. The current video frame is the video frame currently being processed.
Step 202, controlling a hardware decoder to decode the current video frame, and if the hardware decoder decodes the error code block, terminating the decoding of the current video frame to generate a decoded frame and an initial position of the error code block.
Illustratively, after receiving the nth frame of video frame, the hardware is solvedThe decoder decodes the n frame data, wherein the decoding may include entropy decoding, inverse transformation, motion compensation and other steps, and may perform error code detection in the entropy decoding stage, if the current frame has error codes, terminate the decoding of the frame in advance, record the start position of the error code block, which is denoted as BLKerrGenerating a decoded frame, wherein the decoded frame can comprise YUV data and block-level prediction information, the block-level prediction information can comprise a prediction mode, a motion vector, a reference frame index and the like, and the decoded frame is used for an error code compensation step. The present embodiment does not limit the specific method for decoding and error detection of the hardware decoder.
Illustratively, the current time point when decoding is completed is recorded as t1. If the current frame has no error code, the decoding process of the current frame will continue until the frame is completely decoded to obtain a decoded frame, and the error code information shows that the error code is 0, and then the next frame of video frame is received, and the process starts from step 201 again.
Fig. 3 is a flowchart of a video stream error concealment method according to yet another embodiment of the present application, as shown in fig. 3, in the embodiment shown in fig. 1 of the present application, step 102 may include:
Illustratively, a video frame receiving time t is obtained through step 2010The decoding completion time t is obtained by step 2031If the total decoding time T of each frame of image is 1000 ms/frame rate, for example, if the frame rate is 30, T is 1000/30 ms or 33.3ms, and the ratio T of the decoding time to the total decoding time of each frame of image is calculated from the video frame receiving time, the decoding completion time, and the total decoding time of each frame of image (T ═ T { (T) } T { (T } is the total decoding completion time of each frame of image is the video frame of the video frame) is calculated1-t0)/T。
Exemplary, BLKtotalIs the total number of coded blocks of a frame of image, obtained when decoding the current video frame, by stepsStep 202 obtains the error code block occurrence start position BLKerrCalculating the ratio of the code error block to the ratio radio (BLK)total-BLKerr)/BLKtotal。
Fig. 4 is a schematic diagram of a frame image error detection result according to an embodiment of the present application, and from fig. 4, the total number of coded blocks BLK of the frame image can be seentotal10 x 8-80 block, the start position BLK of the error code block appearserrAfter the 45 th block, the error code block ratio radio may be calculated to be (80-45)/80 to 0.5625.
And 303, calculating an error code complexity value according to the decoding time ratio, the error code block ratio, a pre-stored decoding time ratio coefficient and a pre-stored error code block ratio coefficient.
For example, the decoding time ratio t is obtained in step 301, the code error block ratio radio is obtained in step 302, and the formula for calculating the code error complexity value I may be: i (t, ratio) ═ α × t + β ratio, where α is a decoding time ratio coefficient, β is a code error block ratio coefficient, and α and β may be empirical values obtained by repeated experiments or values calculated by linear regression. In the present embodiment, the specific method for obtaining α and β is not limited, and any method may be used as long as the relationship between I and the coefficients α and β can be obtained.
In step 103, the pre-stored correspondence relationship between the error code complexity value and the error code compensation algorithm is already stored before the execution of the video stream error code concealment method, and is used for selecting the corresponding error code compensation algorithm, and in the running process of the software program, the error code compensation algorithm corresponding to the error code complexity value can be found from the pre-stored correspondence relationship between the error code complexity value and the error code compensation algorithm according to the error code complexity value calculated in step 102, so as to obtain the corresponding error code compensation algorithm.
Further, in the correspondence relationship between the pre-stored error code complexity value and the error code compensation algorithm: the larger the error code complexity value is, the larger the corresponding error code complexity is, and in the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm: the N error code complexity values which are ordered from large to small sequentially correspond to N error code compensation algorithms which are ordered from small to large according to the complexity, so that the larger the error code complexity value is, the smaller the corresponding error code compensation algorithm complexity is, wherein N is a positive integer which is larger than 1.
Illustratively, in order to ensure the real-time property of video error concealment, the error compensation algorithm with smaller complexity needs to be selected from candidate error compensation algorithms when the error complexity value I is larger, otherwise the real-time property requirement of video decoding cannot be guaranteed. The pre-stored correspondence relationship between the error code complexity value and the error code compensation algorithm may be a pre-stored correspondence table between the error code complexity value and the error code compensation algorithm.
Fig. 5 is a schematic diagram of a pre-stored error complexity value and error compensation algorithm mapping table according to an embodiment of the present application, where in fig. 5, a0-AK-1Representing K error compensation algorithms of low to high complexity, I0-IK-1An artificially assigned error complexity value, wherein I0>I1>...IK-1Wherein, the error complexity value of the current frame calculated by step 102 is assumed to be IcurError code complexity value I0-IK-1And error code compensation algorithm A0-AK-1The correspondence of (a) may be: when I iscur≥I0Time-to-error compensation algorithm A0When I is1≤Icur＜I0Time-to-error compensation algorithm A1When I is2≤Icur＜I1Time-to-error compensation algorithm A2And so on. Then, when the error complexity value IcurSatisfy In<Icur<In-1Then the current frame selects AnThe algorithm performs error code compensation. The error code compensation effect of the K error code compensation algorithms selected here is proportional to the complexity thereof, i.e. the compensation effect of the error code compensation algorithm with high complexity is better than that with low complexity. The embodiment does not limit the specific form of the correspondence between the pre-stored error code complexity value and the error code compensation algorithm, and the error code complexity value I is used to select the appropriate error code compensation algorithm as long as the real-time requirement can be met.
Illustratively, after the corresponding error compensation algorithm is selected, the error compensation is performed on the decoded frame by the selected error compensation algorithm. The error code compensation algorithm may be an error code compensation algorithm known in the art. The embodiment does not limit the specific method of error compensation.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Fig. 6 is a schematic structural diagram of a video stream error concealment apparatus according to an embodiment of the present application, where the video stream error concealment apparatus is disposed in a terminal device, and as shown in fig. 6, the video stream error concealment apparatus may include: a decoding module 41, a calculation module 42 and a compensation module 43;
the decoding module 41 is configured to receive a current video frame, control a hardware decoder to decode the current video frame and detect an error code, and obtain a decoded frame and error code information;
a calculating module 42, configured to calculate an error code complexity value according to the error code information if the error code information indicates that the current video frame has an error code block;
and the compensation module 43 is configured to select a corresponding error code compensation algorithm according to the error code complexity value based on a correspondence between a pre-stored error code complexity value and the error code compensation algorithm, and perform error code compensation on the decoded frame by using the selected error code compensation algorithm.
The decoding module 41 may include: the receiving submodule is used for receiving the current video frame and recording the receiving time of the video frame; the decoding submodule is used for controlling the hardware decoder to decode the current video frame, if the hardware decoder decodes the error code block, the decoding of the current video frame is stopped, and a decoding frame and an initial position of the error code block are generated; and the completion submodule is used for recording the decoding completion time.
Among other things, the calculation module 42 may include: the first calculation submodule is used for calculating a decoding time ratio according to the video frame receiving time, the decoding completion time and the total decoding time of each frame of image; the second calculation submodule is used for calculating the occupied ratio of the error code blocks according to the starting position of the error code blocks and the total code block number of the current video frame; and the third calculation submodule is used for calculating the error code complexity value according to the decoding time ratio, the error code block ratio, the prestored decoding time ratio coefficient and the prestored error code block ratio coefficient.
For example, the video stream error concealment apparatus may be a chip, and the chip may integrate a hardware decoder, a memory and a processor, where the memory is pre-stored with an error compensation algorithm and a computer program, and when the chip is in operation, the technical solution of the embodiment of the method shown in fig. 1 in this specification may be executed.
It is to be understood that the above chip may be a processor chip, or may be another chip integrated with a hardware decoder, a memory, and a processor, and the present embodiment does not limit the type of the chip.
The video stream error concealment apparatus provided in the embodiment shown in fig. 6 can be used to implement the technical solution of the method embodiment shown in fig. 1 in this specification, and the implementation principle and technical effects of the apparatus can be further referred to the related description in the method embodiment.
Fig. 7 is a schematic structural diagram of a terminal device according to an embodiment of the present application, and as shown in fig. 7, the terminal device may include at least one processor, at least one hardware decoder communicatively connected to the processor, and at least one memory communicatively connected to the processor, where: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the video stream error concealment method provided by the embodiments shown in fig. 1 to 5 in the present specification.
In the embodiment corresponding to fig. 7, the at least one hardware decoder may also be integrated into the processor, and the processor invoking the program instruction may also be capable of executing the video stream error concealment method provided in the embodiments shown in fig. 1 to 5 in this specification, and this embodiment does not limit the form of the terminal device.
The terminal device may be an intelligent electronic device such as a smart phone, a tablet computer, or a notebook computer, and the form of the terminal device is not limited in this embodiment.
For example, fig. 7 illustrates a schematic structure diagram of a terminal device by taking a smart phone as an example, as shown in fig. 7, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, a hardware decoder 150, and the like.
It is to be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation to the terminal device 100. In other embodiments of the present application, terminal device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.
The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.
A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.
The processor 110 executes programs stored in the internal memory 121 to perform various functional applications and data processing, for example, implementing the video stream error concealment method provided by the embodiments of fig. 1 to 5 of the present application. The control hardware decoder decodes and detects the error code of the current video frame, may be used to call the hardware decoder 150 to decode and detect the error code of the current video frame, wherein the corresponding error code compensation algorithm is selected to perform error code compensation on the decoded frame, and may be used to call an error code compensation algorithm program stored in the processor 110 or stored in the internal memory 121 to perform error code compensation.
In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.
The wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 130, the wireless communication module 140, a modem processor, a baseband processor, and the like.
The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in terminal device 100 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
The mobile communication module 130 may provide a solution including 2G/3G/4G/5G wireless communication applied on the terminal device 100. The mobile communication module 130 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module 130 can receive the electromagnetic wave from the antenna 1, and filter, amplify, etc. the received electromagnetic wave, and transmit the electromagnetic wave to the modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna 1 to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the same device as at least some of the modules of the processor 110.
The wireless communication module 140 may provide a solution for wireless communication applied to the terminal device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The wireless communication module 140 may be one or more devices integrating at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 140 may also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into electromagnetic waves through the antenna 2 to radiate the electromagnetic waves.
In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 130 and the antenna 2 is coupled to the wireless communication module 140, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), General Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), Wideband Code Division Multiple Access (WCDMA), time-division code division multiple access (time-division code division multiple access, TD-SCDMA), Long Term Evolution (LTE), LTE, BT, GNSS, WLAN, NFC, FM, and/or IR technologies, etc. The GNSS may include a Global Positioning System (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and/or a Satellite Based Augmentation System (SBAS).
The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.
The internal memory 121 may be used to store computer-executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, a phonebook, etc.) created during use of the terminal device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like. The processor 110 executes various functional applications of the terminal device 100 and data processing by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor. The embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the video stream error concealment method provided in the embodiments shown in fig. 1 to 5 in this specification.
The computer-readable storage medium described above may take any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM) or flash memory, an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present description may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
In the description of embodiments of the invention, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present specification, "a plurality" means at least two, e.g., two, three, etc., unless explicitly defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present description in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present description.
The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.
It should be noted that the terminal referred to in the embodiments of the present application may include, but is not limited to, a Personal Computer (PC), a Personal Digital Assistant (PDA), a wireless handheld device, a tablet computer (tablet computer), a mobile phone, an MP3 player, an MP4 player, and the like.
In the several embodiments provided in this specification, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
In addition, functional units in the embodiments of the present description may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (10)
1. A video stream error code masking method is applied to a processor of a terminal device, and is characterized in that the terminal device is provided with a hardware decoder and a pre-stored error code compensation algorithm, and the method comprises the following steps:
receiving a current video frame, and controlling a hardware decoder to decode and detect an error code of the current video frame to obtain a decoded frame and error code information;
if the error code information indicates that the current video frame has an error code block, calculating an error code complexity value according to the error code information;
and selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm.
2. The method of claim 1, wherein the error information comprises a video frame receiving time, a start position of an error block, and a decoding completion time, and the receiving the current video frame, the controlling the hardware decoder to decode and perform error detection on the current video frame, and obtaining the decoded frame and the error information comprises:
receiving a current video frame and recording the receiving time of the video frame;
controlling a hardware decoder to decode the current video frame, and if the hardware decoder decodes the error code block, terminating the decoding of the current video frame to generate a decoded frame and an initial position of the error code block;
the decoding completion time is recorded.
3. The method of claim 1, wherein if the error information indicates that there is a block error in the current video frame, the calculating the error complexity value according to the error information comprises:
calculating a decoding time ratio according to the video frame receiving time, the decoding completion time and the total decoding time of each frame of image;
calculating the occupied ratio of the error code blocks according to the initial position of the error code blocks and the total code block number of the current video frame;
and calculating the error code complexity value according to the decoding time ratio, the error code block ratio, a prestored decoding time ratio coefficient and a prestored error code block ratio coefficient.
4. The method of claim 1, wherein the greater the error complexity value, the greater the corresponding error complexity, and wherein the pre-stored error complexity value corresponds to an error compensation algorithm, wherein: the N error code complexity values which are ordered from large to small sequentially correspond to N error code compensation algorithms which are ordered from small to large according to the complexity, so that the larger the error code complexity value is, the smaller the corresponding error code compensation algorithm complexity is, wherein N is a positive integer which is larger than 1.
5. An apparatus for concealing errors in a video stream, comprising:
the decoding module is used for receiving the current video frame, controlling the hardware decoder to decode and detect the error code of the current video frame and obtaining the decoded frame and the error code information;
the calculation module is used for calculating an error code complexity value according to the error code information if the error code information indicates that the current video frame has an error code block;
and the compensation module is used for selecting a corresponding error code compensation algorithm according to the error code complexity value based on the corresponding relation between the pre-stored error code complexity value and the error code compensation algorithm, and performing error code compensation on the decoded frame by adopting the selected error code compensation algorithm.
6. The apparatus of claim 5, wherein the error information comprises a video frame receiving time, a block error occurrence start position, and a decoding completion time, and the decoding module comprises:
the receiving submodule is used for receiving the current video frame and recording the receiving time of the video frame;
the decoding submodule is used for controlling the hardware decoder to decode the current video frame, if the hardware decoder decodes the error code block, the decoding of the current video frame is stopped, and a decoding frame and an initial position of the error code block are generated;
and the completion submodule is used for recording the decoding completion time.
7. The apparatus of claim 5, wherein the computing module comprises:
the first calculation submodule is used for calculating a decoding time ratio according to the video frame receiving time, the decoding completion time and the total decoding time of each frame of image;
the second calculation submodule is used for calculating the occupied ratio of the error code blocks according to the starting position of the error code blocks and the total code block number of the current video frame;
and the third calculation submodule is used for calculating the error code complexity value according to the decoding time ratio, the error code block ratio, the prestored decoding time ratio coefficient and the prestored error code block ratio coefficient.
8. A terminal device, comprising:
at least one processor, at least one hardware decoder communicatively coupled to the processor, and at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the video stream error concealment method according to any one of claims 1 to 4.
9. A terminal device, comprising:
at least one processor having at least one hardware decoder integrated therein, and at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the video stream error concealment method according to any one of claims 1 to 4.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the method of error concealment of a video stream according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111101332.5A CN113810721B (en) | 2021-09-18 | 2021-09-18 | Video stream error concealment method, device, terminal equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111101332.5A CN113810721B (en) | 2021-09-18 | 2021-09-18 | Video stream error concealment method, device, terminal equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113810721A true CN113810721A (en) | 2021-12-17 |
| CN113810721B CN113810721B (en) | 2023-07-25 |
Family
ID=78939984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111101332.5A Active CN113810721B (en) | 2021-09-18 | 2021-09-18 | Video stream error concealment method, device, terminal equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113810721B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102769747A (en) * | 2012-06-29 | 2012-11-07 | 中山大学 | A hierarchical distributed video encoding and decoding method and system based on parallel iteration |
| US20130028325A1 (en) * | 2011-07-29 | 2013-01-31 | Canon Kabushiki Kaisha | Method and device for error concealment in motion estimation of video data |
| CN103957413A (en) * | 2014-01-13 | 2014-07-30 | 南京达鹏信息技术有限公司 | Real-time error-code concealment method and device for mobile network video communication application |
| CN104734818A (en) * | 2013-12-23 | 2015-06-24 | 联芯科技有限公司 | Self-adaption switchover method and device of MIMO receiving algorithm |
-
2021
- 2021-09-18 CN CN202111101332.5A patent/CN113810721B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130028325A1 (en) * | 2011-07-29 | 2013-01-31 | Canon Kabushiki Kaisha | Method and device for error concealment in motion estimation of video data |
| CN102769747A (en) * | 2012-06-29 | 2012-11-07 | 中山大学 | A hierarchical distributed video encoding and decoding method and system based on parallel iteration |
| CN104734818A (en) * | 2013-12-23 | 2015-06-24 | 联芯科技有限公司 | Self-adaption switchover method and device of MIMO receiving algorithm |
| CN103957413A (en) * | 2014-01-13 | 2014-07-30 | 南京达鹏信息技术有限公司 | Real-time error-code concealment method and device for mobile network video communication application |
Non-Patent Citations (1)
| Title |
|---|
| "基于自然场景统计的无参考视频业务QoE模型研究", 《中国优秀硕士论文全文数据库》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113810721B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2377735C2 (en) | Method, device and system for efficient encoding and decoding video data | |
| JP7604493B2 (en) | Multi-screen collaboration method and system, and electronic device | |
| CN117376615A (en) | Screen-throwing display method and electronic equipment | |
| US9978384B2 (en) | Audio decoding using modulator-demodulator | |
| CN113346911A (en) | Compression encoding method, electronic device, and storage medium | |
| CN113993226B (en) | Service processing method and device in terminal equipment supporting double cards | |
| CN113473013A (en) | Display method and device for beautifying effect of image and terminal equipment | |
| CN116368460A (en) | Audio processing method and device | |
| CN114205336A (en) | Cross-device audio playing method, mobile terminal, electronic device and storage medium | |
| CN116052701A (en) | Audio processing method and electronic equipment | |
| CN113810721B (en) | Video stream error concealment method, device, terminal equipment and readable storage medium | |
| CN116665692B (en) | Voice noise reduction method and terminal equipment | |
| CN113971969A (en) | Recording method, device, terminal, medium and product | |
| CN111770332A (en) | Frame insertion processing method, frame insertion processing device, storage medium and electronic device | |
| CN104053220A (en) | Multi-card multi-standby 3G functional module based on PCM | |
| CN116321085A (en) | Short-distance communication method and device and electronic equipment | |
| CN115482821A (en) | A voice control method, device, electronic equipment and storage medium | |
| CN113950037A (en) | Audio playing method and terminal equipment | |
| CN114697953B (en) | Network system determining method and device | |
| US12212619B2 (en) | Terminal device interaction method and apparatus | |
| CN117714969B (en) | A sound effect processing method, device and storage medium | |
| CN118450239B (en) | Frequency modulation method and related device | |
| CN113141665B (en) | Method and device for receiving system-on-demand message and user equipment | |
| CN118400814A (en) | Interference processing method, device, chip module, electronic equipment and storage medium | |
| CN115994573A (en) | Method and device for generating RNN model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |