CN115550709A - Data processing method and electronic device - Google Patents

Abstract

The application provides a data processing method and electronic equipment. The data processing method comprises the following steps: the first video data frame is read from an output buffer of a decoding module of the electronic device. Then, a second video data frame written into the first buffer before the first video data frame is determined, a first number of video data frames are spaced between the second video data frame and the first video data frame, and the first buffer is used for storing the video data frames to be sent to a display system of the electronic device. Next, a first time difference is determined based on a first time instant at which a first frame of video data is to be written to the first buffer and a second time instant at which a second frame of video data is to be written to the first buffer. When the first time difference is less than or equal to a current vertical synchronization period of the display system, the first frame of video data is discarded. Therefore, the time length of the video data frames waiting for display can be reduced by controlling the number of the video data frames to be sent and displayed in one Vsync period, and the display time delay is reduced.

Description

Data processing method and electronic device

technical field

The present application relates to the field of terminal equipment, and in particular to a data processing method and electronic equipment.

Background technique

At present, in the scene where video data is transmitted between two electronic devices through the network, and the receiving-end electronic device displays the content corresponding to the video data, the receiving-end electronic device receives the video data, decodes the video data and directly sends it to the display , but there is a large display delay of some video data frames, and the user experience is poor.

Contents of the invention

Embodiments of the present application provide a data processing method and an electronic device, so as to reduce the display delay of video data frames.

In some scenarios, due to the instability of network transmission, the decoding and display time of the video data by the electronic device at the receiving end is unstable. The SurfaceFlinger (graphics synthesizer) of the receiving-end electronic device refreshes the screen according to the Vsync cycle (ie, the vertical synchronization cycle). Therefore, the decoded video data of each frame needs to be displayed in order by further waiting for the Vsync signal to be displayed by SurfaceFlinger. In this way, the display delay of some video data frames is very large.

In a first aspect, the embodiment of the present application provides a data processing method. The data processing method is applied to an electronic device, and the method includes: reading the first video data frame from the output buffer of the decoding module of the electronic device, and the output buffer is used to store the decoded data decoded by the decoding module from the encoded data sent by the source electronic device frame of video data. Then, it is determined that the second video data frame written into the first buffer before the first video data frame is spaced by a first number of video data frames between the second video data frame and the first video data frame, and the first buffer is used for storing A frame of video data to be sent to the display system of the electronic device. Next, a first time difference is determined according to the first moment when the first video data frame is to be written into the first buffer and the second moment when the second video data frame is to be written into the first buffer. When the first time difference is less than or equal to the current vertical synchronization period of the display system, the first video data frame is discarded. In this way, if the video data frames to be displayed within one Vsync period are buffered and reach a certain number (the number is equal to the sum of the first number and 1), the electronic device discards the video data frames, thereby making the buffering within one Vsync period The number of video data frames to be displayed does not exceed the first number. By controlling the number of buffered video data frames to be displayed within one Vsync period, the waiting time for the video data frames to be displayed can be reduced, thereby reducing the display delay.

According to the first aspect, in an implementation manner, the data processing method may further include: reading the third video data frame from the output buffer of the decoding module of the electronic device, and determining that the first buffer is written before the third video data frame The fourth video data frame, the first number of video data frames are separated between the fourth data frame and the third video data frame. Then, according to the third moment of writing the third video data frame to the first buffer and the fourth moment of writing the fourth video data frame to the first buffer, determine the second time difference, when the second time difference is greater than the current display system In the vertical synchronization period, the third video data frame is written into the first buffer. In this way, if the video data frame is written into the first cache moment, the time difference with the writing moment of the (first quantity+1) video data frame written into the first cache before the video data frame exceeds one Vsync period, the writing of this video data frame will not cause the number of video data frames to be displayed in the cache in a Vsync cycle to exceed the first number, so the video data frame can be written into the first cache, so that the subsequent The video data frames are sent to the display system for display.

According to the first aspect, in an implementation manner, when the second time difference is greater than the current vertical synchronization cycle of the display system, after writing the third video data frame into the first buffer, it further includes: recording the third video data frame into the first buffer. The moment of the frame of video data. In this way, it can be determined whether the (first number+1)th video data frame after the third video data frame can be written into the first buffer according to the writing moment of the third video data frame, so as to ensure that within one Vsync period The number of cached video data frames to be displayed does not exceed the first number, thereby reducing display delay.

According to the first aspect, in an implementation manner, it also includes: when the time difference between the current moment and the moment when the video data frame was sent to the display system last time is equal to the current vertical synchronization period, reading from the first cache Write the earliest target video data frame, and send the target video data frame to the display system. In this way, the video data frame is sent to the display system according to the current vertical synchronization cycle, which is consistent with the screen refresh frequency of the display system, which can reduce the capacity requirement of the cache of the display system and reduce the number of frames lost by the display system.

According to the first aspect, in an implementation manner, after sending the target video data frame to the display system, the method further includes: deleting the target video data frame from the first cache. In this way, deleting the video data frames that have been sent for display from the first cache can release the storage resources of the first cache in time, so that the first cache has enough space to store the video data frames subsequently decoded by the decoding module.

According to the first aspect, in an implementation manner, before reading the first video data frame from the output buffer of the decoding module of the electronic device, it further includes: reading the fifth video data from the output buffer of the decoding module of the electronic device frame, when the number of video data frames written into the first buffer is less than or equal to the first number, write the fifth video data frame into the first buffer. At the initial stage of data transmission from the source end to the receiver end, the first buffer is empty, and there is no need to discard video data frames at this time. When the video data frames in the first buffer reach the first number, then determine whether the subsequent video data frames are written into the first cache.

According to the first aspect, in an implementation manner, before reading the fifth video data frame from the output buffer of the decoding module of the electronic device, it also includes: receiving the encoded data sent by the electronic device at the source end, the encoded data is the video data If the frame is encoded, write the encoded data into the input buffer of the decoding module, so that the decoding module decodes the encoded data to obtain the decoded video data frame, and store the decoded video data frame into the output buffer of the decoding module. In this way, the electronic device at the receiving end can continuously receive and decode the encoded data sent by the electronic device at the source end.

According to the first aspect, in an implementation manner, after discarding the first video data frame when the first time difference is less than or equal to the current vertical synchronization period of the display system, the method further includes: deleting the first video data frame from the output buffer. In this way, by deleting the discarded video data frames from the output buffer, the storage resources of the output buffer can be released in time, so as to prevent the video data frames decoded by the decoding module from covering the video that has not yet been displayed due to insufficient remaining storage space of the output buffer Data Frame.

According to the first aspect, in an implementation manner, when the second time difference is greater than the current vertical synchronization cycle of the display system, after writing the third video data frame into the first buffer, further comprising: deleting the third video data from the output buffer frame. In this way, the video data frames that have been written into the first buffer are deleted from the output buffer, and the storage resources of the output buffer can be released in time, so as to prevent the video data frames decoded by the decoding module from being overwritten due to insufficient remaining storage space in the output buffer. Send video data frame for display.

According to the first aspect, in an implementation manner, the first quantity is less than or equal to the capacity of the first cache. In this way, it can be ensured that the first buffer space can store at least the first number of video data frames, ensuring a certain display quality.

In a second aspect, the present application provides an electronic device, the electronic device includes: a memory and a processor, the processor is coupled to the memory, the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device performs the following steps: Read the first video data frame from the output buffer of the decoding module of the electronic device; the output buffer is used to store the video data frame decoded by the decoding module from the encoded data sent by the source electronic device, and it is determined before the first video data frame The second video data frame written into the first cache, the second video data frame and the first video data frame are separated by a first number of video data frames, and the first cache is used to store the video to be sent to the display system of the electronic device Data frame, according to the first moment of writing the first video data frame to the first buffer and the second moment of writing the second video data frame to the first buffer, determine the first time difference, when the first time difference is less than or equal to the display The current vertical synchronization period of the system, discarding the first video data frame.

According to the second aspect, in an implementation manner, when the program instruction is executed by the processor, the electronic device is made to perform the following steps: read the third video data frame from the output buffer of the decoding module of the electronic device, and determine A fourth video data frame is written into the first buffer before the video data frame, and a first number of video data frames are separated between the fourth data frame and the third video data frame. Then, according to the third moment of writing the third video data frame to the first buffer and the fourth moment of writing the fourth video data frame to the first buffer, determine the second time difference, when the second time difference is greater than the current display system In the vertical synchronization period, the third video data frame is written into the first buffer. In this way, if the video data frame is written into the first cache moment, the time difference with the writing moment of the (first quantity+1) video data frame written into the first cache before the video data frame exceeds one Vsync period, the writing of this video data frame will not cause the number of video data frames to be displayed in the cache in a Vsync cycle to exceed the first number, so the video data frame can be written into the first cache, so that the subsequent The video data frames are sent to the display system for display.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is caused to perform the following steps: record the moment when the third video data frame is written into the first cache. In this way, it can be determined whether the (first number+1)th video data frame after the third video data frame can be written into the first buffer according to the writing moment of the third video data frame, so as to ensure that within one Vsync period The number of cached video data frames to be displayed does not exceed the first number, thereby reducing display delay.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is made to perform the following steps: the time difference between the current moment and the last time the video data frame is sent to the display system is equal to the current vertical In the case of a synchronous cycle, the target video data frame with the earliest writing time is read from the first cache, and the target video data frame is sent to the display system. In this way, the video data frame is sent to the display system according to the current vertical synchronization cycle, which is consistent with the screen refresh frequency of the display system, which can reduce the capacity requirement of the cache of the display system and reduce the number of frames lost by the display system.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is caused to perform the following steps: delete the target video data frame from the first cache. In this way, deleting the video data frames that have been sent for display from the first cache can release the storage resources of the first cache in time, so that the first cache has enough space to store the video data frames subsequently decoded by the decoding module.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is made to perform the following steps: read the fifth video data frame from the output buffer of the decoding module of the electronic device, and send the first When the number of video data frames written in the cache is less than or equal to the first number, write the fifth video data frame into the first cache. At the initial stage of data transmission from the source end to the receiver end, the first buffer is empty, and there is no need to discard video data frames at this time. When the video data frames in the first buffer reach the first number, then determine whether the subsequent video data frames are written into the first cache.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is caused to perform the following steps: receive the encoded data sent by the source electronic device, the encoded data is obtained by encoding the video data frame, and convert the The encoded data is written into the input buffer of the decoding module, so that the decoding module decodes the encoded data to obtain decoded video data frames, and stores the decoded video data frames in the output buffer of the decoding module. In this way, the electronic device at the receiving end can continuously receive and decode the encoded data sent by the electronic device at the source end.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is caused to perform the following steps: delete the first video data frame from the output buffer. In this way, by deleting the discarded video data frames from the output buffer, the storage resources of the output buffer can be released in time, so as to prevent the video data frames decoded by the decoding module from covering the video that has not yet been displayed due to insufficient remaining storage space of the output buffer Data Frame.

According to the second aspect, in an implementation manner, when the program instructions are executed by the processor, the electronic device is caused to perform the following steps: delete the third video data frame from the output buffer. In this way, the video data frames that have been written into the first buffer are deleted from the output buffer, and the storage resources of the output buffer can be released in time, so as to prevent the video data frames decoded by the decoding module from being overwritten due to insufficient remaining storage space in the output buffer. Send video data frame for display.

According to the second aspect, in an implementation manner, the first quantity is less than or equal to the capacity of the first cache. In this way, it can be ensured that the first buffer space can store at least the first number of video data frames, ensuring a certain display quality.

In a third aspect, the present application provides a computer-readable storage medium, including a computer program. When the computer program runs on an electronic device, the electronic device executes the data processing method in any one of the foregoing first aspects.

Description of drawings

FIG. 1 is a schematic structural diagram of an electronic device 100 exemplarily shown;

FIG. 2 is a software structural block diagram of an electronic device 100 according to an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a data processing process in the application scenario shown in FIG. 3 exemplarily shown;

FIG. 5 is a schematic diagram of a flow process of data in the data processing method of the embodiment of the present application;

Fig. 6 is a schematic diagram of timing in the video frame processing process exemplarily shown.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.

The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than describing a specific order of the target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

In this paper, the display delay refers to the difference between the moment when the video data frame is displayed on the electronic device at the receiving end and the moment at which the video data frame is displayed on the electronic device at the source end.

Herein, the Vsync period is a vertical synchronization period, which may also be called a screen refresh period. There is a reciprocal relationship between the Vsync period and the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync period is 16.6ms (milliseconds), and when the screen refresh rate is 120Hz, the Vsync period is 8.3ms.

The data processing method in the embodiment of the present application can be applied to a screen projection scenario. In the screen projection scenario, the first electronic device acts as the source, and after encoding the video data, it can be transmitted to the second electronic device through the network, such as Wi-Fi, Bluetooth, etc., and the second electronic device is the receiving end. After decoding the video data, the second electronic device sends the decoded video data to a display system for display (referred to as display sending).

In the traditional solution, the video data is sent to display immediately after being decoded. The SurfaceFlinger of the electronic device at the receiving end displays video data frame by frame according to the Vsync period. The video frames that need to be displayed are all waiting in SurfaceFlinger's cache. The lower the sorting, the longer the video frame waits, and the greater the display delay.

For example, suppose there are 5 video frames waiting to be displayed in the cache. If the first video frame needs to wait for 1 Vsync cycle to be displayed, then the second video frame needs to wait for 2 Vsync cycles, and the third video frame needs to wait for 3 Vsync cycles, the fourth video frame needs to wait for 4 Vsync cycles, and the fifth video frame needs to wait for 5 Vsync cycles. It can be seen that the lower the video frame is, the longer the waiting time is, and the greater the display delay is. If there are a lot of waiting video frames, the display delay of the later video frames will be very large, and the user experience will be poor.

In an example, the first electronic device may be, for example, a PC (Personal Computer, personal computer), and the PC may be installed with a Windows system. The second electronic device may be a mobile phone, a tablet, etc. installed with an Android (Android) system.

In another example, the first electronic device may be, for example, a mobile phone or a tablet installed with an Android system, and the second electronic device may be a PC installed with a Windows system.

In another example, the first electronic device may be, for example, a mobile phone, a tablet, etc. installed with an Android system, and the second electronic device may also be a mobile phone, a tablet, etc. installed with an Android system.

The embodiment of the present application provides a data processing method, which can reduce the display delay of the second electronic device and improve user experience.

The data processing method in the embodiment of the present application may be applied to the aforementioned second electronic device. The structure of the second electronic device may be as shown in FIG. 1 .

FIG. 1 is a schematic structural diagram of an electronic device 100 exemplarily shown. It should be understood that the electronic device 100 shown in FIG. 1 is only an example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, and two or more components may be combined , or can have different component configurations. The various components shown in FIG. 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

1, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 , a display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In this embodiment of the present application, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

FIG. 2 is a schematic block diagram showing the software structure of the electronic device 100 according to the embodiment of the present application.

The layered architecture of the electronic device 100 divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system may include an application program layer, an application program framework layer, and a kernel layer.

The application layer can consist of a series of application packages.

As shown in FIG. 2 , the application package of the application layer of the electronic device 100 may include modules such as service, service management, service setting, transmission module, decoding logic module, and display module.

Among them, the business module is used to realize specific business functions. The service management module is used to realize the management function corresponding to the service corresponding to the service module. The business management module is used to realize the setting function corresponding to the business corresponding to the business module.

For example, the service module may be a wireless screen projection module.

Wherein, the decoding logic module is used to execute the data processing method of the present application. For the specific functions of the decoding logic module, please refer to the detailed description in the subsequent embodiments of this document.

Wherein, the display module is used for displaying video data.

As shown in Figure 2, the application framework layer may include basic capabilities, video codec, Socket and other modules.

Wherein, the basic capability module is used to provide various APIs (Application Programming Interface, application programming interface) that may be used when building an application program.

The video codec module is used to implement the encoding and decoding functions of video data. The video codec module may include an encoding module and a decoding module. The encoding module is used for encoding video data, and the decoding module is used for decoding video data. The video codec module may be a module for encoding and decoding video data in a hardware manner. Wherein, the encoding logic module of the application layer can call the encoding module in the video codec module for encoding, and the decoding logic module of the application layer can call the decoding module in the video codec module for decoding.

Socket (socket) is an abstraction of endpoints for two-way communication between application processes on different hosts in the network. A socket is one end of process communication on the network, providing a mechanism for application layer processes to exchange data using network protocols. From the perspective of its position, the socket connects to the application process and connects to the network protocol stack, which is the interface for the application program to communicate through the network protocol, and the interface for the application program to interact with the network protocol stack.

Wherein, the application framework layer may also include SurfaceFlinger (not shown in FIG. 2 ).

The kernel layer is the layer between hardware and software.

As shown in Figure 2, the kernel layer may include modules such as a sensor driver, a Wi-Fi driver, and a USB driver.

It can be understood that the layers in the software structure shown in FIG. 2 and the components included in each layer do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

FIG. 3 is a schematic diagram of an application scenario of the data processing method according to the embodiment of the present application. Referring to FIG. 3 , the application scenario of this embodiment includes a source end and a receiver end, both of which are electronic devices. The software architecture of the receiving end may be shown in FIG. 2 , which will not be repeated here. The software architecture at the source end may include an application program layer, an application program framework layer, and a kernel layer. Wherein, the application program layer at the source end may include modules such as business, business management, business setting, transmission module, screen capture module, and coding logic module. The application framework layer at the source can have the same structure as the application framework layer at the receiving end, and the kernel layer at the source can have the same structure as the kernel layer at the receiving end. Please refer to the aforementioned description of the software structure shown in FIG. 2 , which will not be repeated here.

FIG. 4 is a schematic diagram of a data processing process in the application scenario shown in FIG. 3 exemplarily shown. Referring to Figure 4, in this embodiment, the data processing process may include the following steps:

S1. After the service at the source end is started, the service module at the source end sends a service start notification to the screen capture module at the source end.

In this embodiment, the source end is a mobile phone A installed with an Android system, the receiver is a tablet B installed with an Android system, and the source end projects a video to the receiver end as an example for illustration.

When the user turns on the wireless projection function of mobile phone A, search for devices that can receive the screen projection of mobile phone A in mobile phone A, and obtain the list C of available devices that can receive the screen projection of mobile phone A, and the available device list C includes tablet B. Then, the user can select the tablet B in the searched available device list C as the receiving device for screen projection this time. Next, a wireless connection, such as a Wi-Fi connection or a Bluetooth connection, is established between the mobile phone A and the tablet B.

After the wireless connection between mobile phone A and tablet B is successfully established, the wireless screen projection module in mobile phone A sends a service start notification to the screen capture module in mobile phone A. The service start notification is used to instruct the screen capture module to capture video data on the screen.

After the wireless connection between mobile phone A and tablet B is successfully established, the decoding logic module in tablet B creates the first thread, the second thread, the third thread and the first cache.

Wherein, the capacity N of the first buffer needs to be greater than the capacity of the output buffer (OutputBuffer) of the decoding module in the tablet B. N represents the maximum number of frames that can be stored in the first cache.

In practical applications, the capacity of the first cache can be set according to experience. For example, if the difference between the decoding speed of the decoding module in tablet B and the screen refresh frequency of tablet B is small, a first buffer with a smaller capacity may be set. On the contrary, if the decoding speed of the decoding module in the tablet B is greatly different from the refresh rate of the screen of the tablet B, a larger-capacity first buffer can be set.

S2. The screen capture module of the source terminal captures the video data on the screen of the source terminal.

For example, the screen capture module of mobile phone A starts to capture video data on the screen of mobile phone A after receiving the service start notification. The screen capture module can capture video data at a certain frame rate, such as 60FPS.

S3. The screen capture module at the source end sends video data to the encoding logic module at the source end.

For example, the screen capture module of mobile phone A sends captured video data to the encoding logic module of mobile phone A.

S4. The encoding logic module at the source calls the encoding module to encode the video data to obtain encoded data.

Here, the coding module is a module in the video codec module at the source end.

Wherein, the coding module can code the video data based on the video coding standard H.264. The coding module can also use any coding method among JPEG (Joint Photographic Experts Group, Joint Photographic Experts Group), H.261, MPEG (Moving Picture Experts Group, Dynamic Picture Experts Group), etc. to perform coding.

In another example, the encoding module may also use more than two encoding methods to perform mixed encoding.

It should be noted that the above is only a schematic description of the encoding method adopted by the encoding module, and other encoding methods are not excluded, and the embodiment of the present application does not limit the encoding method adopted by the encoding module.

S5. The encoding logic module at the source end sends the encoded data to the transmission module at the source end.

Wherein, in an example, the transmission module may be a Wi-Fi module. In another example, the transmission module may be a Bluetooth module.

Assuming that both mobile phone A and tablet B are equipped with Wi-Fi modules and Bluetooth modules, when the wireless connection between mobile phone A and tablet B is a Wi-Fi connection in wireless projection, the coding logic module of mobile phone A sends The Wi-Fi module sends encoded data. If the wireless connection between mobile phone A and tablet B is a Bluetooth connection in the wireless screen projection, the encoding logic module of mobile phone A sends encoded data to the Bluetooth module of mobile phone A.

S6. The transmission module at the source end sends the encoded data to the transmission module at the receiving end.

For example, if the wireless connection between mobile phone A and tablet B is a Wi-Fi connection in wireless screen projection, the Wi-Fi module of mobile phone A sends coded data to the Wi-Fi module of tablet B.

If the wireless connection between mobile phone A and tablet B is a Bluetooth connection in wireless screen projection, the Bluetooth module of mobile phone A sends coded data to the Bluetooth module of tablet B.

Due to the instability of network transmission, the number of frames received by the receiving end within one Vsync period is not fixed. For example, in the first Vsync cycle, the receiving end receives 1 frame of data; in the second Vsync cycle, the receiving end receives 2 frames of data; in the third Vsync cycle, the receiving end receives 1 frame of data... .

S7. The transmission module at the receiving end sends the encoded data to the decoding logic module at the receiving end.

For example, after receiving the encoded data sent by the Wi-Fi module of mobile phone A, the Wi-Fi module of tablet B can send the encoded data to the decoding logic module of tablet B.

S8. The encoding logic module at the receiving end receives the encoded data through the created first thread.

S9. The encoding logic module at the receiving end sends the encoded data to the input buffer of the decoding module through the first thread.

Here, the decoding module is a module in the video codec module at the source end. The decoding module itself has two buffers, one is the input buffer (InputBuffer), and the other is the output buffer (OutputBuffer). The aforementioned first cache does not belong to the decoding module, and the first cache is a cache created by the decoding logic module.

S10. The decoding module at the receiving end decodes the coded data to obtain video data, which is stored in the output buffer of the decoding module.

After the decoding module decodes the video data from the encoded data, it will immediately store the video data in the output buffer OutputBuffer of the decoding module.

S11. The decoding logic module at the receiving end obtains video data from the output buffer of the decoding module through the created second thread, and determines whether to store the acquired video data in the created first buffer according to the preset display threshold M.

The second thread can monitor the output buffer OutputBuffer of the decoding module, and once video data is written into the output buffer OutputBuffer, immediately read the video data in the output buffer OutputBuffer into the first buffer. Then, the output buffer OutputBuffer can delete the video data that the second thread has finished reading.

The video data read by the second thread from the output buffer OutputBuffer each time may be one or more frames.

Wherein, the display threshold M represents the maximum number of display frames within one Vsync period. The display threshold M can be preset, and M is less than or equal to N. Wherein, the value of M can be determined according to the requirement on time delay. When the required time delay is small, the value of M can be set to a small value; when the time delay is required to be large, the value of M can be set to a large value. In this way, the number of frames sent for display in one Vsync cycle is controlled below M frames.

In the embodiment of the present application, determining whether to store the acquired video data into the created first cache according to the preset display threshold M may include:

Obtain the time T1 when the current video data frame enters the first cache;

Obtain the time T2 when the M-1th frame video data before the current video data frame enters the first buffer;

Compare the difference between T1 and T2 with the Vsync period to obtain the comparison result;

If the comparison result indicates that the difference between T1 and T2 is greater than the Vsync period, then the current video data frame is stored in the first buffer; if the comparison result indicates that the difference between T1 and T2 is less than or equal to the Vsync period, then the current video data frame is discarded , the current video frame will not be stored in the first buffer.

S12. The decoding logic module at the receiving end periodically reads video data from the first cache through the created third thread, and the period is a Vsync period.

For example, if the Vsync period of tablet B is 16.6ms, then the third thread reads a frame of video data from the first buffer every 16.6ms. If the Vsync period of tablet B is 8.3ms, then the third thread reads a frame of video data from the first buffer every 8.3ms.

It should be noted that the above numerical values of the Vsync period are only illustrative, and do not limit the Vsync period of the embodiment of the present application. In practical applications, the actual Vsync period may be determined according to the screen refresh rate adopted by SurfaceFlingerVsync in the second electronic device.

S13. The decoding logic module at the receiving end sends the video data to the display module through the third thread according to the Vsync period.

For example, if the Vsync period of tablet B is 16.6ms, then the third thread sends a frame of video data to the display module every 16.6ms. If the Vsync period of tablet B is 8.3ms, then the third thread sends a frame of video data to the display module every 8.3ms.

It should be noted that, within the same Vsync cycle, the time t1 when the third thread reads the video data frame from the first cache is earlier than the time t2 when the third thread sends the video data to the display module.

After the display module receives the video data frame, it can call the releaseOutputBuffer method. The releaseOutputBuffer method calls the MediaCodec of the Android display system, and the MediaCodec calls SurfaceFlinger. SurfaceFlinger displays the interface corresponding to the video data frame.

FIG. 5 is a schematic diagram illustrating a data transfer process in the data processing method of the embodiment of the present application. Referring to FIG. 4 , as shown in FIG. 5 , the first thread receives the encoded data transmitted from the source, and then stores the encoded data in the input buffer of the decoding module. Then, the decoding module decodes the data in the input buffer, and stores the decoded video data in the output buffer of the decoding module. Then, the second thread obtains the video data frame from the output buffer of the decoding module, if the time T1 when the video data frame enters the first buffer and the time T2 when the M-1 frame video data before the video data frame enters the first buffer If the difference t0 between T1 and T2 is less than or equal to the Vsync cycle, then the video data frame is discarded, and the video data frame will not be stored in the first buffer. stored in the first cache. Next, the third thread reads video data frame by frame from the first buffer according to the Vsync cycle, and upon receiving the Vsync signal, the third thread transmits the read video data to SurfaceFlinger for display. After SurfaceFlinger receives the video data, it displays the video image frame by frame according to the Vsync cycle.

In this embodiment of the present application, by controlling the number of buffered video data frames to be displayed within one Vsync cycle, the waiting time for the video data frames to be displayed can be reduced, thereby reducing the display delay. This is illustrated below with the timing during video frame processing.

Fig. 6 is a schematic diagram of timing in the video frame processing process exemplarily shown. All time axes in Figure 6 are aligned. Please refer to Figure 6, in the first Vsync cycle, the second thread reads the first frame of video data from the output buffer at t1, reads the second frame of video data from the output buffer at t2, and reads the second frame of video data from the output buffer at t3. The third frame of video data is read from the output buffer, the fourth frame of video data is read from the output buffer at time t4, and the fifth frame of video data is read from the output buffer at time t5. In the second Vsync period, the second thread reads the sixth frame of video data from the output buffer at time t6.

Please continue to refer to Figure 6, the first frame of video data enters the first buffer at t1', the second frame of video data enters the first buffer at t2', the third frame of video data enters the first buffer at t3', and the fourth frame of video data enters the first buffer at t3'. A frame of video data enters the first buffer at time t4', a fifth frame of video data enters the first buffer at time t5', and a sixth frame of video data enters the first buffer at time t6'.

Assume that the SurfaceFlinger cache has a capacity of 6 frames. In the traditional scheme, the first frame of video data, the second frame of video data, the third frame of video data, the fourth frame of video data, the fifth frame of video data, and the sixth frame of video data are all sent to the SurfaceFlinger cache, and then SurfaceFlinger follows The Vsync period is displayed frame by frame. In this way, the display time interval between the sixth frame of video data and the first frame of video data is 5 Vsync cycles, that is, the sixth frame of video data needs to wait for 5 Vsync cycles to be displayed after the first frame of video data is displayed, and the sixth frame of video The display delay of the video data following the data will be greater than the display delay of the sixth frame of video data, and as the number of frames increases later, the display delay will become larger and larger.

Herein, the moment when the video data frame enters the first cache refers to the moment when the video data frame will be written into the first cache, but whether it is actually written into the first cache still needs to be judged according to the follow-up of this embodiment. If it is determined to discard the video data frame, then the video data frame will not be written into the first cache, if it is determined to write the video data frame into the first cache, then the video data frame will be written at the moment when the determined video data frame enters the first cache into the first cache.

Please continue to refer to Fig. 6, according to the data processing method of the embodiment of the present application, assuming that the display threshold M=5, since the moment t5' when the fifth frame of video data enters the first buffer and the moment when the first frame of video data enters the first buffer The difference between t1' is less than one Vsync period, the second thread discards the fifth frame of video data, and the fifth frame of video data will not be stored in the first buffer. The moment t6' when the sixth frame of video data enters the first cache is the same as the first frame of video data (because the fifth frame of video data is discarded, the first M-1 frame of video data of the sixth frame of video data is the first frame of video data) The difference between the times t1' entering the first buffer is greater than one Vsync period, so the second thread writes the sixth frame of video data into the first buffer. In this case, the sixth frame of video data can be displayed after waiting for 4 Vsync periods after the first frame of video data is displayed. It can be seen that, compared with the traditional solution, the display delay of the sixth frame of video data is reduced. And, each frame of video data after the sixth frame of video data will meet the condition after being judged according to the display threshold M (this condition refers to the moment when the video data frame enters the first cache and the preceding M-1 frame of the video data frame The difference between the moment when the video data enters the first buffer is greater than one Vsync period) will be written into the first buffer. In this way, it is ensured that the number of frames waiting to be sent for display in any time window whose duration is equal to one Vsync period does not exceed the display threshold M, thereby effectively reducing the display delay of video data frames.

It should be noted that the above data processing method is only one embodiment of the data processing method of the present application, and the data processing method of the present application can also adopt other embodiments.

For example, 3 processing threads are used in the foregoing embodiments, and a new cache, that is, the first cache is constructed, while in other embodiments of the present application, 2 processing threads can also be used, and there is no need to construct the above-mentioned first cache. cache.

In an embodiment using two processing threads, thread 1 can be the same as the aforementioned first thread, and another thread 2 can be used to read video data from the output buffer of the decoding module and send it to display according to the Vsync period. For example, assuming that the display threshold M=5, in the first Vsync cycle, the electronic device at the receiving end receives the 1st to 5th frames of video data; in the second Vsync cycle, the receiving end receives the 6th to 9th frames of video data; On the 3rd Vsync cycle.... Thread 2 constructs a display sending cycle, and the duration of the display sending cycle is equal to the duration of the Vsync cycle.

In the first display cycle, thread 2 reads the first frame of video data from the output buffer of the decoding module and sends it to SurfaceFlinger; in the second display cycle, thread 2 reads the second frame of video data from the output buffer of the decoding module. Frame video data and send it to SurfaceFlinger... In the 5th sending cycle, thread 2 reads the 5th frame video data from the output buffer of the decoding module, if the moment of sending the 5th frame video data to SurfaceFlinger is the same as sending the 5th frame video data to SurfaceFlinger If the time difference of one frame of video data is less than or equal to the Vsync period, thread 2 discards the fifth frame of video data. Next, thread 2 reads the sixth frame of video data from the output buffer of the decoding module. If the difference between the moment of sending the sixth frame of video data to SurfaceFlinger and the moment of sending the first frame of video data to SurfaceFlinger is greater than the Vsync period, the thread 2 Send the sixth frame of video data to SurfaceFlinger. Then, thread 2 reads the 7th frame of video data from the output buffer of the decoding module. If the difference between the moment of sending the 7th frame of video data to SurfaceFlinger and the moment of sending the 2nd frame of video data to SurfaceFlinger is less than or equal to the Vsync period, Thread 2 discards frame 7 video data...and so on.

It can be seen that in the embodiment of the present application, the number of frames waiting to be sent for display in each Vsync period will not exceed the threshold M for display. In this way, by controlling the number of buffered video data frames to be displayed within one Vsync cycle, the waiting time for the video data frames to be displayed is reduced, thereby reducing the display delay.

The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory is coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device described above The data processing method to execute.

It can be understood that, in order to realize the above functions, the electronic device includes hardware and/or software modules corresponding to each function. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

This embodiment also provides a computer storage medium, in which computer instructions are stored, and when the computer instructions are run on the electronic device, the electronic device is made to execute the above related method steps to implement the data processing method in the above embodiment.

This embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the above related steps, so as to implement the data processing method in the above embodiment.

In addition, the embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein, the memory is used to store instructions executed by a computer, and when the device is running, the processing The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the data processing methods in the above method embodiments.

Wherein, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the corresponding method provided above The beneficial effects in the method will not be repeated here.

Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be assigned by different Completion of functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or It may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component shown as a unit may be one physical unit or multiple physical units, which may be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above contents is within the scope of the present application.

If an integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application. The above-mentioned storage medium includes: U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk, and other various media that can store program codes.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

The steps of the method or algorithm described in connection with the disclosure of the embodiments of the present application may be implemented in hardware, or in the form of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read Only Memory, ROM), erasable programmable read-only memory ( Erasable Programmable ROM, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically EPROM, EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC.

Those skilled in the art should be aware that, in the foregoing one or more examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims (11)

1. A data processing method is applied to electronic equipment and comprises the following steps:

reading a first video data frame from an output buffer of a decoding module of the electronic device, wherein the output buffer is used for storing the video data frame decoded by the decoding module from the encoded data sent by the source-end electronic device;

determining a second video data frame written into a first cache before the first video data frame, wherein a first number of video data frames are spaced between the second video data frame and the first video data frame, and the first cache is used for storing video data frames to be sent to a display system of the electronic equipment;

determining a first time difference according to a first time when the first video data frame is written into the first cache and a second time when the second video data frame is written into the first cache;

discarding the first video data frame when the first time difference is less than or equal to a current vertical synchronization period of the display system.

2. The method of claim 1, further comprising:

reading a third video data frame from an output buffer of a decoding module of the electronic device;

determining a fourth frame of video data written to the first buffer before the third frame of video data, the fourth frame of data being spaced from the third frame of video data by the first number of frames of video data;

determining a second time difference according to a third time when the third video data frame is written into the first cache and a fourth time when the fourth video data frame is written into the first cache;

and when the second time difference is larger than the current vertical synchronization period of the display system, writing the third video data frame into the first cache.

3. The method of claim 2, wherein after writing the third frame of video data to the first buffer when the second time difference is greater than a current vertical synchronization period of the display system, further comprising:

and recording the time when the third video data frame is written into the first buffer.

4. The method of claim 1, further comprising:

under the condition that the time difference between the current time and the time of sending the video data frame to the display system last time is equal to the current vertical synchronization period, reading the target video data frame with the earliest writing time from the first cache;

and sending the target video data frame to the display system.

5. The method of claim 4, wherein after transmitting the target frame of video data to the display system, further comprising:

deleting the target video data frame from the first buffer.

6. The method of claim 1, wherein prior to reading the first frame of video data from the output buffer of the decoding module of the electronic device, further comprising:

reading a fifth video data frame from an output buffer of a decoding module of the electronic device;

writing the fifth frame of video data to the first buffer if the number of frames of video data written to the first buffer is less than or equal to the first number.

7. The method of claim 6, wherein before reading the fifth frame of video data from the output buffer of the decoding module of the electronic device, further comprising:

receiving encoded data sent by the source electronic device, wherein the encoded data is obtained by encoding a video data frame;

writing the coded data into an input buffer of the decoding module so that the decoding module decodes the coded data to obtain a decoded video data frame;

and storing the decoded video data frame to an output buffer of the decoding module.

8. The method according to any of claims 1-7, further comprising, after discarding the first frame of video data when the first time difference is less than or equal to a current vertical synchronization period of the display system:

deleting the first video data frame from the output buffer.

9. The method of claim 2, wherein after writing the third frame of video data to the first buffer when the second time difference is greater than a current vertical synchronization period of the display system, further comprising:

deleting the third frame of video data from the output buffer.

10. The method of claim 1, wherein the first amount is less than or equal to a capacity of the first buffer.

11. An electronic device, comprising:

a memory and a processor, the memory coupled with the processor;

the memory stores program instructions that, when executed by the processor, cause the electronic device to perform the data processing method of any one of claims 1 to 10.

Images