CN117372238A - Registration function method, video frame processing method, device, equipment and storage medium - Google Patents

Abstract

The embodiments of the present application disclose a registration function method, a video frame processing method, a registration function device, a video frame processing device, computer equipment and a computer-readable storage medium. Among them, this application can register the registration function provided by the user in the layer compositor. The registered function will expose the interface in the form of a video processing function so that it can be called at any time. During the image processing process, the video processing function in the image synthesizer can be directly called, and the video memory resources are used for video processing. The video data does not need to be converted multiple times between the memory and the video memory. Compared with the existing technology, this application can support higher frame rate output under the same hardware configuration, ensure better video output effects, and improve performance and user experience.

Description

Registration function method, video frame processing method, device, equipment and storage medium

Technical field

The present application belongs to the technical field of computer video processing, and in particular relates to a registration function method, a video frame processing method, a registration function device, a video frame processing device, computer equipment and a computer-readable storage medium.

Background technique

With the rapid development and popularization of 5G, more and more apps running on mobile terminals have the function of optimizing the displayed images. Existing methods need to go through the following cumbersome process when processing video frames: Step 1: App controls to open the camera; Step 2: Gets the camera to collect video frames; Step 3: Reads the video frames from the Surface on the camera to the memory and converts them into pre-formatted video frames; Step 4: Write the pre-formatted video frames in the memory to the Surface you applied for, and then use the relevant principles of shaders to process the video frames, that is, video editing (such as: face recognition, Beauty, big eyes and face slimming, filters, etc.); Step 5: Fuse the processed Surface with the desktop screen; Step 6: Give the fused Surface to Surfaceflinger; Step 7: Deliver the screen in Surfaceflinger to opengl or screen hardware for display. The above steps need to be carried out every frame. If the APP requires 30fps, the above process needs to be carried out 30 times per second. The above steps are very cumbersome. The video frames are converted multiple times between memory and video memory, which is particularly wasteful of performance. As a result, the frame rate of the current mainstream live broadcast platforms is not high during live broadcast, and often freezes. At the same time, it also affects the CPU and memory of the mobile phone hardware. It occupies a large area, causing serious heating of the phone and increasing power consumption. Therefore, how to improve the processing efficiency of video frames is an urgent technical problem that needs to be solved by those skilled in the art.

The preceding description is intended to provide general background information and does not necessarily constitute prior art.

Contents of the invention

Based on this, it is necessary to address the above problems and propose a registration function method, video frame processing method, registration function device, video frame processing device, computer equipment and computer-readable storage medium, which can process video frames more efficiently.

This application solves its technical problems by adopting the following technical solutions:

This application provides a registration function method, which is applied to video equipment and includes the following steps: perform an initialization operation, and the objects of the initialization operation include at least one of an open graphics library, a video memory, and a texture handle; obtain the registration function provided by the user, and parse Register the function to obtain the first shader; perform a registration operation on the first shader, the registration operation includes compiling the shader code and binding it in the layer compositor; encapsulate the first shader that completes the registration operation as a video processing function, And expose the interface to the outside, so that the video application can directly call the video processing function through the interface.

In an optional embodiment of the present application, after performing the initialization operation, the method further includes: obtaining a second shader, which is owned by the layer compositor itself; and performing a registration operation on the second shader so as to create the second shader in the image. Register the second shader in the layer synthesizer; encapsulate the first shader that has completed the registration operation as a video processing function, and expose the interface to the outside, including: encapsulate the first shader and the second shader that have completed the registration operation as video Process functions and expose interfaces to the outside world.

In an optional embodiment of the present application, encapsulating the first shader that has completed the registration operation into a video processing function includes: serializing the first shader that has completed the registration operation into the processing pipeline of the open graphics library; obtaining the completed registration operation The external texture handle provided by the first shader; encapsulate the texture handle and processing pipeline into video processing functions.

This application also provides a video frame processing method, which is applied to video applications and includes the following steps: obtaining the first video data from the first layer, which is the layer of the camera module; sending the first video data To the layer compositor, the layer compositor completes the registration through the method described above; the video processing function in the layer compositor is called to control the layer compositor to process the first video data according to the video processing function. The video processing function is the video processing function described above; reads and outputs the video frames processed by the video processing function.

In an optional embodiment of the present application, after obtaining the first video data from the first layer, the method further includes: obtaining the second video data from the second layer, and the second layer is the layer of the application module; Perform screen fusion processing on the first video data and the second video data to obtain the third video data; send the third video data to the layer synthesizer, and call the video processing function to control the layer synthesizer to process the third video data according to the video processing function. Three video data; read and output the video frames processed by the video processing function.

In an optional embodiment of the present application, reading and outputting the video frames processed by the video processing function includes: determining whether there is a demand for streaming; if there is no demand for streaming, delivering the video frames to an open graphics library or screen The hardware displays; if there is a demand for streaming, the video frames are delivered to the open graphics library or screen hardware for display. At the same time, the video frames are read and transmitted to the associated streaming media server, so that the streaming media server can push the video.

This application also provides a registration function device, including: an initialization module, used to perform an initialization operation, the object of the initialization operation includes at least one of an open graphics library, a video memory, and a texture handle; a registration module, used to obtain the user-provided The registration function parses the registration function to obtain the first shader; performs a registration operation on the first shader. The registration operation includes compiling the shader code and binding it in the layer compositor; the encapsulation module is used to complete the registration operation of the third shader. A shader is encapsulated as a video processing function and exposes an interface to the outside, so that the video application can directly call the video processing function through the interface.

This application also provides a video frame processing device, including: an acquisition module, used to acquire the first video data from the first layer, where the first layer is the layer of the camera module; and a sending module, used to send the first video data to the camera module. The video data is sent to the layer synthesizer, and the layer synthesizer completes the registration through the method described above; the processing module is used to call the video processing function in the layer synthesizer, and uses the video memory resources to process the first video data; video processing The function is the video processing function described above; the output module is used to read and output the video frames processed by the video processing function.

This application also provides a computer device, including a processor and a memory: the processor is used to execute the computer program stored in the memory to implement the aforementioned method.

This application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the aforementioned method is implemented.

Adopting the embodiments of this application has the following beneficial effects:

This application has the ability to register user-provided registration functions in the layer compositor. The registered function will expose the interface in the form of a video processing function so that it can be called at any time. During the image processing process, the video processing function in the image synthesizer can be directly called, and the video memory resources are used for video processing. The video data does not need to be converted multiple times between the memory and the video memory. Compared with the existing technology, this application can support higher frame rate output under the same hardware configuration, ensure better video output effects, and improve performance and user experience.

The above description is only an overview of the technical solutions of the present application. In order to have a clearer understanding of the technical means of the present application, they can be implemented in accordance with the contents of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand. , the following is a detailed description of the preferred embodiments, together with the accompanying drawings. It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present application.

Description of the drawings

In order to explain the embodiments of the present application or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

in:

Figure 1 is a schematic flowchart of a registration function method provided by an embodiment;

Figure 2 is a schematic flowchart of a video frame processing method provided by an embodiment;

Figure 3 is a schematic diagram of the internal functional module connection relationship of a registration function device provided by an embodiment;

Figure 4 is a schematic diagram of the connection relationship between internal functional modules of a video frame processing device according to an embodiment;

FIG. 5 is a schematic structural block diagram of a computer device according to an embodiment.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

In the existing technology, the camera module of the device collects the video data and places it in the memory. The device can only process frames of the video data in the memory, but cannot process the entire picture. And during the processing, data is often converted multiple times between memory and video memory, which is time-consuming and labor-intensive, resulting in a lot of unnecessary resource consumption. To this end, this application proposes a registration function method and a video frame processing method, so that the video processing function provided by the user is registered in the layer compositor of the device, and the interface is exposed for calling at any time. Therefore, the video application can directly call this interface and set up its own picture optimization related scheme to call the registered picture optimization code when rendering the device picture to optimize the picture and avoid the picture being transferred and converted between various modules or between memory and memory. , greatly reducing time and energy consumption. In order to clearly describe a registration function method and a video frame processing method provided by this embodiment, please refer to Figures 1 to 2. The video frame processing method is implemented by steps S210 to S240.

Specifically, the method provided in this application can be applied to a video device, and in a preferred embodiment, the video device can be a mobile terminal. Specific forms of mobile terminals include, but are not limited to, mobile phones, tablet computers, personal digital assistants (English: personal digital assistant, abbreviation: PDA), mobile Internet devices (English: mobile Internet device, abbreviation: MID) and wearable devices (such as smart phones). watches) etc. Applications (such as APP) can be used on mobile terminals to process and display corresponding information and perform corresponding operations, thereby improving usage efficiency.

Among them, the method provided by this application can develop pluggable components in the layer compositor (Surfaceflinger), so that the corresponding video processing function can be run once when the video application is started. There is no need to call it every frame, and there is no need for frames to be stored in the video memory and memory. Conversion between to improve video processing efficiency. To this end, it is necessary to register a function in the layer compositor to realize the insertion of the component. A registration function method is proposed, which is applied to the video device to realize the settings at the hardware system level, so that all video applications in the video device can call the registered Function, specifically, the registration function method includes steps S110 to S130.

Step S110: Perform an initialization operation. The objects of the initialization operation include at least one of an open graphics library, a video memory, and a texture handle.

In one embodiment, before registering the function, the initialization operation is to return the video device to the initialization state, and the specific execution object includes at least one of an open graphics library (OpenGL), a video memory, and a texture handle. Specifically, you can also register and allocate a video memory size for this function, so that it is ready for storing vertex data, texture data, etc. Load a texture image and create a texture handle: You can use a third-party library (such as STB Image, FreeImage) to load a texture image, and use the glTexImage2D() function to create a texture object. You can then create a texture handle using the glGenTextures() function and bind it to the open graphics library context using the glBindTexture() function to complete the initialization operation. And it is worth noting that the specific steps and implementation methods of the initialization operation may vary slightly depending on the environment and requirements. For example, in actual applications, operations such as error handling and resource cleaning can also be performed to ensure the correctness and stability of the program. The actual operation is set according to the specific application environment of the video device, and this application will not elaborate further.

Step S120: Obtain the registration function provided by the user, parse the registration function to obtain the first shader, and perform a registration operation on the first shader. The registration operation includes compiling the shader code and binding it in the layer compositor.

Step S130: Encapsulate the first shader that has completed the registration operation as a video processing function, and expose the interface to the outside, so that the video application can directly call the video processing function through the interface.

In one embodiment, after performing the initialization operation, the method further includes: obtaining a second shader, which is owned by the layer compositor itself; and performing a registration operation on the second shader to register it within the layer compositor. Register the second shader; encapsulate the first shader that has completed the registration operation as a video processing function, and expose the interface to the outside, including: encapsulate the first shader and the second shader that have completed the registration operation as video processing functions, and Expose the interface to the outside world.

In one embodiment, registering a user-provided registration function in the layer compositor necessarily requires mutation binding of the corresponding shader. So when the user writes shader-related code and provides it to the video device in the form of a registered function. The video device can parse the registration function to obtain the first shader. The first shader is the shader in the registration function and is a video processing solution set by the user, such as a beauty solution and a filter solution. And control the layer compositor to compile the first shader related code and register the binding implementation function.

Furthermore, the layer compositor not only processes the content specified by the user, but also needs to process other images. Therefore, while compiling and binding the first shader, you also need to control the layer compositor to obtain all of its own second shaders and also perform registration operations, that is, compile and bind the relevant code. After the layer compositor completes the registration operation of the first shader and the second shader, the registration function provided by the user is embedded in the layer compositor, and the layer compositor can also set according to the user registration function. The input video data is processed.

In one embodiment, encapsulating the first shader that has completed the registration operation into a video processing function includes: serializing the first shader that has completed the registration operation into a processing pipeline of the open graphics library; and obtaining the first shader that has completed the registration operation. The external texture handle provided by the processor; the texture handle and processing pipeline are encapsulated into video processing functions.

In one implementation, the layer compositor that completes the registration operation also needs to encapsulate the function and expose the interface, so that subsequent video applications can directly call the control layer compositor through the interface to process video data. Specifically, the encapsulation process may be to serialize the first shader that has completed the registration operation into a processing pipeline of the open graphics library. The same processing pipeline includes not only the first shader, but also the second shader, making the processing pipeline of the layer compositor more complete. Afterwards, the external texture handle provided by the first shader and the second shader is extracted and packaged into a video processing function together with the processing pipeline. Through the above steps, it is also necessary to expose the video memory handle, so that users can register or read video frames, and register their own beauty scheme as a callback. This setting makes the video processing algorithm provided by the customer callback when the layer compositor renders. Implement video processing operations on the current frame.

Therefore, when starting the video application, the video device can execute the above steps once to implement the registration function method. By running it once, there is no need to call each frame during subsequent video processing, and there is no frame conversion between video memory and memory, which improves the efficiency and effect of video processing.

Furthermore, after executing the method described above, it is necessary to execute the video frame processing provided by this application and applied to video applications, so as to reasonably use the layer synthesizer and achieve efficient video processing.

Step S210: Obtain the first video data from the first layer, which is the layer of the camera module.

Step S220: Send the first video data to the layer compositor, and the layer compositor completes the registration through the method described above.

In one embodiment, after obtaining the first video data from the first layer, the method further includes: obtaining the second video data from the second layer, where the second layer is a layer of the application module; converting the first video perform picture fusion processing on the data and the second video data to obtain the third video data; send the third video data to the layer synthesizer, and call the video processing function to control the layer synthesizer to process the third video data according to the video processing function; Read and output the video frames processed by the video processing function.

In one embodiment, video applications are applications related to processing video data in video devices, especially those that require real-time processing, including but not limited to camera software, video recording software, beauty software, live broadcast software, etc. The video application no longer needs to call the video data from the memory like the existing technology, but controls the first video data collected by the first layer (Surface) in the camera module and sends it directly to the layer synthesizer for processing. The layer compositor directly utilizes video memory resources for processing, eliminating the need for memory conversion processing.

In addition, as mentioned earlier, the layer compositor not only needs to process the video data collected by the camera module, but also needs to process and render other data, such as desktop images. The video data such as the desktop screen is also the second video data in the second layer, and the second layer is the layer of the application module. The first video data and the second video data are subjected to screen fusion processing to obtain third video data.

Step S230: Call the video processing function in the layer compositor to control the layer compositor to process the first video data according to the video processing function. The video processing function is the video processing function described above.

Step S240: Read and output the video frames processed by the video processing function.

In one implementation, reading and outputting the video frames processed by the video processing function includes: determining whether there is a demand for streaming; if there is no demand for streaming, delivering the video frames to an open graphics library or screen hardware for display; If there is a demand for streaming, the video frames are delivered to the open graphics library or screen hardware for display. At the same time, the video frames are read and transmitted to the associated streaming media server, so that the streaming media server can push the video.

In one implementation, the third video data obtained by the layer compositor will be directly processed using video memory resources. Since the layer compositor has completed the previous registration function method steps, the image compositor already contains the encapsulated video processing function. When a video application processes video data, it needs to call the layer compositor for processing. In this case, the video processing function in the layer compositor can be called. This allows the layer compositor to actually process video data according to the requirements of the video processing function, thereby achieving the video effects set by the user, including but not limited to beautification, filters, noise reduction and other effects. The processed video data will be output in the form of video frames.

For the output process, the layer compositor can deliver the processed video frames to the open graphics library or screen hardware for display. During the output process, you can also determine whether the video application receives a push request. If there is a demand for streaming, while delivering the video frames to the open graphics library or screen hardware for display, it also controls sending the video frames to the associated streaming media server for streaming. If not, the video frames are simply delivered to the Open Graphics Library or screen hardware for display. Through output, users can see the processed video frames in real time on the video device, such as videos that have been beautified or filtered. After registering the function of the layer compositor, and using the layer compositor with the registered function to process the video data, the video frame can be converted between memory and video memory multiple times, and a higher frame rate can be supported under the same hardware configuration. , improve performance and user experience.

Therefore, this application is able to register user-provided registration functions in the layer compositor. The registered function will expose the interface in the form of a video processing function so that it can be called at any time. During the image processing process, the video processing function in the image synthesizer can be directly called, and the video memory resources are used for video processing. The video data does not need to be converted multiple times between the memory and the video memory. Compared with the existing technology, this application can support higher frame rate output under the same hardware configuration, ensure better video output effects, and improve performance and user experience.

Figure 3 shows a schematic diagram of the internal functional module connection relationship of a registration function device provided by an embodiment. The registration function device 30 includes: an initialization module 31 , a registration module 32 and an encapsulation module 33 . The initialization module 31 is configured to perform an initialization operation. The objects of the initialization operation include at least one of an open graphics library, a video memory, and a texture handle. The registration module 32 is used to obtain the registration function provided by the user, parse the registration function to obtain the first shader, and perform a registration operation on the first shader. The registration operation includes compiling the shader code and binding it in the layer compositor. The encapsulation module 33 is used to encapsulate the first shader that has completed the registration operation into a video processing function, and expose the interface to the outside, so that the video application can directly call the video processing function through the interface.

Figure 4 shows a schematic diagram of the connection relationship between internal functional modules of a video frame processing device provided by an embodiment. The video frame processing device 40 includes: an acquisition module 41, a sending module 42, a processing module 43 and an output module 44. The acquisition module 41 is used to acquire the first video data from the first layer, where the first layer is the layer of the camera module. The sending module 42 is used to send the first video data to the layer synthesizer, and the layer synthesizer completes the registration through the method described above. The processing module 43 is used to call the video processing function in the layer synthesizer and use the video memory resources to process the first video data; the video processing function is the video processing function described above. The output module 44 is used to read and output video frames processed by the video processing function.

Figure 5 shows an internal structure diagram of a computer device in one embodiment. Specifically, the computer device may be a terminal or a server. As shown in Figure 5, the computer device includes a processor, a memory and a network interface connected through a system bus. Among them, memory includes non-volatile storage media and internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, it can cause the processor to implement the method described above. A computer program may also be stored in the internal memory. When the computer program is executed by the processor, it may cause the processor to perform the method described above. Those skilled in the art can understand that the structure shown in Figure 5 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. The specific computer equipment can May include more or fewer parts than shown, or combine certain parts, or have a different arrangement of parts.

In one embodiment, this application also proposes a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, it causes the processor to perform the steps of the aforementioned method,

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through computer programs. The programs can be stored in a non-volatile computer-readable storage medium. , when the program is executed, it may include the processes of the above-mentioned method embodiments. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims (10)

1. A registration function method, characterized in that, applied to video equipment, including the following steps: Perform an initialization operation, where the objects of the initialization operation include at least one of an open graphics library, a video memory, and a texture handle; Obtain the registration function provided by the user, parse the registration function to obtain the first shader; perform a registration operation on the first shader, the registration operation includes compiling the shader code and binding it in the layer compositor; The first shader that completes the registration operation is encapsulated as a video processing function, and an interface is exposed to the outside, so that the video application directly calls the video processing function through the interface. 2. The registration function method according to claim 1, characterized in that, after the initialization operation is performed, the method further includes: Obtain a second shader, which is owned by the layer compositor itself; Perform a registration operation on the second shader to register the second shader within the layer compositor; The first shader that completes the registration operation is encapsulated as a video processing function, and an interface is exposed to the outside, including: The first shader and the second shader that have completed the registration operation are encapsulated as the video processing function, and interfaces are exposed to the outside. 3. The registration function method according to claim 1, wherein the first shader that completes the registration operation is encapsulated as a video processing function, including: Serialize the first shader that has completed the registration operation into the processing pipeline of the open graphics library; Obtain the external texture handle provided by the first shader that completes the registration operation; The texture handle and the processing pipeline are encapsulated into the video processing function. 4. A video frame processing method, characterized in that it is applied to video applications and includes the following steps: Obtain the first video data from the first layer, where the first layer is the layer of the camera module; Send the first video data to a layer compositor, and the layer compositor completes registration through the method described in any one of claims 1 to 3; Call the video processing function in the layer compositor to control the layer compositor to process the first video data according to the video processing function, the video processing function being the method described in any one of claims 1 to 3 Video processing function; Read and output the video frames processed by the video processing function. 5. The video frame processing method according to claim 4, wherein after obtaining the first video data from the first layer, the method further includes: Obtain the second video data from the second layer, where the second layer is the layer of the application module; Perform screen fusion processing on the first video data and the second video data to obtain third video data; Send the third video data to the layer compositor and call the video processing function to control the layer compositor to process the third video data according to the video processing function; Read and output the video frames processed by the video processing function. 6. The video frame processing method according to claim 4 or 5, wherein the reading and outputting of the video frames processed by the video processing function includes: Determine whether there is a demand for push streaming; If there is no demand for streaming, the video frames are delivered to the open graphics library or screen hardware for display; If there is a demand for streaming, the video frames are delivered to the open graphics library or screen hardware for display, and the video frames are read and transmitted to the associated streaming media server so that the streaming media server can push the video. flow. 7. A registration function device, characterized in that it includes: An initialization module, configured to perform an initialization operation. The objects of the initialization operation include at least one of an open graphics library, a video memory, and a texture handle; A registration module, used to obtain the registration function provided by the user, parse the registration function to obtain the first shader; perform a registration operation on the first shader, the registration operation includes compiling the shader code and binding it to the In layer compositor; An encapsulation module, configured to encapsulate the first shader that has completed the registration operation into a video processing function, and expose an interface to the outside, so that the video application can directly call the video processing function through the interface. 8. A video frame processing device, characterized in that it includes: An acquisition module, configured to acquire the first video data from the first layer, where the first layer is the layer of the camera module; A sending module, configured to send the first video data to a layer synthesizer, which has completed registration through the method described in any one of claims 1 to 3; A processing module, configured to call the video processing function in the layer synthesizer and use video memory resources to process the first video data; the video processing function is the video processing function described in any one of claims 1 to 3; An output module is used to read and output video frames processed by the video processing function. 9. A computer device, characterized by comprising a processor and a memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 6. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented or The method of any one of claims 4 to 6.