CN114697392A - Image processing method, image processing device, computer equipment and storage medium - Google Patents

Abstract

The embodiment of the invention discloses an image processing method, an image processing device, computer equipment and a storage medium; the embodiment of the invention displays the interactive interface of the application program; uploading the interaction information acquired by the interaction interface to a server; receiving image coding data returned by the server according to the interactive information; extracting a reference frame corresponding to the number from a local memory according to the number; and decoding according to the reference frame and the associated frame data to obtain a current display image and displaying the current display image in the interactive interface. Therefore, the reference frames used for encoding and decoding the display image are stored in the server and the terminal respectively, the server is prevented from transmitting the key frame image corresponding to the reference frames to the terminal, network load is more balanced during data transmission, the transmitted data volume is reduced, and operation efficiency of cloud application on the user terminal is improved.

Description

Image processing method, device, computer equipment and storage medium

technical field

The present invention relates to the field of Internet technologies, and in particular, to an image processing method, device, computer equipment and storage medium.

Background technique

In recent years, with the development of Internet technology and the improvement of network communication speed, cloud applications have developed rapidly. Cloud application refers to the application that the terminal interacts with the service cloud, the terminal operation is synchronized with the cloud, and the data occupying the local space is also retained through the cloud backup. The cloud application collects the interactive data input by the user through the terminal, transmits the interactive data collected by the terminal to the cloud for processing, and the cloud returns the processed result to the terminal for display. In this way, the user terminal only needs to have basic data collection and display functions to run applications that require relatively powerful graphics computing and data processing capabilities, releasing the user's dependence on high-configuration terminals.

However, when a cloud application is currently used, the cloud returns the running result to the terminal for display. Due to the large amount of data transmitted, the running efficiency of the cloud application on the user terminal is relatively slow.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide an image processing method, apparatus, computer equipment and storage medium, which improve the efficiency of image processing and reduce the amount of data transmission.

A first aspect of the present application provides an image processing method, including:

Displaying an interactive interface of the application, the interactive interface is used to receive interactive information;

uploading the interactive information collected by the interactive interface to the server;

Receive the image encoding data returned by the server according to the interaction information, where the image encoding data includes the reference frame number and associated frame data determined when the server encodes the displayed image, and the associated frame data is used to calculate the displaying difference data between the image and the reference frame;

Extract the reference frame corresponding to the number from the local memory according to the number;

The currently displayed image is obtained by decoding the reference frame and the associated frame data, and the currently displayed image is displayed in the interactive interface.

A second aspect of the present application provides an image processing method, including:

Receive interactive information uploaded by the terminal;

generating a display image corresponding to the interaction information according to the interaction information;

Determine the target reference frame corresponding to the display image from the reference frames stored in the memory and determine the number corresponding to the target reference frame;

determining the associated frame data of the display image relative to the target reference frame, where the associated frame data is calculated difference data between the display image and the target reference frame;

sending the number and the associated frame data to the terminal, so that the terminal retrieves the target reference frame corresponding to the number from the local storage of the terminal according to the number, and combines the associated frame The data generates a display image and displays the display image.

A third aspect of the present application provides an image processing device, including:

a display unit, configured to display an interactive interface of the application, the interactive interface is used to receive interactive information;

an uploading unit, configured to upload the interactive information collected by the interactive interface to the server;

a receiving unit, configured to receive the image encoding data returned by the server according to the interaction information, the image encoding data including the reference frame number determined when the server encodes the display image and the relative value of the display image to the reference frame The associated frame data of the frame, the associated frame data is the difference data between the calculated display image and the reference frame;

an extraction unit, configured to extract a reference frame corresponding to the number from the local memory according to the number;

A decoding unit, configured to decode the reference frame and the associated frame data to obtain a currently displayed image, and display the currently displayed image in the interactive interface.

A fourth aspect of the present application provides an image processing apparatus, including:

a receiving unit, configured to receive the interaction information uploaded by the terminal;

a generating unit, configured to generate a display image corresponding to the interaction information according to the interaction information;

a first determining unit, configured to determine a target reference frame corresponding to the display image from the reference frames stored in the memory and determine a number corresponding to the target reference frame;

a second determining unit, configured to determine associated frame data of the display image relative to the target reference frame, where the associated frame data is calculated difference data between the display image and the target reference frame;

a sending unit, configured to send the serial number and the associated frame data to the terminal, so that the terminal retrieves the target reference frame corresponding to the serial number from the local storage of the terminal according to the serial number, and A display image is generated and displayed in combination with the associated frame data.

A fifth aspect of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the first computer program when the processor executes the computer program The aspect or the steps in the image processing method provided in the second aspect.

A sixth aspect of the present application provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute the steps in the image processing method provided in the first aspect or the second aspect.

A seventh aspect of the present application provides a computer program product or computer program, wherein the computer program product or computer program includes computer instructions, and the computer instructions are stored in a storage medium. The processor of the computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the image processing method provided in the first aspect or the second aspect.

In the embodiment of the present application, the interactive interface of the application program is displayed, and the interactive interface is used for receiving interactive information; uploading the interactive information collected by the interactive interface to the server; The number of the reference frame and the associated frame data determined when the image is encoded. The associated frame data is the difference data between the displayed image and the reference frame; the reference frame corresponding to the number is extracted from the local memory according to the number; The associated frame data is decoded to obtain the currently displayed image, and the currently displayed image is displayed in the interactive interface. In this way, by storing the reference frame locally in the terminal, when the server sends image data to the terminal, it only needs to send the reference frame number and associated frame data, avoiding the sending of the key frame corresponding to the reference frame. Thus, the amount of data transmitted from the server to the terminal is reduced, the efficiency of the terminal receiving image data is improved, and the operation efficiency of the cloud application in the user terminal is further improved.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1a is a schematic diagram of a data transmission scenario in a cloud application scenario provided by the present invention.

FIG. 1b is a schematic diagram of another scenario of data transmission in a cloud application scenario provided by the present invention.

FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present invention.

FIG. 3 is another schematic flowchart of an image processing method provided by an embodiment of the present invention.

Fig. 4a is a schematic diagram of a payment terminal of a cloud application of smart face-scanning payment provided by the present application.

FIG. 4b to FIG. 4e are schematic diagrams of reference frames of the smart face-swiping payment cloud application provided by the present application.

FIG. 4f is a schematic diagram of the process of encoding the displayed image by the cloud server in the smart face-swiping payment cloud application provided by the present application.

FIG. 5 is another schematic flowchart of an image processing method provided by an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 7 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 8 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 9 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 10 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 11 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 12 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 13 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 14 is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention.

FIG. 15 is a schematic structural diagram of a computer device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

Embodiments of the present invention provide an image processing method, apparatus, computer equipment, and storage medium, which are used to solve the technical problem of slow cloud application processing efficiency due to a large amount of data transmitted from a server to a terminal in a cloud application scenario. Among them, cloud applications may include cloud education, cloud conference, cloud social networking, cloud gaming, and cloud medical treatment.

Please refer to FIG. 1a , which is a schematic diagram of a data transmission scenario in a cloud application scenario; it includes a server A, at least one terminal B (only two are shown in the figure), and a memory connected to the server A. The server A and the terminal B may be connected through a communication network, the communication network includes a wireless network and a wired network, wherein the wireless network includes one or more of a wireless wide area network, a wireless local area network, a wireless metropolitan area network, and a wireless personal network The combination. The network includes network entities such as routers and gateways, which are not shown in the figure. The terminal B can exchange information with the server A through the communication network. As shown in the figure, the user uploads the interaction information collected on the interaction interface of the terminal B to the server A through the terminal B. Server A receives the interaction information, generates a display image corresponding to the interaction information according to the interaction information, encodes the generated display image, and sends the encoded encoded data to terminal B. After receiving the encoded data sent by the server, terminal B decodes the encoded data to obtain a display image, and then displays the display image on the interactive interface.

Server A can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, Middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

Wherein, the terminal B may be a smart cash register device, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. There may be one terminal B or multiple terminals, and the number of terminals does not limit the protection scope of the solution of the present application. Various applications required by users can be installed in the terminal, including applications of cloud applications such as intelligent self-service payment, cloud games or cloud education.

As shown in Figure 1b, another schematic diagram of data transmission in the cloud application scenario provided by the present application, in this application scenario, the server is a cloud server C, and the cloud server C includes multiple Android operating systems (only shown in the figure). 3 out), one or more Android applications can be installed in each Android system. The number of terminal devices D connected to the cloud server may be one or more. Specifically, multiple operating systems in the cloud server C may respectively correspond to one terminal device D, or may only correspond to one terminal device D. When application 1 is running in terminal device D, application 1 receives the interaction information input by the user through terminal device D, and sends the interaction information input by the user to the Android system 1 running application 1 in the cloud server, and the Android system in cloud server C. The application 1 running in 1 generates a display video according to the received interaction information, encodes the display video to obtain a video stream, and then returns the video stream to the terminal device D. The central processor in the terminal device D decodes the received video stream to obtain a display video, and displays the display video on the display screen of the application 1 running in the device 1 .

It should be noted that the schematic diagrams of the data transmission scenarios in the cloud application scenarios shown in FIGS. 1a and 1b are only examples, and the data transmission scenarios in the cloud application scenarios described in the embodiments of the present application are for the purpose of more clearly illustrating the technology of the present application. The solution does not constitute a limitation on the technical solutions provided in this application. Those of ordinary skill in the art know that with the evolution of image processing and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

Based on the above implementation scenarios, detailed descriptions are given below.

Example 1

The embodiments of the present application will be described from the perspective of an image processing apparatus, and the image processing apparatus may be integrated in a terminal. The terminal may be a smart cash register device, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like. As shown in FIG. 2, which is a schematic flowchart of an image processing method provided by this application, the processing method includes:

Step 101 , displaying an interactive interface of the application, where the interactive interface is used to receive interactive information.

The applications are cloud application applications loaded in the terminal, including but not limited to cloud applications such as smart face-swiping payment cloud applications, cloud games, and cloud conferences. The interactive interface of the application is an interface used to display interactive information in the terminal. For example, in the smart face payment cloud application, the interactive interface can display prompt information prompting the user to face the camera; in cloud games, the interactive interface can display the game interface and operation area information. The interactive interface of the application may also be an interface in the terminal for receiving interactive information. For example, in the smart face payment cloud application, the interactive interface can receive the touch information input by the user to confirm the payment as the interactive information; in the cloud game, the interactive interface can receive the user's manipulation operation information as the interactive information. Of course, the interactive interface also includes a camera component for collecting image information, a microphone component for collecting sound information, a gyroscope component for collecting terminal motion status, and a sensor component for collecting temperature and humidity information and other information collection components . The information collected by these information collection components is also used as interactive information.

Step 102: Upload the interaction information collected by the interaction interface to the server.

Among them, as mentioned above, the interactive interface includes not only a display screen for displaying interactive information and receiving screen touch information, but also other information collection components. But in currently running cloud applications, not all interactive information that can be collected needs to be collected. For example, in the smart face-swiping payment cloud application, the user's face image information and the touch information input by the user on the display screen of the interactive interface need to be collected as interactive information. In cloud game applications, the touch information input by the user on the display screen of the interactive interface, the motion state information of the terminal, and the voice control information input by the user need to be collected as interaction information. That is, the interaction information to be collected can be determined according to the cloud application actually running on the terminal, and then the collected interaction information can be sent to the server.

Step 103: Receive the image encoding data returned by the server according to the interaction information, the image encoding data includes the reference frame number and associated frame data determined when the server encodes the displayed image, and the associated frame data is to calculate the difference between the displayed image and the reference frame. data.

Wherein, after the server receives the interaction information sent by the terminal, the server generates the display image that should be displayed on the interactive interface of the terminal at this time according to the operation logic of the application program and the received interaction information, and then sends the generated display image to the terminal, so that the terminal displays the display image on the interactive interface. It can be understood that the display image may be a single frame of display image, or may be a continuous multi-frame display image, and the continuous multiple-frame display image constitutes a video stream. Since the image or video stream generated by the server has a large amount of data, directly sending the image or video stream generated by the server to the terminal will occupy network bandwidth. Moreover, when the server is connected to multiple terminals, sending display images or video streams to multiple terminals at the same time will cause network congestion, which in turn will reduce the speed at which terminals receive displayed images or video streams, causing network delays or freezes to end users. Wait for a bad experience.

In the related art, in order to reduce the amount of data transmitted by the server to the terminal and speed up the transmission speed of the displayed image or video stream, before the displayed image or video stream is sent, the server encodes and compresses the displayed image or video stream, thereby reducing the size of the displayed image or video stream. The amount of transmitted data; after the terminal receives the encoded encoded data, it decodes the encoded data to obtain the displayed image and video stream. Specifically, to encode and compress the displayed image or video stream, an IPB encoding method may be used to encode and compress the displayed image or video stream. Among them, I frame (intra-frame coding frame) is a full-frame compression coding frame, which compresses the displayed image information of the whole frame. It can be compressed without referring to other pictures. When decoding, only the data of the I frame can be reconstructed. full image. Since the I frame retains the complete data of one frame of picture, the compression rate of the I frame encoding is low, and the common I frame compression rate is 7. P frame (forward predictive coding frame) is the difference between the current frame image and the previous I frame or P frame, that is, the P frame is a difference frame, and the P frame does not have complete picture data, only the picture difference from the previous I frame or P frame. The data. During encoding, the P frame takes the I frame or the P frame as the reference frame, and calculates the difference frame data of the current frame image relative to the reference frame. The difference frame data contains the predicted value and motion vector for each point in the reference frame. When decoding, the terminal obtains the sample value of each point in the P frame according to the received prediction value of each point and the motion vector combined with the information of the point in the reference frame, thereby obtaining the complete current frame image. Compared with the I frame, the compression rate of the P frame is higher, and the common compression rate of the P frame is 20. The B frame (bidirectional predictive interpolation coding frame) is a bidirectional difference frame, that is, the B frame records the difference between the image of the current frame and the previous and previous frames. The compression rate of the B frame is higher, and the common B frame compression rate is 50.

In the related art, the first frame image of a scene is generally encoded as an I frame, and then the encoded I frame is used as a reference frame to encode P and B frames of subsequent frames. Among them, if the P frame is used as the reference frame for many times during decoding, decoding errors will spread. Therefore, in order to ensure the accuracy of data transmission, not only the first frame image of each scene is encoded according to the I frame, but if a scene is maintained for a long time, it is often set to perform every preset time period in the scene. Encode the I frame once and transmit the encoded I frame to the terminal for decoding and display. Due to the low compression rate of the I-frame encoding, the I-frame data still has a large amount of data during transmission.

In order to further reduce the amount of data transmitted by the server to the terminal, the image processing method provided by the embodiments of the present application all use the I frame as a reference frame to perform P frame encoding when the server encodes the displayed image. The I frame here is not a fixed I frame, but will change with the usage scene of the application. Here the I frame is the reference frame for editing the P frame. An application has multiple usage scenarios, that is, there are multiple reference frames, and each reference frame has a unique number corresponding to it. In this way, the display image of the application is encoded, that is, the P-frame encoding is performed with the reference frame corresponding to the currently displayed image, that is, the difference data between the currently displayed image and its corresponding reference frame is calculated, and the difference data is compressed to be associated frame data. In this way, the result obtained by encoding the currently displayed image is the associated frame data corresponding to the currently displayed image. The terminal receives the associated frame data returned by the server and the number of the reference frame used for encoding, and can extract the reference frame corresponding to the number from the reference frame stored locally by the terminal according to the number, and then decode the associated frame data to obtain a display image. In this way, the data transmission from the server to the terminal only needs to transmit the number of the reference frame and the associated frame data. Moreover, the associated frame data is encoded by the P frame, the compression rate is high, and the amount of data is small.

Step 104: Extract the reference frame corresponding to the number from the local memory according to the number.

Wherein, after receiving the reference frame number and associated frame data returned by the server, the terminal searches for the reference frame corresponding to the reference frame number from the local memory of the terminal. In the image processing method provided by the present application, a set of reference frame data corresponding to the application program is stored in the local memory of the terminal and the memory of the server respectively, and the reference frame data includes a plurality of reference frames and a serial number corresponding to each reference frame . The reference frame data corresponding to the application program stored in the local memory of the terminal is the same as the reference frame data corresponding to the application program stored in the server. That is, the reference frame extracted by the terminal according to the received serial number is the same as the reference frame used when the server performs P-frame encoding on the displayed image of the current frame.

Step 105: Decode the currently displayed image according to the reference frame and the associated frame data, and display the currently displayed image in the interactive interface.

Wherein, after the terminal extracts the corresponding reference frame according to the serial number sent by the server, and decodes by using the reference frame and the received associated frame data, the currently displayed image can be obtained. The current display image is the same as the display image generated by the server according to the interaction information. After decoding to obtain the currently displayed image, the terminal displays the currently displayed image in the interactive interface.

In some embodiments, the image processing method provided by the present application may further include the following steps:

1. Determine a preset number of reference frames from the interface image displayed on the interactive interface;

2. Number each reference frame according to the preset rules;

3. Associate each reference frame with the corresponding serial number and store it in the local memory of the terminal;

4. Upload each reference frame and its corresponding serial number to the server, so that the server associates each reference frame with the corresponding serial number and stores it in the storage space of the server.

Wherein, the above steps may be performed in any step before step 103 or before any previous step. The interface image displayed on the interactive interface includes the interface image displayed in each frame of the interactive interface in the entire running process of the application. The interface image displayed in each frame of the interactive interface can be obtained according to the historical running data of the application. A preset number of reference frames can be determined from the interface images displayed on the interactive interface. One frame of the interface image can be extracted from the interface image every preset frame and subjected to I-frame encoding to obtain the reference frame, or the reference frame can be coded according to other set rules. Sure.

In some embodiments, a preset number of reference frames are determined from interface images displayed on the interactive interface, including:

a. Obtain each frame of the interface image displayed corresponding to the interactive interface of the application running;

b. Classify the interface images according to the image similarity to obtain a plurality of interface image sets;

c. Extract any frame of interface image from each interface image set as a reference frame.

Wherein, after each frame of interface images displayed corresponding to the interactive interface of the application program is obtained according to the historical data of the operation of the application program, these interface images are classified. Specifically, these interface images can be classified according to the application scenarios corresponding to the interface images or the image similarity of the interface images, and after the classification, multiple interface image sets can be obtained. Each interface image set includes at least one frame of interface image, and then one frame of interface image is extracted from each interface image set, and I-frame encoding is performed on these interface images to obtain multiple reference frames.

In some embodiments, the interface images are classified according to the image similarity to obtain a plurality of interface image sets, including:

Image recognition algorithm is used to perform image recognition on each interface image, and the corresponding recognition feature of each interface image is obtained;

The interface images are classified according to the similarity of the recognition features, and multiple interface image sets are obtained.

Wherein, an image recognition algorithm is used to perform image recognition on each interface image, and specifically, a convolutional neural network can be used to perform image recognition on the interface image to obtain the recognition feature of each interface image. Then the similarity calculation is performed on the identification features, and the interface images are classified into multiple interface image sets according to the similarity of the identification features.

In some embodiments, the image processing method provided by the embodiments of the present application further includes:

When receiving the application program update instruction sent by the server, update the application program;

Extract a preset number of reference frames from the interface image displayed on the interactive interface of the updated application and number each reference frame according to the preset rules;

Associate each reference frame with the corresponding number and store it in the local memory of the terminal;

Upload each reference frame and its corresponding serial number to the server, so that the server stores each reference frame and the corresponding serial number in the storage space of the server in association with each other.

Wherein, after the version of the application program is updated, the server sends an application program update instruction to the terminal, and the terminal automatically updates according to the update instruction sent by the application program. Since the interface image displayed by the updated version of the application may change during runtime, the terminal will automatically extract the reference frame according to the interface image displayed by the updated application's interactive interface, and re-extract the reference frame. numbered and stored in the terminal's local memory. While numbering and storing the re-extracted reference frame, delete the reference frame and numbering data corresponding to the original application program saved in the terminal. After that, the re-extracted reference frame and its serial number are sent to the server, so that the server can update the previously stored reference frame data according to the received re-extracted reference frame and its serial number.

As can be seen from the above description, the image processing method provided by the embodiments of the present application displays the interactive interface of the application program, and the interactive interface is used to receive interactive information; upload the interactive information collected by the interactive interface to the server; receive the information returned by the server according to the interactive information Image encoding data, the image encoding data includes the number of the reference frame determined when the server encodes the displayed image and the associated frame data. The associated frame data is the difference data between the displayed image and the reference frame; extracted from the local storage according to the number. The reference frame corresponding to the serial number; the currently displayed image is obtained by decoding the reference frame and the associated frame data, and the currently displayed image is displayed in the interactive interface. In this way, by saving the reference frame locally in the terminal, when the server sends image data to the terminal, it only needs to send the reference frame number and associated frame data, thereby avoiding the sending of the reference frame. Thus, the amount of data transmitted from the server to the terminal is reduced, the efficiency of the terminal receiving image data is improved, and the operation efficiency of the cloud application in the user terminal is further improved.

Embodiment 2

According to the method described in the first embodiment, further detailed description will be given below by taking an example. In this embodiment, the image processing method is described with the terminal as the execution subject.

Please refer to FIG. 3 , which is another schematic flowchart of an image processing method provided by an embodiment of the present application. The method includes the following steps:

Step 201, the terminal determines a preset number of reference frames from the interface image displayed on the application interactive interface, and numbers the reference frames.

Wherein, as mentioned above, the application may be a cloud application. Specifically, it can be cloud applications such as cloud games, cloud conferences, or smart face payment cloud applications. Then the terminal can be a user terminal for cloud games, cloud conferences, or a payment terminal for smart face payment cloud applications. In the embodiments of the present application, the following application programs may be described by taking the smart face-scanning payment cloud application as an example. Then the terminal here can be a payment terminal for smart face-scanning payment. In the image processing method provided by the present application, the payment terminal first acquires the interface image displayed on the interactive interface of the payment terminal by the smart face-swiping payment cloud application. Specifically, a video stream of a historical running process of the smart face-swiping payment cloud application can be obtained, and then the video stream can be divided into several frames of interface images. After obtaining the interface images, the payment terminal extracts a preset number of frames of interface images from these several frames of interface images, and then performs I-frame encoding on these interface images to obtain a preset number of reference frames.

In some embodiments, the terminal extracts a preset number of reference frames from the interface image displayed on the application interactive interface, including:

1. The terminal sorts each frame of interface images displayed on the application interactive interface;

2. Determine the interface image of the previous frame of each frame of interface image according to the sorting order;

3. When the interface image does not have the interface image of the previous frame, determine the interface image of the current frame as the reference frame; when the interface image has the interface image of the previous frame, obtain the similarity between the interface image and the interface image of the previous frame;

4. When the similarity is less than the preset threshold, determine the interface image of the current frame as the reference frame.

Wherein, after the payment terminal acquires several frames of interface images according to the video stream of the historical running of the application, it sorts the acquired interface images according to the sequence of the running process of the application. Perform I-frame encoding on the first frame of the interface image in the sorted image sequence to obtain a reference frame, and then compare the similarity between each other frame of interface image and the previous frame of interface image. When the similarity of the images is less than a preset threshold, it can be determined that the application scene corresponding to the interface image of the current frame has changed with the application scene corresponding to the interface image of the previous frame, and then the interface image of the current frame is subjected to I-frame encoding to obtain a reference. frame. In this way, each frame of the interface image in the image sequence is determined, so as to determine all the reference frames of the smart face-swiping payment cloud application. When all the reference frames are determined, these reference frames are extracted, and each reference frame is numbered according to the sorting order in the image sequence.

Specifically, as shown in FIG. 4a , which is a schematic diagram of a payment terminal of the smart face-swiping payment cloud application provided by the present application, when the smart face-swiping payment cloud application is running, the display interface of the payment terminal displays the smart face-swiping payment cloud application. The interface image displayed in the interactive interface changes in real time with the process of face recognition and payment operations. The payment terminal acquires the video stream displayed on the display interface of the payment terminal during the face-swiping payment process, and splits the video stream into several frames of interface images. The payment terminal then sorts several frames of interface images obtained by splitting in the order of face-swiping payment to obtain a sequence of interface images. A reference frame is obtained by performing I-frame encoding on the first frame of the interface image of the interface image sequence, as shown in FIG. 4b, and it is determined that FIG. 4b is a reference frame of the smart payment cloud application. For other frame interface images in the image sequence, the image similarity between the current frame image and the previous frame image is obtained respectively. If the image similarity between the current frame image and the previous frame image is less than the preset threshold, the current frame image is encoded as a reference frame of the smart face brush payment cloud application. Then, all reference frames of the smart payment cloud application are determined one by one through this method. As shown in Fig. 4c, Fig. 4d and Fig. 4e, all are the smart payment cloud application reference frames determined according to the foregoing method. After the reference frame of the smart payment cloud application is determined, each reference frame is numbered according to the face-swiping payment operation process. For example, it is determined that the reference frame number corresponding to FIG. 4b is 01, the reference frame number corresponding to FIG. 4c is 02, the reference frame number corresponding to FIG. 4d is 03, and the reference frame number corresponding to FIG. 4e is 04.

Step 202, the terminal associates each reference frame with the corresponding serial number and stores it in the local memory of the terminal.

The payment terminal associates and stores the extracted reference frame with the corresponding serial number, and stores it in the local memory of the payment terminal, so that subsequent reference frames corresponding to the serial number are extracted from the memory according to the serial number of the reference frame.

Step 203, the terminal uploads each reference frame and the corresponding serial number to the server, so that the server associates each reference frame and the corresponding serial number and stores it in the storage space of the server.

Wherein, the payment terminal not only stores the extracted reference frame and the corresponding serial number in the local memory of the terminal, but also uploads each reference frame and the corresponding serial number to the server for storage. Among them, the server running the smart face-scanning payment cloud application may be a cloud server.

Step 204, the terminal displays the interactive interface of the application.

Wherein, when the user touches the display icon of the smart face-swiping payment cloud application on the display interface of the payment terminal, the terminal sends an operation instruction to the cloud server to open the smart face-swiping payment cloud application, and the cloud server starts the loaded smart payment according to the operation instruction. Scan the face payment cloud application, and return the display screen of the application to the terminal for display, and the display screen is the interactive interface of the smart face payment cloud application. The user can input interactive information on the interactive interface, and the interactive interface can also display the interactive content processed and output by the server according to the interactive interface.

Step 205, the terminal uploads the collected interaction information to the server, so that the server generates a display image according to the interaction information.

Among them, the payment terminal can collect interactive information through the interactive interface, and can also collect images, video, voice, motion status, temperature and humidity and other interactive information input by the user according to other components such as cameras, microphones, gyroscopes, and sensors installed on the payment terminal. . After the payment terminal collects the interaction information, it uploads the collected interaction information data to the cloud server, and the cloud server generates the corresponding display image according to the interaction information uploaded by the payment terminal and combined with the current running scene of the smart face payment cloud application.

Step 206, the terminal receives the image encoding data returned by the server.

Wherein, after the cloud server generates a corresponding display image according to the interaction data uploaded by the payment terminal and the operation scenario of the smart face-swiping payment cloud application, it uses the reference frame uploaded by the payment terminal to the cloud server in step 203 to perform P-frame encoding, that is, the display is determined to be displayed. The difference data of the image relative to the corresponding reference frame is compressed to obtain the associated frame data. The cloud server needs to determine the reference frame corresponding to the displayed image before encoding the P-frame of the displayed image. Specifically, all reference frames stored in the memory of the cloud server may be compared with the displayed image for similarity, and the reference frame with the highest similarity with the displayed image is determined as the reference frame corresponding to the displayed image. After the reference frame corresponding to the displayed image is determined, the number corresponding to the reference frame is determined according to the stored correspondence between the reference frame and the number. As shown in FIG. 4f , it is a schematic diagram of the process of encoding the displayed image by the cloud server in the smart face-scanning payment operation application provided by the embodiment of the present application. As shown in the figure, the server determines the reference frame corresponding to each frame of the generated display image, and then uses the reference frame corresponding to each frame of the display image to perform P-frame encoding on each frame of the display image to obtain the P frame corresponding to each frame of the display image. (associated frame data) video stream. In this way, after the cloud server performs P-frame encoding on the displayed image to obtain the associated frame data, the associated frame data and the reference frame number are sent to the payment terminal as image encoding data, and the payment terminal receives the image encoding data returned by the cloud server, and then processes the image encoding data returned by the cloud server. The image encoded data is decoded.

Step 207, the terminal extracts the reference frame corresponding to the number from the local memory according to the number.

Wherein, after receiving the image encoding data returned by the cloud server, the payment terminal determines the number of the reference frame used when displaying the image encoding from the image encoding data returned by the server, and then extracts the reference frame from the reference frame stored locally by the payment terminal according to the number. The reference frame corresponding to the number.

Step 208: The terminal decodes the reference frame and the associated frame data to obtain the currently displayed image, and displays the currently displayed image in the interactive interface.

Wherein, after extracting the reference frame corresponding to the serial number from the local memory, the payment terminal decodes the reference frame corresponding to the displayed image and the associated frame data to obtain the displayed image, where the displayed image is the current displayed image. The payment terminal then displays the currently displayed image obtained by decoding in the interactive interface. It is understandable that in the face-swiping stage of the smart face-swiping payment cloud application, the terminal continuously collects interactive information and sends it to the cloud server, and the cloud server also generates continuous frames of display images according to the interactive data uploaded by the terminal in real time. That is, the cloud server generates a video stream according to the interaction information, and then the cloud server performs P-frame encoding on each frame of the displayed image in the video stream, and real-time encodes the reference frame number used for encoding each frame of the displayed image and the correlation obtained by the P-frame encoding. The frame data is returned to the payment terminal for decoding and display. Therefore, the smart face-swiping payment cloud application will also display the corresponding video stream in the interactive interface during the face-swiping stage.

As can be seen from the above description, the image processing method provided by the embodiments of the present application displays the interactive interface of the application program, and the interactive interface is used to receive interactive information; upload the interactive information collected by the interactive interface to the server; receive the information returned by the server according to the interactive information Image encoding data, the image encoding data includes the number of the reference frame determined when the server encodes the displayed image and the associated frame data. The associated frame data is the difference data between the displayed image and the reference frame; extracted from the local storage according to the number. The reference frame corresponding to the serial number; the currently displayed image is obtained by decoding the reference frame and the associated frame data, and the currently displayed image is displayed in the interactive interface. In this way, by saving the reference frame locally in the terminal, when the server sends image data to the terminal, it only needs to send the reference frame number and associated frame data, thereby avoiding the sending of the reference frame. Thus, the amount of data transmitted from the server to the terminal is reduced, the efficiency of the terminal receiving image data is improved, and the operation efficiency of the cloud application in the user terminal is further improved.

Embodiment 3

The embodiments of the present application will be described from the perspective of an image processing apparatus, and the image processing apparatus may be integrated in a server. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, and cloud communications. , middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. As shown in FIG. 5, it is a schematic flowchart of an image processing method provided by this application, and the processing method includes:

Step 301: Receive interaction information uploaded by a terminal.

The terminal is a terminal that is connected to the server and performs data interaction. For example, in a cloud game scenario, the server is a cloud server, and the terminal is a game user terminal of the cloud game. In the smart face payment cloud application scenario, the terminal is the payment terminal. When the application program runs on the server, the terminal continuously collects the interaction information input by the user, and the server generates processing results according to the interaction information and returns the processing results to the terminal for display. Multiple application programs can be run in the server at the same time, so there can be one terminal or multiple terminals that are connected to the server and perform data interaction. In this embodiment, an example of running an application is used for description. It can be understood that, when multiple application programs are run in the server at the same time, the image processing method of each application program is the same as that in this embodiment.

Step 302: Generate a display image corresponding to the interaction information according to the interaction information.

The application program running in the server performs corresponding processing according to the interaction information uploaded by the terminal. For example, when the application is a cloud game, the server receives the operation instructions uploaded by the terminal to control the tasks in the game, and integrates these operation instructions into the game program, so that the characters in the game move according to the operation instructions received by the user terminal. Or attack, and generate corresponding animation and video stream data. It can be understood that animation and video stream data are composed of multiple frames of continuous display images, so the essence is that the server generates corresponding interactive images according to the interactive information uploaded by the terminal. The information continuously generates multiple frames of display images.

Step 303: Determine the target reference frame corresponding to the display image from the reference frames stored in the memory and determine the number corresponding to the target reference frame.

After generating the corresponding display image according to the interaction information uploaded by the terminal, the server encodes the generated display image to reduce the amount of data transmitted. Specifically, the server first determines the corresponding display image from the reference frames stored in the memory. target reference frame. Then use the target reference frame to perform P-frame coding on the displayed image.

In some embodiments, determining the target reference frame corresponding to the display image from the reference frames stored in the memory includes:

1. Compare the displayed image with the reference frames stored in the memory one by one to determine the similarity between the displayed image and each reference frame;

2. Determine the reference frame with the highest similarity with the displayed image as the target reference frame.

Wherein, after the server generates the display image according to the interaction information uploaded by the terminal, the similarity calculation is performed between the display image and all reference frames stored in the memory, and the similarity between the display image and each reference frame is determined. Specifically, the similarity calculation between the displayed image and the reference frame may be performed based on a preset similarity calculation algorithm. After the similarity between the displayed image and each reference frame is determined, the reference frame with the highest similarity with the displayed image is determined as the target reference frame corresponding to the displayed image.

In some embodiments, when the display image displayed by the interactive interface of the application is relatively single and fixed. For example, in a smart face payment cloud application, the display images displayed on the interactive interface mainly include: face brushing interface, mobile phone number input page, payment confirmation page, and payment result page, etc., which are relatively fixed and single interfaces. Then the reference frames for these applications can be bound with each application scenario, and a corresponding reference frame is determined for each application scenario. For example, for a face brushing application scenario, a reference frame corresponding to a face brushing application scene may be determined. In this way, when it is determined according to the interaction information input by the terminal that the application scene corresponding to the currently displayed display interface is the face brushing application scene, the reference frame corresponding to the face brushing application scene can be directly called as the target reference frame to perform P-frame encoding on the displayed image. In this way, there is no need to calculate the similarity between the displayed image and all the stored reference frames, and then determine the target reference frame according to the similarity. Thus, the efficiency of encoding the image by the server is improved.

Further, since the reference frame is stored in association with its corresponding number in the memory, after the target reference frame corresponding to the display image is determined, the number corresponding to the target reference frame can be determined at the same time.

Step 304: Determine the associated frame data of the display image relative to the target reference frame, where the associated frame data is calculated difference data between the display image and the target reference frame.

Wherein, after the target reference frame corresponding to the display image is determined, the display image is P-frame encoded by using the target reference frame corresponding to the display image. Specifically, the difference data of the display image relative to the target reference frame is calculated, and then the calculated difference data is compressed to obtain the associated frame data. Here, since the target image is P-frame encoded, the compression ratio of the difference data is 20.

Step 305: Send the serial number and the associated frame data to the terminal, so that the terminal retrieves the reference frame corresponding to the serial number from the local storage of the terminal according to the serial number, and generates and displays the display image in combination with the related frame data.

The server sends the number of the target reference frame corresponding to the P-frame encoding of the determined display image and the associated frame data obtained by performing the P-frame encoding according to the target reference frame to the terminal together. According to the serial number, the terminal extracts the target reference frame corresponding to the serial number from the local storage of the terminal, and then decodes it in combination with the associated frame data to obtain a display image, and displays the display image. In this way, since consistent reference frames are stored in both the server and the terminal, the server only needs to transmit the reference frame number and the associated frame data encoded by the P frame, and the terminal can decode the reference frame number and the associated frame data to obtain a display image. The transmission of I frame data or reference frame data is avoided, the amount of data transmitted from the server to the terminal is reduced, and the efficiency of the terminal receiving image data is improved. On the other hand, traffic consumption caused by data transmission is also reduced, and resources are saved.

In some embodiments, the image processing methods provided by the above embodiments may further include the following steps:

1. Identify the target application for uploading interactive information;

2. Acquire the key frame used when performing image encoding on the interface image displayed by the target application. The key frame is the frame data of the original frame image when image encoding is performed.

3. Determine a preset number of reference frames from the key frames, and number each reference frame.

4. Associate each reference frame with a corresponding number and store it in a storage location in the memory corresponding to the target application.

5. Send each reference frame and the corresponding serial number to the terminal, so that the terminal associates and stores each reference frame and the corresponding serial number in the local memory of the terminal.

Wherein, the implementation timing of the above steps may be before the server performs P-frame encoding on the generated display image. Specifically, when there are multiple running application programs in the server, the target application program for uploading the interaction information is first determined according to the uploaded interaction data or the upload path of the interaction data. In the related art, the IPB encoding method is used to encode and transmit the displayed image. Therefore, the server can call the encoding history data in the past IPB encoding, and extract the I-frame data obtained by encoding in the I-frame encoding method in the historical data, and these I-frame data can be called key frame data. The key frame data is the frame data of the original frame picture when encoding, and the compression rate of the key frame data is low, generally 7. The server acquires all the I-frame data in the historical data of a complete running process encoded by the target application according to IPB, and then extracts a preset number of reference frames from the acquired I-frame (key frame) data, and analyzes each Reference frames are numbered. Then, the server stores each reference frame and the corresponding serial number in the memory of the server in association with each other, and sends each reference frame and the corresponding serial number to the terminal running the target application program at the same time, so that the terminal can store the reference frame and serial number in the terminal in association with each other. in local storage.

In some embodiments, a predetermined number of reference frames are determined from the key frames, including:

a. Classify key frames according to similarity to obtain multiple key frame sets.

b. Extract a key frame from each key frame set as a reference frame.

Wherein, in order to ensure that the decoded image of the encoded data is not distorted during encoding according to IPB, one I-frame encoding needs to be performed every preset time. Therefore, it is possible to encode multiple I-frames in the same application scenario, but the differences between the I-frames in the same application scenario are small. If these I-frames are all stored as reference frames, the storage space in the memory will be lost. Excessive occupation. Therefore, in this embodiment of the present application, the acquired key frames may be classified first to obtain multiple key frame sets. Then extract a key frame from each key frame set as a reference frame. In this way, it is possible to avoid excessive occupation of the storage space of the server and the terminal due to too many similar reference frames, thereby saving the storage space of the terminal and the server.

According to the above description, in this embodiment of the present application, the interaction information uploaded by the terminal is received; the display image corresponding to the interaction information is generated according to the interaction information; the target reference frame corresponding to the display image is determined from the reference frames stored in the memory, and the target reference frame is determined. The number corresponding to the frame; determine the associated frame data of the display image relative to the target reference frame, and the associated frame data is to calculate the difference data between the display image and the target reference frame; send the number and the associated frame data to the terminal, so that the terminal can change from The reference frame corresponding to the serial number is retrieved from the local storage of the terminal, and a display image is generated and displayed in combination with the associated frame data. In this way, when the server transmits the display image to the terminal, it only needs to transmit the number corresponding to the reference frame and the associated frame data obtained by encoding the display image according to the reference frame, and the terminal can retrieve the corresponding number from the local memory of the terminal according to the number. The reference frame is decoded in combination with the associated frame data to obtain the display image. Therefore, the transmission of key frame data is avoided, the amount of data transmitted from the server to the terminal in the cloud application is reduced, the data transmission speed is accelerated, and the operation efficiency of the cloud application on the terminal is improved.

Embodiment 4

In order to better implement the image processing method of one of the above embodiments, the embodiment of the present invention further provides an image processing apparatus, and the image processing apparatus may be integrated in a terminal, and the terminal may include an intelligent payment terminal, a smart phone, a tablet computer , laptop and/or personal computer, etc.

For example, as shown in FIG. 6, which is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application, the image processing apparatus may include a display unit 401, an uploading unit 402, a receiving unit 403, an extracting unit 404, and a decoding unit 305, as follows:

A display unit 401, configured to display an interactive interface of an application, and the interactive interface is used to receive interactive information;

An uploading unit 402, configured to upload the interaction information collected by the interaction interface to the server;

The receiving unit 403 is used for receiving the image encoding data returned by the server according to the interaction information, the image encoding data including the reference frame number and associated frame data determined when the server encodes the display image, and the associated frame data is the calculation of the display image and the reference frame. difference data between;

Extraction unit 404, for extracting the reference frame corresponding to the number from the local memory according to the number;

The decoding unit 405 is configured to decode the currently displayed image according to the reference frame and the associated frame data, and display the currently displayed image in the interactive interface.

In some embodiments, as shown in FIG. 7 , which is another schematic structural diagram of the image processing apparatus provided in the present application, the image processing apparatus further includes:

a first determining unit 406, configured to determine a preset number of reference frames from the interface images displayed on the interactive interface;

Numbering unit 407, for numbering each reference frame according to a preset rule;

The first storage unit 408 is used for storing each reference frame in the local memory of the terminal in association with the corresponding serial number;

The first sending unit 409 is configured to send each reference frame and the corresponding serial number to the server, so that the server associates and stores each reference frame and the corresponding serial number in the storage space of the server.

In some embodiments, as shown in FIG. 8 , which is another schematic structural diagram of an image processing apparatus provided in an embodiment of the present application, the determining unit 406 includes:

The acquisition subunit 4061 is used to acquire each frame of interface image displayed corresponding to the interactive interface in which the application is running;

The classification subunit 4062 is used to classify the interface images according to the image similarity to obtain a plurality of interface image sets;

The extraction subunit 4063 is configured to extract any frame of interface image from each interface image set and encode it to obtain a reference frame.

In some embodiments, the classification subunit 4062 is also used to:

Image recognition algorithm is used to perform image recognition on each interface image, and the corresponding recognition feature of each interface image is obtained;

The interface images are classified according to the similarity of the recognition features, and multiple interface image sets are obtained.

In some embodiments, as shown in FIG. 9 , which is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present application, the determining unit 406 includes:

Sorting subunit 4064, used for sorting each frame of interface images displayed in the application interactive interface;

Determining subunit 4065, for determining the interface image of the previous frame of each frame of interface image according to the sorting order;

The encoding subunit 4066 is used to encode the interface image of the current frame as a reference frame when the interface image does not have the interface image of the previous frame; when the interface image has the interface image of the previous frame, obtain the difference between the interface image and the interface image of the previous frame. Similarity; when the similarity is less than the preset threshold, encode the current frame interface image as a reference frame.

In some embodiments, as shown in FIG. 10 , which is another schematic structural diagram of an image processing apparatus provided in an embodiment of the present application, the image processing apparatus further includes:

An update unit 410, configured to update the application when receiving the application update instruction sent by the server;

The second determining unit 411 is configured to determine a preset number of reference frames from the interface image displayed on the interactive interface of the updated application and number each reference frame according to a preset rule;

The second storage unit 412 is configured to associate and store each reference frame with the corresponding serial number in the local memory of the terminal;

The second sending unit 413 is configured to send each reference frame and its corresponding serial number to the server, so that the server associates each reference frame with the corresponding serial number and stores it in the storage space of the server.

During specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

As can be seen from the above, in this embodiment, the display unit 401 is used to display the interactive interface of the application, and the interactive interface is used to receive interactive information; the uploading unit 402 uploads the interactive information collected by the interactive interface to the server; the receiving unit 403 receives the return from the server according to the interactive information The image encoding data, the image encoding data includes the number of the reference frame determined when the server encodes the display image and the associated frame data, and the associated frame data is to calculate the difference data between the displayed image and the reference frame; The reference frame corresponding to the serial number is extracted from the memory; the decoding unit 405 decodes the reference frame and the associated frame data to obtain the currently displayed image and displays the currently displayed image in the interactive interface. In this way, by saving the reference frame locally in the terminal, when the server sends image data to the terminal, it only needs to send the reference frame number and associated frame data, thereby avoiding the sending of the reference frame. Thus, the amount of data transmitted from the server to the terminal is reduced, the efficiency of the terminal receiving image data is improved, and the operation efficiency of the cloud application in the user terminal is further improved.

Embodiment 5

In order to better implement the image processing method in the third embodiment, the present application also provides an image processing device, which can be integrated in a server, and the server can be an independent physical server, or can be composed of multiple physical servers. It can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and network acceleration services (Content Delivery Network, CDN), as well as cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

For example, as shown in FIG. 11 , which is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present application, the image processing apparatus may include: a receiving unit 501 , a generating unit 502 , a first determining unit 503 , and a second determining unit 504 And the sending unit 505, as follows:

a receiving unit 501, configured to receive interaction information uploaded by a terminal;

a generating unit 502, configured to generate a display image corresponding to the interaction information according to the interaction information;

The first determining unit 503 is used to determine the target reference frame corresponding to the display image from the reference frame stored in the memory and determine the number corresponding to the target reference frame;

The second determining unit 504 is used to determine the associated frame data of the display image relative to the target reference frame, where the associated frame data is the difference data between the calculated display image and the target reference frame;

The first sending unit 505 is configured to send the serial number and the associated frame data to the terminal, so that the terminal can retrieve the target reference frame corresponding to the serial number from the local memory of the terminal according to the serial number, and generate a display image in combination with the related frame data and display the image. to display.

In some embodiments, as shown in FIG. 12 , which is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present application, the image processing apparatus further includes:

An identification unit 506, configured to identify the target application for uploading the interaction information;

Obtaining unit 507 is used to obtain the key frame used when the interface image displayed by the target application program is subjected to image encoding, and the key frame is the frame data of the original frame picture when the image encoding is performed;

A third determining unit 508, configured to determine a preset number of reference frames from the key frames, and number each reference frame;

The storage unit 509 is used to associate and store each reference frame with the corresponding number in the storage location corresponding to the target application program in the memory;

The second sending unit 510 is configured to send each reference frame and the corresponding serial number to the terminal, so that the terminal associates and stores each reference frame and the corresponding serial number in the local memory of the terminal.

In some embodiments, as shown in FIG. 13 , which is another schematic structural diagram of the image processing apparatus provided by the embodiments of the present application, the third determining unit 508 includes:

The classification subunit 5081 is used to classify the key frames according to the similarity to obtain a plurality of key frame sets;

The extraction subunit 5082 is used for extracting a key frame from each key frame set as a reference frame.

In some embodiments, as shown in FIG. 14 , which is another schematic structural diagram of an image processing apparatus provided in an embodiment of the present application, the first determining unit 503 includes:

The comparison subunit 5031 is used to compare the displayed image with the reference frames stored in the memory one by one to determine the similarity between the displayed image and each reference frame;

The first determination subunit 5032 is used to determine the reference frame with the highest similarity with the displayed image as the target reference frame;

The second determination subunit 5033 is configured to determine the number corresponding to the target reference frame stored in the memory.

During specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

It can be seen from the above that in this embodiment, the interaction information uploaded by the terminal is received by the receiving unit 501; the generating unit 502 generates a display image corresponding to the interaction information according to the interaction information; the first determining unit 503 determines and displays the image from the reference frame stored in the memory The corresponding target reference frame and determine the number corresponding to the target reference frame; the second determination unit 504 determines the associated frame data of the display image relative to the target reference frame, and the associated frame data is to calculate the difference data between the display image and the target reference frame; the first The sending unit 505 sends the serial number and the associated frame data to the terminal, so that the terminal retrieves the reference frame corresponding to the serial number from the local storage of the terminal according to the serial number, and generates and displays the display image in combination with the related frame data. In this way, when the server transmits the display image to the terminal, it only needs to transmit the number corresponding to the reference frame and the associated frame data obtained by encoding the display image according to the reference frame, and the terminal can retrieve the corresponding number from the local memory of the terminal according to the number. The reference frame is decoded in combination with the associated frame data to obtain the display image. Therefore, the transmission of key frame data is avoided, the amount of data transmitted from the server to the terminal in the cloud application is reduced, the data transmission speed is accelerated, and the operation efficiency of the cloud application on the terminal is improved.

Embodiment 6

An embodiment of the present application further provides a computer device, as shown in FIG. 15 , which is a schematic structural diagram of the server provided by the present application. Specifically:

The computer device may include a processor 601 of one or more processing cores, a memory 602 of one or more storage media, a power supply 603 and an input unit 604 and other components. Those skilled in the art can understand that the computer device structure shown in FIG. 15 does not constitute a limitation to the computer device, and may include more or less components than the one shown, or combine some components, or arrange different components. in:

The processor 601 is the control center of the computer equipment, and uses various interfaces and lines to connect various parts of the entire computer equipment, by running or executing the software programs and/or modules stored in the memory 602, and calling the software programs stored in the memory 602. Data, perform various functions of computer equipment and process data, so as to conduct overall monitoring of computer equipment. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc. , the modem processor mainly deals with wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 601.

The memory 602 can be used to store software programs and modules, and the processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602 . The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, and web page access, etc.), etc.; the storage data area Data and the like created according to the use of the server can be stored. Additionally, memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, memory 602 may also include a memory controller to provide processor 601 access to memory 602 .

The computer equipment also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so that functions such as charging, discharging, and power consumption management are implemented through the power management system. Power source 603 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

The computer device may also include an input unit 604 that may be operable to receive input numerical or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and functional control.

Although not shown, the server may also include a display unit and the like, which will not be described herein again. Specifically, in this embodiment, the processor 601 in the server loads the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 executes them and stores them in the memory 602 to realize various functions, as follows:

Display the interactive interface of the application, the interactive interface is used to receive interactive information; upload the interactive information collected by the interactive interface to the server; receive the image encoding data returned by the server according to the interactive information, and the image encoding data includes the image encoding data determined when the server encodes the displayed image The number of the reference frame and the associated frame data, the associated frame data is the difference data between the calculated display image and the reference frame; the reference frame corresponding to the number is extracted from the local memory according to the number; The reference frame and the associated frame data are decoded to obtain The currently displayed image and the currently displayed image are displayed in the interactive interface.

Or, receive the interaction information uploaded by the terminal; generate a display image corresponding to the interaction information according to the interaction information; determine the target reference frame corresponding to the display image from the reference frames stored in the memory and determine the number corresponding to the target reference frame; determine the display image Relative to the associated frame data of the target reference frame, the associated frame data is the difference data between the calculated display image and the target reference frame; the serial number and the related frame data are sent to the terminal, so that the terminal can retrieve and number from the local memory of the terminal according to the serial number The corresponding reference frame is combined with the associated frame data to generate a display image and display the display image.

It should be noted that the functions implemented by the processors in the computer equipment provided in the embodiments of the present application belong to the same concept as the image processing methods in the above embodiments, and the specific implementation of the above operations can refer to the previous embodiments, which will not be repeated here. .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructions, or by instructions that control relevant hardware, and the instructions can be stored in a storage medium and processed by to load and execute.

To this end, the embodiments of the present invention provide a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute steps in any image processing method provided by the embodiments of the present invention. For example, the instruction can perform the following steps:

Display the interactive interface of the application, the interactive interface is used to receive interactive information; upload the interactive information collected by the interactive interface to the server; receive the image encoding data returned by the server according to the interactive information, and the image encoding data includes the image encoding data determined when the server encodes the displayed image The number of the reference frame and the associated frame data, the associated frame data is the difference data between the calculated display image and the reference frame; the reference frame corresponding to the number is extracted from the local memory according to the number; The reference frame and the associated frame data are decoded to obtain The currently displayed image and the currently displayed image are displayed in the interactive interface.

Or, receive the interaction information uploaded by the terminal; generate a display image corresponding to the interaction information according to the interaction information; determine the target reference frame corresponding to the display image from the reference frames stored in the memory and determine the number corresponding to the target reference frame; determine the display image Relative to the associated frame data of the target reference frame, the associated frame data is the difference data between the calculated display image and the target reference frame; the serial number and the related frame data are sent to the terminal, so that the terminal can retrieve and number from the local memory of the terminal according to the serial number The corresponding reference frame is combined with the associated frame data to generate a display image and display the display image.

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not described herein again.

Wherein, the storage medium may include: a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Since the instructions stored in the storage medium can execute the steps in any image processing method provided by the embodiment of the present invention, it is possible to realize the steps that can be realized by any image processing method provided by the embodiment of the present invention. For the beneficial effects, refer to the foregoing embodiments for details, which will not be repeated here.

Wherein, according to one aspect of the present application, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, and the computer instructions are stored in a storage medium. The processor of the computer device reads the computer instruction from the storage medium, and the processor executes the computer instruction, so that the computer device performs the image processing provided in the various optional implementation manners in the above-mentioned Embodiment 1, Embodiment 2, and Embodiment 3 method.

The image processing method, device, computer equipment, and storage medium provided by the embodiments of the present invention have been described above in detail. The principles and implementations of the present invention are described with specific examples in this paper. The descriptions of the above embodiments are only It is used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, this specification The contents should not be construed as limiting the present invention.

Claims (14)

1. an image processing method, is characterized in that, comprises: Displaying an interactive interface of the application, the interactive interface is used to receive interactive information; uploading the interactive information collected by the interactive interface to the server; Receive the image encoding data returned by the server according to the interaction information, where the image encoding data includes the reference frame number and associated frame data determined when the server encodes the displayed image, and the associated frame data is used to calculate the displaying difference data between the image and the reference frame; Extract the reference frame corresponding to the number from the local memory according to the number; The currently displayed image is decoded according to the reference frame and the associated frame data, and the currently displayed image is displayed in the interactive interface. 2. The method according to claim 1, wherein the method further comprises: Determine a preset number of reference frames from the interface image displayed on the interactive interface; Number each reference frame according to preset rules; storing each reference frame in the local memory of the terminal in association with the corresponding number; Each reference frame and the corresponding number are sent to the server, so that the server stores each reference frame and the corresponding number in the storage space of the server in association with each other. 3. The method according to claim 2, wherein the determining a preset number of reference frames from the interface image displayed on the interactive interface comprises: Acquiring each frame of interface image displayed corresponding to the interactive interface in which the application program runs; classifying the interface images according to the image similarity to obtain a plurality of interface image sets; Extract any frame of interface image from each interface image set and encode it to obtain a reference frame. 4. The method according to claim 3, wherein the interface images are classified according to image similarity to obtain a plurality of interface image sets, comprising: Image recognition algorithm is used to perform image recognition on each frame of interface image, and the corresponding recognition feature of each frame of interface image is obtained; The interface images are classified according to the similarity of the identification features to obtain a plurality of interface image sets. 5. The method according to claim 2, wherein the determining a preset number of reference frames from the interface image displayed on the interactive interface comprises: Sorting each frame of interface images displayed on the application program interactive interface; Determine the interface image of the previous frame of each frame of interface image according to the sorting order; When there is no previous frame of interface image in the interface image, encode the interface image of the current frame as a reference frame; When the interface image has the interface image of the previous frame, obtain the similarity between the interface image and the interface image of the previous frame; When the similarity is less than the preset threshold, the current frame interface image is encoded as a reference frame. 6. The method of claim 1, wherein the method further comprises: When receiving the application program update instruction sent by the server, update the application program; Determine a preset number of reference frames from the interface image displayed in the interactive interface of the updated application and number each reference frame according to a preset rule; storing each reference frame in the local memory of the terminal in association with the corresponding number; Uploading each reference frame and its corresponding serial number to the server, so that the server stores each reference frame and the corresponding serial number in the storage space of the server in association with each other. 7. An image processing method, characterized in that, comprising: Receive interactive information uploaded by the terminal; generating a display image corresponding to the interaction information according to the interaction information; Determine the target reference frame corresponding to the display image from the reference frames stored in the memory and determine the number corresponding to the target reference frame; determining the associated frame data of the display image relative to the target reference frame, where the associated frame data is calculated difference data between the display image and the target reference frame; Sending the number and the associated frame data to the terminal, so that the terminal retrieves the reference frame corresponding to the number from the local storage of the terminal according to the number, and combines the associated frame data with the reference frame A display image is generated and displayed. 8. The method according to claim 7, wherein the method further comprises: identifying the target application that uploaded the interaction information; Obtaining the key frame used when performing image encoding on the interface image displayed by the target application, where the key frame is the frame data of the original frame image when performing image encoding; Determine a preset number of reference frames from the key frames, and number each reference frame; Associate and store each reference frame with a corresponding number in a storage location corresponding to the target application in the memory; Each reference frame and the corresponding number are sent to the terminal, so that the terminal stores each reference frame and the corresponding number in the local memory of the terminal in association with each other. 9. The method according to claim 8, wherein the determining a preset number of reference frames from the key frames comprises: Classify the key frames according to the similarity to obtain a plurality of key frame sets; One key frame is extracted from each of the key frame sets as a reference frame. 10. The method according to claim 7, wherein determining the target reference frame corresponding to the display image from the reference frames stored in the memory and determining the number corresponding to the target reference frame, comprising: comparing the displayed image with the reference frames stored in the memory one by one to determine the similarity between the displayed image and each reference frame; Determine the reference frame with the highest similarity with the displayed image as the target reference frame; A number stored in the memory corresponding to the target reference frame is determined. 11. An image processing device, comprising: a display unit, configured to display an interactive interface of the application, the interactive interface is used to receive interactive information; an uploading unit, configured to upload the interactive information collected by the interactive interface to the server; a receiving unit, configured to receive the image encoding data returned by the server according to the interaction information, the image encoding data including the reference frame number determined when the server encodes the displayed image and associated frame data, the associated frame The data is to calculate the difference data between the display image and the reference frame; an extraction unit, configured to extract a reference frame corresponding to the number from the local memory according to the number; A decoding unit, configured to decode the reference frame and the associated frame data to obtain a currently displayed image, and display the currently displayed image in the interactive interface. 12. An image processing device, comprising: a receiving unit, configured to receive the interaction information uploaded by the terminal; a generating unit, configured to generate a display image corresponding to the interaction information according to the interaction information; a first determining unit, configured to determine a target reference frame corresponding to the display image from the reference frames stored in the memory and determine a number corresponding to the target reference frame; a second determining unit, configured to determine associated frame data of the display image relative to the target reference frame, where the associated frame data is calculated difference data between the display image and the target reference frame; a sending unit, configured to send the serial number and the associated frame data to the terminal, so that the terminal retrieves the target reference frame corresponding to the serial number from the local storage of the terminal according to the serial number, and A display image is generated and displayed in combination with the associated frame data. 13. A computer device, characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements claim 1 when the processor executes the computer program to 6 or the image processing method of any one of claims 7 to 10. 14. A storage medium, characterized in that the storage medium stores a plurality of instructions, the instructions are adapted to be loaded by a processor to execute any one of claims 1 to 6 or claims 7 to 10. steps in an image processing method.

Images