WO2023015808A1 - Digital retina video processing method and apparatus, electronic device, and storage medium - Google Patents

Abstract

The present application discloses a digital retina video processing method and apparatus, an electronic device, and a storage medium. The method comprises: determining an inter-prediction reference frame on the basis of, at least in part, a determined feature-encoded reference frame; and/or determining a feature-encoded reference frame on the basis of at least in part, the determined inter-prediction reference frame; and sending reference frame indication information in a video compression code stream, the reference frame indication information being the association between the feature reference frame and the inter-frame prediction reference frame.

Description

Digital retina video processing method, device, electronic equipment and storage medium technical field

The present application relates to the field of computer image processing, in particular to a digital retinal video processing method, device, electronic equipment and storage medium.

Background technique

Since the concept of digital retina was put forward, it has attracted great attention in the fields of video codec, video surveillance and so on. An important feature of digital retina technology is the simultaneous dual-stream or multi-stream transmission of video streams, visual feature streams, and even summary video streams, which facilitates video retrieval, video analysis, and storage.

However, the inventor found in the process of implementing the related technical solutions of the embodiments of the present invention that at least the following problems exist in the prior art: the digital retina technology requires the front-end camera or a video processing device close to the front-end to have both video feature processing and video compression processing capabilities, The technical complexity and processing power requirements are high, and the inventors found that the selection of the feature reference frame selected in the feature encoding process and the selection of the inter-frame encoding reference frame in the video compression process have at least some results that can be referred to each other Even directly reused. Therefore, if we can design a coordinated mechanism for feature reference frame and inter-frame prediction reference frame selection, it will help reduce the complexity of video processing in digital retinal systems.

Contents of the invention

Aiming at the above-mentioned technical problems in the prior art, the embodiment of the present application proposes a digital retinal video processing method, device, electronic equipment, and computer-readable storage medium to solve the video processing requirements of the front-end camera or close to the front-end in the digital retinal technology. The device has both video feature processing and video compression processing capabilities, which requires high technical complexity and processing capabilities.

The first aspect of the embodiment of the present application provides a digital retinal video processing method, including:

determining an inter prediction reference frame based at least in part on the determined feature coding reference frame; and/or

determining a feature encoding reference frame based at least in part on the determined inter prediction reference frame;

Send reference frame indication information in the video compression coded stream or feature processing coded stream; wherein, the reference frame indication information is the association relationship between the feature reference frame and the inter-frame prediction reference frame, including the reference frame Select instructions.

In some embodiments, the inter-frame prediction reference frame is at least partially replaced by a feature reference frame or formed by combining several feature reference frames.

In some embodiments, the feature reference frame is at least partly replaced by an inter-frame prediction reference frame or formed by a combination of several inter-frame prediction reference frames.

In some embodiments, the reference frame selection indication is an indication that the feature reference frame is determined based on the inter-frame prediction reference frame and an indication that the inter-frame prediction reference frame is determined based on the feature reference frame, including: whether the frame coding mode indicates whether a reference frame is required, At least one of frame encoding mode sequence, forward or backward reference frame and whether to clear reference frame buffer.

In some embodiments, the method also includes:

Feedback a reference frame selection result based on at least one reference frame selection indication.

In some embodiments, the method also includes:

Based on the determined reference frame selection result fed back by the inter-frame prediction reference frame, determine a feature encoding reference frame; and/or

Determine the reference frame for inter-frame prediction based on the reference frame selection result fed back by the determined feature coding reference frame.

The second aspect of the embodiment of the present application provides a digital retinal video processing device, including:

The reference frame determination module determines an inter-frame prediction reference frame based at least in part on the determined feature coding reference frame; and/or determines a feature coding reference frame based at least in part on the determined inter-frame prediction reference frame;

A reference frame indication information sending module, configured to send reference frame indication information in a video compression coded stream or a feature processing coded stream; wherein, the reference frame indication information is the feature reference frame and the inter-frame prediction reference frame The association relationship between, including the reference frame selection indication.

In some embodiments, the inter-frame prediction reference frame is replaced by a feature reference frame or formed by combining several feature reference frames.

In some embodiments, the feature reference frame is formed by replacing or combining several inter-frame prediction reference frames.

In some embodiments, the reference frame selection indication is an indication that the feature reference frame is determined based on the inter-frame prediction reference frame and an indication that the inter-frame prediction reference frame is determined based on the feature reference frame, including: whether the frame coding mode indicates whether a reference frame is required, At least one of frame encoding mode sequence, forward or backward reference frame and whether to clear reference frame buffer.

In some embodiments, the device also includes:

A reference frame feedback module, configured to feed back a reference frame selection result based on at least one reference frame selection indication.

In some embodiments, the reference frame determination module also includes:

A feature coding reference frame determination module, configured to determine a feature coding reference frame based on the reference frame selection result fed back from the determined inter-frame prediction reference frame;

The inter-frame prediction reference frame determination module is configured to determine the inter-frame prediction reference frame based on the reference frame selection result fed back from the determined feature coding reference frame.

A third aspect of the embodiments of the present application provides an electronic device, including:

memory and one or more processors;

Wherein, the memory is connected in communication with the one or more processors, the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors , the electronic device is used to implement the methods described in the foregoing embodiments.

The fourth aspect of the embodiments of the present application provides a computer-readable storage medium on which computer-executable instructions are stored. When the computer-executable instructions are executed by a computing device, they can be used to implement the above-mentioned embodiments. Methods.

A fifth aspect of the embodiments of the present application provides a computer program product, the computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by the computer , it can be used to implement the methods described in the foregoing embodiments.

The technical solution of the embodiment of the present application provides a coordination mechanism for selecting a feature reference frame and an inter-frame prediction reference frame in the digital retina technology, so that the feature encoding reference frame selected in the encoding process is the same as that in the video compression process. At least part of the results of the selected inter-frame prediction reference frames can be referred to each other or even directly used, which is beneficial to reduce the complexity of video processing in the digital retina system.

Description of drawings

The features and advantages of the present application will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the application in any way. In the accompanying drawings:

Fig. 1 is a schematic diagram of a frame interpolation image processing method according to some embodiments of the present application;

FIG. 2 is a schematic diagram of a determination process of an inter-frame prediction reference frame according to some embodiments of the present application;

Fig. 3 is a schematic diagram of a process of sending reference frame indication information in a characteristic coded code stream according to some embodiments of the present application;

Fig. 4 is a schematic diagram of a frame interpolation image processing device according to some embodiments of the present application.

Fig. 5 is a schematic diagram of a logical structure of an electronic device according to some embodiments of the present application;

Fig. 6 is a schematic structural diagram of a general-purpose computer node according to some embodiments of the present application.

Detailed ways

In the following detailed description, numerous specific details of the application are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. It will be apparent, however, to one skilled in the art that the application may be practiced without these details. It should be understood that the terms "system", "device", "unit" and/or "module" used in this application are used as a means to distinguish between different components, elements, parts or assemblies at different levels in a sequential arrangement. method. However, these terms may be replaced by other expressions if the same purpose can be achieved by other expressions.

It will be understood that when a device, unit or module is referred to as being "on," "connected to" or "coupled to" another device, unit or module, it can be directly on the other device, unit or module. connected or coupled to or communicate with other devices, units or modules, or intervening devices, units or modules may be present, unless the context clearly suggests an exception. For example, as used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used in the present application is for describing specific embodiments only, and does not limit the scope of the present application. As shown in the specification and claims of this application, words such as "a", "an", "an" and/or "the" do not refer to the singular, and may also include the plural, unless the context clearly indicates an exception. Generally speaking, the terms "comprising" and "comprising" only suggest the inclusion of clearly identified features, integers, steps, operations, elements and/or components, and such expressions do not constitute an exclusive list, other features, Integrals, steps, operations, elements and/or assemblies may also be included.

These and other features and characteristics, methods of operation, functions of relevant elements of structure, combinations of parts, and economies of manufacture of the present application may be better understood with reference to the following description and drawings, which form a part of the manual. However, it can be clearly understood that the drawings are only for the purpose of illustration and description, and are not intended to limit the protection scope of the present application. It is understood that the drawings are not drawn to scale.

Various structural diagrams are used in this application to illustrate various modifications of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit the present application. The protection scope of the present application shall be determined by the claims.

An important feature of digital retina technology is the simultaneous dual-stream or multi-stream transmission of video streams and visual feature streams, even abstract video streams, which facilitates video retrieval, video analysis and storage, but the existing digital retina technology It is impossible to require the front-end camera or the video processing device close to the front-end to have video feature processing and video compression processing capabilities at the same time, and the relationship between the feature encoding reference frame and the inter-frame prediction reference frame is not reasonably used, which not only increases the complexity of the technology, but also Higher requirements are placed on the processing power of technology.

In view of this, the embodiment of the present application provides a digital retinal video processing method. In the digital retinal technology, a coordination mechanism is provided for the selection of feature reference frames and inter-frame prediction reference frames, so that the selected At least part of the results of the feature coding reference frame and the selected inter-frame prediction reference frame in the video compression process can be referred to each other or even directly used, which is beneficial to reduce the complexity of video processing in the digital retina system. Further, as shown in FIG. 1, in one embodiment of the present application, the frame insertion image processing method includes steps:

Determining an inter-frame prediction reference frame based at least in part on the determined feature encoding reference frame; and/or determining a feature encoding reference frame based at least in part on the determined inter-frame prediction reference frame; sending the reference in the video compression encoding code stream or feature processing code stream Frame indication information.

Among them, in the embodiment of the present application, video compression is mainly realized by eliminating redundant information. Continuous video images have a lot of redundant information in time and space, so by reducing redundant information in video , which can store and transmit video image information more efficiently. Among them, motion compensation technology is a widely used video information redundancy elimination technology, which is widely used in various video codec standards, including MPEG2, MPEG4, H.264, H.265/HEVC, H.266/ VCC. Coded pictures are classified into three types: I-frames, P-frames, and B-frames. The pictures used to predict the coded pictures are called reference frames. Among them, the I frame is an intra-coded frame, which does not need to refer to other frames, and uses the spatial correlation of the video image to perform independent compression coding. The P frame is a forward prediction frame, and the I frame is used as a reference frame to find out the prediction value and motion vector of a "certain point" in the P frame in the I frame, and transmit the prediction difference and the motion vector together. The B frame is a bidirectional interpolation frame, with the preceding I or P frame and the following P frame as reference frames. Early video coding standards only supported a single reference picture. However, starting from H.263+, multi-reference frame technology is supported, while H.264/AVC and HEVC standards can support up to 15 reference frames. The reference frame of the P frame can be an encoded I frame or P frame, and the reference frame of the B frame can be an encoded I frame, P frame or B frame. During the prediction process of the B frame, one or two reference images may be used, and these reference images may be forward images or backward images. It can be seen that selecting an appropriate reference frame is beneficial to reducing non-zero residual data and improving compression efficiency. In one prior art, I-frames, P-frames and B-frames appear in a predetermined pattern sequence, for example, a frame encoding pattern sequence composed of 15 frames has the following form IBBPBBPBBPBBPBB. The frame sequence can also be an adaptive flexible mode, that is, the coding unit will select the most suitable frame type according to the purpose of improving the compression rate. Network transmission errors may cause video decoding to fail, and I frames may need to be resent, which will result in readjustment of the frame encoding mode sequence. The image frame encoding mode determines whether inter-frame encoding is required and determines an appropriate reference frame. The reference frame is determined by matching among other frames within a certain frame number range before or after the current frame at the encoder/decoder. In some cases, when an error occurs in the network transmission, the I frame needs to be resent, the buffered video frame to be sent is cleared, and the video frame after the I frame is used as a reference frame for the video frame sent subsequently.

Further, in an embodiment of the present application, video feature processing may generally include functions such as video feature extraction, feature location compression, and video feature compression. Visual features include information about color, pattern, texture, grayscale, etc. in video frames, and deep learning features can include information such as pedestrian recognition, license plate recognition, traffic accidents, and traffic instruction recognition. In some embodiments, generation and compression of summary video streams are further included. For the convenience of description, in the present invention, video features include any one or more of visual features and machine learning features. The positions or feature positions of the points of interest determined after video feature extraction are provided to the video compression unit. Similar to inter-frame predictive video coding, some feature frames also use intra-frame coding to compress feature value data. Some other feature frames use the temporal correlation of feature values and adopt the inter-frame differential coding mode. Taking a certain frame as a reference frame, comparing the current frame with the reference frame, and encoding the resulting eigenvalue residuals can significantly improve the eigenvalue compression efficiency. The feature frame adopts intra-frame or inter-frame coding, and can also be defined by a predetermined sequence of feature frame coding modes, such as IBBPBBPBBPBBPBB (similar to video coding, define I, P, and B feature frames as intra-frame coding and forward frame respectively) Between differential coding and forward and backward bidirectional differential coding frames). In the present invention, there is a correlation between the feature value compressed reference frame and the video compressed frame.

Specifically, FIG. 2 shows a process of determining a reference frame for inter-frame prediction. In some embodiments of the present invention, a feature processing unit and a video compression unit are included. The feature processing unit uses the inter-frame prediction reference frame determined by the video compression unit as the feature coding inter-frame differential coding reference frame. The feature processing unit needs to send a reference frame request to the video compression unit, including one or more of these indications (feature frame inter-frame or intra-frame coding, or indication of feature frame types such as I, B, P frames, or providing fixed Coding mode sequences such as IBBPP, etc., whether it is necessary to refresh the cache feature reference frame, etc.). The video compression unit determines the encoding mode (I, P, B frame, etc.) of a specific video frame according to the instruction of the feature processing unit, and determines one or more reference frames for the P and B frames and feeds it back to the feature processing unit. If it is indicated that the currently cached reference frame needs to be cleared, the best reference frame is searched after the designated specific frame and the reference frame index is provided to the feature processing unit. One way for the feature processing unit to implement inter-frame differential coding of feature frames is to compare feature values of other frames with the closest reference frame to form feature value residuals. Perform transformation coding (such as DCT transformation) on the feature value, then quantize the transformation coefficient, perform run-length coding and entropy coding after quantization, and finally form a feature code stream. Obviously, the feature processing unit does not need to select an independent reference frame, which significantly reduces the amount of calculation.

Furthermore, in an embodiment based on the present invention, the determination of the inter-frame prediction reference frame is at least based on the result of feature reference frame selection by the feature processing unit. The feature processing unit follows the instructions of the video compression processing unit (inter-frame or intra-frame coding, or indicates with frame types such as I, B, P frames, or provides a fixed encoding mode sequence such as IBBPP, etc., whether it is necessary to refresh the cache reference frame, etc.), Determines the encoding mode for a specific video frame. If a certain video frame is encoded by inter-frame prediction, then the feature processing module determines one or more feature reference frames for the video frame, which are also used for video compression inter-frame prediction reference frames.

One embodiment of the present invention is: comparing the current feature frame with several previous feature frames, and selecting one or more frames with the smallest sum of absolute values of differences as reference frames. For the B frame, it is also necessary to compare the current feature frame with several subsequent sampled/displayed feature frames, and select one or more frames with the smallest sum of absolute values of differences as the backward reference frame. In some cases, the video compression processing unit instructs to clear the feature frames before a certain frame in the cache, and the reference frame can only be selected from the feature frames after the frame. The feature reference frame determined by the feature processing unit is fed back to the video compression unit. The video compression unit directly uses it as a video compression reference frame or at least determines the video compression inter-frame prediction reference frame based on the feature reference frame determination result. A possible implementation is to combine several feature reference frames to form multiple video compression intra-frame reference frames of the same video frame. It can be seen that in this implementation manner, the selection of the reference frame for video compression inter-frame prediction becomes very simple, which is beneficial to reduce the computing load of the processor. In order to reduce the transmission bandwidth and compression space of eigenvalues, eigenvalues need to be compressed. A simple implementation is: compare the eigenvalues of other frames with the closest reference frame in front to form eigenvalue residuals. Perform transformation coding (such as DCT transformation) on the feature value, then quantize the transformation coefficient, perform run-length coding and entropy coding after quantization, and finally form a feature code stream.

The present invention associates the feature reference frame with the video compression inter-frame prediction reference frame, and can also reduce the overhead of the reference frame indication information. The reference frame indication information generally includes one or more reference frame numbers. In one embodiment, the reference frame indication information in the video coded stream is also applicable to the decoding of feature data, and the feature stream will not repeatedly transmit the reference frame indication information.

In one embodiment, the reference frame indication information in the feature coded code stream is also applicable to the decoding of the video compression code stream, and the video code stream will not repeatedly transmit the reference frame indication information. An indication information is required to indicate the above association relationship, including but not limited to: the characteristic reference frame is determined by the video compression reference frame or vice versa, the reference frame indication information is transmitted through the video compression stream or the feature stream. The indication information can be transmitted in any mode of the video compression code stream or the feature information code stream, so that the receiving and processing device can determine the correct reference frame.

Specifically, Fig. 3 shows the process of sending reference frame indication information in the characteristic encoding code stream according to the embodiment of the present application. The indication information indicates the association relationship between the characteristic reference frame and the inter-frame prediction reference frame, including but not limited to: The inter-prediction reference frame is based on the indication determined by the feature reference frame.

In the present invention, the same video processing unit can support multiple video analysis (including video retrieval) applications, and one video analysis application may include multiple video features. The compressed video data and video feature data are sent out through the data sending unit. Data transmission may be through a priority network, such as an Ethernet interface; it may also be a WIFI network or a cellular communication network such as 4G/5G.

Wherein, one, several or all of the compressed video data and feature data are sent to one or more data receiving devices. Data reception can be through wired or wireless (WIFI, cellular network, etc.), which have been described above and will not be repeated here. It is easy to know that different video features may have a unified encoding and decoding and/or feature matching manner, or may adopt independent encoding and decoding and/or feature matching manners.

Fig. 4 is a schematic diagram of a frame interpolation image processing device according to some embodiments of the present application. As shown in Figure 4, the device 400 includes:

The reference frame determining module 410 determines an inter-frame prediction reference frame based at least in part on the determined feature coding reference frame; and/or determines a feature coding reference frame based at least in part on the determined inter-frame prediction reference frame;

The reference frame indication information sending module 420 is configured to send the reference frame indication information in the video compression coded stream or the characteristic processing coded stream; wherein, the reference frame indication information is the characteristic reference frame and the inter-frame prediction reference The association relationship between frames, including reference frame selection instructions.

In some embodiments of the present application, the inter-frame prediction reference frame is replaced by a feature reference frame or formed by combining several feature reference frames.

In some embodiments of the present application, the feature reference frame is formed by replacing or combining several inter-frame prediction reference frames.

In some embodiments of the present application, the reference frame selection indication is an indication that the feature reference frame is determined based on the inter-frame prediction reference frame and an indication that the inter-frame prediction reference frame is determined based on the feature reference frame, including: whether the frame coding mode indication needs At least one of the reference frame, frame coding mode sequence, forward or backward reference frame and whether the reference frame buffer area needs to be cleared.

In some embodiments of the present application, the device also includes:

A reference frame feedback module 430, configured to feed back a reference frame selection result based on at least one reference frame selection indication.

In some embodiments of the present application, the reference frame determination module further includes:

A feature coding reference frame determination module 411, configured to determine a feature coding reference frame based on the reference frame selection result fed back from the determined inter-frame prediction reference frame;

The inter-frame prediction reference frame determination module 412 is configured to determine the inter-frame prediction reference frame based on the reference frame selection result fed back from the determined feature coding reference frame.

Referring to FIG. 5 , it is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 5, the electronic device 500 includes:

memory 530 and one or more processors 510;

Wherein, the memory 530 is connected in communication with the one or more processors 510, the memory 530 stores a program 532 executable by the one or more processors, and the program 532 is executed by the one or more processors. A plurality of processors 510 are executed, so that the one or more processors 510 execute the methods in the foregoing embodiments of the present application.

Specifically, the processor 510 and the memory 530 may be connected through a bus or in other ways. In FIG. 5 , connection through a bus 540 is taken as an example. The processor 510 may be a central processing unit (Central Processing Unit, CPU). The processor 510 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate array (Field-Programmable Gate Array, FPGA) or Other chips such as programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above-mentioned types of chips.

As a non-transitory computer-readable storage medium, the memory 530 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as the cascaded progressive network in the embodiment of the present application. The processor 510 executes various functional applications and data processing of the processor by running non-transitory software programs, programs 532 and functional modules stored in the memory 530 .

The memory 530 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by the processor 510 and the like. In addition, the memory 530 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 530 may optionally include memory located remotely from the processor 510, and these remote memories may be connected to the processor 510 through a network (such as through the communication interface 520). Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

An embodiment of the present application provides a computer-readable storage medium, wherein computer-executable instructions are stored in the computer-readable storage medium, and each step in the foregoing method embodiments is executed after the computer-executable instructions are executed.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described devices and modules can refer to the corresponding descriptions in the foregoing method and/or device embodiments, and details are not repeated here.

Although the subject matter described herein is presented in the general context of being executed in connection with the execution of operating systems and application programs on a computer system, those skilled in the art will recognize that other types of program modules can also be used to execute other programs. accomplish. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. , can also be used in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art or the technical solution, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. For example, typically, the technical solutions of the present application can be implemented and/or propagated through at least one general-purpose computer node 610 as shown in FIG. 6 . In FIG. 6, a general-purpose computer node 610 includes: a computer system/server 612, peripherals 614, and a display device 616; wherein, the computer system/server 612 includes a processing unit 620, an input/output interface 622, a network adapter 624, and The memory 630 usually implements data transmission through a bus inside; further, the memory 630 is usually composed of various storage devices, such as a RAM (Random Access Memory, random access memory) 632, a cache 634, and a storage system (generally composed of one or more large-capacity non- Volatile storage media) 636, etc.; the program 640 that realizes part or all of the functions of the technical solution of the present application is stored in the memory 630, usually in the form of multiple program modules 642.

The aforementioned computer-readable storage medium includes physically volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Indong medium. Computer-readable storage media specifically include, but are not limited to, U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), erasable programmable read-only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash or other solid-state memory technology, CD-ROM, Digital Versatile Disk (DVD), HD-DVD, Blue-Ray or other optical storage device, tape, disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and that can be accessed by a computer.

To sum up, the present application proposes a multi-table splicing method, device, electronic equipment and computer-readable storage medium thereof. The embodiment of this application controls the generation of multi-table features through the visualization of ontology view, and at the same time uses the meta-path to operate and control the transfer of features, effectively realizing the controllable aggregation of features during multi-table splicing, which can be flexible while reducing complexity. Efficiently control multi-table feature spaces.

It should be understood that the above specific implementation manners of the present application are only used to illustrate or explain the principle of the present application, but not to limit the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall fall within the protection scope of the present application. Furthermore, the claims appended to this application are intended to embrace all changes and modifications that come within the scope and metes and bounds of the appended claims, or equivalents of such scope and metes and bounds.

Claims (14)

A digital retina video processing method, characterized in that, comprising: determining an inter prediction reference frame based at least in part on the determined feature coding reference frame; and/or determining a feature encoding reference frame based at least in part on the determined inter prediction reference frame; Send reference frame indication information in the video compression coded stream or feature processing coded stream; wherein, the reference frame indication information is the association relationship between the feature reference frame and the inter-frame prediction reference frame, including the reference frame Select instructions. The method according to claim 1, wherein the inter-frame prediction reference frame is at least partially replaced by a feature reference frame or formed by combining several feature reference frames. The method according to claim 1, wherein the characteristic reference frame is at least partly replaced by an inter-frame prediction reference frame or formed by combining several inter-frame prediction reference frames. The method according to claim 1, wherein the reference frame selection indication is an indication that the feature reference frame is determined based on the inter-frame prediction reference frame and an indication that the inter-frame prediction reference frame is determined based on the feature reference frame, including: a frame coding mode indication At least one of whether a reference frame is needed, a sequence of frame coding modes, a forward or backward reference frame, and whether a reference frame buffer needs to be cleared. The method according to claim 1, further comprising: Feedback a reference frame selection result based on at least one reference frame selection indication. The method according to claim 1, further comprising: Based on the determined reference frame selection result fed back by the inter-frame prediction reference frame, determine a feature coding reference frame; and/or Determine the reference frame for inter-frame prediction based on the reference frame selection result fed back by the determined feature coding reference frame. A digital retinal video processing device, characterized in that it comprises: The reference frame determination module determines an inter-frame prediction reference frame based at least in part on the determined feature coding reference frame; and/or determines a feature coding reference frame based at least in part on the determined inter-frame prediction reference frame; A reference frame indication information sending module, configured to send reference frame indication information in a video compression coded stream or a feature-processed coded stream; wherein, the reference frame indication information is the feature reference frame and the inter-frame prediction reference frame The association relationship among them, including reference frame selection indication. The device according to claim 7, wherein the inter-frame prediction reference frame is formed by replacing a feature reference frame or combining several feature reference frames. The device according to claim 7, wherein the feature reference frame is formed by replacing the inter-frame prediction reference frame or by combining several inter-frame prediction reference frames. The device according to claim 7, wherein the reference frame selection indication is an indication that the feature reference frame is determined based on the inter-frame prediction reference frame and an indication that the inter-frame prediction reference frame is determined based on the feature reference frame, including: a frame coding mode indication At least one of whether a reference frame is needed, a sequence of frame coding modes, a forward or backward reference frame, and whether a reference frame buffer needs to be cleared. The device according to claim 7, wherein the device further comprises: A reference frame feedback module, configured to feed back a reference frame selection result based on at least one reference frame selection indication. The device according to claim 7, wherein the reference frame determining module further comprises: A feature coding reference frame determination module, configured to determine a feature coding reference frame based on the reference frame selection result fed back from the determined inter-frame prediction reference frame; The inter-frame prediction reference frame determination module is configured to determine the inter-frame prediction reference frame based on the reference frame selection result fed back from the determined feature coding reference frame. An electronic device, characterized in that it comprises: memory and one or more processors; Wherein, the memory is connected in communication with the one or more processors, the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors , the electronic device is used to implement the method according to any one of claims 1-6. A computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a computing device, they can be used to implement the method according to any one of claims 1-6.

Images