CN114187403A

CN114187403A - Method, system, electronic device and storage medium for repetitive texture filtering

Info

Publication number: CN114187403A
Application number: CN202111328298.5A
Authority: CN
Inventors: 王成; 丛林
Original assignee: Hangzhou Yixian Advanced Technology Co ltd
Current assignee: Hangzhou Yixian Advanced Technology Co ltd
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2022-03-15
Anticipated expiration: 2041-11-10
Also published as: CN114187403B

Abstract

The present application relates to a method, system, electronic device and storage medium for repetitive texture filtering, wherein the method includes: acquiring an image matching map of a video sequence, performing FH verification on the matching map, obtaining an image containing repeated textures, and The image containing repeated textures is subjected to co-view verification, wherein the repeated textures include continuous repeated textures and a single plane texture; then, hierarchical clustering and convex polygon fitting are performed on the repeated texture images after the co-view verification to obtain Convex polygon seed image containing repeated texture features; finally, depth progressive transfer is performed on the convex polygon seed image containing repeated texture features, and feature matching pairs that fall into convex polygons in the image are filtered to obtain the filtered image matching map . Through the present application, the mismatched images of repeated textures are filtered, and the accuracy of mapping is improved.

Description

Method, system, electronic device and storage medium for repetitive texture filtering

Technical Field

The present application relates to the field of three-dimensional reconstruction techniques, and more particularly, to methods, systems, electronic devices, and storage media for repetitive texture filtering.

Background

In the SFM reconstruction process, the first step is to extract the characteristics of each image, and the second step is to perform characteristic matching and geometric verification of two views on any two images; after the two steps are finished, the images are taken as nodes, the matching relation between the images is edge, and a graph formed by the images is an image matching graph. However, repeated texture feature matching pairs may occur during the reconstruction process, and in order to avoid this, such mismatching maps need to be filtered and deleted.

In the related art, in filtering the mismatch, on one hand, a view graph (orientation graph) is mainly used, that is, a relative orientation between two images is used as an undirected edge, all the images are used as nodes, the graph is constructed, and finally, the mismatch is filtered by using the inconsistency of transmission observation. If there are image matching pairs: A-B, B-C and A-C, assuming that A-C is the matching pair with the highest confidence coefficient, obtaining A-C through A-B-C, calculating the difference between A-C and A-C, and filtering out the error matching A-B and B-C if the difference is too large. However, in the view graph algorithm, it is assumed that the relative orientation view between two images is known, and when the view between two images is solved, it is necessary to know the H matrix or the E matrix (the E matrix is obtained by solving the F matrix in the case of acquiring camera parameters) and then decompose the matrix to obtain the view. However, in a general SFM scene, camera parameters are not known, so an E matrix cannot be obtained, and it is impossible that all image matching pairs are H matrices, so that such algorithms cannot be directly applied to an actual scene. On the other hand, the wrong matching pairs are filtered based on the global feature match algorithm, however, the algorithm may mistakenly delete the matching pairs with large visual angle variation, and still cannot filter the scene with repeated texture. In addition, a track length algorithm can be adopted to carry out weight accumulation on the feature points of the image, and the feature points are sorted according to the weight and registered in sequence; however, this method essentially avoids the mismatch, does not really filter the mismatch, and still cannot solve the problem for the repetitive texture scene.

At present, no effective solution is provided for the problems of lack of repeated texture detection and filtering and lack of practicability in filtering image mismatching occurring in the three-dimensional reconstruction process in the related art.

Disclosure of Invention

The embodiment of the application provides a method, a system, an electronic device and a storage medium for filtering repeated textures, so as to solve the problems of lack of repeated texture detection and filtering and lack of practicability in filtering image mismatching occurring in a three-dimensional reconstruction process in the related art.

In a first aspect, an embodiment of the present application provides a method for repetitive texture filtering, where the method includes:

acquiring an image matching image of a video sequence, performing FH (frequency hopping) verification on the matching image to obtain an image containing repeated textures, and performing common-view verification on the image containing the repeated textures, wherein the repeated textures comprise continuous repeated textures and single plane textures;

performing hierarchical clustering and convex polygon fitting on the repeated texture image subjected to the common-view check to obtain a convex polygon seed image containing repeated texture features;

and carrying out depth progressive transmission on the convex polygon seed image containing the repeated texture features, and filtering the feature matching pairs falling into the convex polygons in the image to obtain a filtered image matching image.

In some of these embodiments, an image matching graph of a video sequence is obtained, performing a FH check on the matching graph comprising:

matching images in a video sequence to obtain different types of matching pairs, and performing FH (frequency hopping) verification on the different types of matching pairs, wherein H verification is performed on the matching relationship of two images in one plane, and F verification is performed on the matching relationship of a plurality of objects which are not in the same plane in three dimensions.

In some embodiments, after FH checking the different types of matching pairs, the method includes:

when the matching pairs of different types can pass the H check or the F check, the images in the video sequence are images without repeated textures;

when the number of matching pairs which can pass the H check and the number of matching pairs which can pass the F check are the same among the different types of matching pairs, determining whether images in the video sequence contain repeated textures;

and when the number of the matching pairs which can pass the H check is larger than that of the matching pairs which can pass the F check, or the number of the matching pairs which can pass the H check is smaller than that of the matching pairs which can pass the F check, the image is an image containing repeated textures.

In some of these embodiments, co-viewing the image containing the repeated texture comprises:

and constructing the feature map of the image containing the repeated textures, and reserving the image with the common visual number of the feature points being more than or equal to a first preset value.

In some embodiments, the performing hierarchical clustering and convex polygon fitting on the repeated texture images after the common view check comprises:

performing hierarchical clustering on the feature points of which the number is greater than or equal to a second preset value in the repeated texture image after the common-view verification to obtain a feature point cluster;

and deleting the feature point clusters with the number of the feature points smaller than the third preset value, performing convex polygon fitting on the reserved feature point clusters, and outputting to obtain the convex polygon seed image containing the repeated texture features.

In a second aspect, an embodiment of the present application provides a system for repetitive texture filtering, the system including:

the verification module is used for acquiring an image matching image of a video sequence, performing FH (frequency hopping) verification on the matching image to obtain an image containing repeated textures, and performing common-view verification on the image containing the repeated textures, wherein the repeated textures comprise continuous repeated textures and single plane textures;

the cluster fitting module is used for performing hierarchical clustering and convex polygon fitting on the repeated texture image subjected to common view verification to obtain a convex polygon seed image containing repeated texture features;

and the transfer filtering module is used for performing depth progressive transfer on the convex polygon seed image containing the repeated texture features and filtering the feature matching pairs falling into the convex polygons in the image to obtain a filtered image matching image.

In some embodiments, the verification module is further configured to match images in a video sequence to obtain different types of matching pairs, and perform FH verification on the different types of matching pairs, where H verification is performed on a matching relationship between two images in one plane, and F verification is performed on a matching relationship between a plurality of objects in three dimensions that are not in one plane.

In some of these embodiments, after FH checks on the matching pairs of the different types,

the checking module is further configured to, when the matching pairs of different types can both pass the H check or can both pass the F check, determine that an image in the video sequence is an image that does not include a repeated texture;

In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor, when executing the computer program, implements the method of repetitive texture filtering according to the first aspect.

In a fourth aspect, embodiments of the present application provide a storage medium having stored thereon a computer program, which when executed by a processor, implements the method of repetitive texture filtering as described in the first aspect above.

Compared with the related art, the method for filtering the repeated texture, provided by the embodiment of the application, comprises the steps of obtaining an image matching graph of a video sequence, carrying out FH (frequency hopping) verification on the matching graph to obtain an image containing the repeated texture, and carrying out common-view verification on the image containing the repeated texture, wherein the repeated texture comprises continuous repeated textures and single plane textures; then, carrying out hierarchical clustering and convex polygon fitting on the repeated texture image subjected to the common-view check to obtain a convex polygon seed image containing repeated texture features; and finally, carrying out depth progressive transmission on the convex polygon seed image containing the repeated texture features, and filtering the feature matching pairs falling into the convex polygons in the image to obtain a filtered image matching image.

According to the method, a plurality of video sequences are adopted for three-dimensional reconstruction, time sequence prior information exists among the video sequences, a potential repeated texture image can be obtained through verification by using a FH (frequency hopping) verification algorithm, a convex polygon of repeated textures is obtained by using a clustering and convex polygon fitting algorithm, and finally, repeated texture feature matching pairs are deleted, so that image error registration is avoided, and the success of image construction is guaranteed. The problems that repeated texture detection and filtering are lacked and practicability is lacked when image mismatching occurring in the three-dimensional reconstruction process is filtered are solved, the mismatching image of the repeated texture is filtered, and the accuracy of image construction is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a flow diagram of a method of repetitive texture filtering according to an embodiment of the present application;

FIG. 2 is a block diagram of a system for repetitive texture filtering according to an embodiment of the present application;

fig. 3 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

The present embodiment provides a method for repetitive texture filtering, and fig. 1 is a flowchart of a method for repetitive texture filtering according to an embodiment of the present application, and as shown in fig. 1, the flowchart includes the following steps:

step S101, obtaining an image matching image of a video sequence, performing FH (frequency hopping) verification on the matching image to obtain an image containing repeated textures, and performing common-view verification on the image containing the repeated textures, wherein the repeated textures comprise continuous repeated textures and single plane textures;

preferably, after acquiring a video sequence image, the present embodiment matches images in the video sequence to obtain different types of matching pairs, for example, acquire an image t in a certain video sequence, match the image t to obtain four different types of matching pairs: t-2 and t, t-1 and t, t +2 and t; and then, performing FH (frequency hopping) verification on the different types of matching pairs, wherein H verification is performed on the matching relationship of the two images in one plane, and F verification is performed on the matching relationship of a plurality of objects which are not in one plane in three dimensions.

Preferably, when the matching pairs of different types can both pass the H check or can both pass the F check, the images in the video sequence are images that do not contain repeated textures. For example, when the 4 matching pairs are verified to obtain 4F or 4H, the image t is "good", that is, the image t is an image containing no repeated texture;

when the number of matching pairs that can pass H check is the same as the number of matching pairs that can pass F check among different types of matching pairs, it cannot be determined whether an image in a video sequence contains a repetitive texture. For example, when the 4 matching pairs are checked to obtain 2F2H, the image t is "unknown", that is, it cannot be determined whether the image t contains repeated texture;

when the number of matching pairs which can pass the H check is larger than that of matching pairs which can pass the F check, or the number of matching pairs which can pass the H check is smaller than that of matching pairs which can pass the F check, the image is an image containing repeated textures. For example, when the 4 matching pairs are verified to obtain 3F1H or 3H1F, the image t is "bad", that is, the image t is an image containing repeated textures.

It should be noted that the repeated texture found by the FH check described above includes a continuous repeated texture and a single plane texture with a null. The repeated texture in the image is found through FH verification, so that the matching risk can be reduced, and the mismatching in the image construction can be reduced. For example, a planar poster is shot, after an image of the poster is obtained, FH verification is carried out on the image, if a continuous F condition occurs, the poster may have few features, or the poster may have a particularly large number of structural textures, and in this case, the single plane texture of the lone zero has no mismatching hazard. However, when checking for FH, if the case of 3F1H occurs, it is likely that the field of view is not large enough when shooting a poster, and most of the area is the poster; or the view is large enough, but the structural texture of the poster is poor, so that the feature point of the poster accounts for a large proportion, at this time, the situation that matching connection is carried out by one poster alone can occur, the matching relation is high in risk, and if the scene has the same poster at other positions, mismatching which seriously affects the correctness of drawing can occur.

Further, after the image containing the repeated texture is obtained through the steps, the image containing the repeated texture is subjected to common-view verification. Specifically, a feature map of an image containing repeated textures is constructed, wherein feature points are nodes, and a feature point matching pair conforming to geometric matching of two views is edge; only images with the common-view number of the feature points being more than or equal to a first preset value are reserved, and repeated textures with very high probability can be extracted through common-view verification;

step S102, carrying out hierarchical clustering and convex polygon fitting on the repeated texture image subjected to common view verification to obtain a convex polygon seed image containing repeated texture features;

preferably, in this embodiment, first, hierarchical clustering is performed on feature points, the number of which is greater than or equal to a second preset value, in the repeated texture image after the common view verification, so as to obtain a feature point cluster, where a clustering threshold is ratio _ min (image _ height, image _ width);

then, deleting the feature point clusters with the number of the feature points smaller than a third preset value, and performing convex polygon fitting on the reserved feature point clusters;

and finally, outputting to obtain the convex polygon seed image containing the repeated texture features.

By the hierarchical clustering algorithm and convex polygon fitting, repeated textures are expressed by regions, and the filtering effect is enhanced;

and step S103, carrying out depth progressive transmission on the convex polygon seed image containing the repeated texture features, and filtering the feature matching pairs falling into the convex polygons in the image to obtain a filtered image matching image.

In this embodiment, the convex polygon seed image including the repeated texture features is subjected to depth progressive transfer, and is transferred to the inside with the depth of 0 to obtain a seed image including more complete convex polygon seed images, and then transferred from the depth of 0 to the depth of 1 to obtain a seed image including the depth of 1 of the convex polygon, and the seed image including the depth of N of the convex polygon is obtained by sequential recursion and transferred from the depth of N-1 to the depth of N. All images containing repeated textures in the image matching graph can be found through the progressive transmission;

further, all the images obtained after the depth transmission are filtered, and the feature matching pairs falling into the convex polygons are filtered to obtain the filtered image matching image, so that the image matching image is ensured to have no mismatching.

Through the steps S101 to S103, in this embodiment, a plurality of video sequences are used for three-dimensional reconstruction, timing sequence prior information exists between each video sequence, a potential repetitive texture image can be obtained through verification using an FH verification algorithm, a convex polygon with repetitive textures is obtained through a clustering and convex polygon fitting algorithm, and finally, a repetitive texture feature matching pair is deleted, thereby avoiding image error registration and ensuring that image construction is successful. The problems that repeated texture detection and filtering are lacked and practicability is lacked when image mismatching occurring in the three-dimensional reconstruction process is filtered are solved, the mismatching image of the repeated texture is filtered, and the accuracy of image construction is improved.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

The present embodiment further provides a system for filtering repeated textures, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the system that has been already described is omitted. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 2 is a block diagram of a system for repetitive texture filtering according to an embodiment of the present application, and as shown in fig. 2, the system includes a checking module 21, a cluster fitting module 22, and a transfer filtering module 23:

the verification module 21 is configured to obtain an image matching graph of a video sequence, perform FH verification on the matching graph to obtain an image including a repeated texture, and perform common-view verification on the image including the repeated texture, where the repeated texture includes a continuous repeated texture and a single planar texture; the cluster fitting module 22 is used for performing hierarchical clustering and convex polygon fitting on the repeated texture image subjected to the common-view check to obtain a convex polygon seed image containing repeated texture features; and the transfer filtering module 23 is configured to perform depth progressive transfer on the convex polygon seed image including the repeated texture features, and filter a feature matching pair falling in the convex polygon in the image to obtain a filtered image matching image.

Through the system, the three-dimensional reconstruction is carried out by adopting a plurality of video sequences, time sequence prior information exists among the video sequences, the checking module 21 can check to obtain a potential repeated texture image by using an FH (frequency hopping) checking algorithm, the cluster fitting module 22 obtains a convex polygon of a repeated texture by using a clustering and convex polygon fitting algorithm, and finally, the transmission filtering module 23 deletes a repeated texture feature matching pair, thereby avoiding image error registration and ensuring that the image construction is successful. The problems that repeated texture detection and filtering are lacked and practicability is lacked when image mismatching occurring in the three-dimensional reconstruction process is filtered are solved, the mismatching image of the repeated texture is filtered, and the accuracy of image construction is improved.

It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.

Note that each of the modules may be a functional module or a program module, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.

The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

In addition, in combination with the method of repeated texture filtering in the above embodiments, the present application embodiment may provide a storage medium to implement. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the methods of repetitive texture filtering in the above embodiments.

In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of repetitive texture filtering. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

In one embodiment, fig. 3 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 3, there is provided an electronic device, which may be a server, and its internal structure diagram may be as shown in fig. 3. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used for providing computing and control capabilities, the network interface is used for communicating with an external terminal through a network connection, the internal memory is used for providing an environment for an operating system and the running of a computer program, the computer program is executed by the processor to realize a repeated texture filtering method, and the database is used for storing data.

Those skilled in the art will appreciate that the architecture shown in fig. 3 is a block diagram of only a portion of the architecture associated with the subject application, and does not constitute a limitation on the electronic devices to which the subject application may be applied, and that a particular electronic device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method of repetitive texture filtering, the method comprising:

2. The method according to claim 1, characterized in that a matching graph of images of a video sequence is obtained, performing a FH check on said matching graph comprising:

3. The method according to claim 2, wherein after FH checking said matching pairs of different types, said method comprises:

4. The method of claim 1, wherein performing a co-view check on the image containing the repeated texture comprises:

5. The method of claim 1, wherein performing hierarchical clustering and convex polygon fitting on the co-view verified repetitive texture image comprises:

6. A system for repetitive texture filtering, the system comprising:

7. The system of claim 6,

the checking module is further configured to match images in the video sequence to obtain different types of matching pairs, and perform FH checking on the different types of matching pairs, where H checking is to check a matching relationship between two images in one plane, and F checking is to check a matching relationship between a plurality of objects in three dimensions that are not in the same plane.

8. The system of claim 7, wherein after FH checks on the different types of matching pairs,

9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is configured to execute the computer program to perform the method of repetitive texture filtering of any one of claims 1 to 5.

10. A storage medium having stored thereon a computer program, wherein the computer program is arranged to perform the method of repetitive texture filtering of any one of claims 1 to 5 when executed.