KR102366172B1 - Method and apparatus for identifying image mathcihng - Google Patents

Abstract

The present disclosure relates to a method for identifying whether an image is matched, and an image matching apparatus for performing a method for identifying whether an image is matched. According to an embodiment, a method of identifying whether an image is matched may include: acquiring a first image patch and a second image patch from lights of different wavelength bands; converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band; extracting a feature vector from the transformed first image patch and the transformed second image patch; and inputting the extracted feature vector into a first neural network model to identify whether the first image patch and the second image patch match; may include

Description

METHOD AND APPARATUS FOR IDENTIFYING IMAGE MATHCIHNG}

The present disclosure relates to a method and apparatus for identifying whether an image is matched. More particularly, it relates to a method and apparatus for identifying whether images are matched using an artificial neural network.

An artificial neural network may refer to a computing device or a method performed by a computing device to implement interconnected sets of artificial neurons. As an example of the artificial neural network, a deep neural network or deep learning may have a multi-layer structure, and each of the layers may be learned according to a plurality of data.

Recently, as the development of artificial neural network technology is activated, a technology for automatically recognizing an image is being actively studied in the field of artificial intelligence. In particular, a cross-spectrum image matching technique for accurately discriminating whether or not matching between multi-spectrum images using light of different wavelength ranges is actively being studied.

However, in a system for processing multi-spectral images using light of different wavelength regions, since each image shows a fairly large difference in image characteristics, there is a problem in that it is difficult to identify whether images of different spectra are matched.

Therefore, there is a demand for technology development for effectively identifying the matching of different spectral images.

Korean Patent Publication No. 2019-0097640

According to an embodiment, a method and an image matching apparatus for identifying whether an image is matched may be provided.

Also, according to an embodiment, a method and an image matching apparatus for identifying whether an image is matched by using at least one neural network model may be provided.

According to an embodiment of the present disclosure for achieving the above-described technical problem, the method comprising: obtaining a first image patch and a second image patch from lights of different wavelength bands; converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band; extracting a feature vector from the transformed first image patch and the transformed second image patch; and inputting the extracted feature vector into a first neural network model to identify whether the first image patch and the second image patch match; A method of identifying whether an image including a match may be provided.

In addition, according to an embodiment, extracting a feature vector from the obtained first image patch and the obtained second image patch; The method further includes, wherein the step of identifying whether the match is performed comprises further inputting a feature vector extracted from each of the obtained first image patch and the obtained second image patch into a first neural network model, whereby the first image patch and There may be provided a method of identifying whether an image is matched, characterized in that it is identified whether the second image patch is matched.

In addition, according to another embodiment of the present disclosure for solving the above technical problem, a memory for storing one instruction; and at least one processor executing the one or more instructions. including, wherein the at least one processor obtains a first image patch and a second image patch from lights of different wavelength bands by executing the one or more instructions, and pre-processes the first image patch and the second image patch. By converting to represent image characteristics of a set wavelength band, extracting a feature vector from the converted first image patch and the converted second image patch, and inputting the extracted feature vector to a first neural network model, the first An image matching apparatus may be provided that identifies whether the first image patch and the second image patch are matched.

In addition, according to another embodiment of the present disclosure for solving the above technical problem, the method comprising: obtaining a first image patch and a second image patch from lights of different wavelength bands; converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band; extracting a feature vector from the transformed first image patch and the transformed second image patch; and inputting the extracted feature vector into a first neural network model to identify whether the first image patch and the second image patch match; A computer-readable recording medium in which a program for executing a method of identifying whether an image is matched, including a computer-readable recording medium, may be provided.

1 is a diagram schematically illustrating a method of identifying whether images are matched, according to an exemplary embodiment.
2 is a flowchart illustrating a method of identifying whether an image is matched by an image matching apparatus according to an exemplary embodiment.
3 is a diagram for describing a method of identifying whether an image is matched by an image matching apparatus according to an exemplary embodiment.
4 is a flowchart illustrating a method of identifying whether an image is matched by an image matching apparatus according to another exemplary embodiment.
5 is a diagram for explaining a method of identifying whether an image is matched by an image matching apparatus according to another exemplary embodiment.
6 is a diagram for describing a structure of an image matching apparatus according to an exemplary embodiment.
7 is a diagram for describing a structure of a neural network model used by an image matching apparatus to convert an image patch, according to an embodiment.
8 is a diagram for explaining the structure of a neural network model used by an image matching apparatus to extract a feature vector from an image patch according to an embodiment.
9 is a diagram for explaining a result of converting an image patch using a neural network model by an image matching apparatus according to an exemplary embodiment.
10 is a block diagram of an image matching apparatus according to an exemplary embodiment.
11 is a block diagram of a server connected to an image matching apparatus according to an exemplary embodiment.
12 is a diagram for comparing performance of a trained neural network model based on a pair of a visible ray image patch and a near-infrared image patch, according to an embodiment.
13 is a diagram for comparing the performance of a neural network model learned based on a pair of a visible light image patch and a thermal image patch according to an embodiment.
14 is a diagram for comparing the performance of a neural network model trained based on a pair of RGB image patches and near-infrared image patches, according to an embodiment.
15 is a diagram for comparing the performance of a neural network model of a general Siamese network structure and a neural network model according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the present disclosure have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, with reference to the accompanying drawings, the embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram schematically illustrating a method of identifying whether images are matched, according to an exemplary embodiment.

According to an embodiment, the image matching apparatus 1000 may include an artificial neural network model 110 . According to an embodiment, the artificial neural network model may include at least one neural network model. The image matching apparatus 1000 may acquire an image patch pair, and identify whether the acquired image patch pair matches by using an artificial neural network model.

According to an embodiment, the image matching apparatus 1000 may extract a feature vector from the image patch to identify whether the image patch pair matches each other, and identify whether the image patch matches each other based on the extracted feature vector. there is. The image matching apparatus 1000 may identify whether the image pairs match based on whether the image patches included in each image pair match. The image matching device 1000 is equipped with an AI program for processing an image using a neural network model, and includes a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a media player, a server, a micro server, including an image shooting function, It may be, but is not limited to, any other mobile or non-mobile computing device.

According to an embodiment, the artificial neural network model used by the image matching apparatus 1000 may refer to a computing system focusing on a biological neural network. Unlike classical algorithms that perform tasks according to predefined conditions, artificial neural networks can learn to perform tasks by considering a large number of samples. An artificial neural network may have a structure in which artificial neurons are connected, and a connection between neurons may be referred to as a synapse. A neuron may process a received signal, and may transmit the processed signal to another neuron through a synapse. The output of a neuron may be referred to as activation, and a neuron and/or synapse may have a weight that can be varied, and the influence of a signal processed by the neuron may increase or decrease depending on the weight. .

For example, the artificial neural network may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values (weights), and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial neural network.

For example, a plurality of weights may be modified and updated so that a loss value or a cost value obtained from the neural network model during the learning process is reduced or minimized. The neural network model according to the present disclosure may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), Restricted (RBM) Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), or deep Q-Networks, but is not limited to the above-described example.

According to an embodiment, the image matching apparatus 1000 may acquire the first image 122 and the second image 142 . According to an embodiment, the first image may be an image corresponding to the first domain 120 , and the second image 142 may be an image corresponding to the second domain 140 . For example, the first domain 120 and the second domain 140 may correspond to a continuous or discontinuous wavelength band used to generate an image. The first domain, the second domain, the third domain, or the fourth domain described herein may be a criterion for classifying a light spectrum region based on a wavelength length of light.

For example, the first domain 120 may represent a spectrum of light belonging to a visible ray region (eg, a region in which the wavelength of light is 380 nm to 780 nm), and the second domain 140 may indicate a near-infrared region (eg, a wavelength of light). It can represent the spectrum of light belonging to this 0.75um ~ 1um region). According to another embodiment, the first domain 120 is a wavelength of the RGB region, and may include a spectrum of light belonging to a region having a wavelength of 400 nm to 500 nm, 450 nm to 630 nm, and 500 to 650 nm, but is limited thereto. It goes without saying that it may be arbitrarily set to indicate different wavelength regions.

According to an embodiment, the first domain and the second domain to which the first image patch and the second image patch used by the image matching apparatus 1000 belong are multi-spectrum spectrums of lights belonging to different wavelength bands, respectively. ) can also be shown.

According to an embodiment, the image matching apparatus 1000 may determine the first image patch 124 from the first image 122 and determine the second image patch 144 from the second image 142 . For example, the image matching apparatus 1000 extracts a first feature point from the first image based on the pixel value of the first image 122 and generates a partial region of the first image including the extracted first feature point. It can be determined by 1 image patch. In addition, the image matching apparatus 1000 extracts a second feature point from the second image based on the pixel value of the second image 142 , and converts a partial region of the second image including the extracted second feature point into the second image. It can be determined by patch 144 . The first image patch determined by the image matching apparatus 1000 may belong to the first domain, and the second image patch may belong to the second domain.

The first image patch and the second image patch acquired by the image matching apparatus 1000 belong to the same domain as each of the first image and the second image, and the first image patch has the same image characteristics as the wavelength band of the first image. With the image characteristics, the second image patch may have the same image characteristics as those of the wavelength band of the second image.

According to an embodiment, the image matching apparatus 1000 may perform the function of transforming the domain of the obtained image patch (102). For example, the image matching apparatus 1000 may obtain the first image patch 124 and the second image patch 144 , and may transform domains to which the obtained first image patch and the second image patch belong. For example, the image matching apparatus 1000 may transform the first image patch so that image characteristics belonging to the second domain appear by transforming the domain of the first image patch, and by transforming the domain of the patch of the second image The second image patch may be transformed so that image characteristics belonging to the first domain appear. The process of converting the domain of the image patch by the image matching apparatus 1000 may correspond to the converting of pixel values in the image patch to indicate image characteristics of a wavelength band corresponding to the target domain.

For example, the image matching apparatus 1000 may convert the first image patch so that the first image patch generated by the light of the first wavelength band exhibits image characteristics of the second wavelength band, and the second wavelength band The second image patch may be converted so that the second image patch generated by the light of the band exhibits image characteristics of the first wavelength band. According to an embodiment, the image characteristic of the second wavelength band may mean the intensity, level, pattern, etc. of pixel values in images generated based on light belonging to the second wavelength band, and the image of the first wavelength band The characteristic may mean intensity, level pattern, etc. of pixel values in images generated based on light belonging to the first wavelength band.

The image matching apparatus 1000 may perform a feature vector extraction 104 function using a predetermined image patch. For example, the image matching apparatus 1000 may extract a feature vector from an acquired image patch or an image patch converted to another domain. In addition, the image matching apparatus 1000 may perform the matching identification 106 function of determining whether an image patch pair is matched by using a feature vector extracted from each image patch. For example, the image matching apparatus 1000 may obtain a matching result 152 regarding whether an image patch pair is matched based on the extracted feature vector. For example, the matching result obtained by the image matching apparatus 1000 may include a matching result expressed in a binary number, and may further include information on a probability value for a matching degree of an image patch pair.

According to an embodiment, the image matching apparatus 1000 may identify whether the entire image including each image patch matches, based on a result of identifying whether the image patch pairs match. Matching according to the present disclosure may indicate the similarity of positions of pixels in an image, pixel values at the corresponding positions, and variations in pixel values in a predetermined region.

In general, since images generated from light having spectra of different wavelength bands have a significant difference in both pixel level and intensity, matching of images belonging to spectra of different wavelength bands A convolution-based neural network model of a Siamese network structure can be used to accurately identify whether or not However, since the convolution-based neural network model of the general Siamese network structure does not accurately process the difference in pixel intensity and pixel level between images generated by light of different wavelength bands, the spectrum of different wavelength bands There was a limit in identifying whether the images belonging to .

However, the image matching apparatus 1000 according to the present disclosure does not extract a feature vector directly from image patches, but converts the domains of the image patches, and then uses a neural network model of a dual siamese network structure, By extracting feature vectors from the image patches having the converted domain and the image patches having the unconverted domain, it is possible to accurately identify whether images generated from light having different wavelength bands match.

According to an embodiment, the image matching apparatus 1000 may be connected to the server 2000 . According to an embodiment, the image matching apparatus 1000 may work with the server 2000 to identify whether an image patch pair is matched. The server 2000 may include other computing devices capable of transmitting/receiving data to and from the image matching device by being connected to the image matching device 1000 through a network. According to an embodiment, the server 2000 includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, It may be connected to the image matching apparatus 1000 through a satellite communication network and a combination thereof. In addition, according to an embodiment, the server 2000 is a data communication network in a comprehensive sense that allows each network constituent entity (eg, an image matching device and a server) shown in FIG. 1 to communicate smoothly with each other, and is a wired Internet. , wireless Internet and mobile wireless communication networks themselves.

2 is a flowchart illustrating a method of identifying whether an image is matched by an image matching apparatus according to an exemplary embodiment.

In S210 , the image matching apparatus 1000 may obtain a first image patch and a second image patch from lights of different wavelength bands. For example, the image matching apparatus 1000 may include a first image (eg, a visible ray image) generated by lights belonging to a visible ray wavelength region and a second image (eg, a near-infrared image) generated by lights belonging to a near-infrared wavelength region. , extracting feature points based on the amount of change in pixel values in each acquired image, and obtaining a first image patch and a second image patch from each of the first image and the second image based on the extracted feature points. can

According to another embodiment, the image matching apparatus 1000 acquires a first image from a visible ray camera that captures a visible ray image connected to the image matching apparatus, and acquires a second image from a near infrared camera that captures a near infrared image In addition, the first image patch and the second image patch may be respectively obtained from the obtained first image and the second image, respectively.

In S220 , the image matching apparatus 1000 may convert the first image patch and the second image patch to represent image characteristics of a preset wavelength band. For example, the image matching apparatus 1000 converts the first image patch so that the first image patch generated by the light of the first wavelength band exhibits image characteristics of the second wavelength band, and The second image patch may be converted so that the second image patch generated by the light exhibits image characteristics of the first wavelength band. That is, the image matching apparatus 1000 may convert the domain of the first image patch from the first domain to the second domain, and may convert the domain of the second image patch from the second domain to the first domain.

However, according to another embodiment, the image matching apparatus 1000 converts the first image patch so that the first image patch exhibits image characteristics of the third wavelength band, and the second image patch is an image of the third wavelength band The second image patch may be transformed to exhibit the characteristics. That is, the image matching apparatus 1000 may convert the domain of the first image patch from the first domain to the third domain, and may convert the domain of the second image patch from the second domain to the third domain. As described above, the operation of the image matching apparatus 1000 converting the domain of the image patch may correspond to the operation of converting the pixel values of the image patch to indicate image characteristics of the target wavelength band corresponding to the target domain. there is.

According to an embodiment, the image matching apparatus 1000 may convert the domain of the image patch by using a neural network model previously learned to convert the domain to which the image patch belongs to the target domain. For example, when the first image patch representing the image property of the first wavelength band is input, the image matching apparatus 1000 generates a second neural network model trained to output the image patch representing the image property of the second wavelength band. It can be used to transform the first image patch. Also, when the second image patch representing the image property of the second wavelength band is input, the image matching apparatus 1000 uses the third neural network model trained to output the image patch representing the image property of the first wavelength band. It is also possible to transform the second image patch.

In S230 , the image matching apparatus 1000 may extract a feature vector from the converted first image patch and the converted second image patch. For example, the image matching apparatus 1000 extracts a first feature vector from a first image patch transformed using a pre-trained fourth neural network model, and a second transformed second neural network model using a pre-trained fifth neural network model. A fourth feature vector may be extracted from the image patch. According to an embodiment, the fourth neural network model and the fifth neural network model may be a convolution-based neural network model that is pre-trained in order to extract a feature vector from an image patch belonging to a predetermined domain.

In general, as a feature based on the similarity of pixel values in an image patch to extract a feature vector from an image, a SIFT, SURF, FAST feature extraction algorithm, etc. may be used. However, these feature extraction algorithms have limitations in the change of pixel intensity and texture inconsistency in a multi-spectral image. In order to overcome these limitations, multi-spectral SIFT and methods for analyzing statistical image characteristics of RGB-near-infrared pairs have been disclosed, but there is still a limit to accurate feature extraction for multi-spectral image matching.

However, as will be described later, the image matching apparatus 1000 according to the present disclosure does not extract the feature vector directly from the image patch, but transforms the domain of the image patch, and the feature vector extracted from the image patch of the converted domain and By using all the feature vectors extracted from image patches in the untransformed domain, matching between different spectral images can be accurately performed.

In S240 , the image matching apparatus 1000 determines whether the first image patch and the second image patch are matched by inputting feature vectors respectively extracted from the transformed first image patch and the transformed second image patch to the first neural network model. can be identified. For example, the image matching apparatus 1000 includes, as an element, the square of a distance value for each element of a first feature vector extracted from the transformed first image patch and a fourth feature vector extracted from the transformed second image patch. The polymerization vector may be generated by determining a difference vector to be used and concatenating the determined difference vector, the first feature vector, and the fourth feature vector. The image matching apparatus 1000 may input the polymerization vector to the pre-trained first neural network model, and identify whether the first image patch and the second image patch match based on an output value of the first neural network model.

3 is a diagram for describing a method of identifying whether an image is matched by an image matching apparatus according to an exemplary embodiment.

Referring to FIG. 3 , a process in which the image matching apparatus 1000 according to an exemplary embodiment identifies whether an image is matched will be described in more detail. As described above, the image matching apparatus 1000 may acquire the first image patch 302 belonging to the first domain and the second image patch 312 belonging to the second domain. According to an embodiment, the first domain and the second domain may mean a spectrum of light belonging to a visible ray region, and the second domain may mean a spectrum of light belonging to a near-infrared region. However, the present invention is not limited thereto, and the first domain may indicate a spectrum of light belonging to a visible ray region and the second domain may indicate a spectrum of light belonging to a thermal infrared region.

The image matching apparatus 1000 may convert the first image patch 302 and the second image patch 312 to represent image characteristics of a preset wavelength band. For example, the image matching apparatus 1000 converts the first image patch so that the first image patch generated by the light of the first wavelength band exhibits image characteristics of the second wavelength band, and The second image patch may be converted so that the second image patch generated by the light exhibits image characteristics of the first wavelength band. However, according to another embodiment, the image matching apparatus 1000 converts the first image patch so that the first image patch exhibits image characteristics of the third wavelength band, and the second image patch converts the second image patch into the image characteristic of the third wavelength band. As described above with reference to FIG. 2, the second image patch can be transformed to represent the image characteristics.

According to an embodiment, the converting of the image patch to indicate the image characteristic of the target wavelength band by the image matching apparatus 1000 may include converting pixel values of the image patch to convert the current domain of the image patch into the target domain. It can correspond to the action.

Also, according to an embodiment, the image matching apparatus 1000 may convert the domain of the image patch by using a neural network model previously learned to convert the domain to which the image patch belongs to the target domain. For example, the image matching apparatus 1000 receives a second neural network model ( 304) may be used to transform the first image patch. Also, when the second image patch representing the image property of the second wavelength band is input, the image matching apparatus 1000 has a third neural network model 314 trained to output an image patch representing the image property of the first wavelength band may be used to transform the second image patch.

However, according to another embodiment, when the first image patch representing the image property of the first wavelength band is input, the image matching apparatus 1000 is trained to output the image patch representing the image property of the third wavelength band. The first image patch may be transformed using the second neural network model 304 , and when a second image patch representing the image property of the second wavelength band is input, the image patch representing the image property of the third wavelength band The second image patch may be transformed using the third neural network model 314 trained to output .

That is, the image matching apparatus 1000 according to the present disclosure may convert domains of image patches belonging to different domains into a common domain, but domains of image patches may be exchanged so that domains of image patches belonging to different domains are exchanged. You can also convert

The image matching apparatus 1000 obtains a converted first image patch 306 by transforming the first image patch using the second neural network model 304 , and uses the third neural network model 314 to obtain the second image patch. By transforming the two image patches, a transformed second image patch 316 may be obtained. The image matching apparatus 1000 may extract a first feature vector 308 from the transformed first image patch 306 and extract a second feature vector 318 from the transformed second image patch 316 . . For example, although not shown in FIG. 3 , the image matching apparatus 1000 uses at least one pre-trained convolution-based neural network model to obtain a first feature vector 308 from a transformed first image patch 306 . ), and a second feature vector 318 may be extracted from the transformed second image patch 316 .

According to an embodiment, the image matching apparatus 1000 determines a difference vector including, as an element, the square of the difference value for each element of the first feature vector and the second feature vector, and based on the determined difference vector, the first neural network The model can be trained to output the similarity of image patches. For example, in the image matching apparatus 1000, the process of learning the first neural network model that outputs the matching degree of the image patch as a probability value is to metric the neural network model based on the distance difference between the first feature vector and the second feature vector. It may correspond to the process of learning (320).

4 is a flowchart illustrating a method of identifying whether an image is matched by an image matching apparatus according to another exemplary embodiment.

2 to 3, the image matching apparatus 1000 uses feature vectors extracted from each of the first image patch, the second image patch, the converted first image patch, and the converted second image patch, A registration of the first image patch and the second image patch may be identified. Hereinafter, a method of identifying whether an image is matched according to another exemplary embodiment will be described in detail with reference to FIGS. 4 to 5 .

In S410 , the image matching apparatus 1000 may obtain a first image patch and a second image patch from lights of different wavelength bands. According to an embodiment, the first image patch may be an image of a partial region in the first image generated by light belonging to a first wavelength band, and the second image patch may be a second image patch generated by light belonging to the second wavelength band. 2 It may be a partial area image within the image. The first image patch may be a region image corresponding to the first domain, and the second image patch may be a region image corresponding to the second domain. Since S410 may correspond to S210 of FIG. 2 , a detailed description thereof will be omitted.

In S420 , the image matching apparatus 1000 may convert the first image patch and the second image patch to represent image characteristics of a preset wavelength band. According to an embodiment, the image matching apparatus 1000 is a second neural network trained to output an image patch representing an image property of a second wavelength band when a first image patch representing an image property of a first wavelength band is input. The model 304 may be used to transform the first image patch. Also, when the second image patch representing the image property of the second wavelength band is input, the image matching apparatus 1000 has a third neural network model 314 trained to output an image patch representing the image property of the first wavelength band can be used to transform the second image patch.

That is, the image matching apparatus 1000 according to the present disclosure converts the domain of the first image patch from the first domain to the second domain, and converts the domain of the second image patch from the second domain to the first domain. It is possible to convert the first image patch to indicate the image characteristic belonging to the second domain, and the second image patch to indicate the image characteristic belonging to the first domain.

In S430 , the image matching apparatus 1000 may extract a feature vector from the transformed first image patch and the transformed second image patch. For example, the image matching apparatus 1000 extracts a first feature vector from the converted first image patch by inputting the converted first image patch to a fourth neural network model, and uses the converted second image patch By inputting the fifth neural network model, it is possible to extract a fourth feature vector from the transformed second image patch.

In S440 , the image matching apparatus 1000 may extract a feature vector from the obtained first image patch and the obtained second image patch. For example, the image matching apparatus 1000 extracts a second feature vector from the obtained second image patch by inputting the obtained second image patch to a sixth neural network model, and the obtained first image patch By inputting to the seventh neural network model, it is possible to extract a third feature vector from the obtained first image patch.

In step S460 , the image matching apparatus 1000 sets a first feature vector extracted from each of the obtained first image patch, the obtained second image patch, the converted first image patch, and the converted second image patch. By input to the neural network model, it is possible to identify whether the first image patch and the second image patch match.

For example, the image matching apparatus 1000 may determine a first distance difference between a first feature vector extracted from the first transformed image patch and a second feature vector extracted from the second image patch, and from the first image patch A second distance difference between the extracted third feature vector and the fourth feature vector extracted from the transformed second image patch may be determined. The image matching apparatus 1000 inputs the difference vector regarding the first distance difference and the second distance difference to the first neural network model, so that the probability regarding the degree of matching of the first image patch and the second image patch from the first neural network model value can be obtained. The image matching apparatus 1000 may identify whether the first image patch and the second image patch match based on the probability value output from the first neural network model.

5 is a diagram for explaining a method of identifying whether an image is matched by an image matching apparatus according to another exemplary embodiment.

A process in which the image matching apparatus 1000 according to an embodiment identifies whether an image is matched will be described in more detail with reference to FIG. 5 . As described above, the image matching apparatus 1000 may acquire the first image patch 502 belonging to the first domain 501 and the second image patch 504 belonging to the second domain 503 . The image matching apparatus 1000, in addition to the converted first image patch 522 and the converted second image patch 524, differently from those described above with reference to FIGS. 2 to 3 , includes the first image patch 502 and the second image patch. Whether the first image patch and the second image patch match may be identified by further using the feature vector extracted from the patch 504 .

For example, the image matching apparatus 1000 obtains a first image patch 502 belonging to the first domain 502, and when an image patch showing image characteristics according to the first domain is input, the input image patch is A transformed first image patch 522 is obtained by transforming the first image patch 502 using a second neural network model 512 that transforms the input image patch to represent image characteristics according to the second domain. can do.

According to an embodiment, the first domain 502 may mean a spectral region to which lights of the first wavelength band belong. The image matching apparatus 1000 obtains a first image from lights of a first wavelength band, and converts a partial region of the first image including feature points extracted from the first image based on pixel values in the first image to the first image. It can be decided by patch.

Also, according to an embodiment, the image matching apparatus 1000 acquires the second image patch 504 belonging to the second domain 503, and when an image patch showing image characteristics according to the second domain is input, the input The transformed second image patch by transforming the second image patch 504 using the third neural network model 514 that transforms the input image patch so that the converted image patch exhibits image characteristics according to the first domain (524) can be obtained.

According to an embodiment, the second domain 503 may mean a spectral region to which lights of the second wavelength band belong. The image matching apparatus 1000 obtains a second image from lights of a second wavelength band, and converts a partial region of the second image including feature points extracted from the second image based on pixel values in the second image to the second image. It can be decided by patch.

According to an embodiment, the above-described first domain and the second domain may represent a spectrum of light belonging to a visible ray region and a spectrum of light belonging to a near-infrared region, respectively. However, according to another embodiment, the first domain and the second domain may represent a spectrum of light belonging to a visible ray region and a spectrum of light belonging to a thermal infrared region, respectively. Also, according to an embodiment, the first domain and the second domain may represent a spectrum of light belonging to each RGB region and a spectrum of light belonging to a near-infrared region, respectively.

According to another embodiment, the first image patch and the second image patch acquired by the image matching apparatus 1000 may have image characteristics of images generated by light in the visible light region and images generated by light in the near infrared region, respectively. may include their characteristics. However, according to another embodiment, the first image patch and the second image patch may include image characteristics of images generated by light in the visible light region and characteristics of images generated by light belonging to the thermal infrared region, respectively. may be Also, according to an embodiment, the first image patch and the second image patch may include characteristics of images generated by light belonging to each RGB region and characteristics of images generated by light belonging to a near-infrared region, respectively. there is.

Also, according to an embodiment, characteristics of images according to the present disclosure may mean intensity, level, and minimum and maximum ranges of pixel values in the image. For example, intensities, levels, and minimum and maximum ranges of pixel values in a near-infrared image patch photographed by a near-infrared camera and a visible light image patch photographed by a visible light camera may be different from each other.

According to an embodiment, the image matching apparatus 1000 uses a fourth neural network model trained in advance to extract a feature vector from an image patch belonging to the second domain, and receives the first feature from the converted first image patch 522 . A vector 532 may be extracted. Also, the image matching apparatus 1000 extracts the second feature vector 534 from the second image patch 504 by using a sixth neural network model trained in advance to extract the feature vector from the image patch belonging to the second domain. can do. According to an embodiment, the converted first image patch and the second image patch may be image patches representing image characteristics according to the second domain, and the fourth neural network model and the sixth neural network model share a weight with each other. Siamese) may be a neural network model of a network structure.

Also, according to an embodiment, the image matching apparatus 1000 uses a seventh neural network model trained in advance to extract a feature vector from an image patch belonging to the first domain, and the third feature from the first image patch 502 is used. A vector 536 may be extracted. In addition, the image matching apparatus 1000 uses a fifth neural network model trained in advance to extract a feature vector from an image patch belonging to the first domain, and a fourth feature vector 538 from the converted second image patch 524 . can be extracted. According to an embodiment, the converted second image patch and the first image patch may be image patches representing image characteristics according to the first domain, and the seventh neural network model and the fifth neural network model share a weight with each other. Siamese) may be a neural network model of a network structure.

The image matching apparatus 1000 according to an embodiment of the present disclosure may include a pair of neural network models of a Siamese network structure (dual Siamese Network), and the pair of neural network models of a Siamese network structure is a first type of Siamese network neural network model. and a Siamese network neural network model of the second time.

The image matching apparatus 1000 may determine a first difference vector 542 including, as an element, the square of the distance values for each element of the first feature vector 532 and the second feature vector 534 . For example, the image matching apparatus 1000 determines, as the first difference vector 542 , a vector including, as an element, the square of the distance difference value for each element of the first feature vector 532 and the second feature vector 534 . may be Also, according to an embodiment, the image matching apparatus 1000 may generate a second difference vector 544 including, as an element, the square of the distance values for each element of the third feature vector 536 and the fourth feature vector 538 . can decide For example, the image matching apparatus 1000 determines, as the second difference vector 544 , a vector including, as an element, the square of the distance difference value for each element of the third feature vector 536 and the fourth feature vector 538 . may be

When the first difference vector 542 and the second difference vector 544 are input, the image matching apparatus 1000 generates a first neural network model to output a probability value indicating the degree of matching of the first image patch and the second image patch. can learn For example, the image matching apparatus 1000 may train the first neural network model using a metric learning 546 technique in order to identify similarity between images. The image matching apparatus 1000 uses the first neural network model trained to identify the degree of similarity between the first image patch and the second image patch based on the first difference vector and the second difference vector, Matching of the second image patch may be identified.

The image matching apparatus 1000 according to an embodiment of the present disclosure may identify whether images to which each image patch belongs by identifying whether image patches having spectra of different wavelength regions are matched. there is. Also, according to an embodiment, the image matching apparatus 1000 may acquire images belonging to different spectra by acquiring continuous image sequences according to the passage of time. The image matching apparatus 1000 may identify matching between images belonging to different spectra by identifying whether image patches belonging to different spectra are matched.

When the image matching apparatus 1000 according to the present disclosure identifies whether multi-spectrum images are matched, and it is determined that multi-spectral images are matched, between the matched multi-spectrum images The matching of can be expressed as cross-spectrum image matching.

6 is a diagram for describing a structure of an image matching apparatus according to an exemplary embodiment.

According to an embodiment, the image matching apparatus 1000 may identify whether a pair of image patches is matched by using a plurality of neural network models. Hereinafter, for convenience, the first image patch 602 corresponds to a visible domain 601 corresponding to the spectrum of light belonging to the visible ray region, and the second image patch 604 corresponds to the spectrum of light belonging to the near-infrared region. A case corresponding to the corresponding near-infrared domain (NIR domain, 603) will be described on the assumption that.

For example, the image matching apparatus 1000 may acquire the first image patch 602 of the visible ray domain 601 and the second image patch of the near-infrared domain 604 . The image matching apparatus 1000 may be connected to a visible ray camera and a near-infrared camera, and may acquire a first image patch and a second image patch from the visible ray camera and the near-infrared camera, respectively.

The image matching apparatus 1000 may convert the visible ray domain, which is the domain of the first image patch, into a near-infrared domain by using the second neural network model 612 . For example, the operation of the image matching apparatus 1000 converting the domain of the first image patch from the visible ray domain to the near-infrared domain is such that the first image patch belonging to the visible ray domain exhibits image characteristics according to the near-infrared domain; It may correspond to an operation of converting a pixel value of the first image patch. In the second neural network model 612 used by the image matching apparatus 1000, when an image patch belonging to the visible ray domain is input, the input image patch exhibits image characteristics according to the near-infrared domain. Pixel values of the input image patch It may be a neural network model trained to transform According to an embodiment, the second neural network model is a U-Net model including 10 blocks, and may include an input layer, an encoding block, a decoding block, and an output layer. The structure of the second neural network model will be described in more detail with reference to FIG. 7 .

The image matching apparatus 1000 may acquire the converted first image patch 622 representing image characteristics according to the near-infrared domain by converting the first image patch belonging to the visible ray domain. The converted first image patch may not be an image patch captured by a camera including a near-infrared image sensor, but may be an image patch converted to exhibit near-infrared image characteristics. In addition, the transformed first image patch may have a similar appearance to that of the first image patch input to the second neural network model, as well as maintaining low-level feature values. Although the converted first image patch may have slightly different pixel intensity and level from the first image patch, the image matching apparatus 1000 may accurately identify the registration of the multi-spectral image by using the converted first image patch. there is.

Also, according to an embodiment, the image matching apparatus 1000 may convert the near-infrared domain, which is the domain of the second image patch, into a visible ray domain by using the third neural network model 614 . For example, the image matching apparatus 1000 converts the domain of the second image patch from the near-infrared domain to the visible ray domain so that the second image patch belonging to the near-infrared domain exhibits image characteristics according to the visible ray domain. , may correspond to an operation of converting pixel values of the second image patch. When an image patch belonging to the near-infrared domain is input, the third neural network model 614 used by the image matching apparatus 1000 calculates the pixel values of the input image patch so that the input image patch exhibits image characteristics according to the near-infrared domain. It can be a neural network model that has been trained to transform. According to an embodiment, the third neural network model is a U-Net model including 10 blocks, and may include an input layer, an encoding block, a decoding block, and an output layer. The structure of the third neural network model will be described in more detail with reference to FIG. 7 .

The image matching apparatus 1000 may obtain the converted second image patch 624 representing image characteristics according to the visible ray domain by converting the second image patch belonging to the near-infrared domain. The converted second image patch may not be an image captured by a camera including a visible light image sensor, but may be an image patch converted to exhibit visible light image characteristics.

The image matching apparatus 1000 may extract a feature vector from the first image patch 602 , the second image patch 604 , the transformed first image patch 622 , and the transformed second image patch 624 . . For example, the image matching apparatus 1000 uses the fourth neural network model 626 and the sixth neural network model 628 trained in advance to extract feature vectors from image patches belonging to the near-infrared domain, and the converted first image A first feature vector 632 and a second feature vector 634 may be extracted from the patch 622 and the second image patch, respectively. The fourth neural network model and the sixth neural network model may be pre-trained to extract feature vectors from image patches belonging to the near-infrared domain, and may be convolutional neural network models of a Siamese network structure sharing weights.

The image matching apparatus 1000 uses the seventh neural network model 629 and the fifth neural network 631 trained in advance to extract feature vectors from image patches belonging to the visible ray domain, the first image patch 602 and the transformation A third feature vector 636 and a fourth feature vector 638 may be extracted from the second image patch 624 , respectively. The seventh neural network model and the fifth neural network model may be pre-trained to extract feature vectors from image patches belonging to the visible ray domain, and may be convolutional neural network models of a Siamese network structure sharing weights.

According to an embodiment, the above-described fourth neural network model, fifth neural network model, sixth neural network model, and seventh neural network model are based on a correlation between pixel information of an image patch and a kernel of a preset size when an image patch is input. It may be a convolution-based neural network model that outputs determined feature values in a vector form.

The image matching apparatus 1000 may determine a first difference vector 642 including, as an element, a distance difference value for each element of the first feature vector 632 and the second feature vector 634 . Also, the image matching apparatus 1000 may determine a second difference vector 644 including, as an element, a distance difference value for each element of the third feature vector 636 and the fourth feature vector 638 . The image matching apparatus 1000 may generate a first polymerization vector 646 by concatenating the first difference vector 642 , the first feature vector 632 , and the second feature vector 634 .

According to an embodiment, the image registration apparatus 1000 concatenates the second difference vector 644 , the third feature vector 636 , and the fourth feature vector 638 , so that the second polymerization vector 644 . can create The image matching apparatus 1000 may generate the third polymerization vector 648 by concatenating the first polymerization vector 646 and the second polymerization vector 648 .

The image matching apparatus 1000 inputs the generated third polymerization vector 648 to the pre-trained first neural network model, and a similarity score ( A similarity score, 658) can be obtained. According to an embodiment, the first neural network model may further include an activation function connected to a plurality of fully connected layers 652 , 654 , and 656 and an output terminal of the fully connected network.

For example, the fully connected layer 652 receives the third polymerization vector 648 generated from the feature vectors, transforms the dimension of the input third polymerization vector, and converts the third polymerization vector of the transformed dimension. The next fully connected layer 654 and the fully connected layer 656 are sequentially transferred. In the fully connected layer 656, a pre-trained non-linear sigmoid activation function may be further connected to generate a similarity score from the polymerization vector.

According to an embodiment, the first neural network model, the second neural network model, the third neural network model, the fourth neural network model, The fifth neural network model, the sixth neural network model, and the seventh neural network model may be implemented as a single neural network model as a Spectral-Invariant Matching Network (SPIMNet).

According to an embodiment, the above-described first neural network model, the second neural network model, the third neural network model, The fourth neural network model, the fifth neural network model, the sixth neural network model, and the seventh neural network model include a first loss function related to a degree of matching of the first image patch and the second image patch and the transformed first Layers and weights related to connection strength between layers may be corrected and updated so that the second loss function related to the transformation degree of the image patch and the transformed second image patch is minimized.

According to an embodiment, when the neural network model used by the image matching apparatus 1000 is implemented as a spectral invariant image matching model as a single neural network model, weights in the spectral invariant image matching model are also included in the first image patch and the second The first loss function regarding the degree of matching of the two image patches and the second loss function regarding the degree of conversion of the transformed first image patch and the converted second image patch may be modified and updated to be minimized. According to an embodiment, the first loss function and the second loss function may be included in the third loss function by being connected as one term.

That is, the image matching apparatus 1000 according to an embodiment of the present disclosure may modify and update weights in the neural network models in the image matching apparatus 1000 so that the third loss function expressed as Equation 1 below is minimized. there is.

In Equation 1, L _SPIMNet is a third loss function including a first loss function and a second loss function, and L _matching is used to learn a similarity level of the first image patch and the second image patch. is a first loss function, yL _conversion is a second loss function used to learn the image matching apparatus 1000 to transform domains of image patches. The image matching apparatus 1000 may modify and update weights in the neural network models in the image matching apparatus so that the third loss function according to Equation 1 is minimized.

는 하나의 트레이닝 이미지에 대한 이미지 정합 장치(1000)가 이용하는 스펙트럼 불변 이미지 정합 모델의 출력 값이며, y는 학습 데이터의 클래스를 나타낸다. 예를 들어, y가 1인 경우, 제1 이미지 패치 및 제2 이미지 패치의 정합을 나타내고, y가 0이면, 제1 이미지 패치 및 제2 이미지 패치가 정합되지 않음을 나타낼 수 있다.Here, L _matching is a binary cross-entropy function, and is a first loss function used to learn a similarity level of the first image patch and the second image patch,

is an output value of a spectral invariant image matching model used by the image matching apparatus 1000 for one training image, and y represents a class of training data. For example, when y is 1, it may indicate that the first image patch and the second image patch are matched, and when y is 0, it may indicate that the first image patch and the second image patch are not matched.

은 L1 손실이며,

는 perceptual 손실이다. Perceptual 손실이 사용되지 않으면, Lconversion 손실은 정합(matched) 레이블링된 패치 쌍들이 misaligned 되는 경우, 상당히 증가할 수 있다. 상기 수학식 3에서,

은 Perceptual 손실을 결정하기 위해, ImageNet 데이터 셋에 기초하여 미리 학습된 VGG19 network일 수 있다. 또한, 상기 수학식 3에서, xvis는 가시광선 도메인에 속하는 제1 이미지 패치이고, xnir은 근적외선 도메인에 속하는 제2 이미지 패치를 나타내며, xtvis는 가시광선 도메인에 속하는 이미지 패치가 입력되면, 근적외선 도메인에 따른 이미지 특성을 나타내도록, 입력된 이미지 패치를 변환하는 제2 신경망 모델로부터 출력된, 변환된(translated) 제1 이미지 패치이고, xtnir은 근적외선 도메인에 속하는 이미지 패치가 입력되면, 가시광선 도메인에 따른 이미지 특성을 나타내도록, 입력된 이미지 패치를 변환하는 제3 신경망 모델로부터 출력된, 변환된(translated) 제2 이미지 패치를 나타낸다.In Equation 3, L _conversion is a second loss function used to learn the image matching apparatus 1000 to transform domains of image patches,

is the L1 loss,

is a perceptual loss. If perceptual loss is not used, Lconversion loss can increase significantly if matched labeled patch pairs are misaligned. In Equation 3 above,

may be a pre-trained VGG19 network based on the ImageNet data set to determine the perceptual loss. Also, in Equation 3, xvis is a first image patch belonging to the visible ray domain, xnir indicates a second image patch belonging to the near-infrared domain, and xtvis is an image patch belonging to the visible ray domain. A first image patch that is translated and output from a second neural network model that transforms an input image patch to show image characteristics according to It represents a translated second image patch output from the third neural network model that transforms the input image patch to represent image characteristics.

는 0.1,

는 30으로 설정될 수 있으나, 이에 제한되는 것은 아니며, 제3 손실 함수를 최소화하기 위해 학습 과정에서 다르게 설정될 수도 있다. 예를 들어, 이미지 정합 장치(1000)는 이미지 데이터 셋에 기초하여 이미지 정합 장치 내 신경망 모델을 훈련하는 과정에서 L_conversion이 최소화 되도록, 이미지 신경망 모델 내 가중치들을 수정 및 갱신할 수 있다. L_conversion은, 제2 신경망 모델 및 제3 신경망 모델로부터 각각 출력된 변환된 제1 이미지 패치 및 변환된 제2 이미지 패치가, 제1 이미지 패치 및 제2 이미지 패치와 유사해질수록, 작아질 수 있다.According to one embodiment,

is 0.1,

may be set to 30, but is not limited thereto, and may be set differently in the learning process to minimize the third loss function. For example, the image matching apparatus 1000 may modify and update weights in the image neural network model so that L _conversion is minimized during training of the neural network model in the image matching apparatus based on the image data set. L _conversion may be smaller as the converted first image patch and the converted second image patch output from the second neural network model and the third neural network model, respectively, become similar to the first image patch and the second image patch. .

7 is a diagram for describing a structure of a neural network model used by an image matching apparatus to convert an image patch, according to an embodiment.

The structures of the second neural network model and the third neural network model used by the image matching apparatus 1000 to transform an image patch to represent image characteristics according to a target domain will be described in detail with reference to FIG. 7 . Hereinafter, for convenience, the first image patch 602 corresponds to a visible domain 601 corresponding to the spectrum of light belonging to the visible ray region, and the second image patch 604 corresponds to the spectrum of light belonging to the near-infrared region. A case corresponding to the corresponding near-infrared domain (NIR domain, 603) will be described on the assumption that.

The image matching apparatus 1000 may transform the first image patch using the second neural network model 712 so that the first image patch belonging to the visible ray domain exhibits image characteristics in the near-infrared domain. According to an embodiment, when an image patch belonging to the visible ray domain is input, the second neural network model 712 may be a neural network model that transforms the input image patch to represent image characteristics of the near-infrared domain. According to an embodiment, the second neural network model 712 may be a U-net including 10 blocks.

According to an embodiment, the second neural network model 712 may include a plurality of convolutional layers 722 , 724 , and 728 and convolutional transponse layers 726 . More specifically, the second neural network model 712 may include an input layer, an encoding block, a decoding block, and an output layer. According to an embodiment, each of the encoding blocks may include a convolutional layer, a batch norm layer 732 , an instance norm layer 734 , and a ReLU activation layer 736 , and each decoding block is a convolution transpose layer. , 726 ), a batch norm layer 732 , an instance norm layer 734 , and a ReLU activation layer 736 .

According to an embodiment, the batch norm layer may normalize the average and variance values of the feature vector values in the neural network model in batch units. In addition, the instance norm layer may normalize the average value and variance value of the feature vectors in the neural network model in a channel unit rather than a batch unit in the neural network model. In addition, the Relu activation layer may output 0 when the output values of each block are less than 0, and output the corresponding output value when the output values of each block are greater than 0. In addition, the tanh activation layer is generated by transforming the sigmoid activation layer. When the output value of each block is used as the input value of the tanh function, the output value of the tanh function can be used as the output of the activation layer.

According to an embodiment, the second neural network model 712 may include five encoding blocks, and the number of filters of the convolutional layers in each encoding block may be gradually increased, such as 64, 128, 256, 256. 256. can In addition, in the four decoding blocks in the second neural network model 712 , the number of filters of the convolutional preposition layers may be gradually reduced, such as 256, 256, 128, 64. According to an embodiment, the filter (or kernel kernel) size of the convolutional layers in the encoding block and the decoding block is 4*4, the interval stride at which the filters are moved is 2, and padding for preventing output data reduction of the convolutional layers ) value may be 2.

According to an embodiment, the image matching apparatus 1000 uses the third neural network model 714 to make the second image patch belonging to the near-infrared domain exhibit image characteristics according to the visible ray domain, to make the second image patch. can be converted According to an embodiment, the third neural network model 714 may be a U-net including 10 blocks. The third neural network model 714 may have the same structure as the second neural network model 714 except for the structure of the last block. The structure of the last block in the second neural network model and the third neural network model may vary depending on the type of target domain that each neural network model targets.

For example, since the third neural network model 714 is a model for converting an image patch corresponding to the near-infrared domain to represent image characteristics according to the visible ray domain when an image patch corresponding to the near-infrared domain is input, the target domain is the visible ray domain. It can be set as a domain. Accordingly, in the case of the visible ray domain, since a total of three channels are required, the number of filters in the convolutional layer of the last block of the third neural network model may be three. Conversely, since the second neural network model 712 is a model for converting an image patch corresponding to the visible ray domain to show image characteristics according to the near-infrared domain when an image patch is input, the target domain is set to the near-infrared domain. can be Accordingly, the number of filters in the convolutional layer of the last block of the second neural network model may be 1.

In addition, a Tanh activation layer 738 may be connected to each end of the second neural network model 712 and the third neural network model 714 . As described above, the Tanh activation layer 738 may perform a normalizing function so that the range of the image patch output from each of the second neural network model and the third neural network model becomes [-1, 1].

8 is a diagram for explaining the structure of a neural network model used by an image matching apparatus to extract a feature vector from an image patch according to an embodiment.

According to an embodiment, the image matching apparatus 1000 may extract a feature vector from an image patch using a convolutional neural network model of a dual-Siamese network structure. For example, as described above with reference to FIG. 6 , the image matching apparatus 1000 extracts a first feature vector from a first image patch converted using a fourth neural network model, and uses a sixth neural network model to extract a second feature vector. A second feature vector is extracted from the image patch, a third feature vector is extracted from the first image patch using a seventh neural network model, and a fourth feature vector is extracted from the converted second image patch using a fifth neural network model. can be extracted. According to an embodiment, each of the fourth neural network model, the sixth neural network model, the seventh neural network model, and the fifth neural network model may include the neural network model structure 802 shown in FIG. 8 .

The fourth neural network model and the sixth neural network model used by the image matching apparatus 1000 are a first type of convolutional neural network model of a dual-Siamese network structure that shares weights with each other, and the seventh neural network model and the sixth neural network model are 5 The neural network model may be a second type of convolutional neural network model of a dual-Siamese network structure that shares weights with each other. That is, the image matching apparatus 1000 extracts a feature vector from each of the first image patch, the second image patch, the converted first image patch, and the converted second image patch using the convolutional neural network model of the dual Siamese network structure. can do.

According to an embodiment, each of the fourth neural network model, the sixth neural network model, the fifth neural network model, and the seventh neural network model used by the image matching apparatus 1000 to extract a feature vector from the image patch or the converted image patch is a total It may include eight blocks, and each block may include a convolutional layer, a batch norm layer, an instance norm layer, and a ReLU activation layer. Since the configuration of each layer may correspond to that described above with reference to FIG. 7 , a detailed description thereof will be omitted. In addition, according to an embodiment, the number of filters, filter size, and stride of the convolutional layer in each block are (32, 3*3, 1), (64, 3*3, 1), (128, 3*) for each block. 3, 1), (128, 5*5, 2), (256, 3*3, 1), (256, 5*5, 2), (256, 3*3, 1) and (256, 5* 5, 2) can be set.

The image matching apparatus 1000 may configure the first difference vector based on the square value of the absolute value of the distance difference between elements of the first feature vector extracted from the converted first image patch and the second feature vector extracted from the second image patch. can be decided Also, the image matching apparatus 1000 may be configured to perform a second function based on the square value of the absolute value of the distance difference between elements of the third feature vector extracted from the first image patch and the fourth feature vector extracted from the converted second image patch. The difference vector can be determined. The image matching apparatus 1000 may obtain a similarity score regarding the degree of matching between image patch pairs by inputting the determined first difference vector and the second difference vector into the first neural network model trained to determine the similarity between images. .

FIG. 9 is a diagram for explaining a result of converting an image patch using a neural network model by an image matching apparatus according to an exemplary embodiment.

Referring to FIG. 9 , when an image patch belonging to the visible ray domain is input, the image patch output from the second neural network model that converts the input image patch and the near infrared ray so that the input image patch exhibits image characteristics according to the near infrared domain When an image patch belonging to a domain is input, an image patch output from the third neural network model that transforms the input image patch is shown so that the input image patch exhibits image characteristics according to the visible ray domain.

In more detail, the image patches shown in FIG. 9 minimize a loss function (eg, a first loss function) in which a second loss function (eg, an Lconversion loss according to Equation 3) is removed from the third loss function. Output image patches of the second neural network model and the third neural network model that have been trained to do so may be represented.

For example, when the image patches 912 , 916 , and 922 are input to the second neural network model and the third neural network model that are trained to minimize only the first loss function obtained by removing the second loss function from the third loss function , image patches 913 , 918 , and 924 converted from the second neural network model and the third neural network model may be output. The transformed image patches 913 , 918 , and 924 have a large difference in appearance from the original image because they do not include an object or parts of an object similar to the original image patch, as shown in FIG. 9 . can In addition, the transformed image patches 913, 918, and 924 output from the second neural network model and the third neural network model trained to minimize only the first loss function are image information necessary to identify whether the image patch pairs are matched. There may be little to no

However, unlike the embodiment shown in FIG. 9 , the image matching apparatus 1000 according to an embodiment of the present disclosure includes a first loss function for determining a matching degree of a patch and a conversion between the image patch and the converted image patch By modifying and updating the weights of the neural network models in the image matching device to minimize the third loss function including all the second loss functions that determine there is.

10 is a block diagram of an image matching apparatus according to an exemplary embodiment.

As shown in FIG. 10 , the image matching apparatus 1000 may include a processor 1400 and a memory 1402 . However, not all illustrated components are essential components, and the image registration apparatus 1000 may be implemented by more components than the illustrated components, and the image registration apparatus 1000 may be implemented with fewer components than the illustrated components. may be implemented. According to an embodiment, the image matching apparatus 1000 may further include a communication unit (not shown) in addition to the processor 1400 and the memory 1402 .

The processor 1400 may generally control the overall operation of the image matching apparatus 1000 .

According to an embodiment, the processor 1400 according to the present disclosure executes programs stored in the memory 1402 to perform the function of the image matching apparatus 1000 described in FIGS. 1 to 9 . In addition, the processor 1400 may be composed of one or a plurality of processors, and the one or the plurality of processors is a general-purpose processor such as a CPU, AP, DSP (Digital Signal Processor), etc., a graphics-only processor such as a GPU, or artificial intelligence (AI). ) may be a dedicated processor. According to an embodiment, when the processor 1400 includes a general-purpose processor, an artificial intelligence processor, and a graphics-only processor, the artificial intelligence processor may be implemented as a general-purpose processor or a chip separate from the graphics-only processor.

According to an embodiment, when the processor 1400 is implemented as a plurality of processors or a graphics-only processor or an artificial intelligence-only processor, at least some of the plurality of processors or the graphics-only processor or the artificial intelligence-only processor may be configured in the image matching apparatus 1000 . and any electronic device or server connected to the image matching apparatus 1000 may be mounted.

For example, the processor 1400 executes programs stored in the memory 1402 to obtain a first image patch and a second image patch from lights of different wavelength bands, and the first image patch and the second image By converting a patch to represent image characteristics of a preset wavelength band, extracting a feature vector from the converted first image patch and the converted second image patch, and inputting the extracted feature vector to a first neural network model , it is possible to identify whether the first image patch and the second image patch match.

According to an embodiment, the processor 1400 extracts a feature vector from the obtained first image patch and the obtained second image patch, and from each of the obtained first image patch and the obtained second image patch. By further inputting the extracted feature vector into the first neural network model, it is possible to identify whether the first image patch and the second image patch match.

According to an embodiment, the processor 1400 obtains a first image from lights of a first wavelength band and a second image from lights of a second wavelength band, and each pixel value in the first image and the second image Based on , a feature point may be extracted from each of the first image and the second image, and a partial region of the image including the extracted feature point may be obtained as the first image patch and the second image patch.

According to an embodiment, the processor 1400 uses a second neural network model that outputs an image patch representing image characteristics of a second wavelength band when a first image patch representing image characteristics of a first wavelength band is input. , using a third neural network model that transforms the first image patch and outputs an image patch representing image characteristics of the first wavelength band when a second image patch representing image characteristics of the second wavelength band is input , the second image patch may be transformed.

According to an embodiment, the processor 1400 extracts a first feature vector from the transformed first image patch by inputting the transformed first image patch to a fourth neural network model, and the transformed second image patch. By inputting ? into the fifth neural network model, it is possible to extract a fourth feature vector from the transformed second image patch.

According to an embodiment, the processor 1400 extracts a second feature vector from the obtained second image patch by inputting the obtained second image patch to a sixth neural network model, and the obtained first image patch By inputting to the seventh neural network model, it is possible to extract a third feature vector from the obtained first image patch.

According to an embodiment, the processor 1400 determines a first difference vector including, as an element, the square of the distance values for each element of the first feature vector and the second feature vector, and the third feature vector and the second feature vector A second difference vector including the square of the distance value for each element of the 4 feature vector as an element may be determined. As described above, the first neural network model may output a matching probability value between the first image patch and the second image patch based on the first difference vector and the second difference vector.

The communication unit (not shown) may include one or more components that allow the image matching apparatus 1000 to communicate with another device (not shown) and the server 2000 . The other device (not shown) may be a computing device such as the image matching device 1000 or a sensing device, but is not limited thereto. For example, the communication unit (not shown) may include a short-distance communication unit and a mobile communication unit.

Short-range wireless communication unit, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, near field communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee communication unit, infrared (IrDA, infrared) It may include a data association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, and the like, but is not limited thereto. The mobile communication unit transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network.

According to an embodiment, the communication unit (not shown) may transmit the first image patch and the second image patch generated by lights of different wavelength bands to the server under the control of the processor. According to an embodiment, the communication unit may receive a visible ray image patch, a thermal image patch, a near-infrared image patch, and the like from the visible ray camera, the thermal imager, and the near-infrared camera, respectively. Also, according to an embodiment, the communication unit (not shown) may receive information about the image matching result determined by the server for the image patch pairs from the server.

The memory 1402 may store a program for processing and control of the processor 1400 , and may store data input to or output from the image matching apparatus 1000 . In addition, the memory 1402 is a first neural network model, a second neural network model, a third neural network model, a fourth neural network model, a fifth neural network model, and a sixth Information on the neural network model and the seventh neural network model may be further stored. According to an embodiment, the memory 1402 and the like may further store the layers in the artificial neural network model in the image matching apparatus trained in advance to minimize the third loss function and weight values related to the connection strength between the layers.

The memory 1402 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

11 is a block diagram of a server connected to an image matching apparatus according to an exemplary embodiment.

According to an embodiment, the server 2000 may include a communication unit 2100 , a database 2200 , and a processor 2300 .

The communication unit 2100 may correspond to the communication unit (not shown) of the image matching apparatus 1000 described above. For example, the communication unit 2100 may receive, from the image matching apparatus 1000 , layers of the neural network model, information on zoad included in layers, or weight values relating to connection strength of layers in the neural network.

The database 2200 may correspond to the memory 1402 of the image matching apparatus shown in FIG. 10 . For example, the database 2200 may store a program for processing and controlling the processor 2300 , and may further store data input to or output from the image matching apparatus 1000 . there is. In addition, the database 2200 may further store information about the layers constituting the neural network model, the nodes included in the layers, and weights related to the connection strength of the layers, which are used to identify whether the images match. . In addition, when the weights in the artificial neural network model used to identify whether the images match or not are modified and updated, the database 2200 may further store information about the modified and updated weights.

The processor 2300 typically controls the overall operation of the server 2000 . For example, the processor 2300 may control the DB 2200 and the communication unit 2100 in general by executing programs stored in the DB 2200 of the server 2000 . Also, the processor 2300 may perform some of the operations of the image matching apparatus 1000 in FIGS. 1 to 10 by executing programs stored in the DB 2100 .

For example, the processor 2300 acquires a first image patch and a second image patch generated from lights of different wavelength bands, and sets the image characteristics of the first image patch and the second image patch in a preset wavelength band. The first image patch and the second image patch are transformed to represent It is also possible to identify whether the image patches are matched.

12 is a diagram for comparing performance of a trained neural network model based on a pair of a visible ray image patch and a near-infrared image patch, according to an embodiment.

12 shows several types of artificial neural network models 902 trained based on a pair of visible light image patches and near infrared image patches, by category (904, 906, 908, 910, 912, 914, 916, 918) shows the comparison result of the false positive rate recall (or sensitivity) 95. DA connected after the name of each model means data augmentation. It can be seen that, among the neural network models shown in FIG. 12 , the spectral invariant image matching model (Ours DA shown in FIG. 12 ) used by the image matching apparatus 1000 according to the present disclosure exhibits the best performance.

13 is a diagram for comparing the performance of a neural network model learned based on a pair of a visible light image patch and a thermal image patch according to an embodiment.

13 shows a comparison result of false positive rate recall (or sensitivity) 95 and false positive rate recall (or sensitivity) 99 as performance indicators of several neural network models trained based on a pair of visible light image patches and thermal image patches. . Referring to FIG. 13 , a spectral invariant image matching model (shown in FIG. 13 ) used by the image matching apparatus 1000 according to the present disclosure among several neural network models learned based on a pair of a visible light image patch and a thermal image patch Ours DA) shows the best performance in both the false positive rate recall (or sensitivity) 95 and the false positive rate recall (or sensitivity) 99 indicators.

That is, the spectral invariant image registration model used by the image registration apparatus 1000 according to the present disclosure to identify whether an image is matched is not only a visible light image patch and a near-infrared image patch pair, but also a visible light image patch and a thermal image image. It can be seen that the patch pair also shows the best performance. Accordingly, the image matching apparatus 1000 according to the present disclosure may be used in various fields such as stereo matching in a night image, face and object recognition using a multi-spectral image, and pedestrian detection.

14 is a diagram for comparing the performance of a neural network model trained based on a pair of RGB image patches and near-infrared image patches, according to an embodiment.

13 shows performance comparison results of various types of neural network models trained based on a pair of RGB image patches and near-infrared image patches in order to identify whether images are matched. The spectral invariant image registration model (Ours DA shown in FIG. 14 ) used by the image matching apparatus 1000 according to the present disclosure is, False positive rate recall (or sensitivity) 95, False positive rate recall (or sensitivity) 97 and False positive It can be seen that it shows the best performance in all of rate recall (or sensitivity) 99.

15 is a diagram for comparing the performance of a neural network model of a general Siamese network structure and a neural network model according to an embodiment of the present disclosure.

Referring to FIG. 15 , similarity scores 1502 output from a neural network model of a general Siamese network structure to identify whether images are matched, a neural network used by the image matching apparatus 1000 described above in FIGS. 3 to 4 . The similarity score 1504 output from the model and the similarity score 1506 output from the neural network model used by the image matching apparatus 1000 described above in FIGS. 4 to 5 are shown. Referring to FIG. 15 , as a neural network model having a dual Siamese network structure, it can be confirmed that the spectral invariant image matching model including both the second neural network model and the third neural network model has the highest similarity score.

The method according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present disclosure, or may be known and available to those skilled in the art of computer software.

In addition, according to the embodiment, a computer program apparatus including a recording medium storing a program for performing another method may be provided. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiment of the present disclosure has been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improved forms of the present disclosure are also provided by those skilled in the art using the basic concept of the present disclosure as defined in the following claims. belong to the scope of the right.

Claims (20)

In the method of identifying whether an image is matched,
obtaining a first image patch and a second image patch from lights of different wavelength bands;
converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band;
extracting a feature vector from the obtained first image patch and the obtained second image patch;
extracting a feature vector from the transformed first image patch and the transformed second image patch; and
By inputting feature vectors extracted from each of the obtained first image patch, the obtained second image patch, the converted first image patch, and the converted second image patch to a first neural network model, the first image identifying whether the patch and the second image patch match; A method comprising delete The method of claim 1, wherein the obtaining
obtaining a first image from lights of a first wavelength band and a second image from lights of a second wavelength band;
extracting a feature point from each of the first image and the second image based on respective pixel values in the first image and the second image; and
acquiring a partial region of an image including the extracted feature points as the first image patch and the second image patch; A method comprising The method of claim 1, wherein the converting comprises:
converting the first image patch by using a second neural network model that outputs an image patch representing image characteristics of a second wavelength band when a first image patch representing image characteristics of a first wavelength band is input; and
converting the second image patch by using a third neural network model that outputs an image patch representing the image characteristic of the first wavelength band when a second image patch representing the image characteristic of the second wavelength band is input; ; A method comprising 5. The method of claim 4, wherein the extracting of a feature vector from the transformed first image patch and the transformed second image patch comprises:
extracting a first feature vector from the transformed first image patch by inputting the transformed first image patch into a fourth neural network model; and
extracting a fourth feature vector from the transformed second image patch by inputting the transformed second image patch into a fifth neural network model; A method comprising The method of claim 5, wherein extracting a feature vector from the obtained first image patch and the obtained second image patch comprises:
extracting a second feature vector from the obtained second image patch by inputting the obtained second image patch into a sixth neural network model; and
extracting a third feature vector from the obtained first image patch by inputting the obtained first image patch into a seventh neural network model; A method comprising 7. The method of claim 6,
The fourth neural network model and the sixth neural network model share a weight with respect to layers in the fourth neural network model and the sixth neural network model and a connection strength between the layers,
The fifth neural network model and the seventh neural network model are a neural network of a siamese network structure in which layers within the fifth neural network model and the seventh neural network model and a weight related to a connection strength between the layers are shared. A method, characterized in that it is a model. The method of claim 6, wherein the step of identifying whether the match
determining a first difference vector including, as an element, the square of the distance values for each element of the first feature vector and the second feature vector; and
determining a second difference vector including, as an element, the square of the distance values for each element of the third feature vector and the fourth feature vector; further comprising,
The method of claim 1, wherein the first neural network model outputs a probability value that the first image patch and the second image patch match based on the first difference vector and the second difference vector. 7. The method of claim 6,
The first neural network model, the second neural network model, the third neural network model, the fourth neural network model, the fifth neural network model, the sixth neural network model, and the seventh neural network model,
A layer such that a first loss function related to the degree of matching of the first image patch and the second image patch and a second loss function related to the degree of conversion of the first transformed image patch and the transformed second image patch are minimized. and updating the weights regarding connection strength between layers and layers. 7. The method of claim 6,
The fourth neural network model, the fifth neural network model, the sixth neural network model, and the seventh neural network model are determined based on the correlation between pixel information of the image patch and a kernel of a preset size when an image patch is input. A method that is a convolutional neural network model that outputs the feature values to be used in a vector form. The method of claim 4, wherein the second neural network model and the third neural network model are
An encoder including a plurality of convolutional layers for encoding the image patch and a decoder for outputting an image patch representing image characteristics of a predetermined wavelength band by decoding the encoded image patch. An image matching device comprising:
a memory for storing one instruction; and
at least one processor executing the one or more instructions; including,
The at least one processor by executing the one or more instructions,
obtaining a first image patch and a second image patch from lights of different wavelength bands,
converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band;
extracting a feature vector from the obtained first image patch and the obtained second image patch;
extracting a feature vector from the transformed first image patch and the transformed second image patch;
By inputting feature vectors extracted from each of the obtained first image patch, the obtained second image patch, the converted first image patch, and the converted second image patch to a first neural network model, the first image An image matching device for identifying whether a patch and the second image patch are matched. delete 13. The method of claim 12, wherein the at least one processor comprises:
obtaining a first image from lights of a first wavelength band and a second image from lights of a second wavelength band;
extracting a feature point from each of the first image and the second image based on each pixel value in the first image and the second image;
An image matching apparatus for acquiring a partial region of an image including the extracted feature points as the first image patch and the second image patch. 13. The method of claim 12, wherein the at least one processor comprises:
When a first image patch representing an image characteristic of a first wavelength band is input, the first image patch is converted using a second neural network model that outputs an image patch representing an image characteristic of a second wavelength band;
Transforming the second image patch by using a third neural network model that outputs an image patch representing the image characteristic of the first wavelength band when a second image patch representing the image characteristic of the second wavelength band is input Phosphorus, an image matching device. 16. The method of claim 15, wherein the at least one processor comprises:
By inputting the transformed first image patch into a fourth neural network model, extracting a first feature vector from the transformed first image patch,
By inputting the transformed second image patch into a fifth neural network model, the fourth feature vector is extracted from the transformed second image patch. 17. The method of claim 16, wherein the at least one processor comprises:
By inputting the obtained second image patch into a sixth neural network model, a second feature vector is extracted from the obtained second image patch,
By inputting the obtained first image patch into a seventh neural network model, extracting a third feature vector from the obtained first image patch. 18. The method of claim 17,
The fourth neural network model and the sixth neural network model share a weight with respect to layers in the fourth neural network model and the sixth neural network model and a connection strength between the layers,
The fifth neural network model and the seventh neural network model are a neural network of a siamese network structure in which layers within the fifth neural network model and the seventh neural network model and a weight related to a connection strength between the layers are shared. An image matching device, characterized in that the model. 18. The method of claim 17, wherein the at least one processor comprises:
determining a first difference vector including, as an element, the square of the distance values for each element of the first feature vector and the second feature vector;
determining a second difference vector including, as an element, the square of the distance values for each element of the third feature vector and the fourth feature vector;
The first neural network model outputs a probability value that the first image patch and the second image patch are matched based on the first difference vector and the second difference vector. obtaining a first image patch and a second image patch from lights of different wavelength bands;
converting the first image patch and the second image patch to exhibit image characteristics of a preset wavelength band;
extracting a feature vector from the obtained first image patch and the obtained second image patch;
extracting a feature vector from the transformed first image patch and the transformed second image patch; and
By inputting feature vectors extracted from each of the obtained first image patch, the obtained second image patch, the converted first image patch, and the converted second image patch to a first neural network model, the first image identifying whether the patch and the second image patch match; A computer-readable recording medium recording a program for executing a method of identifying whether an image is matched or not, including a computer-readable recording medium.

Images