CN118537227A - Iterative interactive reference type stereo image super-resolution reconstruction method and system - Google Patents

Abstract

The invention discloses an iterative interactive reference type stereo image super-resolution reconstruction method and system, wherein the method comprises the following steps: acquiring two images to be reconstructed; inputting two images to be reconstructed into a trained image reconstruction model to obtain a high-resolution left view stereoscopic image and a high-resolution right view stereoscopic image; the method comprises the steps that a trained image reconstruction model adopts a plurality of information perception modules to extract features in views of two images to be reconstructed; the model also simulates the dependency relationship between views based on a pixel matching module and a tile matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; projecting the matched dictionary into a high-resolution space, and performing high-resolution view information interaction by using feature graphs of two views as cross references; and the model also re-weights the characteristic diagram of the view through a supervision side output modulator to realize block level matching.

Description

Iterative interactive reference stereo image super-resolution reconstruction method and system

Technical Field

The present invention relates to the technical field of super-resolution image reconstruction, and in particular to an iterative interactive reference stereo image super-resolution reconstruction method and system.

Background Art

For depth estimation tasks, binocular stereo images are still important information carriers in practical applications. How to obtain high-resolution binocular stereo images has become an important issue of concern to academia and industry, and stereo image super-resolution (SSR) has also attracted more and more attention. Most existing SSR methods focus on the correspondence between views in the low-resolution (LR) space, while ignoring the reference guidance between high-quality super-resolution images.

In general, existing SSR work can be divided into two main lines: 1) Exploring pixel-level correspondence matching in LR space through the disparity attention mechanism, the disadvantage of which is that it pays little attention to high-resolution (HR) correspondence. This may lead to the loss of a large amount of details in the LR image, thus limiting the utilization of cross-view information. In addition, the degradation of the LR image itself (such as noise and blur) will also seriously affect the matching accuracy. 2) Introducing additional disparity maps as priors, which mainly relies on additional depth estimation, making the network complex and bloated. At the same time, inaccurate prior depth estimation will have an adverse effect on correspondence matching.

Summary of the invention

In order to solve the deficiencies of the prior art, the present invention provides an iterative interactive reference stereo image super-resolution reconstruction method and system;

On the one hand, an iterative interactive reference stereo image super-resolution reconstruction method is provided, comprising:

Acquire two images to be reconstructed; the two images to be reconstructed include: a low-resolution left-view stereo image and a low-resolution right-view stereo image;

Input the two images to be reconstructed into the trained image reconstruction model to obtain a high-resolution left-view stereo image and a high-resolution right-view stereo image;

Among them, the trained image reconstruction model uses several information perception modules to extract features within the view of the two images to be reconstructed; the model also simulates the dependency between views based on the pixel matching module and the block matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; the matching dictionary is projected into the high-resolution space, and high-resolution view information interaction is performed by using the feature maps of the two views as mutual references; the model also re-weights the feature maps of the views through the supervisory side output modulator to achieve block-level matching.

On the other hand, an iterative interactive reference stereo image super-resolution reconstruction system is provided, comprising:

An acquisition module is configured to: acquire two images to be reconstructed; the two images to be reconstructed include: a low-resolution left-view stereo image and a low-resolution right-view stereo image;

A reconstruction module is configured to: input the two images to be reconstructed into the trained image reconstruction model to obtain a high-resolution left-view stereo image and a high-resolution right-view stereo image;

Among them, the trained image reconstruction model uses several information perception modules to extract features within the view of the two images to be reconstructed; the model also simulates the dependency between views based on the pixel matching module and the block matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; the matching dictionary is projected into the high-resolution space, and high-resolution view information interaction is performed by using the feature maps of the two views as mutual references; the model also re-weights the feature maps of the views through the supervisory side output modulator to achieve block-level matching.

On the other hand, an electronic device is provided, comprising:

a memory for non-transitory storage of computer-readable instructions; and

a processor for executing the computer readable instructions,

When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

The above technical solution has the following advantages or beneficial effects:

(1) This paper proposes a Reference-based Iterative Interaction for Stereo Image Super-Resolution (RIISSR), which uses reference-based iterative matching of pixels and patches (called ^P2 -Matching) to establish cross-view and cross-resolution correspondences for SSR. Specifically, this paper first designs parallel cascaded information-aware blocks (IPBs) to extract hierarchical contextual features from different views. Pixel matching is embedded between two parallel IPBs to exploit cross-view interactions in the low-resolution space. Then, iterative patch matching is performed using super-resolution stereo image pairs as mutual references, and high-resolution (HR) correspondences are learned using cross-scale patch recursive properties to achieve SSR performance. In addition, this paper introduces a supervised side output modulator (SSOM) to reweight local intra-view features and generate intermediate super-resolution images, thereby seamlessly connecting the two matching mechanisms. Experimental results on four datasets demonstrate the superior performance of the proposed RIISSR network.

(2) To achieve the SSR task, an iterative interactive reference based stereo image super-resolution network (RIISSR) is proposed, which provides a new paradigm for stereo image super-resolution reconstruction to achieve powerful cross-view interaction in a mutually iterative reference mode.

(3) A ^P2 -Matching mechanism is proposed to iteratively implement pixel-level matching and tile-level matching to learn cross-view and cross-resolution correspondences. At the same time, a cross-view matching dictionary is innovatively constructed to perform tile matching using the cross-scale tile recursive feature.

(4) An information-aware module (IPB) is proposed to unify channel attention, large kernel attention, and feature redistribution to extract information-rich hierarchical features within the view. In addition, a SSOM with high-resolution stereo image ground-truth supervision is designed to bridge the gap between pixel-level matching and patch-level matching, thereby improving super-resolution performance.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

FIG1 is a stereo image super-resolution network based on iterative cross-reference according to Embodiment 1;

FIG2 is a schematic diagram of the internal structure of a first iteration information sensing unit according to Embodiment 1;

FIG3 is a schematic diagram of the internal structure of a pixel matching module according to the first embodiment;

FIG4 is a schematic diagram of the internal structure of the first block matching module in accordance with the first embodiment;

FIG5 is a schematic diagram of the internal structure of the supervisory side output modulator of Example 1;

FIG6 is a schematic diagram of the internal structure of the fusion module of Example 1;

FIG. 7 is a visualization result of different image super-resolution methods in Example 1.

DETAILED DESCRIPTION

It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

Embodiment 1

This embodiment provides an iterative interactive reference stereo image super-resolution reconstruction method;

Iterative interactive reference stereo image super-resolution reconstruction method, comprising:

S101: Acquire two images to be reconstructed; the two images to be reconstructed include: a low-resolution left-view stereo image and a low-resolution right-view stereo image;

S102: inputting the two images to be reconstructed into the trained image reconstruction model to obtain a high-resolution left-view stereo image and a high-resolution right-view stereo image;

Among them, the trained image reconstruction model uses several information perception modules to extract features within the view of the two images to be reconstructed; the model also simulates the dependency between views based on the pixel matching module and the block matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; the matching dictionary is projected into the high-resolution space, and high-resolution view information interaction is performed by using the feature maps of the two views as mutual references; the model also re-weights the feature maps of the views through the supervisory side output modulator to achieve block-level matching.

Furthermore, the low resolution specifically refers to an image with a resolution less than or equal to 400 pixels × 450 pixels;

The high resolution specifically refers to an image with a resolution less than or equal to 1600 pixels×1800 pixels.

The low resolution and high resolution are relative.

The stereo image refers to a pair of images from different perspectives captured by two cameras at two locations parallel to the scene and only horizontally offset.

The left-view stereoscopic image refers to the image captured by the left-viewing camera in the obtained stereoscopic image.

The right-view stereo image refers to an image captured by a right-view camera in the obtained stereo image. Furthermore, the trained image reconstruction model includes:

Two parallel branches: the first branch and the second branch;

The first branch comprises: a first convolutional layer, M information perception modules, a first upsampling layer, a first supervisory side output modulator, a first block matching module and a first adder connected in sequence;

The second branch includes: a second convolutional layer, M information perception modules, a second upsampling layer, a second block matching module, a second supervisory side output modulator and a second adder connected in sequence;

The first convolutional layer is used to input a low-resolution left-view stereo image;

The second convolutional layer is used to input a low-resolution right-view stereo image;

The output end of the i-th information perception module of the first branch is connected to the input end of the corresponding i-th pixel matching module; the output end of the i-th pixel matching module is connected to the input end of the i+1-th information perception module of the first branch;

The output end of the i-th information perception module of the second branch is connected to the input end of the corresponding i-th pixel matching module; the output end of the i-th pixel matching module is connected to the input end of the i+1-th information perception module of the second branch;

An output terminal of the first supervisory-side output modulator is connected to an input terminal of the second block matching module;

An output terminal of the second supervisory-side output modulator is connected to an input terminal of the first block matching module;

An output end of the first block matching module and an output end of the second supervisory side output modulator are both connected to an input end of the fusion module;

The output end of the fusion module is connected to the input end of the first adder and the input end of the second adder respectively;

An input end of the first convolutional layer is connected to an input end of the first adder through a first upsampling layer;

An input end of the second convolutional layer is connected to an input end of the second adder through a second upsampling layer;

The first adder outputs a high-resolution left-view stereo image;

The second adder outputs a high-resolution right-view stereo image.

Further, as shown in FIG2 , the information perception module includes:

The perceptual information extractor and refinement feed-forward network are connected sequentially;

The perceptual information extractor comprises a first normalization layer, a first point-by-point convolution layer, a first depth-separable convolution layer, a first channel partitioning gating unit, a channel attention layer, a second point-by-point convolution layer, and a third adder connected in sequence; the channel attention layer is connected in parallel with the large core attention layer; the input end of the third adder is also connected to the input end of the first normalization layer;

The refined feedforward network comprises a second normalization layer, a third point-by-point convolution layer, a second depth-separable convolution layer, a second channel partitioning gating unit, a fourth point-by-point convolution layer and a fourth adder connected in sequence; the input end of the fourth adder is also connected to the input end of the second normalization layer;

The input end of the first normalization layer is the input end of the information perception module, and the output end of the fourth adder is the output end of the information perception module; the input end of the second normalization layer is connected to the output end of the third adder.

Furthermore, the channel attention layer includes: an average pooling layer, a fifth point-by-point convolutional layer and a first multiplier connected in sequence, and the input end of the average pooling layer is connected to the input end of the first multiplier.

Furthermore, the large core attention module includes: a 5*5 convolutional layer, a 7*7 convolutional layer, a sixth point-by-point convolutional layer and a second multiplier connected in sequence, and the input end of the 5*5 convolutional layer is connected to the input end of the second multiplier.

Furthermore, the internal structure of the first channel division gating unit is consistent with that of the second channel division gating unit, and the first channel division gating unit includes:

The channel partitioning layer divides each input feature map into several slices, performs element-by-element product on adjacent slices, and concatenates the obtained products in series to obtain an output feature map.

It should be understood that the information perception module (IPB) is a transformer-style convolutional block consisting of a perception information extractor and a refinement feedforward network. The two modules each contain two channel partitioning gates for redistributing features between channels.

Furthermore, the information perception module is used to:

Assume that the input features of the perceptual information extractor are The number of channels is C; the perceptual information extractor first uses 1×1 point-by-point convolution and 3×3 depth-separable convolution to extract local features from the layer-normalized feature X The formula is:

Among them, Conv _PW (·) represents point-by-point convolution, that is, convolution with a convolution kernel of 1×1, Conv _DW (·) is a depth-wise separable convolution, in which each channel performs convolution operations independently, and LN (·) is layer normalization.

Then, use the first channel partition gating unit (CS) to Divide into n slices along the channel dimension.

The features between channels are associated in pairs to form n/2 groups, and a nonlinear gating mechanism is used to integrate the information:

in, Indicates that by operating the (n-1)th and nth channels and The (n/2)th output obtained is, is the GELU activation function, ⊙ is the element-wise product; all groups The channel features of are merged and reordered to obtain redistributed features

Then, large kernel attention (LKA) and channel attention (CA) layers are used to capture long-term dependencies and global spatial distribution, respectively. The refined feed-forward network controls the information flow through channel-wise gating, allowing each channel to focus on recovering fine details that are complementary to those of other channels.

It should be understood that point-by-point convolution is ordinary convolution with a 1×1 convolution kernel, and depth-wise separable convolution is to independently assign a convolution kernel to each channel of the feature for convolution operation, and the number of parameters is greatly reduced compared to ordinary convolution.

It should be understood that the channel attention layer is used to calculate the global statistics of the feature map to strengthen the focus on important features. At the same time, the large kernel attention layer uses larger convolution kernels to capture the long-range dependencies of the image within the view, thereby strengthening the focus on local information. The combination of these two attention mechanisms enables the perceptual information extractor to effectively model the hierarchical information contained in the input image and accurately restore texture details.

It should be understood that, through the information perception module and the overall integration stack, the present invention improves the performance of the model and greatly improves the processing speed of the model. The information perception module is deployed in parallel and is used to extract the internal features of the left and right views. Therefore, through the information perception module, the corresponding internal features of the view are obtained. and

It should be understood that most of the previous stereo image super-resolution reconstruction methods calculate pixel-level feature similarity from low-resolution images for stereo matching. However, the resolution limitation of the low-resolution spatial domain itself and the image degradation make it difficult for a single pixel-level match to fully convey complementary information, which hinders the recovery of high-resolution fine-grained details. Although some methods suggest improving stereo correspondence by estimating high-resolution disparity maps, the additional disparity estimation network will increase the computational overhead and significantly increase the number of network parameters. In addition, inaccurate prior disparity estimates will in turn have an adverse effect on the correspondence between the left and right views.

The present invention rethinks stereo image correspondence from the perspective of reference reconstruction and proposes a P ² -Matching method, which regards the stereo information complement between the left and right views as a mutual reference mode and learns cross-view and cross-resolution stereo correspondence through reference-based pixel matching and tile-level matching.

Furthermore, as shown in FIG3 , the pixel matching module is used to:

Assumptions and are the low-resolution left-view and right-view stereo pairing features extracted by the i-th information perception module, which are regarded as reference features of the other view.

First, the stereo pair features are layer-normalized and then fed into point-by-point convolution, with the output of the current view as the query and the features of the other view as the key. Therefore, for the reconstruction of the left view, the following representation is obtained:

Among them, Conv _PW (·) represents point-by-point convolution, LN (·) represents layer normalization, Q _L represents the generated left view query, K _R represents the generated right view key, and V _R represents the generated point-by-point convolution. value.

For each pixel in the right view, we calculate its similarity score with all possible pixels in the left view and generate a cross attention map A _R→L :

Among them, Softmax is the normalized activation function, is the matrix multiplication, (K _R ) ^T is the transpose of the right view key value. After multiplying the right view feature value _VR matrix to calculate the score, the right feature is updated using the supplementary information of the left view

Symmetrically, for the left feature _AR→L only needs to be transposed into the cross attention map AR _→R to generate the complementary features of the left view to the right view.

Among them, ( _AR→L ) ^T is the transposition of the right cross attention, that is, A _L→R , and V _L represents the point-by-point convolution. value.

It should be understood that in order to overcome the loss of fine-grained details in the low-resolution feature space domain, in addition to pixel-level matching, the present invention also designs tile-level matching, which uses the feature representation of the high-resolution space to provide higher accuracy in high-frequency areas to guide high-resolution reconstruction. The whole process includes matching dictionary construction and super-resolution tile transmission.

Furthermore, as shown in FIG4 , the internal working process of the first block matching module is consistent with that of the second block matching module. The first block matching module is used to:

Expand the left and right view features into feature blocks;

Based on the matching dictionary, the similarity score and tile position of the current view are queried to transfer the super-resolution reference features to the current view;

Therefore, the first iteration from the left view to the right view is expressed as:

in, represents the projection matrix learned by sequential 1×1 point convolution and 3×3 depth convolution, is the high-resolution feature of the left view after projection, γ _R is a trainable channel scaling parameter, is based on right Perform a rearranged query operation. is the similarity score obtained from the query, is the feature after upsampling the right view to high resolution, It is the optimized right view feature.

In the second iteration from the right view to the left view, the guiding relationship is reversed, and the improved right view features are used. As a reference. According to formulas (9) and (10), we can get the feature of the left view More informative optimization features

It should be understood that, combined with pixel-level matching and tile-level matching mechanisms, the model proposed in the present invention can achieve sufficient cross-view and cross-resolution interaction and improve the accuracy of complementary information exchange. The correlation between views is modeled based on the cross-scale tile reproduction characteristics, because the disparity between two views and their similar tiles tend to reproduce multiple times at different scales. For example, given a pair of low-resolution images, their differences in low-resolution or high-resolution space are consistent, and similar tiles can be found. Therefore, a LR-LLR (downsampled low-resolution image) mapping can be constructed to learn the correspondence between cross-scale views, and then similar tiles are aggregated in the high-resolution space. Since most of the calculations are performed between low-resolution images and LLR images, this method is both efficient and effective.

Furthermore, the dictionary construction process includes:

For low-resolution input images First downsample it to a lower resolution Then for Each tile in Compute similarity score graph

Among them, i and j indicate that the center of the block is located in the i-th row (i corresponds to the h dimension) and the j-th column, l and r correspond to the left and right views respectively, and Norm represents the normalization operation. and A collection of tiles representing the left view and the right view respectively.

Then, based on the constructed similarity score graph Search along the epipolar line in the low-resolution image on the left If we want to find similar tiles in , then the values and positions of the elements in the dictionary on the left are:

in, and Representation tile The maximum similarity score and position of It means to find the maximum value, Indicates the location of the maximum value.

Transpose the matching dictionary of the left view to construct another dictionary for the right view, using and express.

In order to provide high-quality super-resolution features for block matching, after sub-pixel convolution, the upsampled features It is fed into the supervised side-out modulation module (SSOM) to improve the feature representation.

Further, as shown in FIG5 , the internal structures of the first supervisory side output modulator and the second supervisory side output modulator are consistent, and the first supervisory side output modulator is used to:

Left feature after upsampling First, it is mapped to the image layer through channel point convolution.

Then, the left feature Image after bicubic interpolation Add together to restore an initial super-resolution rough image

in, represents bicubic interpolation upsampling, The left low-resolution image, Conv _1×1 is a 1×1 convolution. It is remapped to the feature space through 3×3 convolution, and the residual nonlinear gate operation is performed on the obtained features to generate feature Used as reference features in patch matching to facilitate the generation of right view features.

Among them, Sigmoid is the activation function, ⊙ is the pixel-level dot product, and Conv _3×3 is a 3×3 convolution.

For the predicted Based on the high-resolution ground-truth image The feature structure similarity and pixel distance provide clear optimization supervision information.

Will and Projected into the same feature space, the structural similarity score (StructuralSimilarity Index Measurement, SSIM) is used as the screening criterion:

Then retain the top K channels with the highest similarity scores:

in, Represents high-resolution features based on ground-truth values and To optimize the output modulator of the first supervision side, point-wise convolution and residual connection are used to recover the new image. It should be understood that based on the parallel first supervisory side output modulator and the second supervisory side output modulator, the proposed RIISSR network generates optimized super-resolution stereo features and Patch matching is performed as super-resolution references to each other.

Furthermore, as shown in FIG6 , the fusion module includes:

First, the super-resolution feature The high-resolution reference feature to its right Cascaded along the channel dimension and then sent to the convolution layer to generate The learnable parameters β and γ are of the same size;

Then reference the feature in the right view Use layer normalization, where and They are The mean and standard deviation of:

The updated γ _R and β _R are merged into the normalized features as weight parameters In, by Finally, the fusion module output

Furthermore, the internal functions of the first upsampling module and the second upsampling module are consistent, and the first upsampling module is used to improve feature resolution by bilinear interpolation.

Furthermore, the training process of the trained image reconstruction model includes:

Constructing a training set, wherein the training set is a left view and a right view stereo image of a known reconstructed image;

The training set is input into the image reconstruction model to train the model. When the total loss function value of the model no longer decreases or the number of iterations exceeds the set number, the training is stopped to obtain the trained image reconstruction model.

Furthermore, the total loss function L _total of the model is expressed as:

L _total ＝L _SO +L _MSE +λ ₃ ·L _FC (20)

Among them, λ ₃ is set to 0.01;

The supervision of the output modulator on the supervisory side is denoted as L _SO , and its formula is:

in, and represents the high-resolution stereo image truth, and represents the stereo image pair after super-resolution of the entire network, ‖·‖ ² is the binary norm, λ ₁ and λ ₂ are set to 0.01 and 0.02 respectively, and N is the number of pixel values.

L _MSE is the MSE loss:

in, Represents the output of the left and right views, represents a high-resolution stereo image ground-truth pair;

The frequency loss L _FC is defined as:

Here, FFT(·) is fast Fourier transform, and the constant ε in all experiments is set to 1×10 ^-6 .

In fact, the stereo information complementation between the left and right views can be regarded as a mutual reference mode. Using the reconstructed SR image as the reference image of another view helps to better reconstruct the image of another perspective, because high-resolution features can provide richer texture details, which is conducive to the use of cross-view information. In view of this, the present invention realizes the stereo image super-resolution SSR task from a new perspective, that is, it is modeled as a reference-based image super-resolution reconstruction (Ref-SR) task, and proposes a reference-based interactive reference stereo image super-resolution reconstruction method (Reference-basedIterative Interaction for Stereo Image Super-Resolution, RIISSR).

RIISSR regards the stereo images of the left view and the right view as each other's reference images, and learns the correspondence between views in the left view and the right view space to achieve full interaction. To this end, RIISSR proposes a ^P2 -Matching method, which simulates the dependency between views through reference-based pixel matching and block matching. Specifically, when performing pixel-level matching between two different views, the present invention first designs a parallel weight-sharing Information Perception Block (IPB) to extract the intra-view information features of the two views. IPB unifies channel attention, large core attention, and feature redistribution to achieve powerful feature representation.

Between two parallel IPBs, the present invention calculates the feature similarity of each pixel in the current view by referring to all possible differences in the other view, thereby learning low-resolution stereo correspondence. By stacking parallel IPBs and pixel matching, the model can form an internal crossover, internal iteration mechanism.

As for the tile-level matching, the present invention utilizes the cross-scale tile recursive property of stereo images (ie, similar image structure information can be well preserved in images of different resolutions) to measure their tile similarities, thereby generating a matching dictionary for query.

The subsequent matching dictionary will be further projected into the high-resolution space, and the network will interact with the high-resolution view information by using the super-resolution results of the two views as mutual references. The super-resolution-low-resolution information transfer form breaks the resolution limitation of the traditional SSR method that can only interact in the low-resolution space, greatly increasing the amount of effective information.

In addition, the present invention introduces a supervised side output modulator (SSOM), which can re-weight the super-resolution local features of the view and provide high-resolution stereo image ground truth supervision for the super-resolution image to achieve patch-level matching. With the support of SSOM, pixel-level matching and patch-level matching are achieved, thereby improving the accuracy and stereo consistency of SSR.

The overall framework of the stereo image super-resolution network based on iterative cross-reference is shown in Figure 1. It is a two-stream architecture consisting of three key parts: the information perception module (IPB), which is used to extract the features of the left and right views; the ^P2 -Matcing matching module, which consists of pixel-level matching and patch-level matching, is used to learn the correspondence between stereo images; and the supervised side output modulator (SSOM), which is used to improve the representation of intra-view super-resolution features and restore the high-resolution reference view. Given a low-resolution stereo image pair The goal of RIISSR is to super-resolve them into high-quality, high-resolution versions In the SSR task, both intra-view information and inter-view information play a vital role in super-resolution reconstruction. In order to extract the spatial features of the left and right views, after performing 3×3 convolution to map the image features to 64 channels, M IPBs are stacked in parallel cascade to obtain the low-resolution features of the corresponding views with a deeper network extraction. Due to the particularity of stereo images, the IPBs of the two views adopt a simple weight sharing strategy and are strictly symmetrical. As analyzed above, the left and right view features of the low-resolution spatial domain are not accurate enough, so their feature extraction and cross-view interaction require multiple iterative updates to continuously correct the errors caused by resolution and original degradation, and provide more accurate reference features for subsequent cross-resolution tile-level matching.

To this end, the present invention embeds pixel-level matching between any two parallel IPBs, performs cross-view interaction and aggregates complementary information within the view. By iteratively deploying IPBs with pixel matching function, the network can form an internal crossover and internal iteration mechanism, thereby improving the network's feature exchange capability and generating enhanced low-resolution stereo features. and

Afterwards, the present invention uses a sub-pixel convolution (Pixel-Shuffle) operation to and Perform super-resolution processing to obtain paired super-resolution features and

The supervised side output modulator (SSOM) re-weights the super-resolution features of the left and right views and provides ground-truth supervision of high-resolution stereo images based on feature structure similarity and pixel distance to emphasize more discriminative features and generate intermediate-level SR images. Subsequent tile-level matching utilizes super-resolution results and We use them as cross-references to learn high-resolution stereo correspondences, leveraging the property that similar patches recur across scales to facilitate matching.

In order to generate the final super-resolution stereo image, the present invention uses a similar spatial adaptive module in MASA as the fusion module of the network (see Figure 1), which can remap the distribution of high-resolution reference features in one view to another view. As shown in Figure 2, the RIISSR model of the present invention makes an intuitive comparison of different 4× super-resolution methods on the Flickr1024 and Middlebury validation datasets. In the first group of complex outdoor scenes, the RIISSR proposed by the present invention accurately restores the texture of the windowsill, showing clearer edges and details. The single-image super-resolution algorithm SwinIR and the binocular super-resolution algorithm in the first row are basically unable to restore the texture details, showing a completely blurred state. Although the single-image SwinIR algorithm has better quantitative performance than the binocular algorithm PASSRnet, its visual reconstruction details are not as good. The binocular super-resolution method in the second row restores certain rough details. In the reconstruction of the "sword" scene in the room, RIISSR also significantly generates fine details at the dents of the basket, and the sharpness in the lower right corner is even better than HR. Visual performance. Similar to outdoor scenes, the methods in the first row cannot effectively reconstruct the scene details, while the methods in the second row have better visual effects. The results show that compared with the existing state-of-the-art SSR methods, RIISSR has higher reconstruction performance and excellent indoor and outdoor scene restoration effects. Figure 7 shows the visualization results of different image super-resolution methods in Example 1.

Embodiment 2

This embodiment provides an iterative interactive reference stereo image super-resolution reconstruction system;

Iterative cross-reference stereo image super-resolution reconstruction system, including:

An acquisition module is configured to: acquire two images to be reconstructed; the two images to be reconstructed include: a low-resolution left-view stereo image and a low-resolution right-view stereo image;

A reconstruction module is configured to: input the two images to be reconstructed into the trained image reconstruction model to obtain a high-resolution left-view stereo image and a high-resolution right-view stereo image;

Among them, the trained image reconstruction model uses several information perception modules to extract features within the view of the two images to be reconstructed; the model also simulates the dependency between views based on the pixel matching module and the block matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; the matching dictionary is projected into the high-resolution space, and high-resolution view information interaction is performed by using the feature maps of the two views as mutual references; the model also re-weights the feature maps of the views through the supervisory side output modulator to achieve block-level matching.

It should be noted that the acquisition module and the reconstruction module correspond to steps S101 to S102 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

Embodiment 3

This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Embodiment 4

This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. The iterative interactive reference type stereo image super-resolution reconstruction method is characterized by comprising the following steps of:

acquiring two images to be reconstructed; the two images to be reconstructed comprise: a low resolution left view stereoscopic image and a low resolution right view stereoscopic image;

inputting two images to be reconstructed into a trained image reconstruction model to obtain a high-resolution left view stereoscopic image and a high-resolution right view stereoscopic image;

The method comprises the steps that a trained image reconstruction model adopts a plurality of information perception modules to extract features in views of two images to be reconstructed; the model also simulates the dependency relationship between views based on a pixel matching module and a tile matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; projecting the matched dictionary into a high-resolution space, and performing high-resolution view information interaction by using feature graphs of two views as cross references; and the model also re-weights the characteristic diagram of the view through a supervision side output modulator to realize block level matching.

2. The iterative cross-reference stereoscopic image super-resolution reconstruction method of claim 1, wherein the information sensing module is configured to:

Assume that the input features of the perceptual information extractor are The number of channels is C; the perception information extractor firstly uses 1×1 point-by-point convolution and 3×3 depth separable convolution to extract local features from the layer normalized features XThe formula is:

Conv _PW (·) represents point-by-point convolution, namely convolution with a convolution kernel of 1×1, conv _DW (·) is convolution with separable depth, each channel independently carries out convolution operation of convolution operation, and LN (·) is layer normalization;

Then, the gating unit is divided by using the first channel Dividing into n slices along a channel dimension;

The characteristics among channels are associated in pairs to form n/2 groups, and a nonlinear gating mechanism is adopted to integrate information:

Wherein, Indicating the pass through operation of the (n-1) th and n-th channelsAndThe (n/2) th output obtained,Is GELU the activation function, as by element product; all groups are combinedChannel feature merging and reordering to obtain redistributed featuresThen, capturing long-term dependence and global spatial distribution respectively by using the large-core attention layer and the channel attention layer; the refined feed-forward network controls the information flow through channel division gating.

3. The iterative cross-reference stereoscopic image super-resolution reconstruction method of claim 1, wherein the pixel matching module is configured to:

Assume that AndThe low-resolution left view angle and the low-resolution right view angle three-dimensional pairing features extracted by the ith information perception module are respectively regarded as reference features of another view;

Firstly, carrying out layer normalization processing on three-dimensional pairing features, and then respectively sending the three-dimensional pairing features into point-by-point convolution, wherein the output of the current view is used as a query and the features of the other view are used as key values; for the reconstruction of the left view, there is the following representation:

Wherein Conv _PW (s)) represents a point-by-point convolution, LN (s)) represents layer normalization, Q _L represents a generated left view query, K _R represents a generated right view key, and V _R represents the generation of the point-by-point convolution A value; for each pixel in the right view, calculate its similarity score to all possible pixels in the left view, generating a cross attention map a _R→L:

wherein Softmax is the normalized activation function, For matrix multiplication, (K _R)^T is the transposition of the key value of the right view; after the score is calculated by matrix multiplication with the characteristic value V _R of the right view, the characteristic of the right side is updated by the supplementary information of the left view

Symmetrically, for the left-hand featureA _R→L need only be transposed to cross-attention to FIG. A _L→R, thereby generating the left view to right view supplemental features

Wherein (A _R→L)^T is the transpose of the right cross-attention A _L→R,V_L represents the point-wise convolution generationValues.

4. The iterative cross-referenced stereoscopic image super-resolution reconstruction method of claim 1, wherein the tile matching module is configured to: unfolding the left view feature and the right view feature into feature blocks;

Inquiring the similarity score and the block position of the current view based on the matching dictionary, and transferring the reference feature SR to the current view; thus, the first iteration from left view to right view is expressed as:

Wherein, Representing a projection matrix learned by sequential 1 x 1 point convolution and 3 x 3 depth convolution,Is the left-view high-resolution feature after projection, gamma _R is a trainable channel scaling parameter,Is based onFor a pair ofA query operation of the rearrangement is performed such that,For the similarity score obtained by the query,For right view upsampling to high resolution features,The right view characteristics after optimization;

in the second iteration from right view to left view, the guiding relationship is reversed, utilizing the improved right view features As a reference; according to formulas (9) and (10), the characteristics of the original left view are obtainedOptimization features with greater information content

5. The iterative cross-referenced stereoscopic image super-resolution reconstruction method of claim 4, wherein the dictionary construction process comprises:

for low resolution input images Downsampling it first to a lower resolutionThen for the followingEach tile in (1)Computing a similarity score graph

Where i, j denote that the center of the tile is located in the ith row and the jth column, l, r correspond to the left and right views, respectively, norm denotes the normalization operation,AndTile sets representing left and right views, respectively; then, according to the constructed similarity score graphSearching for its left low resolution image along epipolar lineThe values and positions of the elements in the left dictionary are then:

Wherein, AndRepresenting a tileIs used to determine the maximum similarity score and location of (1),Indicating that the maximum value is to be found,Representing the position of the maximum value; transpose the matched dictionary of the left view to construct another dictionary of the right view usingAndAnd (3) representing.

6. The iterative cross-reference stereoscopic image super-resolution reconstruction method of claim 1, wherein the supervisory side output modulator is configured to:

Upsampled left-hand feature Firstly, mapping to an image layer through channel point convolution;

Then, the left side feature With bicubic interpolated imagesAdding to restore an initial super-resolution coarse image

Wherein,Indicating that bicubic interpolation upsampling is performed,Conv _1×1 is a1×1 convolution for the left low resolution image; re-mapping it to feature space by 3 x 3 convolution, performing residual nonlinear gate operation on the obtained features, and generatingFeatures (e.g. a character)Is used as a reference feature in tile matching to facilitate the generation of right view features;

wherein Sigmoid is the activation function, by pixel-level dot product, conv _3×3 is the 3 x 3 convolution;

For prediction According to and high resolution truth imageProviding explicit optimization supervision information for feature structure similarity and pixel distance;

Will be AndProjected into the same feature space and used as a screening criterion with a structural similarity score:

The top K channels with highest similarity scores are then kept:

Wherein, Representing high resolution features based on truth valuesAnd (3) withTOP-K selection of structural similarity scores between; restoring new images using point-wise convolution and residual connection

7. The iterative interactive reference type stereo image super-resolution reconstruction system is characterized by comprising the following components:

An acquisition module configured to: acquiring two images to be reconstructed; the two images to be reconstructed comprise: a low resolution left view stereoscopic image and a low resolution right view stereoscopic image;

A reconstruction module configured to: inputting two images to be reconstructed into a trained image reconstruction model to obtain a high-resolution left view stereoscopic image and a high-resolution right view stereoscopic image;

The method comprises the steps that a trained image reconstruction model adopts a plurality of information perception modules to extract features in views of two images to be reconstructed; the model also simulates the dependency relationship between views based on a pixel matching module and a tile matching module; the pixel matching module forms internal intersection and internal iteration; the block matching module generates a matching dictionary; projecting the matched dictionary into a high-resolution space, and performing high-resolution view information interaction by using feature graphs of two views as cross references; and the model also re-weights the characteristic diagram of the view through a supervision side output modulator to realize block level matching.

8. An electronic device, comprising:

A memory for non-transitory storage of computer readable instructions; and

A processor for executing the computer-readable instructions,

Wherein the computer readable instructions, when executed by the processor, perform the method of any of the preceding claims 1-6.

9. A storage medium, characterized by non-transitory storage of computer readable instructions, wherein the instructions of the method of any of claims 1-6 are performed when the non-transitory computer readable instructions are executed by a computer.

10. A computer program product comprising a computer program for implementing the method of any of the preceding claims 1-6 when run on one or more processors.