CN108268815B - Method and device for understanding image scene - Google Patents

Abstract

An embodiment of the present invention provides a method for understanding an image scene, including: acquiring an original image of a scene; performing a convolution operation on the original image to obtain a convolution output; and processing the convolution output through a global convolution network , obtaining a processing result; and performing boundary refinement on the processing result to obtain an image scene understanding result. The embodiment of the present invention utilizes a global convolutional network to effectively increase the effective receptive field, and further utilizes boundary refinement to increase the discrimination of the boundary, thereby effectively improving the overall performance of the system.

Description

Method and device for image scene understanding

technical field

The present invention relates to the field of video surveillance, and more particularly, to a method and device for image scene understanding.

Background technique

The concept of deep learning originated from the research of artificial neural network. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to discover distributed feature representations of data. In computer vision and related fields, emerging deep learning methods have made great progress compared to traditional methods in the past. Convolutional neural network (CNN) is a machine learning model under deep supervised learning, and it is the core operation of deep learning.

Scene understanding has important applications in the field of video surveillance. Traditional scene understanding systems are often implemented through fully convolutional networks, but this implementation method does not take into account the limitations of the effective receptive field. In general, the theoretical receptive field of a fully convolutional network is the entire image, but the actual effective receptive field is often a limited area. Objects within this limited area can be well understood, but objects located outside this limited area will have larger errors. For example: a small car can be understood by a good segmentation, but a large truck may be understood as a combination of different objects because of its size. On the other hand, the traditional method has a great misjudgment for the boundary of the object. For example, some people leaning on the car will be understood as the body. It can be seen that the existing method will lead to a decrease in the overall performance of the system.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above-mentioned problems. The present invention provides a method for image scene understanding, which can improve the overall performance of the system.

According to a first aspect of the present invention, a method for image scene understanding is provided, comprising:

Get the original image of the scene;

performing a convolution operation on the original image to obtain a convolution output; and

The convolution output is processed through a global convolution network to obtain a processing result;

The boundary refinement is performed on the processing result to obtain the result of image scene understanding.

Exemplarily, performing a convolution operation on the original image to obtain a convolution output, including:

The original image is subjected to a convolution operation through N convolutional neural networks to obtain N-way convolution outputs;

Wherein, the spatial dimensions of the N-way convolution outputs are not equal to each other, and the spatial dimensions of the N-way convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive integer greater than 1.

Exemplarily, the convolution output is processed through a global convolution network to obtain a processing result, including:

The N channels of convolution outputs are respectively processed through the N global convolution networks, and N processing results corresponding to the N channels of convolution outputs are obtained.

Exemplarily, the N channels of convolution outputs are respectively processed through the N global convolution networks, and N processing results corresponding to the N channels of convolution outputs are obtained, including:

Perform multiple convolution on the i-th convolution output in the N-way convolution outputs, add the outputs of the multiple convolutions, and determine the result of the addition as the same as the i-th convolution output. The i-th processing result corresponding to the road convolution output;

Among them, the value of i ranges from 1 to N.

Exemplarily, performing boundary refinement on the processing result to obtain an image scene understanding result, including:

Perform boundary refining on the N processing results respectively, and obtain N boundary refining results corresponding to the N processing results one-to-one;

According to the N boundary refinement results, the image scene understanding result is obtained.

Exemplarily, performing boundary refining on the N processing results, respectively, to obtain N boundary refining results corresponding to the N processing results one-to-one, including:

Correcting the i-th processing result in the N processing results, combining the corrected result with the i-th processing result to obtain the i-th boundary refining result;

Among them, the value of i ranges from 1 to N.

Exemplarily, obtaining the image scene understanding result according to the N boundary refining results, including:

A fusion operation is performed on the N boundary refinement results to obtain the image scene understanding result.

Exemplarily, before the acquiring the original image, the method further includes:

Train the global convolutional network.

According to a second aspect of the present invention, there is provided an image scene understanding device, comprising:

The acquisition module is used to acquire the original image of the scene;

a convolution module, configured to perform a convolution operation on the original image acquired by the acquisition module to obtain a convolution output;

a global convolution network module for processing the convolution output obtained by the convolution module through a global convolution network to obtain a processing result; and

The boundary refining module is configured to perform boundary refining on the processing result obtained by the global convolutional network module to obtain the result of image scene understanding.

Exemplarily, the convolution module is used for:

The original image is subjected to a convolution operation through N convolutional neural networks to obtain N-way convolution outputs;

Wherein, the spatial dimensions of the N-way convolution outputs are not equal to each other, and the spatial dimensions of the N-way convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive integer greater than 1.

Exemplarily, the global convolutional network module is used for:

The N channels of convolution outputs are respectively processed through the N global convolution networks, and N processing results corresponding to the N channels of convolution outputs are obtained.

Exemplarily, the i-th global convolutional network in the N global convolutional networks is used for:

Perform multiple convolution on the i-th convolution output in the N-way convolution outputs, add the outputs of the multiple convolutions, and determine the result of the addition as the same as the i-th convolution output. The i-th processing result corresponding to the road convolution output;

Among them, the value of i ranges from 1 to N.

Exemplarily, the boundary refining module includes N boundary refining sub-modules and fusion sub-modules:

The N boundary refining sub-modules are used to perform boundary refining on the N processing results respectively, and obtain N boundary refining results corresponding to the N processing results one-to-one;

The fusion sub-module is configured to refine the results according to the N boundaries to obtain the image scene understanding result.

Exemplarily, the ith boundary refining submodule in the N boundary refining submodules is used for:

Correcting the i-th processing result in the N processing results, combining the corrected result with the i-th processing result to obtain the i-th boundary refining result;

Among them, the value of i ranges from 1 to N.

Exemplarily, the fusion submodule is used for:

A fusion operation is performed on the N boundary refinement results to obtain the image scene understanding result.

Exemplarily, the apparatus further includes a training module for: training the global convolutional network.

The apparatus described in the second aspect can be used to implement the image scene understanding method of the foregoing first aspect.

According to a third aspect of the present invention, there is provided a computer chip including a processor and a memory. The memory stores an instruction code, the processor is configured to execute the instruction code, and when the processor executes the instruction code, the method for understanding an image scene described in the first aspect can be implemented.

The embodiment of the present invention utilizes a global convolutional network to effectively increase the effective receptive field, and further utilizes boundary refinement to increase the discrimination of the boundary, thereby effectively improving the overall performance of the system.

Description of drawings

The above and other objects, features and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention, and constitute a part of the specification, and together with the embodiments of the present invention, they are used to explain the present invention, and do not limit the present invention. In the drawings, the same reference numbers generally refer to the same components or steps.

1 is a schematic block diagram of an electronic device according to an embodiment of the present invention;

FIG. 2 is a schematic flowchart of a method for image scene understanding according to an embodiment of the present invention;

3 is a schematic flowchart of a method for image scene understanding according to an embodiment of the present invention;

4 is another schematic flow chart of a method for image scene understanding according to an embodiment of the present invention;

5 is a schematic flowchart of a method for global convolutional network processing according to an embodiment of the present invention;

6 is a schematic flowchart of a method for boundary refining processing according to an embodiment of the present invention;

7 is a schematic block diagram of an image scene understanding apparatus according to an embodiment of the present invention;

FIG. 8 is another schematic block diagram of an image scene understanding apparatus according to an embodiment of the present invention.

Detailed ways

In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of the embodiments of the present invention, and it should be understood that the present invention is not limited by the example embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

The embodiment of the present invention may be applied to an electronic device, and FIG. 1 is a schematic block diagram of the electronic device according to the embodiment of the present invention. The electronic device 10 shown in FIG. 1 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, an image sensor 110, and one or more non-image sensors 114, these components being connected via a bus Systems 112 and/or other forms of interconnection. It should be noted that the components and structures of the electronic device 10 shown in FIG. 1 are only exemplary and not restrictive, and the electronic device may also have other components and structures as required.

The processor 102 may include a central processing unit (Central Processing Unit, CPU) 1021 and/or a graphics processing unit (Graphics Processing Unit, GPU) 1022, or include other forms of processing with data processing capability and/or instruction execution capability unit, and may control other components in the electronic device 10 to perform desired functions.

The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory 1041 and/or non-volatile memory 1042 . The volatile memory 1041 may include, for example, a random access memory (Random Access Memory, RAM) and/or a cache memory (cache). The non-volatile memory 1042 may include, for example, a read-only memory (Read-Only Memory, ROM), a hard disk, a flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, which may be executed by the processor 102 to implement various desired functions. Various application programs and various data, such as various data used and/or generated by the application program, etc. may also be stored in the computer-readable storage medium.

The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.

The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

The image sensor 110 may capture user-desired images (eg, photos, videos, etc.) and store the captured images in the storage device 104 for use by other components.

Exemplarily, the electronic device 10 may be implemented as an image acquisition terminal such as a smart phone, a tablet computer, an access control system, and the like.

The electronic device in the embodiment of the present invention may be a device in the field of video surveillance, and the specific form of the device is not limited in the embodiment of the present invention.

FIG. 2 is a schematic flowchart of a method for understanding an image scene according to an embodiment of the present invention. The method shown in FIG. 2 includes:

S101, acquiring an original image of a scene.

In one example, the original image containing the scene may be an image acquired in real time. For example, the original image may be a frame of image in a video captured by a camera or a picture captured by a camera. In other examples, the original image containing the scene can also be an image from any source. Here, the original image including the scene may be video data or picture data.

In this embodiment of the present invention, the spatial dimension of the original image may be expressed as w×h. As an example, it can be assumed that the spatial dimension of the original image is 512×512. The original image may also have a third dimension. For example, the third dimension is 1 (indicating that the corresponding image is a grayscale image) or 3 (indicating that the corresponding image is an RGB color image). It should be understood that, according to the nature of the image, The number of the third dimension may also be other values, which are not limited in the present invention.

S102, perform a convolution operation on the original image to obtain a convolution output.

Exemplarily, a convolution operation can be performed directly on the original image to obtain a convolution output. However, due to the large dimension of the original image, this operation also requires relatively high hardware such as processors.

Exemplarily, S102 may include: performing a convolution operation on the original image through N convolutional neural networks to obtain N convolution outputs. Wherein, the spatial dimensions (w×h) of the N-way convolution outputs are not equal to each other, and the spatial dimensions of the N-way convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive value greater than 1. Integer.

Generally, the form of the convolution output is in the form of a tensor, so it can be understood that the dimension of the convolution output is the dimension of the tensor. Exemplarily, the dimension of the convolution output can be represented as w×h×c, where the third dimension c represents the channel dimension, which can also be referred to as the channel dimension.

For example, taking N=4 as an example, it is assumed that four convolution operations are performed on the original image through four convolutional neural networks to obtain four convolution outputs. As shown in Figure 3, the four convolution operations are performed in sequence. are S1021, S1022, S1023 and S1024, and the dimensions of the 4-way convolution outputs are: 128×128×256, 64×64×512, 32×32×1024, and 16×16×2048, respectively. It can be seen that the spatial dimension (w dimension × h dimension (respectively 128 × 128, 64 × 64, 32 × 32, 16 × 16)) in the dimensions of the four-way convolution output is smaller than the spatial dimension of the original image (w dimension ×h dimension (512×512)). The third dimension (c dimension) of the convolution output (in this case, 256, 512, 1024, and 2048, all representing the number of channels of the convolution output).

Optionally, before S1021, a convolution operation may also be performed on the original image, for example, convolution operation 0 is performed first, as shown in S1020 in FIG. 4 , the output dimension of the convolution operation 0 is 256×256×64 . In this way, the dimension of the convolution output can be gradually changed, so that the processing accuracy of the process can be guaranteed.

S103, processing the convolution output through a global convolutional network (Global Convolutional Network, GCN) to obtain a processing result.

Wherein, the global convolutional network (GCN) may also be referred to as a global convolutional neural network (Global Convolutional Neural Network, GCNN), which is not limited in the present invention.

Optionally, adding a global convolutional network at the back end of the S102 convolution operation can increase the effective receptive field. In a convolutional neural network, the receptive field can be defined as: the area of influence on the feature map output by each layer of the convolutional neural network due to changes in pixels in the original image.

Exemplarily, if N channels of convolution outputs are obtained in S102, S103 may include: processing the N channels of convolution outputs through the N global convolution networks, respectively, to obtain a one-to-one correspondence with the N channels of convolution outputs. The N processing results of .

It can be seen that for N-way convolution outputs, N processing results can be obtained in parallel.

For example, taking N=4 as an example, that is, assuming a 4-way convolution operation, as shown in FIG. 3 , then S103 may include 4-way global convolutional neural network processing S1031-S1034 corresponding to the 4-way convolution output. And, in the obtained 4 processing results, the w dimension and h dimension in the dimension of each processing result are consistent with the corresponding dimension of the corresponding convolution output, and the c dimension in the dimension of each processing result represents the global volume The number of types of objects that the product network can recognize. That is to say, the w dimension in the dimension of each processing result is consistent with the w dimension of the corresponding convolution output, the h dimension in the dimension of each processing result is consistent with the h dimension of the corresponding convolution output, and each processing result The c dimension of the dimensions represents the number of kinds of objects that the global convolutional network can recognize.

As an example, the dimension of the convolution output of convolution operation 1S1021 is 128×128×256, then, in the dimension of the processing result of the corresponding GCN 1 processing S1031, the w dimension of the processing result is equal to the w dimension of the corresponding convolution output dimension, namely 128; the h dimension of the processing result is equal to the h dimension of the corresponding convolution output, namely 128; the c dimension of the processing result is equal to the number of object types that can be identified by the global convolutional network, for example 21. Thus, the dimension of the processing result processed by GCN 1 is 128×128×21. The dimension determination method of the processing results from GCN 2 processing to GCN 4 processing is similar to that of GCN1 processing, and will not be listed one by one here. Wherein, the value of the c dimension represents the number of types of objects that the global convolutional network can recognize, for example, 21. It should be understood that the value of the c dimension can be set according to practical applications, which is not limited in the present invention.

Specifically, processing the N-way convolution outputs through the N global convolutional networks respectively, to obtain N processing results corresponding to the N-way convolution outputs one-to-one, may include: processing the N-way convolution outputs. The i-th convolution output in the N-way convolution outputs is subjected to multi-channel convolution, the outputs of the multi-channel convolution are added, and the result of the addition is determined as the i-th channel convolution Output the corresponding i-th processing result. Among them, the value of i ranges from 1 to N.

It should be understood that each of the N global convolutional networks may include multiple convolution branches to perform multiple convolutions, wherein each convolution branch performs a corresponding convolution operation, wherein the multiple convolution branches It can be performed in parallel, and the number of multiple convolutions is not limited, for example, 2-way convolution, or 3-way convolution, etc. The computing capability and the like are preset, which is not limited in the present invention. In addition, any one of the multiple convolutions may include at least one convolution operation, and the number of convolution operations performed by any two of the multiple convolutions may be the same or different. For example, a multi-way convolution is a 2-way convolution, the first convolution in the 2-way convolution can include 2 convolution operations, and the second convolution in the 2-way convolution can include 1 or 2 convolutions Product operation, etc.

For example, if the multi-channel convolution is a 2-channel convolution, and each channel of convolution includes 2 convolution operations. Then, regarding any GCN processing (any one of GCN 1 to GCN 4 ), the specific process thereof may be shown in FIG. 5 . Assuming that the dimension of the convolution output received by GCNi (1≤i≤N) is w1×h1×c1, then perform (a) and (b) in parallel: (a), perform two convolution operations in sequence, the first The input c-dimension of the second convolution operation is c1, the output c-dimension is 21, the input c-dimension of the second convolution operation is 21, and the output c-dimension is also 21. Do the first convolution operation in the direction, and the second convolution operation in the vertical direction; (b), perform two convolution operations in turn, the input c dimension of the first convolution operation is c1, and the output c dimension is 21, the c-dimension of the input of the second convolution operation is 21, and the c-dimension of the output is also 21. In one embodiment, the first convolution operation can be performed in the vertical direction, and the second convolution operation can be performed in the horizontal direction. Convolution operation.

The results of (a) and (b) are further summed (Sum) to obtain the processing result, and the dimension of the processing result is w1×h1×21, for example, for GCN1 processing, the dimension of the processing result is 128×128×21 ; for GCN2 processing, the dimension of the processing result is 64×64×21; for GCN3 processing, the dimension of the processing result is 32×32×21; for GCN4 processing, the dimension of the processing result is 16×16×21.

It can be seen that some underlying network information learned from general pictures in S102 is very effective for scene understanding. The convolutional neural network in S102 can be implemented by resnet, which can effectively utilize the characteristics of resnet itself, and the global convolutional network in S103 can also be implemented by resnet, which can effectively increase the effective receptive field and improve the system performance. performance.

S104 , performing boundary refinement (Boundary Refinement) on the processing result to obtain an image scene understanding result.

Optionally, linear correction may be performed on the processing result, and the linear correction result may be combined with the processing result, so as to obtain the result of image scene understanding.

Exemplarily, if N processing results are obtained in S103, S104 may include:

The N processing results are respectively subjected to boundary refining to obtain N boundary refining results corresponding to the N processing results one-to-one; and the image scene understanding result is obtained according to the N boundary refining results.

It can be seen that for N processing results, N boundary refining results can be obtained in parallel.

For example, taking N=4 as an example, that is, assuming that there are 4 processing results, as shown in FIG. 3 , then S104 may include 4 boundary refining processes S1041-S1044 corresponding to the 4 processing results. In addition, among the four obtained boundary refining results, the dimension of each boundary refining result is consistent with the dimension of the corresponding processing result.

As an example, the dimension of the boundary refinement result obtained by the boundary refinement process S1041 (ie, BR 1 ) is the same as that of the GCN 1 processing result, which is 128×128×21. The dimensions of the boundary refining results of the boundary refining processes S1042 to S1044 (ie, BR 2 to BR 4 ) can be determined similarly, and are not listed one by one here.

Specifically, performing boundary refining on the N processing results respectively to obtain N boundary refining results corresponding to the N processing results one-to-one, which may include: performing the i-th processing result on the N processing results. Correction is performed, and the corrected result is combined with the i-th processing result to obtain the i-th boundary refining result. The value of i ranges from 1 to N.

Alternatively, the corrections may be linear corrections and non-linear corrections. The present invention does not limit the specific execution method of the correction.

For example, the correction may include sequential convolution operations, Rectified Linear Unit (ReLU) operations, and convolution operations. Then, the correction may include the first convolution operation, the ReLU operation and the second convolution operation in sequence as shown in S1040 in FIG. 6 .

As shown in FIG. 6 , assuming that the dimension of the processing result is w1×h1×21, the dimension of the boundary refinement result is still w1×h1×21. The first convolution operation, the ReLU operation, and the second convolution operation are sequentially included in S1040. Among them, the c-dimension of the input of the first convolution operation is 21, and the c-dimension of the output is 21; the c-dimension of the input of the ReLU operation is 21, and the c-dimension of the output is 21; the c-dimension of the input of the second convolution operation is 21. The c dimension of the output is 21. The value of the c-dimension represents the number of kinds of recognized objects. Exemplarily, the size of the convolution kernel of the first convolution operation, the ReLU operation and the second convolution operation may be 3×3. It should be understood that other values may be taken as the size of the convolution kernel according to the needs of the calculation. , such as 5×5, 7×7, etc., which are not limited in the present invention.

The corrected result is combined with the corresponding processing result (summation operation) to obtain the corresponding boundary refinement result. The dimension of the obtained boundary refinement result is equal to the dimension of the corresponding processing result.

Specifically, obtaining the image scene understanding result according to the N boundary refining results may include: performing a fusion operation on the N boundary refining results to obtain the image scene understanding result.

As shown in FIG. 3 , N=4, and S1045 includes: performing a fusion operation on the four boundary refining results.

Illustratively, the fusion operation may include an addition operation and an enlargement operation. In this embodiment of the present invention, the zoom-in operation may also be referred to as a deconvolution operation, which is not limited in the present invention.

Exemplarily, performing a fusion operation on the N boundary refining results to obtain the image scene understanding result may include: performing a fusion operation on the N boundary refining results to obtain a Score Map, and then based on The Score Map obtains the result of scene understanding of the image.

Since the dimensions of the N boundary refining results are not equal to each other, a fusion operation is performed on the N boundary refining results, including: repeating the following (a1) until only one result remains: (a1) Enlarging the result of the lowest dimension To the next-lowest dimension, and then add the result of the next-lowest dimension. Then, the spatial dimension (w×h) of the remaining result is enlarged to the same spatial dimension (w×h) as the original image to obtain the Score Map, and then the result of the image scene understanding is obtained based on the Score Map.

For example, assuming N=4, the dimensions of the N boundary refinement results are: 128×128×21, 64×64×21, 32×32×21, and 16×16×21, respectively. Then the fusion operation can be as shown in Figure 4, which can include:

1. First, perform S1051, and perform the enlargement operation on the result of the lowest dimension of 16×16×21 (that is, the boundary refining result of S1044), and enlarge it to the next lowest dimension of 32×32×21; then perform S1052, and the enlarged result is The results of the next lower dimension (ie, the boundary refinement results of S1043) are added.

2. After performing the above 1, the lowest dimension is 32×32×21. Execute S1053, perform an enlargement operation on the result of the lowest dimension of 32×32×21 at this time (that is, the result of S1052), and enlarge it to the next lowest dimension of 64×64×21 at this time; and then execute S1054, and the enlarged result and The results of the next lower dimension at this time (that is, the boundary refinement results of S1042 ) are added.

3. After performing the above 2, the lowest dimension is 64×64×21. Execute S1055, perform an enlargement operation on the result of the lowest dimension of 64×64×21 at this time (that is, the result of S1054), and enlarge it to the next lowest dimension of 128×128×21 at this time; and then execute S1056, and the enlarged result and The results of the next lower dimension at this time (that is, the boundary refinement results of S1041 ) are added.

4. After performing the above 3, the four boundary refinement results have been merged, and only one result remains, that is, the result of S1056 with a dimension of 128×128×21. Since the dimension of the original image is 512×512×21, the result of S1056 with a dimension of 128×128×21 is enlarged to a dimension of 512×512×21 to obtain a score map (ScoreMap).

Exemplarily, in the above 4, the result of the dimension 128×128×21 is enlarged to the dimension 512×512×21, which may include S1057 and S1058 as shown in FIG. The step-by-step amplification is not limited in the present invention.

Also, for example, before or after each zoom-in operation, a boundary refinement operation may also be included. This can increase the discriminativeness of the boundary.

Exemplarily, boundary refining operations are included between the above-mentioned 1 and 2, 2 and 3, and 3 and 4 (ie, after the addition operation and before the enlargement operation), such as S10530, S10550 and S10570 in FIG. 4 . A boundary refinement operation is included after each enlargement operation in 4 above, such as S10580 and S10590 in FIG. 4 . This can increase the discriminativeness of the boundary.

Exemplarily, in this embodiment of the present invention, before the method shown in FIG. 2 , the method may further include: training the global convolutional network.

It should be noted that the embodiments of the present invention do not limit the training process and method. For example, M sample images may be acquired, and training is performed based on the M sample images, so as to obtain the parameters of the global convolutional network. For example, N global convolutional networks can be trained at the same time, for example, M sample images can be obtained, and the spatial dimensions of the M sample images are equal, such as 512×512; N global convolutional networks are trained to obtain the parameters of N global convolutional networks. Among them, M is a positive integer. Wherein, the category of each object in the sample image is known.

Optionally, a global convolutional network can be fine-tuned on a relevant dataset for scene understanding based on pretrained sample images obtained from the ImageNet dataset. For example, the information in the large data set can be used to train the information in the small data set, and the scene understanding can be performed based on the small data set after training, which can not only eliminate some useless information in the large data set, but also can speed up the speed of convergence , thereby improving the processing efficiency.

It can be understood that, for the convolutional neural network in step S102 and the global convolutional network in step S103, the method may include: training the convolutional neural network and the global convolutional network.

It can be seen that in the embodiment of the present invention, a global convolutional network and boundary refinement are combined, wherein the global convolutional network can effectively increase the effective receptive field, and the boundary refinement can increase the discriminability of object boundaries. Compared with the traditional scene understanding algorithm, the method of the embodiment of the present invention can meet the performance requirements of scene understanding, so that the overall performance of the system can be greatly improved.

FIG. 7 is a schematic block diagram of an image scene understanding apparatus according to an embodiment of the present invention. The apparatus 70 shown in FIG. 7 includes an acquisition module 701 , a convolution module 702 , a global convolution network module 703 and a boundary refinement module 704 . Each of the modules can respectively execute various steps/functions of the image scene understanding method described above in conjunction with FIGS. 2-6 . Only the main functions of each module of the apparatus 700 for understanding the image scene are described below, and the details that have been described above are omitted.

The acquiring module 701 is used for acquiring the original image of the scene.

The convolution module 702 is configured to perform a convolution operation on the original image obtained by the obtaining module 701 to obtain a convolution output.

The global convolution network module 703 is configured to process the convolution output obtained by the convolution module 702 through a global convolution network to obtain a processing result. as well as

The boundary refining module 704 is configured to perform boundary refining on the processing result obtained by the global convolutional network module 703 to obtain the image scene understanding result.

Exemplarily, the convolution module 702 may be specifically configured to perform a convolution operation on the original image through N convolutional neural networks to obtain N channels of convolution outputs. Wherein, the spatial dimensions of the N-channel convolution outputs are not equal to each other, and the spatial dimensions of the N-channel convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive integer greater than 1.

Exemplarily, the global convolutional network module 703 may be specifically configured to: separately process the N-way convolution outputs through the N global convolutional networks to obtain N corresponding to the N-way convolutional outputs one-to-one. processing results.

Exemplarily, the i-th global convolutional network in the N global convolutional networks may be used to: perform multiple convolution on the i-th convolution output in the N-channel convolution outputs, and convert the The outputs of the multi-channel convolution are added, and the result of the addition is determined as the ith processing result corresponding to the output of the ith channel of convolution. Among them, the value of i ranges from 1 to N.

Illustratively, the boundary refinement module 704 may include N boundary refinement sub-modules and fusion sub-modules. The N boundary refining sub-modules may be used to perform boundary refining on the N processing results respectively, and obtain N boundary refining results corresponding to the N processing results one-to-one. The fusion sub-module may be configured to refine the results according to the N boundaries to obtain the image scene understanding result.

Exemplarily, the i-th boundary refining sub-module in the N boundary refining sub-modules may be used to: correct the i-th processing result in the N processing results, and compare the corrected result with the result. The i-th processing results are combined to obtain the i-th boundary refining result. Among them, the value of i ranges from 1 to N.

Exemplarily, the fusion sub-module may be used to: perform a fusion operation on the N boundary refinement results to obtain the image scene understanding result. The fusion operation includes: repeating the following (a1) until only one result remains: (a1) Enlarging the result of the lowest dimension to the next lower dimension, and then adding the result of the second lower dimension. Then, the spatial dimension (w×h) of the remaining result is enlarged to the same spatial dimension (w×h) as the original image to obtain the Score Map, and then the result of the image scene understanding is obtained based on the Score Map.

Exemplarily, as shown in FIG. 8 , the apparatus 70 further includes a training module 700, which can be used for: training the global convolutional network.

The apparatus 70 shown in FIG. 7 and FIG. 8 can be used to implement the image scene understanding method shown in the foregoing FIG. 2 to FIG. 6 .

In addition, an embodiment of the present invention also provides another apparatus for understanding image scenes, the apparatus may include a processor and a memory, wherein the memory is used to store an instruction code, and when the processor executes the instruction code, the aforementioned FIG. 2 to FIG. 2 can be implemented. The method for image scene understanding shown in Figure 6.

In addition, an embodiment of the present invention further provides an electronic device, and the electronic device may include the apparatus 70 shown in FIG. 7 or FIG. 8 .

In the embodiment of the present invention, a global convolutional network module and a boundary refining module are combined, wherein the global convolutional network module can effectively increase the effective receptive field, and the boundary refining module can increase the discriminability of object boundaries. Compared with the traditional scene understanding algorithm, the embodiment of the present invention can meet the performance requirements of scene understanding, so that the overall performance of the system can be greatly improved.

Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above-described example embodiments are exemplary only, and are not intended to limit the scope of the invention thereto. Various changes and modifications can be made therein by those of ordinary skill in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or May be integrated into another device, or some features may be omitted, or not implemented.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the description of the exemplary embodiments of the invention, various features of the invention are sometimes grouped together , or in its description. However, this method of the invention should not be interpreted as reflecting the intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the corresponding claims reflect, the invention lies in the fact that the corresponding technical problem may be solved with less than all features of a single disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

It will be understood by those skilled in the art that all features disclosed in this specification (including the accompanying claims, abstract and drawings) and any method or apparatus so disclosed may be used in any combination, except that the features are mutually exclusive. Processes or units are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although some of the embodiments described herein include certain features, but not others, included in other embodiments, that combinations of features of different embodiments are intended to be within the scope of the invention within and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the article analysis device according to the embodiment of the present invention. The present invention may also be implemented as apparatus programs (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet sites, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

The above is only the specific embodiment of the present invention or the description of the specific embodiment, and the protection scope of the present invention is not limited thereto. Any changes or substitutions should be included within the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (16)

1. a method for image scene understanding, is characterized in that, comprises: Get the original image of the scene; Perform a convolution operation on the original image to obtain a convolution output; The convolution output is processed through a global convolution network to obtain a processing result, wherein the global convolution network includes a plurality of convolution branches to perform multiple convolutions, and each convolution includes at least one convolution operation , wherein the multiple convolution branches have the same input, and the processing result is obtained after summing the multiple branch outputs of the multiple convolution branches; and Perform boundary refinement on the processing result to obtain an image scene understanding result, wherein the boundary refinement includes modifying the processing result, and combining the processing result with the corrected result, and the correction includes linear correction and at least one of nonlinear corrections. 2. The method of claim 1, wherein the performing a convolution operation on the original image to obtain a convolution output, comprising: The original image is subjected to a convolution operation through N convolutional neural networks to obtain N-way convolution outputs; Wherein, the spatial dimensions of the N-way convolution outputs are not equal to each other, and the spatial dimensions of the N-way convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive integer greater than 1. 3. The method of claim 2, wherein the convolution output is processed through a global convolution network to obtain a processing result, comprising: The N channels of convolution outputs are respectively processed through the N global convolution networks, and N processing results corresponding to the N channels of convolution outputs are obtained. 4 . The method according to claim 3 , wherein the N-way convolution outputs are respectively processed by the N global convolutional networks to obtain a one-to-one correspondence with the N-way convolution outputs. 5 . N processing results of , including: Perform multiplex convolution on the i-th convolution output in the N-channel convolution outputs, add the outputs of the multiplex convolution, and determine the result of the addition to be the same as the i-th convolution output. The i-th processing result corresponding to the road convolution output; Among them, the value of i ranges from 1 to N. 5. The method according to claim 3, wherein the process result is subjected to boundary refinement to obtain an image scene understanding result, comprising: Perform boundary refining on the N processing results respectively, and obtain N boundary refining results corresponding to the N processing results one-to-one; According to the N boundary refinement results, the image scene understanding result is obtained. 6. The method of claim 5, wherein the N processing results are respectively subjected to boundary refining to obtain N boundary refining results corresponding to the N processing results one-to-one, comprising: Correcting the i-th processing result in the N processing results, combining the corrected result with the i-th processing result to obtain the i-th boundary refining result; Among them, the value of i ranges from 1 to N. 7. The method according to claim 5, wherein, obtaining the image scene understanding result according to the N boundary refining results, comprising: A fusion operation is performed on the N boundary refinement results to obtain the image scene understanding result. 8. The method according to any one of claims 1 to 7, wherein before the acquiring the original image, the method further comprises: Train the global convolutional network. 9. A device for image scene understanding, comprising: The acquisition module is used to acquire the original image of the scene; a convolution module, configured to perform a convolution operation on the original image acquired by the acquisition module to obtain a convolution output; A global convolutional network module, configured to process the convolution output obtained by the convolution module through a global convolutional network to obtain a processing result, wherein the global convolutional network includes a plurality of convolution branches to perform multiple Convolution, and each convolution includes at least one convolution operation, wherein the multiple convolution branches have the same input, and the multiple branch outputs of the multiple convolution branches are summed to obtain the result. the result of said processing; and A boundary refinement module, configured to perform boundary refinement on the processing result obtained by the global convolutional network module to obtain an image scene understanding result, wherein the boundary refinement includes revising the processing result and converting the processing The results are combined with corrected results, the corrections including at least one of linear corrections and non-linear corrections. 10. The apparatus of claim 9, wherein the convolution module is used for: The original image is subjected to a convolution operation through N convolutional neural networks to obtain N-way convolution outputs; Wherein, the spatial dimensions of the N-way convolution outputs are not equal to each other, and the spatial dimensions of the N-way convolution outputs are all smaller than the spatial dimension of the original image, and N is a positive integer greater than 1. 11. The apparatus of claim 10, wherein the global convolutional network module is used for: The N channels of convolution outputs are respectively processed through the N global convolution networks, and N processing results corresponding to the N channels of convolution outputs are obtained. 12. The apparatus of claim 11, wherein the i-th global convolutional network in the N global convolutional networks is used for: Perform multiple convolution on the i-th convolution output in the N-way convolution outputs, add the outputs of the multiple convolutions, and determine the result of the addition as the same as the i-th convolution output. The i-th processing result corresponding to the road convolution output; Among them, the value of i ranges from 1 to N. 13. The apparatus of claim 11, wherein the boundary refining module comprises N boundary refining sub-modules and fusion sub-modules: The N boundary refining sub-modules are used to perform boundary refining on the N processing results respectively, and obtain N boundary refining results corresponding to the N processing results one-to-one; The fusion sub-module is configured to refine the results according to the N boundaries to obtain the image scene understanding result. 14. The apparatus of claim 13, wherein the ith boundary refining submodule in the N boundary refining submodules is used for: Correcting the i-th processing result in the N processing results, combining the corrected result with the i-th processing result to obtain the i-th boundary refining result; Among them, the value of i ranges from 1 to N. 15. The apparatus of claim 13, wherein the fusion submodule is used for: A fusion operation is performed on the N boundary refinement results to obtain the image scene understanding result. 16. The apparatus according to any one of claims 9 to 15, wherein the apparatus further comprises a training module for: Train the global convolutional network.

Images