KR102167655B1 - Neural network training method for utilizing differences between a plurality of images, and method thereof - Google Patents

Abstract

The present invention provides a method and apparatus for training a neural network for extracting features of an image by using data on differences between images. Obtaining a reference image photographed with a first setting for a subject and a first comparison image photographed with a second setting for the subject according to an embodiment of the present invention, from a first neural network learned using the reference image , Obtaining feature data of the reference image, Feature data of the first extracted image from a second neural network learned using a first extracted image formed of data about a difference between the reference image and the first comparison image And training a third neural network using the feature data of the reference image and the feature data of the first extracted image.

Description

A neural network training method using differences between multiple images {NEURAL NETWORK TRAINING METHOD FOR UTILIZING DIFFERENCES BETWEEN A PLURALITY OF IMAGES, AND METHOD THEREOF}

The present invention relates to a method of training a neural network that extracts features of an image using data on differences between a plurality of images. In more detail, in a method of learning a neural network for extracting features of an image, it relates to a method of learning a neural network using data including information on differences between a plurality of images and a plurality of images.

3D Convolutional Neural Networks (CNNs) are typically used to extract features of 3D data, but a large amount of memory and time are required for learning the 3D CNN, so the utility of the 3D CNN is very low.

In order to reduce computing resources consumed in extracting the features of 3D data, a method of extracting features of the 3D data may be considered using a 2D CNN. However, when a feature of 3D data is extracted using the 2D CNN, all information about the 3D data (eg, time of image data or depth of 3D object, etc.) cannot be utilized.

Registered Patent No. 10-1396308 (2014.05.12)

The technical problem to be solved by the present invention is to provide a method and apparatus or system for learning a neural network that accurately extracts features of 3D objects at low cost.

Another technical problem to be solved by the present invention is to provide a method and apparatus or system for learning a neural network for extracting features of an image by using data on differences between a plurality of images.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, a method for learning a neural network according to an embodiment of the present invention includes: obtaining a reference image taken with a first setting for a subject and a first comparison image taken with a second setting for the subject , Obtaining feature data of the reference image from a first neural network learned using the reference image, the first extracted image formed of data on the difference between the reference image and the first comparison image 2 It may include obtaining feature data of the first extracted image from a neural network, and training a third neural network using feature data of the reference image and feature data of the first extracted image.

In order to solve the above technical problem, a neural network learning computer program according to another embodiment of the present invention includes: obtaining a reference image photographed with a first setting for a subject and a first comparison image photographed with a second setting for the subject , Obtaining feature data of the reference image from a first neural network learned using the reference image, the first extracted image formed of data on the difference between the reference image and the first comparison image 2 In order to execute the step of obtaining feature data of the first extracted image from a neural network and training a third neural network using the feature data of the reference image and the feature data of the first extracted image, it can be read by a computer. It can be stored in a storage medium.

In order to solve the above technical problem, a machine learning apparatus according to another embodiment of the present invention includes a memory for storing one or more instructions and a first setting for a subject by executing the stored one or more instructions. Obtaining a reference image and a first comparison image taken with a second setting of the subject, obtaining feature data of the reference image from a first neural network learned using the reference image, and comparing the reference image with the first Obtaining feature data of the first extracted image from a second neural network learned using a first extracted image formed of data about differences between images, and using feature data of the reference image and feature data of the first extracted image Thus, a processor for training the third neural network may be included.

1 is a flowchart illustrating a neural network training method using one reference image and one extracted image according to an embodiment of the present invention.
2 is a flowchart of a neural network training method using one reference image and a plurality of extracted images according to another embodiment of the present invention.
3 is a conceptual diagram for explaining in detail a process of obtaining an image of a subject photographed with various settings in a neural network learning method using an image of a subject according to another embodiment of the present invention.
4 is a conceptual diagram illustrating a method of learning a neural network using depth information on an image of a subject and an image of the subject according to another embodiment of the present invention.
FIG. 5 is a conceptual diagram illustrating a method of designating a reference image and a comparison image in a neural network learning method using a reference image and a comparison image according to another embodiment of the present invention.
6 is a conceptual diagram illustrating a method of obtaining an extracted image using a reference image and a comparison image in a neural network training method using a reference image and an extracted image according to another embodiment of the present invention.
7 is a diagram for explaining a neural network training method using a reference image and one extracted image according to another embodiment of the present invention.
FIG. 8 is a diagram illustrating a neural network learning method including a neural network for outputting data on a feature of a subject by using feature data of one reference image and feature data of one extracted image according to another embodiment of the present invention. It is a drawing.
9 is a view for explaining a neural network training method for extracting a subject feature using data obtained by connecting feature data of a reference image and feature data of an extracted image according to another embodiment of the present invention.
10 is a diagram for explaining a neural network learning method for extracting a feature of a subject using data obtained by fusion of feature data of a reference image and feature data of an extracted image according to another embodiment of the present invention.
11 is a diagram illustrating a method of learning a neural network for extracting object features using one reference image and a plurality of extracted images according to another embodiment of the present invention.
12 is a diagram illustrating a method of learning a neural network for extracting object features using a reference image, a comparison image, and an extracted image according to another embodiment of the present invention.
13 is a diagram for explaining a method of learning a subject feature extraction neural network using extraction data formed based on a difference between feature data of a reference image and feature data of a plurality of comparison images according to another embodiment of the present invention.
14 is a block diagram of an apparatus for learning a neural network according to another embodiment of the present invention.
15 is a hardware configuration diagram of a neural network learning apparatus according to another embodiment of the present invention.
16 is a block diagram of a neural network learning system according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the technical idea of the present invention is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical idea of the present invention, and in the technical field to which the present invention pertains. It is provided to completely inform the scope of the present invention to those of ordinary skill in the art, and the technical idea of the present invention is only defined by the scope of the claims.

In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible even if they are indicated on different drawings. In addition, in describing embodiments of the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically. The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless specifically stated in the phrase.

Also, in describing the constituent elements of the present disclosure, terms such as first and second may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the term.

Hereinafter, a neural network learning method according to an embodiment of the present invention will be described with reference to FIG. 1. The neural network learning method according to the present embodiment uses a plurality of images of the subject in order to extract features of the subject. That is, the feature of the subject is extracted using one reference image photographing the subject and one extracted image. In this case, the extracted image is an image formed based on a difference between the reference image and a comparison image photographing the subject according to a photographing setting different from that of the reference image.

The neural network learning method according to the present embodiment may be performed by a computing device. The computing device is a device that receives a plurality of image datasets photographed with different settings of the same subject as training data, and may generate data defining a learned neural network as a result of performing the neural network learning method. Hereinafter, in describing each operation belonging to the method according to the present embodiment, when the description of the subject is omitted, it may be understood that the subject of the operation is the computing device.

In step S101, a reference image and a first comparison image of a subject to be photographed are obtained. Both the reference image and the first comparison image are data obtained as a result of photographing the same subject, but the reference image and the first comparison image are distinguished in that'settings' used for photographing are different from each other. Hereinafter, it will be described in connection with the'setting'.

The setting related to photographing may mean various photographing environments, such as a setting defining an operation method of the photographing device or designating a relative position with respect to the subject such as a photographing position or a photographing angle. For example, the operation method of the photographing apparatus may be a radiographic method, a panorama photographing method, a tomography method, etc., and the photographing environment may be an angle of a camera, a distance between the subject and the camera, an exposure value, an illumination ratio, a focal length, an angle of view, Various settings for acquiring different images in imaging for the same subject, such as radiation dose in radiography may be included.

The reference image and the first comparison image may be images belonging to a set of 2D images configured to generate a 3D image. That is, the reference image and the first comparison image may be a 2D image for generating the 3D image, not the 3D image itself. The 3D image may be a computerized tomography (CT) image, a digital breast tomosynthesis (DBT) image, and a multi-layer tooth panorama image used in the medical imaging field. When the reference image and the first comparison image belong to a set of 2D images configured to generate a 3D image of a subject, the reference image and the first comparison image are photographed at different angles or the same The focus may be at different depths of the subject based on the photographing direction.

In step S103, the acquired reference image is input to the first neural network.

Here, the first neural network is a neural network that outputs feature data extracted from image data input to the first neural network. For example, the first neural network may be one of a variety of 2D Convolutional Neural Network (CNN) neural networks such as ResNet, VGGNet, or AlexNet.

In step S105, a first extracted image formed by using the reference image and the first comparison image may be input to a second neural network.

The first extracted image may be a data-based image formed based on information about a difference between the reference image and the first comparison image. For example, the first extracted image may be an image formed based on a difference between a pixel value of the reference image and a pixel value of the first comparison image. That is, the pixel value at the first coordinate of the first extracted image may be a value obtained by subtracting the first coordinate pixel value of the first comparison image from the first coordinate pixel value of the reference image.

Also, according to another embodiment of the present invention, depth information of the reference image and depth information of the first comparison image may be further input to the second neural network. The second neural network is a neural network that outputs feature data extracted from image data input to the second neural network.

In an embodiment, the first neural network and the second neural network may be configured with the same settings. That is, the first neural network and the second neural network may have completely identical settings related to the neural network configuration, such as the number of layers, the number of nodes of each layer, and connection settings between nodes of adjacent layers. In addition, when the first neural network and the second neural network are 2D CNNs, the type and number of filters used in the first neural network and the second neural network, and the order of applying each filter may also be the same.

In another embodiment, the reference image input to the first neural network is a complete image corresponding to a specific depth of the subject, whereas the first extracted image input to the second neural network is a difference value between the reference image and the first comparison image In consideration of the fact that the image is an image containing only information about the first neural network and the second neural network, at least some settings may be different from each other.

The first extracted image includes only information about a difference between the reference image and the first comparison image. Therefore, in order to compensate for the fact that the information input to the second neural network is relatively weak compared to the first neural network, information on the difference between the depth of the first comparison image and the depth of the reference image is further inputted to the second neural network. I can. Through this, characteristics of a subject located in a three-dimensional space may be more accurately identified than when only the first extracted image is used as input data of the first neural network.

Depth information may mean information that disappears in addition to width and height when a 3D object is captured as a 2D image. For example, the depth information may mean information on a distance from a photographing point to each area of a subject.

In addition, when a plurality of slice images having different depths according to the photographing direction are input images of a neural network, the depth information may be expressed in the form of a serial number of each slice image. For example, when a body is photographed from the front, but 20 slice images are photographed with different focal lengths, and a continuous serial number is assigned from the frontmost slice image toward the rear slice image, the depth information is It could also mean a number.

In step S107, the third neural network may be trained by inputting the output data of the first neural network and the output data of the second neural network to the third neural network.

The output data of the first neural network is feature data of a reference image, and the output data of the second neural network is feature data of the first extracted image. For example, the feature data output from the first neural network may be a feature map of the reference image, and the feature data output from the second neural network may be a feature map of the first extracted image.

The third neural network may include a fusion layer for fusion of feature data of the reference image and feature data of the first extracted image. In the data output as a result of the'fusion', both feature information of the reference image and feature information of the first extracted image are reflected. That is, the'fusion' may be achieved in various ways in which both the feature information of the reference image and the feature information of the first extracted image can be reflected.

For example, the data output from the fusion layer is that the feature vector of the first extracted image is concatenated with the feature vector of the reference image, or the feature vector of the reference image and the first extracted image are A feature vector may be summed, or a feature vector of the reference image and a feature vector of the first extracted image may be multiplied.

In addition, the third neural network is a fusion network, and may be replaced with various types of neural networks that can be modified by a person skilled in the art. For example, the third neural network may be a tensor fusion network.

In addition, the third neural network according to an embodiment may further include a fully connected layer to perform various tasks such as classification and clustering.

Hereinafter, a neural network training method using one reference image and a plurality of extracted images according to another embodiment of the present invention will be described with reference to FIG. 2. In the embodiment described with reference to FIG. 1, one reference image and one comparison image are used for training of a neural network. On the other hand, in the embodiment described with reference to FIG. 2, one reference image and two comparison images are used for learning of the neural network so that the feature of the subject can be extracted using more information about the subject. In this case, in each of the two comparison images, the subject was photographed with different settings. It goes without saying that the neural network learning method according to the present embodiment may be modified to use three or more comparison images captured by different settings.

In order to avoid redundant description, descriptions of steps S101 to S105 are omitted.

In step S109, a second extracted image is obtained using the second comparison image, and the second extracted image is input to the fourth neural network. The second extracted image is an image formed of data on a difference in pixel values between the second comparison image and the reference image. The fourth neural network outputs feature data of the second extracted image. Similar to what has already been described in relation to the first extracted image, information about a difference between the depth of the second comparison image and the depth of the reference image may be input to the fourth neural network together with the second extracted image.

The second comparison image is a photograph of the subject in a setting different from that of the first comparison image. For example, the difference in depth between the first comparison image and the reference image may be the same as the difference in depth between the second comparison image and the reference image. For example, when a series of sliced images of a subject are taken with different depths according to the shooting direction (IMAGE[0, 1, ... ,N]), the first comparison with the reference image (IMAGE[i]) The stride s between the images IMAGE[i+s] and the stride s between the reference image and the second comparison image IMAGE[is] may be the same. That is, the first comparison image IMAGE[i+s] and the second comparison image IMAGE[i-s] are symmetrical in depth with respect to the reference image IMAGE[i].

In an embodiment, the first neural network, the second neural network, and the fourth neural network may be configured with the same settings. That is, the first neural network, the second neural network, and the fourth neural network may have completely identical settings related to the neural network configuration, such as the number of layers, the number of nodes of each layer, and the connection setting between nodes of adjacent layers. In addition, when the first neural network, the second neural network, and the fourth neural network are 2D CNN, the type and number of filters used in the first neural network, the second neural network, and the fourth neural network, and each filter The order of application may also be the same.

In another embodiment, the reference image input to the first neural network is a complete image corresponding to a specific depth of the subject, whereas the first extracted image input to the second neural network and the second extracted image input to the fourth neural network Is, reflecting that it is an image that contains only information about a difference value from a reference image, the first neural network and the second neural network have at least some settings different from each other, and the first neural network and the fourth neural network are also At least some settings may be different. However, even in this case, since both the first extracted image and the second extracted image are images containing only information about a difference value from the reference image, the second neural network and the fourth neural network may be configured with the same setting. will be.

Meanwhile, in another embodiment, all extracted images formed from information about a difference value between the reference image and each comparison image may be input to the same neural network. For example, a plurality of extracted images may be input to the same neural network. In an embodiment, both the first extracted image and the second extracted image may be input to the second neural network.

Even when the feature data of the first extracted image and the feature data of the second extracted image are acquired by the same neural network, the feature data of the first extracted image, the feature data of the second extracted image, and the feature data of the reference image are all parallel. Can be input to the third neural network. In this case, the third neural network includes feature data of the first extracted image input from the second neural network, feature data of the second extracted image also input from the second neural network, and the feature data input from the first neural network. Feature data of the reference image may be fused.

In step S111, output data of each of the first neural network, the second neural network, and the fourth neural network are input to a third neural network, so that feature data of the reference image, feature data of the first extracted image, and the second neural network are All feature data of the extracted image may be fused, and a task such as classification may be further performed using the fused feature.

A plurality of images of a subject photographed with various settings will be described in detail with reference to FIG. 3. Various reference images and comparison images may be obtained by varying the settings related to photographing of the subject.

According to some embodiments of the present invention,'setting' related to capturing of the first comparison image and'setting' related to capturing of the second comparison image may be information about a camera angle. For example, if the reference image is the image 122 acquired with the 3rd setting 102, the first comparison image and the second comparison image have the same camera angle, but can be acquired by shooting set in different directions. have. Therefore, if the camera angle 105 of the second setting 101 and the camera angle 106 of the fourth setting 103 are the same based on the straight line 102 between the camera position where the reference image 122 was captured and the subject, Each of the image 121 acquired by setting 2 and the image 123 acquired by setting 4 may be designated as a first comparison image and a second comparison image.

Likewise, if the camera angle of setting No. 1 (100) and the camera angle of setting No. 5 (104) are the same based on the straight line 102 between the camera location where the reference image 122 was captured and the subject, setting No. 1 (100) Each of the image 120 obtained by and the image 124 obtained by setting #5 104 may be designated as a first comparison image and a second comparison image. Also, since the number of comparison images is not limited to two, the image 120 obtained by the first setting 100, the image 121 obtained by the second setting 101, and the fourth setting Four or more comparison images including the image 123 obtained by 103 and the image 104 obtained by setting No. 5 104 may be obtained.

A plurality of extracted images are obtained based on each of the plurality of comparison images using the obtained plurality of comparison images, so that a rich amount of neural network training data may be generated using the plurality of extracted images of various combinations.

In addition, in some embodiments of the present invention, the extracted image and depth information corresponding to the extracted image may be further used for learning of a neural network. In this case, the depth information corresponding to the extracted image according to an exemplary embodiment may be a straight line between a camera position at which a reference image is captured and a subject, and an angle difference between a camera position at which the comparison image is captured and a straight line between the subject.

As illustrated in FIG. 4, the depth information may be information on a height difference from a predetermined reference plane to a specific tomographic plane of the subject. Therefore, the difference between the height 113 of the subject corresponding to the reference image and the height 111 of the subject corresponding to the first comparison image is the difference between the height 113 of the subject corresponding to the reference image and the subject corresponding to the second comparison image. The reference image, the first comparison image, and the second comparison image may be designated to have the same value as the difference between the heights 112.

Hereinafter, a method of acquiring a reference image and a comparison image from a plurality of images acquired according to a setting change of continuous shooting will be described with reference to FIG. 5.

For example, a plurality of 2D images 210 may be obtained by continuously adjusting a photographing angle of a subject, and one 2D image among the obtained 2D images 210 may be designated as a reference image.

The reference image according to an exemplary embodiment may be designated as an image including the most information about a subject among a plurality of images. However, in order to secure various neural network training data, each of the plurality of images 210 is designated as a reference image for each task and may be used for neural network training.

In some embodiments of the present invention, when the neural network training task is repeated, the training process of the same neural network may be repeated using a reference image different from the reference image used in the previous neural network training. For each repetitive neural network training process, a reference image including information on a subject that could not be acquired in the previous neural network training process is newly designated, and the neural network may be trained using an extracted image related to each reference image.

In this case, not only information on various subjects included in each reference image but also information on an extracted image running according to the reference image may be used for neural network learning, thereby forming a more advanced neural network.

After the reference image 201 is designated, one or more comparison images 202, 203, 204, and 205 different from the reference image 201 may be designated. According to an embodiment, in order to obtain one extracted image, one reference image and one comparison image may be designated, or one reference image and two or more comparison images may be designated.

As shown in FIG. 5, a plurality of images 210 corresponding to each of a plurality of depth information of a subject may be aligned based on the depth information. According to an embodiment, when the reference image 201 corresponding to the depth '0' of the subject is acquired, the first comparison image 202 corresponding to the depth'-5' of the subject and the depth of the subject'+5' are The corresponding second comparison images 203 may each be designated to have the same depth difference as the reference image 201.

According to another embodiment, the reference image 201 and several pairs of comparison images 202, 203, 204, and 205 having various depth information may be used to obtain the extracted image. For example, when the reference image 201 corresponding to the depth '0' of the subject is acquired, the image 202 corresponding to the depth'-5' of the subject and the image 203 corresponding to the depth'+5' of the subject ) May be obtained as a pair of first and second comparison images, respectively, an image 204 corresponding to the depth of the subject'-10' and an image 205 corresponding to the depth of the subject'+10' Also, a pair of different first and second comparison images may be further obtained.

Hereinafter, a method of acquiring an extracted image using a reference image and a comparison image will be described with reference to FIG. 6.

The extracted image may be an image formed based on a comparison image having'depth information' different from the reference image. The extracted image according to an embodiment may be obtained using a difference in pixel values between the reference image and the comparison image. In another embodiment, the extracted image may be obtained by taking an absolute value from the difference in pixel values between the reference image and the comparison image, and the extracted image may be a black and white image generated based on a numerical value converted to the absolute value. .

According to an embodiment, when one reference image and one comparison image are obtained, one extracted image may be obtained based on a difference between the reference image and the comparison image.

According to another embodiment, when one reference image and two comparison images are obtained, as shown in FIG. 6, based on the difference between the reference image and the first comparison image and the difference between the reference image and the second comparison image, respectively. The first extracted image and the second extracted image may be obtained.

Instead of the comparison image being input to the neural network and used for training the neural network, the extracted image is input to the neural network and used for training the neural network, so that depth information of the subject that cannot be obtained using only the reference image and the comparison image can be further used for the neural network training have.

Hereinafter, a configuration of a subject feature extraction neural network using one reference image and one extracted image will be described in detail with reference to FIG. 7.

First, the feature data 311 of the reference image may be obtained using the first neural network 310 and the feature data 321 of the first extracted image may be obtained using the second neural network 320. The feature data 311 of the reference image and the feature data 321 of the first extracted image may be data including information on a width, a height, and a channel value. It may be a feature map of feature data output from the first neural network and the second neural network.

Thereafter, the feature data 311 of the reference image and the feature data 321 of the first extracted image may be input to a third neural network 322 that fusions the feature data 311 and 312. The third neural network 322 may output a task result map 323 indicating a result of a task performed by using the characteristics of the subject.

As shown in FIG. 8, the feature data output from the neural network 350 that fuses the feature data 331 output from the first neural network 330 and the feature data 341 output from the second neural network 340 is input. The neural network 360 that receives and performs a task such as classification may output a task result map 361.

The task result map 361 is data indicating information related to the feature of a subject that can be extracted using feature data of a 2D image of the subject. For example, the task result map 361 may be data indicating whether cancer has occurred in the human body, which can be extracted using a tomography image of the human body.

Hereinafter, a method of fusing feature data will be described in detail with reference to FIGS. 9 to 10. As shown in FIG. 9, the neural network 372 receiving a plurality of feature data inputs is data in which the plurality of feature data 370 and 371 are concatenated based on a channel axis as a result of feature data fusion. 373) can be created.

Alternatively, as shown in FIG. 10, the neural network 382 receiving a plurality of feature data 380, 381 includes one or more convolution steps and one or more pooling steps, and multiplication, addition, and subtraction, etc. It is also possible to output data 383 having the same size as the input data through the operation of. Feature data 383 output as a result of the feature data fusion is input to a neural network 384 capable of performing any one or more of classification or clustering using the feature data, and from the neural network 384 The task result map 385 may be output.

According to another embodiment of the present invention, as shown in FIG. 11, one reference image and a plurality of extracted images may be used for neural network training.

When a third neural network is trained using the feature data 401 of the reference image, the feature data 411 of the first extracted image, and the feature data 421 of the second extracted image, information on the subject included in the reference image and Information on the subject corresponding to the first extracted image and information on the subject corresponding to the second extracted image can be used for training a third neural network, and depth information of the first extracted image and information on the first extracted image Depth information may be further used for training the third neural network. As described above, the first extracted image and the second extracted image may be formed based on a first comparison image and a second comparison image that are symmetrical in depth with respect to the reference image, respectively.

Therefore, when both the first extracted image and the second extracted image are used in parallel for neural network training, only one extracted image and a reference image are used, such as information related to the characteristic of a subject that repeatedly exists according to a certain depth difference. In the case of training a neural network, more information that cannot be obtained may be obtained.

In addition, according to another embodiment, the feature data 401 of the reference image, the feature data 411 of the first extracted image, and the feature data 421 of the second extracted image may be input to the third neural network with a predetermined weight. .

In an embodiment, the largest weight may be assigned to the feature data 401 of the reference image.

In another embodiment, the weight of the feature data 411 of the first extracted image and the weight of the feature data 421 of the second extracted image may be greater than the weight of the feature data 401 of the reference image. In another embodiment, the weight of the feature data 411 of the first extracted image and the weight of the feature data 421 of the second extracted image may be equally assigned. In addition, weights for the feature data 401 of the reference image, the feature data 411 of the first extracted image, and the feature data 421 of the second extracted image may all be equally designated.

That is, the weights assigned to the feature data of the reference image, the feature data of the first extracted image, and the feature data of the second extracted image may be variously designated according to the feature of the subject to be extracted from the neural network.

Referring to FIG. 12, according to another embodiment of the present invention, a reference image, a first comparison image, and a first extracted image may be used for training a neural network.

When only the reference image and the first extracted image are used as training data for the neural network, most of the information on the subject other than depth information on the subject included in the first extracted image must be obtained from the reference image. Accordingly, by further using the first comparison image as well as the reference image and the first extracted image, information on the subject included only in the first comparison image can be further used for learning of the neural network. The neural network 440 receiving the reference image according to the present embodiment may output feature data 441 of the reference image, and the neural network 450 receiving the first comparison image may output the feature data 441 of the first comparison image. Data 451 may be output. In addition, the neural network 460 receiving the first extracted image may output feature data 461 of the first extracted image.

A method of learning a neural network using extracted data formed based on a difference between feature data of a reference image and feature data of a comparison image will be described with reference to FIG. 13.

According to the present embodiment, the first extracted data 491b may be formed based on the difference between the feature data 490 of the reference image and the feature data 491a of the first comparison image, and the feature data 490 of the reference image The second extracted data 492b may be formed based on a difference between the feature data 492a of the and the second comparison image.

The difference between the feature data 490 of the reference image and the feature data 491a of the first comparison image and the difference between the feature data 490 of the reference image and the feature data 492a of the second comparison image are the characteristics of the reference image. It may be formed based on a subtraction operation of a numerical value included in each of the data 490, the feature data 491a of the first comparison image, and the feature data 492a of the second comparison image. Specifically, any one or more of the width, height, and channel values of the feature data may be used for the subtraction operation.

In this case, the feature data 490 of the reference image, the first extracted data 491b, and the second extracted data 492b may be used as input data of a third neural network for extracting a feature of a subject.

As described above, according to the present disclosure, not only differences between a plurality of images are extracted, but also differences in a feature space may be extracted. In addition, neural network training as described above may be performed using the difference extracted on the feature region.

Hereinafter, the operation of the neural network learning apparatus will be described in detail with reference to FIG. 14.

A subject feature extraction neural network training apparatus 500 according to another embodiment of the present invention includes an image feature extraction neural network unit 502, a subject feature extraction neural network unit 504, and an image feature data DB 513, and some implementations In an example, an image preprocessor 501, a feature data preprocessor 503, a subject image DB 511, and an extracted image DB 512 may be further included.

The image preprocessor 501 may receive a reference image and a comparison image from the subject image DB 511, and form an extracted image by using the difference between the reference image and the comparison image received from the subject image DB 511. I can. The image preprocessor 501 according to an embodiment may obtain an extracted image by using a difference in pixel values between the reference image and the comparison image, and the difference in numerical values between the feature data of the reference image and the feature data of the comparison image Extraction data can also be obtained by using. The obtained extracted image or the extracted data may be stored in the extracted image DB 512.

The image feature extraction neural network unit 502 may output the feature data of the reference image and the feature data of the extracted image by using the reference image and the extracted image, and in some embodiments, the feature data of the comparison image Can be printed. The image feature extraction neural network unit 502 according to an embodiment may also output feature data for the comparison image by using the comparison image. The output feature data may be stored in the image feature data DB 513.

The feature data preprocessor 503 may fusion feature data of an image acquired from the image feature data DB 513. The data fusion may be performed by a fusion network, and feature data concatenated or summed from the neural network based on a channel axis may be output.

The subject feature extraction neural network unit 504 may receive feature data and output a task result map related to the subject feature. The subject feature extraction neural network unit 504 according to an embodiment may designate feature data of a reference image and feature data of the extracted image as weights and use them for neural network training.

15 is a hardware configuration diagram of a neural network learning apparatus according to another embodiment of the present invention.

As shown in FIG. 15, the computing device 500 includes one or more processors 550, a system bus, a network interface 530, and a memory 522 that loads a computer program executed by the processor 550. And, it may include a storage 540 that stores the image DB (541). 15 illustrates some components related to an embodiment of the present disclosure. Accordingly, those of ordinary skill in the art to which the present disclosure belongs may recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 15.

The processor 550 controls the overall operation of each component of the computing device 500. The processor 550 includes at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any type of processor well known in the technical field of the present disclosure. It can be configured to include. The computing device 500 may also include a plurality of processors. The processor 550 may not be a general-purpose processor but may be a processor having a structure specialized for machine learning.

The memory 522 stores various types of data, commands and/or information. The memory 522 may load one or more instructions 523, 524, and 525 from the storage 540 to execute a method/operation according to various embodiments of the present disclosure. The memory 522 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus provides communication functions between components of computing device 500. The bus may be implemented in various types of buses such as an address bus, a data bus, and a control bus.

The network interface 530 supports wired/wireless Internet communication of the computing device 500. In addition, the network interface 530 may support various communication methods other than Internet communication. To this end, the communication interface 530 may be configured to include a communication module well known in the technical field of the present disclosure. In some cases, the network interface 530 may be omitted.

The storage 540 may non-temporarily store the one or more programs (not shown) and the image DB 541. The storage 540 may include a nonvolatile memory such as a flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art to which the present disclosure pertains.

The image DB 541 stores an image of a subject used in the neural network learning of the present invention, an extracted image formed from data on differences between a plurality of images of the subject, and feature data output from a neural network that extracts features of a plurality of images. I can.

A computer program (not shown) loaded into the memory 522 as it is executed by the operating system performs an operation of learning a CNN-based neural network using the training data. As a result of the operation of the computer program, data defining a CNN-based neural network may be stored in the storage 540. The data defining the CNN-based neural network may be transmitted to another computing device through the network interface 530 as a model that generates an output suitable for a learning purpose. The other computing device may perform inferring using the model. As an example of the reasoning, it may be considered to find the location of a lesion in a medical image.

The computer program may include instructions 523, 524, and 525 that, after being loaded into the memory 522, cause the processor 550 to perform the method described with reference to FIGS. 1 to 13. That is, the processor 550 may perform methods according to various embodiments related to the method described with reference to FIGS. 1 to 13 by executing the instructions 523, 524, and 525. In this specification, an instruction refers to a set of computer-readable instructions grouped on the basis of a function, which is a component of a computer program and executed by a processor.

By executing the image difference value extraction instruction 523 according to an embodiment, based on the difference between the feature data of the extracted image and the feature data of the reference image and the extracted image formed based on the difference in pixel values between the reference image and the comparison image. The formed extraction data can be obtained.

By executing the image feature data extraction instruction 524 according to an embodiment, feature data of a reference image and feature data of an extracted image may be obtained, and feature data of a comparison image may be further obtained according to some embodiments.

By executing the neural network training instruction 525 according to an embodiment, a neural network for extracting features of an image and a neural network for fusing the extracted feature data may be trained.

16 is a block diagram of a neural network learning system according to another embodiment of the present invention.

As shown in FIG. 16, the neural network learning system according to the present embodiment may include a subject photographing apparatus 600 and a machine learning apparatus 500. According to an embodiment, the object image analysis result display device 700 may be further included in the neural network learning system according to the present embodiment.

The object photographing device 600 may be a device for photographing a reference image and a comparison image according to some embodiments of the present invention. For example, the subject photographing apparatus is a device that photographs a medical image of a body, and may be a device that photographs an image such as X-ray, CT, MRI, and the like. The subject photographing apparatus 600 provides image data photographed through a network to the machine learning apparatus 500. Since medical images are sensitive personal information, the network may be a network from which external access is blocked. That is, the machine learning device 500 and the subject photographing device 600 may be devices located in the same hospital.

The machine learning apparatus 500 of FIG. 16 may be understood to be the same as that shown in FIG. 15. That is, the machine learning apparatus 500 accumulates image data provided from the subject photographing apparatus 600, and when the machine learning performance criteria are satisfied, the machine learning apparatus 500 outputs output data suitable for the purpose of machine learning using the newly accumulated image data. You will be able to learn more highly. In this process, the neural network learning method described with reference to FIGS. 1 to 13 is performed.

The definition data of the model learned by the machine learning device 500 may be transmitted to the object image analysis result display device 700. Unlike the subject photographing apparatus 600 and the machine learning apparatus 500, the subject image result display apparatus 700 may be a computing device located outside a hospital in which the subject photographing apparatus 600 is installed. The subject image analysis result display device 700 receives and stores the definition data of the model from the machine learning device 500, and inputs an analysis target image to the model to obtain analysis result data, and renders the analysis result data. And, by displaying the result on the screen, it will be possible to display the inference result for the medical image.

The methods according to the embodiments of the present invention described so far are described as being combined or combined into one operation of all components constituting the embodiments of the present invention above the computer-readable, technical idea of the present disclosure. This is not necessarily limited to this embodiment. That is, as long as it is within the scope of the object of the present disclosure, one or more of the components may be selectively combined and operated.

Although the operations are illustrated in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order shown or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the above-described embodiments should not be understood as necessitating such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, the present disclosure may be implemented in other specific forms without changing the technical spirit or essential features of those of ordinary skill in the art. I can understand that there is. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of protection of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the technical ideas defined by the present disclosure.

Claims (20)

In the neural network learning method performed by a computing device,
Obtaining a reference image photographed with respect to a subject and a first comparison image photographed with a photographing setting different from that of the reference image with respect to the subject;
Generating a first difference image formed based on a difference between the reference image and the first comparison image;
Acquiring feature data of each of the reference image and the first difference image; And
And learning a neural network that outputs a specific task result by using the feature data of the reference image and the feature data of the first difference image.
How to learn a neural network. The method of claim 1,
Obtaining, for the subject, a second comparison image photographed with a shooting setting different from each of the reference image and the first comparison image;
Generating a second difference image formed based on a difference between the reference image and the second comparison image; And
Further comprising the step of acquiring feature data of the generated second difference image
How to learn a neural network. The method of claim 2,
The step of learning the neural network
And learning the neural network using feature data of each of the reference image, the first difference image, and the second difference image.
How to learn a neural network. The method of claim 2,
The difference in depth with respect to the subject between the first comparison image and the reference image is equal to the difference in depth with respect to the subject between the second comparison image and the reference image,
How to learn a neural network. The method of claim 2,
The difference in depth with respect to the subject between the first comparison image and the reference image is different from the difference in depth with respect to the subject between the second comparison image and the reference image,
How to learn a neural network. The method of claim 1,
The reference image and the comparison image are included in a plurality of images constituting the 3D image of the subject.
How to learn a neural network. The method of claim 6,
The 3D image is
Comprising at least one of a computerized tomography (CT) image and a digital breast tomosynthesis (DBT) image
How to learn a neural network. The method of claim 1,
The above shooting settings are
Including at least one of a shooting angle, a shooting distance, and a depth focus for the subject
How to learn a neural network. The method of claim 8,
The subject is
Including body parts,
The above shooting settings are
Including at least one of a photographing angle, a photographing distance, and a depth focus for the body part
How to learn a neural network. In the neural network learning method performed by a computing device,
Obtaining a reference image photographed with respect to a subject and a plurality of comparison images photographed with a photographing setting different from that of the reference image with respect to the subject;
Generating a plurality of difference images formed based on differences between the reference image and each of the plurality of comparison images;
Acquiring feature data of each of the reference image and the plurality of difference images; And
And learning a neural network that outputs a specific task result using feature data of the reference image and feature data of each of the plurality of difference images.
How to learn a neural network. A memory that stores one or more instructions; And
By executing the stored one or more instructions, a reference image photographed for a subject and a first comparison image photographed with a photographing setting different from the reference image for the subject are obtained,
Generating a first difference image formed based on a difference between the reference image and the first comparison image,
Acquiring feature data of each of the reference image and the first difference image,
And a processor that trains a neural network to output a specific task result by using the feature data of the reference image and the feature data of the first difference image.
Machine learning device. The method of claim 11,
The processor is
For the subject, a second comparison image taken with a shooting setting different from each of the reference image and the first comparison image is obtained,
Generating a second difference image formed based on the difference between the reference image and the second comparison image,
Obtaining feature data of the generated second difference image
Machine learning device. The method of claim 12,
The processor is
Learning the neural network using feature data of each of the reference image, the first difference image, and the second difference image
Machine learning device. The method of claim 12,
The difference in depth with respect to the subject between the first comparison image and the reference image is equal to the difference in depth with respect to the subject between the second comparison image and the reference image.
Machine learning device. The method of claim 12,
The difference in depth with respect to the subject between the first comparison image and the reference image is different from the difference in depth with respect to the subject between the second comparison image and the reference image,
Machine learning device. The method of claim 11,
The reference image and the comparison image are included in a plurality of images constituting the 3D image of the subject.
Machine learning device. The method of claim 16,
The 3D image is
Comprising at least one of a computerized tomography (CT) image and a digital breast tomosynthesis (DBT) image
Machine learning device. The method of claim 11,
The above shooting settings are
Including at least one of a shooting angle, a shooting distance, and a depth focus for the subject
Machine learning device. The method of claim 18,
The subject is
Including body parts,
The above shooting settings are
Including at least one of a photographing angle, a photographing distance, and a depth focus for the body part
Machine learning device. A memory that stores one or more instructions; And
By executing the stored one or more instructions, a reference image photographed for a subject and a plurality of comparison images photographed with a photographing setting different from the reference image for the subject are obtained,
Generating a plurality of difference images formed based on differences between the reference image and each of the plurality of comparison images,
Acquiring feature data of each of the reference image and the plurality of difference images,
And a processor for learning a neural network that outputs a specific task result by using feature data of the reference image and feature data of each of the plurality of difference images.
Machine learning device.

Images