KR102456461B1 - Analysis method and system for micro structures of steel using deep learning - Google Patents

Abstract

The present invention provides a method for analyzing the microstructure of steel using deep learning that enables quantitative information to be obtained by clearly numerically quantifying the fraction of a specific phase and the boundary between phases in a microstructure image of steel that can be obtained with an optical microscope, a scanning electron microscope, etc. A system, comprising the steps of: (a) inputting microstructure image information of steel into an input layer; (b) extracting the microstructure image information input to the input layer as a primary feature map from the convolution layer; (c) Linking the first-order feature map to the second-order feature map in which the first-order feature map is secondarily convolved in the first-order density block so that the information of the initial stage can lead to the information of the later stage ( concatenation) and accumulating the first cumulative feature map; and (d) reducing the resolution of the first-order cumulative feature map in a first-order transition down.

Description

Steel microstructure analysis method and system using deep learning {Analysis method and system for micro structures of steel using deep learning}

The present invention relates to a method and system for analyzing the microstructure of steel using deep learning, and more particularly, clearly numerical values for the fraction of a specific phase and the boundary between phases in microstructure images of steel that can be obtained with an optical microscope, a scanning electron microscope, etc. It relates to a method and system for analyzing the microstructure of steel using deep learning to obtain quantitative information.

In general, the microstructure of steel refers to a spatial structure formed by different phases inside a material, and important quality characteristics of a steel material are closely related to this microstructure.

For example, in a dual phase steel, in which two types of phases of martensite of a hard structure and ferrite of a soft structure coexist, the fraction and shape of the two phases are important factors determining strength and formability. Therefore, the phase analysis of the microstructure of steel products is an essential work process for the steel type development experiment process or quality evaluation. In phase analysis of microstructures, there are usually qualitative methods for judging tissues by observing them under a microscope, and quantitative methods using analytical instruments.

The former method was usually determined by an observer's empirical judgment after obtaining an image of the shape of a tissue, such as an optical microscope (OM) or a scanning electron microscope (SEM).

However, since the microstructure of various and complex patterns is generated according to the alloy composition and heat treatment method of steel, the analysis of the microstructure requires the experience of a skilled expert and takes a lot of time.

In addition, even if experts have qualitatively distinguished a specific phase in a tissue image, additional work is required to quantify the fraction of each phase. Therefore, there are cases where a quantitative method using the aforementioned analysis device is needed, and here is mainly an X-ray diffraction (XRD) analysis method or an Electron Backscatter Diffraction (EBSD) analysis method. can be used

These existing manual analysis methods require additional specimen processing procedures, and may require additional cost, time, and manpower, such as the use of analysis equipment and analysis methods.

In order to solve this existing problem, in the prior art, for faster and more accurate microstructure image analysis, a technique for analyzing the microstructure image obtained in the microscopic observation step is not the experience of a skilled person, but a deep learning method has been developed. There is an old room.

In the field of computer vision, various approaches exist according to the purpose of image processing, and representative methods include classification and semantic segmentation.

Classification is a technique for determining which label an input image corresponds to among several labels. Since the entire input image corresponds to one label, it can be used only in the case of a single phase organization in the steel organization. On the other hand, segmentation is a technique in which multiple objects in an image are distinguished by pixel units, and the same label is marked with each other. Since several phases coexist in the steel microstructure, it should be approached as a segmentation technique that can be applied when two or more objects exist.

A representative artificial neural network widely known in the segmentation technique is a Fully Convolutional Network (FCN). The fully convolutional neural network can be replaced with a 1ㅧ1 convolutional layer with the same function instead of the fully-connected layer of the last stage of the VGG artificial neural network developed for the classification technique.

Through this, the final segmentation result can be obtained by maintaining the disappearing spatial information and extending it to the deconvolution operation at the end.

1 is a photograph showing an inference map result obtained by analyzing an existing microstructure image of steel by deep learning.

As shown in FIG. 1 , an original image and a corresponding map image (ground truth) are required for the learning and evaluation image of this segmentation technique.

However, as shown in FIG. 1 , when such a perfect convolutional neural network is used, as a result of inference, many failed cases may occur.

FIG. 2 is a chart showing the average accuracy and the average union-to-intersection ratio according to the inference result of FIG. 1 .

In the conventional case, as a result of learning by classifying five phases of ferrite, pearlite, martensite, tempered martensite and bainite as labels, the average accuracy (Global Accuracy) is 93.94 percent, and the average IOU (Mena Intersection over union) ) was calculated as 67.84.

However, as in the conventional case, most of the failure cases occurred in a phase that is difficult to identify even with the experience of a skilled person, such as tempered martensite and bainite. Therefore, it was necessary to develop a new deep learning algorithm in order to secure higher analysis accuracy.

Korean Patent Application No. 10-2018-016394

The present invention is intended to solve various problems, including the above problems, and enables faster analysis rather than automating the task of human judgment through deep learning by microscopic observation of phase analysis of microstructures. Higher classification accuracy can be obtained by using FC-DenseNet, an artificial neural network that is more advanced than the FCN applied to microstructures, and the fraction of specific phases and An object of the present invention is to provide a method and system for analyzing the microstructure of steel using deep learning that makes it possible to obtain additional quantitative information by clearly quantifying the boundary between phases. However, these problems are exemplary, and the scope of the present invention is not limited thereto.

Steel microstructure analysis method using deep learning according to the spirit of the present invention for solving the above problems, (a) inputting the microstructure image information of the steel to an input layer (Input Layer); (b) extracting the microstructure image information input to the input layer as a primary feature map from the convolution layer; (c) Linking the first-order feature map to the second-order feature map in which the first-order feature map is secondarily convolved in the first-order density block so that the information of the initial stage can lead to the information of the later stage ( concatenation) and accumulating the first cumulative feature map; and (d) reducing the resolution of the first-order cumulative feature map in a first-order transition down.

In addition, according to the present invention, after step (d), (e) the first-order cumulative feature map with reduced resolution in the second-density block so that the information of the initial stage can lead to the information of the later stage is a tertiary control accumulating the second-order feature map or the first-order feature map as a second-order cumulative feature map by linking the voluted third-order feature map; and (f) reducing the resolution of the secondary cumulative feature map in the secondary transition down.

According to the present invention, after step (f), (g) extracting the second-order cumulative feature map with reduced resolution in the third-density block as a fourth-order convolutional fourth-order feature map; and (h) increasing the resolution of the fourth-order feature map in the first-order transition up.

In addition, according to the present invention, after step (h), (i) the fourth-order feature map whose resolution is increased in the fourth-density block so that the information of the initial stage can lead to the information of the later stage is a fifth-order convolution At least any one or more of the 4th feature map, the 3rd feature map, the 2nd feature map, and the 1st feature map is selected and concatenated to the selected 5th feature map and accumulated as a 3rd order cumulative feature map making; and (j) increasing the resolution of the tertiary cumulative feature map in the secondary transition-up.

In addition, according to the present invention, after step (j), (k) the third-order cumulative feature map whose resolution is increased in the fifth-density block so that the information of the initial stage can lead to the information of the later stage is the sixth-order By selecting and concatenating at least one of the 5th feature map, the 4th feature map, the 3rd feature map, the 2nd feature map, and the 1st feature map to the voluted 6th feature map. accumulating into a fourth-order cumulative feature map; and (l) outputting the inference map from the deconvolution layer to the output layer so that the microstructure of the steel can be divided and determined.

On the other hand, the steel microstructure analysis system using deep learning according to the spirit of the present invention for solving the above problems, an input layer for inputting the microstructure image information of the steel (Input Layer); a convolution layer for extracting the microstructure image information input to the input layer as a primary feature map; 1 of accumulating the first-order feature map by concatenating the first-order feature map with the second-order convolutional second-order feature map of the first-order feature map so that the information of the initial stage can lead to the information of the later-stage Dense Block; and a first-order transition down that reduces the resolution of the first-order cumulative feature map.

In addition, according to the present invention, the first-order cumulative feature map with reduced resolution so that the information of the initial stage can lead to the information of the later-stage is added to the secondary feature map or the first a second density block accumulating a second-order cumulative feature map by associating the second-order feature map; and a secondary transition down for reducing the resolution of the secondary cumulative feature map.

In addition, according to the present invention, the third density block for extracting the second-order cumulative feature map with reduced resolution as a fourth-order convolutional fourth-order feature map; and a first-order transition up that increases the resolution of the fourth-order feature map.

In addition, according to the present invention, at least the quaternary feature map, the 3 a fourth density block for accumulating a third-order cumulative feature map by selecting and concatenating one or more of the second-order feature map, the second-order feature map, and the first-order feature map; and a secondary transition up for increasing the resolution of the tertiary cumulative feature map.

In addition, according to the present invention, at least the 5th-order feature map, the 3rd-order cumulative feature map whose resolution is increased so that the information of the initial stage can lead to the information of the later-stage is added to the 6th-order convolutional 6th-order feature map a fifth-order density block for accumulating a fourth-order cumulative feature map by selecting and concatenating one or more of a fourth-order feature map, the third-order feature map, the second-order feature map, and the first-order feature map; and a deconvolution layer that outputs an inference map as an output layer so that the microstructure of steel can be divided and determined.

According to some embodiments of the present invention made as described above, a faster analysis is possible rather than by automating the task of determining the phase of a microstructure by a person through a microscopic observation method through deep learning, and Higher classification accuracy can be obtained by using FC-DenseNet, an artificial neural network that is more advanced than the applied complete convolutional neural network, and the fraction of a specific phase and the boundary between phases are clearly digitized in microstructure images usually obtained from optical microscopes and scanning electron microscopes. This has the effect of obtaining additional quantitative information. Of course, the scope of the present invention is not limited by these effects.

1 is a photograph showing an inference map result obtained by analyzing an existing microstructure image of steel by deep learning.
FIG. 2 is a chart showing the average accuracy and the average union-to-intersection ratio according to the inference result of FIG. 1 .
3 is a flowchart illustrating a method for analyzing a steel microstructure using deep learning according to some embodiments of the present invention.
4 is a conceptual diagram illustrating a method for analyzing a steel microstructure using deep learning of FIG. 3 .
5 is a block diagram illustrating a steel microstructure analysis system using deep learning of FIG. 3 .
6 is a diagram illustrating an example of a neural network structure of a steel microstructure analysis system using deep learning according to some embodiments of the present invention.
7 is a chart showing the accuracy of the steel microstructure analysis method system using deep learning of FIG.

Hereinafter, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Examples of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art, and the following examples may be modified in various other forms, and the scope of the present invention is as follows It is not limited to an Example. Rather, these embodiments are provided so as to more fully and complete the present disclosure, and to fully convey the spirit of the present invention to those skilled in the art. In addition, in the drawings, the thickness or size of each layer is exaggerated for convenience and clarity of description.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings schematically illustrating ideal embodiments of the present invention. In the drawings, variations of the illustrated shape can be envisaged, for example depending on manufacturing technology and/or tolerances. Accordingly, embodiments of the inventive concept should not be construed as limited to the specific shape of the region shown in the present specification, but should include, for example, changes in shape caused by manufacturing.

3 is a flowchart illustrating a method for analyzing a steel microstructure using deep learning according to some embodiments of the present invention, FIG. 4 is a conceptual diagram illustrating a method for analyzing a steel microstructure using deep learning of FIG. 3 , and FIG. 5 is a block diagram illustrating a steel microstructure analysis system using deep learning of FIG. 3 .

First, as shown in FIGS. 3 to 5 , the steel microstructure analysis method using deep learning according to some embodiments of the present invention includes (a) input layer (IL) of microstructure image information (M) of steel ) (Input Layer), (b) extracting the microstructure image information (M) input to the input layer (IL) from the convolution layer (CL) to the first feature map (M1) and (c) a secondary feature map in which the primary feature map M1 is secondarily convolved in the primary density block DB1 (Dense Block) so that the information of the early stage can lead to the information of the later stage. Concatenating the first feature map M1 to (M2) and accumulating the first cumulative feature map T1; (d) the first in the first transition down TD1 (Transition Down); reducing the resolution of the cumulative feature map T1, and (e) the primary cumulative feature map T1 whose resolution is reduced in the secondary density block DB2 so that the information of the early stage can lead to the information of the later stage. ) by linking the second-order feature map (M2) or the first-order feature map (M1) to the third-order convolutional third-order feature map (M3) and accumulating as a second-order cumulative feature map (T2); (f) reducing the resolution of the second-order cumulative feature map (T2) in the second-order transition down (TD2), and (g) the second-order cumulative feature map with reduced resolution in the third-density block (DB3) ( T2) extracting the quaternary convolutional quaternary feature map (M4), and (h) increasing the resolution of the quaternary feature map (M4) in the primary transition up (TU1) (Transition Up) Step and (i) a fifth-order feature map in which the fourth-order feature map (M4) with an increased resolution in the fourth-density block (DB4) is a fifth-order convolution so that the information of the initial stage can lead to the information of the later stage (M5) at least any one of the fourth-order feature map (M4), the third-order feature map (M3), the second-order feature map (M2), and the first-order feature map (M1) Selecting one or more and accumulating them into a tertiary cumulative feature map (T3) by concatenating them, and (j) increasing the resolution of the tertiary cumulative feature map (T3) in the secondary transition up (TU2). and (k) a 6th-order feature map in which the 3rd-order cumulative feature map (T3), whose resolution is increased in the 5th-density block (DB5), is 6th-order convolution so that the information of the initial stage can lead to the information of the later stage In (M6) at least the fifth-order feature map (M5), the fourth-order feature map (M4), the third-order feature map (M3), the second-order feature map (M2), and the first-order feature map (M1) Selecting one or more and accumulating them into a quaternary cumulative feature map (T4) by selecting and concatenating them, and (l) deconvolutional layer (DCL) of the inference map (MM) so that the microstructure of steel can be divided and discriminated ) to the output layer OL.

Here, as shown in FIG. 4, the principle of the steel microstructure analysis method using deep learning according to some embodiments of the present invention will be briefly described. Steel using deep learning according to some embodiments of the present invention The microstructure analysis method uses an artificial neural network called FC-DenseNet, which is a more advanced form than a complete convolutional neural network.

That is, in the existing convolutional neural network, the efficiency increases as the layers become deeper, but when the layers are deeper than a certain level, the learning effect may decrease. This was one of the main reasons for not being able to use a deep network with more than 20 layers in the existing convolutional neural network. The neural network developed to overcome this problem is DenseNet. In order to solve the problem of blurring the data information of the front stage as the layer deepens, DenseNet introduced the concatenation technique, a deep learning technique that adds all the data information of the front stage in the channel direction of the next stage.

Therefore, through this, it is possible to maintain the information of the previous stage to the end of the neural network, so that the learning effect does not decrease even if the layer becomes deeper. In DenseNet, the number of channels added at each stage is the growth rate, and the section of aggregated information is called DenseBlock.

5 is a block diagram illustrating a steel microstructure analysis system using deep learning of FIG. 3 .

As shown in FIG. 5 , the steel microstructure analysis system using deep learning according to some embodiments of the present invention uses the artificial neural network called FC-DenseNet described above, and the microstructure image information (M) of steel is An input layer (IL) (Input Layer) to be input, and a convolution layer (CL) for extracting the microstructure image information (M) input to the input layer (IL) as a primary feature map (M1); By concatenating the primary feature map M1 to the secondary feature map M2 in which the primary feature map M1 is secondarily convolved so that the information of the initial stage can lead to the information of the later stage, A first density block DB1 (Dense Block) accumulated as a first-order cumulative feature map T1, a first-order transition down (TD1) (Transition Down) that reduces the resolution of the first-order cumulative feature map T1, and , the first-order cumulative feature map (T1) with reduced resolution so that the information of the initial stage can lead to the information of the later-stage is the second-order feature map (M2) on the third-order convolutional third feature map (M3) Alternatively, the secondary density block DB2 for linking the primary feature map M1 to accumulate the secondary cumulative feature map T2, and the secondary transition down for reducing the resolution of the secondary cumulative feature map T2 (TD2), the tertiary density block (DB3) in which the resolution-reduced secondary cumulative feature map (T2) is extracted as a quaternary convolutional quaternary feature map (M4), and the quaternary feature map ( The first transition up (TU1) (Transition Up), which increases the resolution of M4), and the fourth-order feature map (M4), the resolution of which is increased so that the information of the initial stage can lead to the information of the later stage, is a fifth-order convolution At least any one or more of the fourth feature map (M4), the third feature map (M3), the second feature map (M2), and the first feature map (M1) in the selected fifth feature map (M5) 4th density accumulated as a 3rd-order cumulative feature map (T3) by selecting and concatenating A block DB4, a secondary transition up TU2 that increases the resolution of the tertiary accumulation feature map T3, and the tertiary accumulation in which the resolution is increased so that information in an early stage can lead to information in a later stage At least the fifth-order feature map M5, the fourth-order feature map M4, the third-order feature map M3, and the second 5th density block DB5 and steel microstructure accumulated as a 4th cumulative feature map T4 by selecting and concatenating any one or more of the primary feature map M2 and the primary feature map M1 It may include a deconvolution layer (DCL) for outputting the inference map (MM) to an output layer (OL) so as to divide and determine.

Therefore, since the information of the initial stage can be continued to the later stage, the information of the previous stage can be maintained to the end of the neural network, thereby preventing the deterioration of the learning effect, and thus the average accuracy and the average union-to-intersection ratio can be greatly improved.

Therefore, it is possible to analyze the phase of microstructures faster by automating the tasks of human judgment through deep learning through microscopic observation, and FC-, an artificial neural network that is more advanced than the complete convolutional neural network applied to steel microstructure Higher classification accuracy can be obtained by using DenseNet, and additional quantitative information can be obtained by clearly quantifying the fraction of a specific phase and the boundary between phases in microstructure images normally obtained by optical microscopy and scanning electron microscopy.

6 is a diagram illustrating an example of a neural network structure of a steel microstructure analysis system using deep learning according to some embodiments of the present invention, and FIG. 7 is the accuracy of the steel microstructure analysis method system using deep learning of FIG. It is a diagram representing

Therefore, in the case of the FC-DenseNet neural network described in FIG. 6, as shown in FIG. 7, both the average accuracy (Global Accuracy) and the average Mena Intersection over union (IOU) are 90% or more even at various growth rates, so the accuracy of the analysis is large. I could see it going up.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

M: microstructure image information
IL: input layer
CL: Convolution Layer
M1: Primary feature map
M2: secondary feature map
M3: Tertiary Feature Map
M4: 4th Feature Map
M5: 5th Feature Map
M6: 6th Feature Map
T1: First-order cumulative feature map
T2: Secondary Cumulative Feature Map
T3: Tertiary Cumulative Feature Map
T4: Quaternary Cumulative Feature Map
DB1: 1st Density Block
DB2: Second Density Blocks
DB3: 3rd Density Block
DB4: 4th Density Block
DB5: 5th Density Block
TD1: 1st transition down
TD2: 2nd transition down
TU1: 1st transition up
TU2: 2nd transition up
MM: Inference Map
DCL: Deconvolution Layer
OL: output layer

Claims (10)

(a) inputting the microstructure image information of the steel to an input layer (Input Layer);
(b) extracting the microstructure image information input to the input layer as a primary feature map from the convolution layer;
(c) Linking the first-order feature map to the second-order feature map in which the first-order feature map is secondarily convolved in the first-order density block so that the information of the initial stage can lead to the information of the later stage ( concatenation) and accumulating the first cumulative feature map;
(d) reducing the resolution of the first-order cumulative feature map in a first-order transition down;
(e) the first-order cumulative feature map with reduced resolution in the second-density block so that the information of the initial stage can lead to the information of the later stage, the second-order feature map or the accumulating a secondary cumulative feature map by associating the primary feature map;
(f) reducing the resolution of the secondary cumulative feature map in the secondary transition down;
(g) extracting the second-order cumulative feature map with reduced resolution in the third-density block as a fourth-order convolutional fourth-order feature map;
(h) increasing the resolution of the quaternary feature map in the primary transition up; including,
(i) at least the quaternary feature map to the fifth-order convolutional fifth-order feature map, in which the fourth-order feature map, whose resolution is increased in the fourth-density block, is added to the fifth-order convolved fifth-order feature map so that the information of the initial stage can lead to the information of the later stage; selecting and concatenating at least one of the tertiary feature map, the secondary feature map, and the primary feature map to accumulate the tertiary cumulative feature map;
(j) increasing the resolution of the tertiary cumulative feature map in the secondary transition up;
(k) at least the fifth-order feature map in which the third-order cumulative feature map, whose resolution is increased in the fifth-density block, is added to the sixth-order convolutional sixth-order feature map, so that the information of the initial stage can lead to the information of the later stage; accumulating into a fourth cumulative feature map by selecting and concatenating one or more of the fourth feature map, the third feature map, the second feature map, and the first feature map; and
(l) outputting the inference map from the deconvolution layer to the output layer so that the microstructure of the steel can be divided and determined; including,
In step (a),
The microstructure image information of the steel is tempered martensite phase or bainite phase image information, the steel microstructure analysis method using deep learning. delete delete delete delete an input layer for inputting microstructure image information of steel;
a convolution layer for extracting the microstructure image information input to the input layer as a primary feature map;
1 of accumulating the first-order feature map by concatenating the first-order feature map with the second-order convolutional second-order feature map of the first-order feature map so that the information of the initial stage can lead to the information of the later-stage Dense Block;
a first transition down for reducing the resolution of the first cumulative feature map;
2 by linking the secondary feature map or the primary feature map to the tertiary feature map in which the primary cumulative feature map with reduced resolution is tertiary convolved so that the information of the initial stage can lead to the information of the later stage a second-density block accumulating into a second-order cumulative feature map;
a secondary transition down for reducing the resolution of the secondary cumulative feature map;
a third density block extracting the second-order cumulative feature map with reduced resolution as a fourth-order convolutional fourth-order feature map;
a first-order transition up for increasing the resolution of the fourth-order feature map;
At least the quaternary feature map, the tertiary feature map, and the second quaternary feature map are at least the quaternary feature map, the tertiary feature map, and the fifth-order convolutional fifth-order feature map with the fourth-order feature map with increased resolution so that the information of the initial stage can lead to the information of the later stage. a fourth-density block for accumulating a third-order cumulative feature map by selecting and concatenating one or more of a feature map and the first-order feature map;
a secondary transition up for increasing the resolution of the tertiary cumulative feature map;
At least the fifth-order feature map, the fourth-order feature map, and the 3 a fifth-order density block for accumulating a fourth-order cumulative feature map by selecting and concatenating one or more of the second-order feature map, the second-order feature map, and the first-order feature map; and
a deconvolution layer that outputs an inference map as an output layer so that the microstructure of steel can be divided and discriminated; including,
The microstructure image information of the steel is tempered martensite phase or bainite phase image information, a steel microstructure analysis system using deep learning. delete delete delete delete

Images