CN110136067A - A real-time image generation method for super-resolution B-ultrasound images - Google Patents

Abstract

The invention discloses a real-time image generation method for super-resolution B-ultrasound images, aiming at the problem that the definition needs to be improved in the process of B-ultrasound real-time image generation. The steps are as follows: establish a loss function based on MSE; prepare two kinds of losses for different stages: use cross entropy loss to classify the gray level of black and white images, initialize the internal features and attention parameters of the network, and use MSE as the refinement result of the later stage build the neural network according to the core block of the deep convolutional neural network; use the block structure to stack to obtain the deep convolutional network to generate a static image; use the Adam optimizer to initialize the cross entropy loss in step b, and wait for the loss rate to decrease. When it is close to flat, the training is terminated, the loss is changed to MSE loss, and the refinement produces enough inference results to improve PSNR; the weights are exported, using static graphs, integrated and run in medical equipment. The invention can input a single frame and output a single frame, and obtain a result better than that of a hand-designed image enhancement algorithm.

Description

A real-time image generation method for super-resolution B-ultrasound images

technical field

The invention relates to an image processing method for medical equipment, in particular to a real-time image generation method for super-resolution B-ultrasound images.

Background technique

B-mode ultrasound image is an image signal with a low signal-to-noise ratio. In order to achieve a higher signal-to-noise ratio, sharper, and more informative images, in addition to increasing the sampling rate of the device itself, it can also be processed by image processing methods. Performance improvements. Super-resolution can significantly reduce the device's A/D circuit performance and accuracy requirements, thereby relying on image processing to make up for hardware deficiencies. Typical super-resolution methods are based on integrating temporal information, that is, treating continuously changing B-Mode images as video streams, and obtaining clearer images through multi-frame synthesis technology. Based on this method, it will directly cause a large delay in the image results, because the latest frame needs to provide information from the previous frames. On the one hand, the multi-frame method leads to a large amount of calculation, and it is impossible to achieve a balance between the effect and the computational burden; on the other hand, the multi-frame synthesis technology requires the image sequence to be almost static to integrate the background and foreground data, which disguisedly increases the frame rate requirements of the device. The usefulness of the technology is lost.

The technical contradiction of super-resolution of Mode images is mainly due to: it is difficult to obtain multiple still images, and the real-time performance of multiple frames is positively correlated with the performance of the device; the delay of multi-frame super-resolution is large, which is not suitable for observation. The present invention proposes a real-time single-frame super-resolution system implementation method, which has strong practical significance for solving the above-mentioned contradictions.

SUMMARY OF THE INVENTION

The invention overcomes the problem that the clarity needs to be improved in the B-ultrasound real-time image generation process in the prior art, and provides a real-time single-frame super-resolution real-time image generation method for super-resolution B-mode ultrasound images.

The technical solution of the present invention is to provide a real-time image generation method for super-resolution B-mode images with the following steps: using a single frame to generate a deep neural network structure and a training method for super-resolution B-Mode images, comprising: The following steps:

Step a, use the single-frame data of the B-ultrasound equipment display as the input of the original data, use the image of multi-frame super-resolution enhancement simultaneously, the two become a pair, repeat this step, and obtain the data for training the neural network of the present invention set;

Step b. Prepare two kinds of losses for different stages: use cross-entropy loss to classify the grayscale of black and white images, initialize the internal features and attention parameters of the network, and use MSE as the loss function of the refinement result in the later stage;

Step c, establish the block structure of the deep convolutional network, generate the static map of the block, and construct the whole network;

Step d. Use the block structure to stack to obtain a deep convolutional network, and generate a static image of the entire network;

Step e. Use the Adam optimizer to initialize the cross-entropy loss in step b. When the loss decline rate is reduced to nearly flat, terminate the training, change the loss to MSE loss, and perform subsequent super-resolution refinement until sufficient inference results are generated. Improve PSNR;

Step f, derive the above network weights, use the static graph of the above network, integrate and run in the medical equipment, and only perform forward propagation.

In the process of running and collecting samples in the step a, directly use the display output image of the B-ultrasound device as the data of the original image, and simultaneously use the high-resolution mode or use the image enhancement software based on multi-frame super-resolution to obtain the enhancement of the image. As a result, both constitute the dataset for the neural network used.

The loss function established in the step b calculates the two-dimensional matrix obtained by the forward propagation of the network, and obtains the loss value as the back propagation of the optimizer. The cross entropy loss initializes the network parameters, distinguishes brightness and implicit spatial features, and then uses MSE loss to train image detail generation;

The formulas of the two losses are as follows: is the MSE loss, where (i, j) is the row and column position of the pixel, For the cross-entropy loss, the pixels are regarded as a one-dimensional sequence, p is the true value, p is the input super-resolution image, and q is the inference result.

In the step c, the internal structure of the block is selected to configure the layer-by-layer connection in the neural network. The network block is divided into three types: conventional feature extraction, downsampling and upsampling. This configuration generates a static map of the network, and then the blocks are stacked in a stacking manner Configure it to form a static map of the entire network.

In the step d, the core block of the deep convolutional neural network is a single-input single-output end-to-end deep convolutional neural network module package, which is a sub-network of the deep convolutional neural network, and the input and output data is N*C* A four-dimensional tensor of H*W, where N is the number of input three-dimensional tensors C*H*W, and the formed network is a static graph that defines the network.

In the step e, two kinds of loss functions and an optimizer based on gradient descent search are used. According to different stages, the cross entropy loss is used first, and then the MSE loss is used. 0.05—0.04, where steps a-e get the weights of the deep convolutional neural network that improves the PSNR of the input image.

The deep neural network trained in the step f stores all the neural network parameters, exports them, and combines the static graph constructed in step d to run in the computing device, and the computing device converts the image output from the display according to the method in step a. As the input of the network, the edge computing unit is integrated in a machine, and the image with higher PSNR, that is, the super-resolution image, is directly obtained by relying on the single frame data.

Compared with the prior art, the present invention has the following advantages for the real-time image generation method for super-resolution B-ultrasound images: the present invention provides a real-time super-resolution image generation method for B-ultrasound image super-resolution, based on the The computing module of the method can be used as a component of an ultrasonic imaging device, and is suitable for portable devices with low signal quality to improve the signal-to-noise ratio or to improve the performance of common devices. Based on machine learning, the present invention needs to prepare samples and their labels in advance, and proposes a convenient way to directly obtain label data, which simplifies the data preparation process. The proposed deep convolutional neural network is used to quickly generate super-resolution B-Mode images, generating images with higher PSNR than the original.

The good performance of the deep convolutional neural network in the present invention is that it can input a single frame and output a single frame, and obtain a result that is better than a hand-designed image enhancement algorithm. Because the computational complexity of deep neural networks is relatively fixed and the parallelization of convolution instructions can achieve very high efficiency. Therefore, local computing can be realized and a software and hardware system can be formed.

Description of drawings

Fig. 1 is the difference schematic diagram of the present invention for the real-time image generation method of super-resolution B ultrasound image and traditional time-domain super-resolution method;

Fig. 2 is the relational schematic diagram of the present invention for the real-time image generation method of super-resolution B-ultrasound image and the output image of B-ultrasound machine;

Fig. 3 is the schematic diagram that the present invention carries out forward propagation calculation to the inputted image of the real-time image generation method for super-resolution B ultrasound images;

4 is a schematic diagram of the internal implementation process of the conventional feature extraction Block440 in the deep convolutional neural network Block of the real-time image generation method for super-resolution B-ultrasound images of the present invention;

5 is a schematic diagram of the internal implementation process of downsampling Block441 in the deep convolutional neural network Block of the real-time image generation method for super-resolution B-ultrasound images of the present invention;

6 is a schematic diagram of the internal implementation process of upsampling Block442 in the deep convolutional neural network Block of the real-time image generation method for super-resolution B-mode ultrasound images of the present invention;

Fig. 7 is the internal realization process schematic diagram of real-time super-resolution deep convolutional neural network 400 in the real-time image generation method for super-resolution B-mode ultrasound images of the present invention;

8 is a schematic flow chart of the present invention for obtaining a super-resolution image from an original input image in a real-time image generation method for super-resolution B-mode ultrasound images.

Detailed ways

The real-time image generation method for super-resolution B-mode images of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments: A deep neural network structure and training method for generating super-resolution B-mode images using a single frame, including the following steps :

Step a, use the single-frame data of the B-ultrasound equipment display as the input of the original data, use the image of multi-frame super-resolution enhancement simultaneously, the two become a pair, repeat this step, and obtain the data for training the neural network of the present invention set;

Step b. Prepare two kinds of losses for different stages: use cross-entropy loss to classify the grayscale of black and white images, initialize the internal features and attention parameters of the network, and use MSE as the loss function of the refinement result in the later stage;

Step c, establish the block structure of the deep convolutional network, generate the static map of the block, and construct the whole network;

Step d. Use the block structure to stack to obtain a deep convolutional network, and generate a static image of the entire network;

Step e. Use the Adam optimizer to initialize the cross-entropy loss in step b. When the loss decline rate is reduced to nearly flat, terminate the training, change the loss to MSE loss, and perform subsequent super-resolution refinement until sufficient inference results are generated. Improve PSNR;

Step f, derive the above network weights, use the static graph of the above network, integrate and run in the medical equipment, and only perform forward propagation.

In the process of running and collecting samples in the step a, directly use the display output image of the B-ultrasound device as the data of the original image, and simultaneously use the high-resolution mode or use the image enhancement software based on multi-frame super-resolution to obtain the enhancement of the image. As a result, both constitute the dataset for the neural network used.

The loss function established in the step b calculates the two-dimensional matrix obtained by the forward propagation of the network, and obtains the loss value as the back propagation of the optimizer. The cross entropy loss initializes the network parameters, distinguishes brightness and implicit spatial features, and then uses MSE loss to train image detail generation;

The formulas of the two losses are as follows: is the MSE loss, where (i, j) is the row and column position of the pixel, For the cross-entropy loss, the pixels are regarded as a one-dimensional sequence, p is the true value, p is the input super-resolution image, and q is the inference result.

In the step c, the internal structure of the block is selected to configure the layer-by-layer connection in the neural network. The network block is divided into three types: conventional feature extraction, downsampling and upsampling. This configuration generates a static map of the network, and then the blocks are stacked in a stacking manner Configure it to form a static map of the entire network.

In the step d, the core block of the deep convolutional neural network is a single-input single-output end-to-end deep convolutional neural network module package, which is a sub-network of the deep convolutional neural network, and the input and output data is N*C* A four-dimensional tensor of H*W, where N is the number of input three-dimensional tensors C*H*W, and the formed network is a static graph that defines the network.

In the step e, two kinds of loss functions and an optimizer based on gradient descent search are used. According to different stages, the cross entropy loss is used first, and then the MSE loss is used. 0.05—0.04, where steps a-e get the weights of the deep convolutional neural network that improves the PSNR of the input image.

The deep neural network trained in the step f stores all the neural network parameters, exports them, and combines the static graph constructed in step d to run in the computing device, and the computing device converts the image output from the display according to the method in step a. As the input of the network, the edge computing unit is integrated in a machine, and the image with higher PSNR, that is, the super-resolution image, is directly obtained by relying on the single frame data.

The specific implementation process of this embodiment is as follows, providing a deep neural network edge computing system for B-ultrasound image super-resolution, which is a computing module for local real-time processing of B-Mode images, using a deep convolutional neural network The network overcomes the shortcomings of traditional super-resolution or modeling methods that rely on temporal information, and the difference from traditional temporal super-resolution methods is shown in Figure 1.

Based on this, a real-time image post-processing system and training method are proposed to solve the problems of high replacement cost of existing equipment and cumbersome post-processing procedures. To realize the system, the detailed implementation is as follows: In order to distinguish from the temporal super-resolution method, Fig. 1 visually presents the difference between the two. After the original image 101 is collected from n frames F ₀ -F _n , the super-resolution image 102 is obtained by processing the time-domain method. In order to obtain the latest time-domain super-resolution image 102 related to the latest frame F _n , it takes time. are: the frame acquisition time of objectively existing F ₀ -F _n : T(F ₀ -F _n ), and the post-processing time T _p .

Since the super-resolution time in the present invention does not depend on the previous frame information, the time consumption is: the objectively existing F ₀ frame acquisition time: T(F ₀ ), and the post-processing time T _p . The difference between super-resolution images 102 and 103 is whether multiple or single frames are used.

In order to more conveniently attach the system to the existing B-ultrasound machine, the images output by the B-ultrasound machine can be directly post-processed. As shown in Fig. 2, there are many equivalent alternatives for the method of obtaining the input image 101, and only a recommended embodiment for obtaining the image without modifying the original hardware is listed here.

The system hardware consists of a signal acquisition module 202, a calculation unit 301, and an image output unit. The signal acquisition module 202 can be used to directly acquire the image signal 201 output by the display. A recommended embodiment is: use an HDMI signal acquisition module to connect to the display interface of the ultrasound machine, obtain the output picture of the ultrasound machine, and then cut the region of interest, as shown in 201 to 202 in Figure 2. The obtained image 101 is transmitted to the computing unit 301, and finally a super-resolution image signal 103 is obtained.

The computing unit is a kind of computer, which is used to load the intermediate expression data of the deep neural network, such as the weight 302 and the network structure 303, and perform forward propagation calculation on the input image 101, and finally obtain the super-resolution image 103, as shown in FIG. 3 shown. As a realization of the hardware system, the computing unit uses the deep convolutional neural network of the present invention to complete super-resolution calculation. The following will focus on explaining the Graph450, Block440-442 of its internal deep convolutional neural network, and the preparation method of its training data. . The internal implementation of the deep convolutional neural network is shown in Figures 4-7.

The deep convolutional neural network implementation in the computing unit is divided into the design of Block (440, 441, 442) and the configuration of Graph 450.

As shown in Figure 4-7, Block itself is an end-to-end structure with a single input and a single output. Block is a module encapsulation of an end-to-end deep convolutional neural network as a deep neural network, and is a sub-network , using channel attention bypass in the block to improve network efficiency. The input and output data is a N*C*H*W four-dimensional tensor, and N is the number of input three-dimensional tensors C*H*W. For the inference process, N is strictly 1 anywhere in the network. For training process, N is a non-zero positive integer, and N is strictly equal to the training batch size anywhere in the network. C represents the number of input/output channels of the block, and C is a non-zero positive integer. According to the properties of Blcok, the number of input and output channels C is not necessarily equal. H and W define the height and width of the two-dimensional tensor, that is, the number of rows and columns.

In order to briefly describe the internal structure and working principle of the deep convolutional neural network in the invention, and considering that N describes the task batch size of the network, N can be any non-zero natural number, so the following descriptions about tensors are all The number of N is not considered.

The network block is divided into three types: conventional feature extraction 440 , down-sampling 441 and up-sampling 442 . Among them, the block of conventional feature extraction uses the channel attention mechanism based on the global strength of the feature, that is, the average value of each channel is obtained, and the weight gain is identified in a small fully connected network. The distribution pattern of the network channel strength represents the input Different tasks, attention mechanism can improve the utilization of network resources and expressive ability under different tasks. The activation function is sigmoid: Therefore, the strength of the channel does not affect the network performance in the case of failure to converge, and the channel utilization can be greatly improved in the case of convergence, that is, in a single layer, not only can the performance be improved according to the pyramid features, but also can be identified according to the input features. The channel distribution of intensity explicitly causes the optimizer to allocate different channel resources to different tasks.

In order to facilitate the description, the key operations 401 to 410 in Block 440 to 442 are explained first: In order to facilitate the following description and avoid repeated descriptions, it should be noted first that K is the edge length of the convolution kernel, and the following convolution kernel shapes are all It is a square, allowing the side length to be 1, that is, 1*1 convolution.

The Padding parameters of all convolution operations are based on the fact that H and W are not changed after convolution, as described below. In this way, Size _padding = (K-1) for each edge. The convolution operation is performed by a sliding window, and the function of Padding here is to ensure that the size of the tensor will not "shrink a circle" after the convolution calculation.

The Padding operation is generally to fill with 0s, but the data of adjacent sides can also be copied, and even the outermost edge is used as the symmetry axis to perform mirror padding. Due to its variability, this embodiment does not require a padding form.

401 is the input tensor of Block, and tensor 406 is obtained by performing dot multiplication calculation 404 on the tensor obtained by tensor 401 after operation 405.

The 402 operation in the block is a global average pooling operation (Global Average Pooling), which takes the C*H*W tensors by channel to take the global average (GAP).

right The summation formula for channel GAP is: The difference between the output tensors 408 and 412 is that 408 is an intermediate variable of the network, and 412 can be an intermediate variable of the network or the final output of the network. The shapes of 408 and 412 need to be set according to the actual size of the tensor in Blcok.

The different Blcoks will be explained below: as shown in Block 440 in Figure 4, 440 is a group of (Block) convolutional neural networks, which are used for multi-scale depth extraction of features, and their scale and configuration are: the input and output tensors are the same size , which is C*H*W.

The input tensor 401 is computed through two paths to obtain a tensor 406. A set of scalars of size C is obtained using operations 402, 403, and 404 is used to perform a channel-C dot product on 406. 406 obtains four tensors through 411, 412, 413, and 414 operations. The four tensors are merged into an output tensor 408 in the channel dimension via 407 .

It needs to be explained in detail that 405 is a 1*1 convolution, which is used to linearly combine C two-dimensional tensors inside 401 to obtain C/4 two-dimensional tensors of the same size, forming a size of tensor 406.

As mentioned above, for relatively shallow networks, in order to enhance the expression ability of Block, an explicit expression 402, 403 and 404 is used to enable Blcok to dynamically adjust the feature strength in the forward propagation process according to the feature strength of different channels. , that is: identify the response intensity of each channel of the input feature 401 , and perform dynamic weight compensation and suppression for 406 by channel. The scale of the sub-network 403 is set as follows, the sub-network 403 is a group of three-layer fully connected networks, the size of the first layer and the third layer is C, and the second is the hidden layer, and the size is C*4.

Operations 411, 412, 413, and 414 are generalized grouping convolution operations. The recommended group size is is a depthwise separable convolution, or in special cases is a typical grouped convolution. The convolution kernel parameters of the convolution operation can be trained. Each convolution operation in Figure 4-Figure 7 is a trainable operation, which will not be repeated below. Its configuration is as follows: the size K of the convolution kernel (Kernel) of 411 is 7, that is, a two-dimensional matrix of size 7*7, which will not be repeated below, the number of output channels is the same as the number of input channels, and the recommended group size is C/4.

The size of the convolution kernel (Kernel) of 412 is 3, and the number of output channels is the same as the number of input channels. The recommended group size is The operation is dilated convolution, also known as hole convolution, which is equivalent to a sieve-like sampling structure, and the number of pixels sampled is 3*3. R is the dilation rate. For small resolution input, R is 1, that is, no pixels are skipped during sliding window sampling. This parameter can be determined according to the actual situation. For example, if R=2, the convolution kernel of 412 is equivalent to a window with a size of 7*7, but 3*3 pixels are still sampled, and 1 pixel is skipped during sampling. If the value of participating sampling pixels in the convolution kernel is 1 and the value of skipping is 0, the two-dimensional representation when R=2 is:

The convolution kernel size K of 413 is 5, the number of output channels is the same as the number of input channels, and the operation is grouped convolution. The recommended group size is The convolution kernel size K of 414 is 3, the number of output channels is the same as the number of input channels, and the operation is grouped convolution. The recommended group size is

Through 411, 412, 413, 414 operations, a total of 4 sizes are obtained tensor.

Here, the operation 407 is defined, and concatenate refers to the operation of arranging multiple tensors in order in a certain dimension and using them as a new tensor. Here, the dimension is the channel dimension. In order to facilitate understanding, an image is expressed as: taking 411 as an example, what is obtained is A two-dimensional tensor of H*W, and the two-dimensional tensors are stacked, that is, a tensor of size C*H*W output by 411. The operation of 407 is to continue to "base" the tensor of 412 to 411 on the basis of 411. The dimension of the "base" is the channel dimension. And so on, 407 merges the tensors of 411, 412, 413, 414 into a tensor 408 of size C*H*W by channel.

The operation of 407 has the same logic in the following, and is also the same in the representation of FIG. 4 to FIG. 7 , so it will not be repeated. In order to avoid repetition, please refer to the above information for the concepts of 401, 407, and 408 used below.

Block440 is a feature extraction structure interspersed in the network, which can effectively improve the network expression ability and is an important idea of the present invention. Similarly, this idea is also embodied in Block441.

As shown in Block 441 in Figure 5: 441 is a group of (Block) convolutional neural networks, the purpose is to downsample and extract features, and its scale and configuration are: the input tensor 401 has a size of C*H*W; the output tensor 408 size is

401 is computed through a path 409. The tensors obtained by 409 are obtained through 415, 416, 417, and 418 operations to obtain four tensors.

The four tensors are merged into the output tensor 408 with the channel dimension through 407. The operation of 409 is that the convolution kernel (Kernel) size K is 3, the number of output channels is the same as the number of input channels, the stride (Stride) is 2, The operation is conventional convolution, and the group size is 1, that is, no grouping is calculated. 409 extracts features by sliding a 3*3 convolution kernel, because the stride is 2, and because the stride is discontinuous, the stride is skipped by 1 pixel, and the sliding window is performed in the HW dimension, so the obtained Tensor size is The tensors obtained by 409 obtain four tensors through the following operations: Operations 415, 416, 417, and 418 are generalized grouping convolution operations. The recommended group size is C, which is a depthwise separable convolution. Special cases below can also be is a typical grouped convolution.

The convolution kernel size K of 415 is 7, the number of output channels is the same as the number of input channels, and the recommended group size is C.

The size of the convolution kernel (Kernel) of 416 is 3, and the number of output channels is the same as the number of input channels. The recommended group size is The operation is dilated convolution. For the definition of dilated convolution, please refer to the relevant description of 412.

441 is the process of downsampling. If the tensor 401 (H, W) input to 441 is too large, it is recommended that R=2 or R=3, then the convolution kernel of 416 is equivalent to a size of 7*7 or 15*15 window, but still sample 3*3 pixels, skip 1 or 3 pixels when sampling.

If the participating sampling pixel value in the convolution kernel is 1 and the skip value is 0, the two-dimensional representation when R=2 is:

When R=2 or R=3, a larger or extremely large window (7*7 or 15*15) can be obtained, that is, the receptive field, which is convenient and efficient to capture airspace features in a large range. 442 is in the front part of the network, and the large receptive field is more important in the front part of the network. 442 also includes the convolution of small receptive fields such as 415, 417, and 418. Combined with the large receptive field information of 416, tensor 410 can perceive multi-scale features in 442 at the same time. Importantly, through this multi-scale design of 441 and 440, the network depth can be effectively reduced.

The multi-scale design of 440 and 441 can expand the expressive ability of the network, and at the same time use very fine-grained grouped convolution or depthwise separable convolution, this design can improve the computational efficiency while ensuring the accuracy and the expressive ability of the network. .

442 is a naive upsampling network. Benefiting from the rich features in the front section and the multi-scale expression ability of 420, 442 only needs upsampling to get richer information in the back section of the network. If you want to improve PSNR or use it for other purposes, you can replace it with a structure based on deconvolution. 442 is only an example of a complete network. The upsampling in the back section of the network can be diversified, and the upsampling method is not protected by the present invention. within the range.

As shown in Block 442 in Figure 6: 442 is a group of (Block) convolutional neural networks, the purpose is to upsample and merge features, and its scale and configuration are: the input tensor 401 has a size of C*H*W; the output tensor 412 size is

401 is computed through a path 410.

410 is a simple operation of Bicubic upsampling and 1*1 convolution merging. At the same time, 1*1 convolution is combined before upsampling to linearly combine the input tensor 401 to obtain tensor 412.

The number of channels in the final output of 412 is 1. In order to overcome the upsampling blur of 412, a simple edge enhancement algorithm can be performed on the network output result to improve PSNR. Because the edge enhancement algorithm is too diverse, the edge enhancement algorithm is not within the protection scope of the present invention. .

As shown in CNN400 in Figure 7, the overall structure of the network is hourglass-shaped, the direction is from left to right, and the input image is single-channel. The meaning of the parameters shown in each Block is as follows: Take the first 441C4 as an example, 441 is the structure of Block441 shown in Figure 5, and C4 is the number of output channels is 4. Take the middle 440C256*20 as an example, 440 is the structure of Block440 shown in Figure 4, C256 is the number of output channels is 256, *20 means that the structure is connected in series with 20. By analogy, the network end 442C1 is the structure of Block 442 shown in Figure 6, and C1 is the output single-channel image.

In order to train the above network, the method of obtaining the training data, the Loss function and the details of the optimizer need to be provided.

As shown in FIG. 8 , the original sample set 501 is the set of collected original inputs 101, and the super-resolution sample 502 is the set of super-resolution images 102 obtained offline by the time domain method. Referring to the schematic diagram of the time domain method in FIG. 1 , 101 and 102 appear in pairs, and the original sample set 501 and the super-resolution sample 502 can be obtained by preparing multiple sheets of 101 and repeating the time domain method. Referring to Figure 8 and Figure 4 to Figure 7, the CNN in Figure 8 refers to the CNN400 in Figure 7, that is, the entire neural network. Referring to Fig. 8, CNN loads the data in the original sample set 501, and the Loss function 503 calculates the result of the forward propagation of the neural network 400 and the corresponding data in the super-resolution sample 502.

In order to facilitate the representation of the Loss function, an image in the original sample set 501 is selected, passed to the neural network 400, and the image obtained after forward propagation is called X, and in the super-resolution sample 502, the image corresponding to 501 The image is called Y.

The Loss function uses MSE (Mean Squared Error): where (i, j) is the row and column position of the pixel.

The Loss function generally uses a regular term to prevent over-fitting. Here, the regular term in Loss is not used, but the Dropout method is used to prevent over-fitting. This is designed in consideration of the particularity of the network.

Dropout is used for all trainable neurons in the neural network 400. The details are as follows: the global dropout probability p _global = 10%, where the dropout probability p ₄₀₃ of 403 in Figure 4 = [0%, 10%], when training Manually decrease gradually according to the training process until floor(C*4*p ₄₀₃ )=0. where floor(x) is an operation that rounds down x.

Thanks to the extensive use of fully automatic optimizers, the optimizer using Adam's method can avoid manual adjustment of the learning rate, and alleviate the problems of training failure caused by too large a learning rate and too long convergence time caused by a too low learning rate. The optimizer 504 used in the present invention is the Adam optimizer, and a recommended parameter is: the initial learning rate is 0.003, and the gradient update parameter beta _1,2 =(0.9, 0.99).

The trained CNN weights 302 can be loaded by any deep learning framework that can parse and instantiate the CNN structure 303, and run in the computing unit 301 shown in Figure 3, and finally form an end-to-end real-time through the relationship shown in Figure 2. super-resolution system.

To sum up: the present invention proposes an overall system architecture, based on which an end-to-end software and hardware system for super-resolution is obtained. In the present invention, a deep convolutional neural network 400 is proposed for rapidly generating super-resolution B-Mode images. The present invention proposes a convenient way to directly obtain the labeling data 502, which simplifies the data preparation process.

The real-time super-resolution deep convolutional neural network 400 for B-ultrasound image super-resolution provided by the present invention has extremely strong practical significance. Further, in combination with the training method and system architecture provided by the present invention, the computing module based on the present invention can be used as a component of an ultrasonic imaging device, and is suitable for portable devices with low signal quality to improve the signal-to-noise ratio or for the performance of common devices. promote.

Although the embodiments of the present invention have been described as detailed and accurate as possible, any equivalent transformations, equivalent replacements, etc. and similar implementation methods of the present invention modified around the idea of the present invention are all within the protection scope of the present invention.

Claims (7)

1. a kind of real-time imaging generation method for super-resolution B ultrasound image, it is characterised in that: generate super-resolution using single frames The deep neural network structure and its training method of rate B-Mode image comprising the steps of:

Step a, use the frame data of B ultrasound device display as the input of initial data, while using multiframe super-resolution The image of enhancing, the two become a pair, repeat this step, obtain the data set for training neural network of the invention；

Step b, prepare two kinds of loss and be used for different phase: carrying out two classification using gray scale of the cross entropy loss to black-and-white image, It initializes network internal feature and pays attention to force parameter, use MSE as the loss function of refinement of rear stage result；

Step c, the block structure for establishing depth convolutional network, generates the static map of block, and building network is whole；

Step d, it is stacked to obtain depth convolutional network using block structure, generates the static map of whole network；

Step e, cross entropy loss in step b is initialized using Adam optimizer, is reduced to loss fall off rate close to flat When smooth, training is terminated, change loss is MSE loss, subsequent super-resolution refinement is carried out, until generating enough reasoning knots Fruit improves PSNR；

Step f, above-mentioned network weight is exported, it is integrated and operate in Medical Devices using the static map of above-mentioned network, only into Row propagated forward.

2. the real-time imaging generation method according to claim 1 for super-resolution B ultrasound image, it is characterised in that: institute Step a is stated during operation and collecting sample, the display of B ultrasound equipment is directly used to export number of the image as original image According to, while the enhancing knot of image is obtained using high-resolution mode or using the image enhancement software based on multiframe super-resolution Fruit, the two constitute the data set for the neural network being used for.

3. the real-time imaging generation method according to claim 1 for super-resolution B ultrasound image, it is characterised in that: institute It states the two-dimensional matrix that the loss function established in step b obtains network propagated forward to calculate, obtains loss value as excellent The backpropagation for changing device first uses cross entropy when using Adam optimizer using the appraisal procedure of average variance and cross entropy Loss initializes network parameter, distinguishes brightness and implicit space characteristics, then carry out image detail generation using MSE loss Training；

The formula of two kinds of loss is as follows:For MSE loss, wherein (i, j) is The column locations of pixel,For cross entropy loss, pixel is considered as one-dimensional sequence, p is true value, p For the super-resolution image of input, q is the reasoning results.

4. the real-time imaging generation method according to claim 1 for super-resolution B ultrasound image, it is characterised in that: institute Stating and selecting Block internal structure in step c is that connection layer-by-layer in neural network is configured, and network B lock points are conventional special Sign is extracted, down-sampling and three kinds of up-sampling, the configuration generate the static map of network, then carry out block according to stack manner Configuration, constitutes the static map of whole network.

5. according to the real-time imaging generation method described in claim 1 for super-resolution B ultrasound image, it is characterised in that: described Depth convolutional neural networks core block is the module encapsulation of the end-to-end depth convolutional neural networks of single-input single-output in step d, Data for the sub-network of depth convolutional neural networks, input and output are the four dimensional tensors of N*C*H*W, and wherein N is input The number of three-dimensional tensor C*H*W, the network of composition are the static maps for defining network.

6. the real-time imaging generation method according to claim 1 for super-resolution B ultrasound image, it is characterised in that: institute It states using two kinds of loss functions and the optimizer based on gradient descent search in step e, according to the different stages, first using intersection Entropy loss reuses MSE loss, and optimizer is iterated the limited calculating of number, and termination condition is that loss drops to 0.05- 0.04, wherein step a-e is improved the weight of the depth convolutional neural networks of input picture PSNR.

7. the real-time imaging generation method according to claim 1 for super-resolution B ultrasound image, it is characterised in that: institute It states trained deep neural network in step f and stores all neural network parameters, exported, and combine step d institute structure The static map built is operated in and is calculated in equipment, at the same calculate image that equipment exports display according to the method for step a as It is higher directly to obtain PSNR by frame data for the input of network, and then the unit that integrated edge calculates in a machine Image, i.e. super-resolution image.