CN111724360B - Lung lobe segmentation method, device and storage medium - Google Patents

Abstract

The invention discloses a lung lobe segmentation method, a device and a storage medium, which relate to the field of medical image processing, wherein the lung lobe segmentation method comprises the following steps: acquiring lung images at multiple moments in the respiratory process; determining a lung image to be segmented in the multi-moment lung images, wherein the lung image outside the image to be segmented is used as a first lung image; fusing the lung images to be segmented by using at least one first lung image to obtain a fused lung image, wherein the at least two first lung images comprise at least one lung image at a moment before the image to be segmented and/or at least one lung image at a moment after the image to be segmented; and segmenting the fused lung image by using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented. So as to solve the problem of poor segmentation effect caused by insufficient characteristic information of the lung (leaf).

Description

A lung lobe segmentation method, device and storage medium

technical field

The invention relates to the field of medical image processing, in particular to a lung lobe segmentation method, device and storage medium.

Background technique

The blunt upper end of the lung is called the lung apex, which protrudes upwards through the upper thorax into the root of the neck. The base is located on the top of the diaphragm. The entrance and exit of lymphatic vessels and nerves are called hilum, and these structures that enter and exit the hilum are wrapped together by connective tissue and are called lung roots. The left lung is divided into upper and lower lobes by an oblique fissure, and the right lung is divided into upper, middle and lower lobes by a horizontal fissure in addition to the oblique fissure.

At present, no matter whether traditional machine learning or deep learning is used to segment lung lobes, the classification is based on the characteristics of the lung (lobe). Therefore, the characteristics of the lung (lobe) are particularly important. If there is enough feature information of a lung (lobe), such It will make the classifier better learn and classify, and better complete the lung lobe segmentation.

Contents of the invention

In view of this, the present invention provides a lung lobe segmentation method, device and storage medium to solve the current problem of insufficient feature information of the lung (lobe), resulting in poor segmentation effect.

In a first aspect, the present invention provides a lung lobe segmentation method, comprising:

Obtain lung images at multiple moments during breathing;

Determining the lung images to be segmented in the lung images at multiple moments, and the lung images other than the images to be segmented as the first lung images;

Using at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image, the at least two first lung images include at least one lung image at a time before the image to be segmented and/or at least one a lung image at a time after the image to be segmented;

The fused lung image is segmented using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented.

Preferably, the method of using the at least two first lung images to fuse the lung image to be segmented to obtain a fused lung image includes:

performing a registration operation of the at least one first lung image to the lung image to be segmented to obtain a lung image to be fused;

Fusing the lung image to be fused with the lung image to be segmented to obtain a fused lung image;

and/or,

The method for determining the lung image to be segmented at a certain moment in the lung images at multiple moments includes:

Calculate lung volumes in the lung images at multiple times, and determine the lung image with the largest lung volume as the lung image to be segmented.

Preferably, the method of fusing the lung image to be fused and the lung image to be segmented to obtain a fused lung image includes:

Determining the weight value of the lung image to be fused;

Obtaining a weighted lung image according to the weight value and the lung image to be fused;

Fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image;

and/or,

The method for determining the lung image to be segmented at a certain moment in the lung images at multiple moments further includes:

Before calculating the lung volumes in the multi-time lung images, respectively extract the left lung and the right lung of the multi-time lung images, and calculate the first volume and the first volume of the left lung in the multi-time lung images respectively For the second volume of the right lung, calculate the lung volumes in the multi-moment lung images according to the first volume and the second volume respectively.

Preferably, the method for determining the weight value of the lung image to be fused includes: determining the registration point of the lung image to be fused, determining that the weight value of the registration point is greater than the weight value of the non-registration point, the registration The feature points other than the quasi-points are non-registration points;

and/or,

The method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image includes: performing sum processing on the weighted lung image and the lung image to be segmented to obtain the Fusion of lung images;

and/or,

The method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image includes: using a first preset neural network to combine the lung image to be fused and the lung image to be fused The lung images are segmented and fused to obtain a fused lung image.

Preferably, the method of performing the registration operation of the at least one first lung image to the lung image to be segmented to obtain the lung image to be fused includes:

extract images from the same position in the at least two first lung images and the image to be segmented, and obtain a lung motion sequence image formed by the images extracted at the same position;

respectively calculating lung displacements of adjacent images in the lung motion sequence images, and performing a registration operation from the at least one first lung image to the lung image to be segmented according to the lung displacements;

and/or,

The lung lobe segmentation method further includes: there are at least two preset lung lobe segmentation models, the features of the lung lobe segmentation images obtained by the preset lung lobe segmentation models are fused to obtain fusion features, and the fusion features are classified. Get the final lung lobe image.

Preferably, the method of extracting images from the same position in the at least two first lung images and the image to be segmented, and obtaining the lung motion sequence image formed by the images extracted at the same position, includes:

determining the number of layers of the multi-moment lung image;

determining the at least two first lung images and the lung images at the same position of the image to be segmented according to the number of layers;

Obtaining the lung motion sequence image according to the lung images at the same position at multiple times;

and/or,

The method for separately calculating the lung displacement of adjacent images in the lung motion sequence images includes:

respectively determining the first forward optical flow of adjacent images in the lung motion sequence images;

determining lung displacements of the adjacent images respectively according to the first forward optical flow;

and/or,

The method of fusing the features of the lung lobe segmentation image obtained by the preset lung lobe segmentation model to obtain fusion features includes:

Splicing the lung lobe segmentation images obtained by the preset lung lobe segmentation model respectively to obtain splicing features, and inputting the splicing features into a second preset neural network for convolution operation to obtain the fusion features.

Preferably, the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images further includes:

respectively determining the first reverse optical flow corresponding to the first forward optical flow;

determining lung displacements of the adjacent images according to the first reverse optical flow and the first reverse optical flow, respectively;

and/or,

The method for separately calculating lung displacements of adjacent images in the lung motion sequence images further includes: respectively performing optical flow optimization processing on the first forward optical flow and the first reverse optical flow to obtain The second forward optical flow corresponding to the first forward optical flow, and the second reverse optical flow corresponding to each of the first reverse optical flows; respectively according to the second forward optical flow and the first Two reverse optical flows determine the lung displacement of the adjacent images.

Preferably, the method for determining the lung displacement of the adjacent image according to the first reverse optical flow and the first reverse optical flow respectively includes:

respectively performing operations on the second forward optical flow and the second reverse optical flow to obtain a corrected optical flow;

The lung displacements of the adjacent images are respectively determined according to the corrected optical flow.

Preferably, performing optical flow optimization processing on the first forward optical flow and the first reverse optical flow respectively, to obtain a second forward optical flow corresponding to each of the first forward optical flows, and The method for the second reverse optical flow corresponding to each of the first reverse optical flow includes:

connecting each of the first forward optical flows to obtain a first connecting optical flow, and connecting each of the first reverse optical flows to obtain a second connecting optical flow;

Perform N times of optical flow optimization processing on the first connected optical flow and the second connected optical flow respectively, to obtain the first optimized optical flow corresponding to the first connected optical flow, and the second optimized optical flow corresponding to the second connected optical flow light flow;

The second forward optical flow corresponding to each first forward optical flow is obtained according to the first optimized optical flow, and the second reverse corresponding to each first reverse optical flow is obtained according to the second optimized optical flow light flow;

Wherein, N is a positive integer greater than or equal to 1.

Preferably, performing N times of optical flow optimization processing on the first connection optical flow and the second connection optical flow respectively includes:

Perform the first optical flow optimization process on the first connected optical flow and the second connected optical flow to obtain the first optimized sub-optical flow corresponding to the first connected optical flow, and the first optimized sub-optical flow corresponding to the second connected optical flow. optimize the sub-optical flow; and

Perform the (i+1)th optical flow optimization process on the i-th optimized sub-optical flow of the first connected optical flow and the second connected optical flow respectively, to obtain the (i+1)th optimized sub-optical flow corresponding to the first connected optical flow sub-optical flow, and the i+1th optimized sub-optical flow corresponding to the second connected optical flow;

Wherein, i is a positive integer greater than 1 and less than N; through the Nth optimization process, the Nth optimized sub-optical flow of the first connected optical flow obtained is determined as the first optimized optical flow, and will be obtained The Nth optimized sub-optical flow of the second connected optical flow is determined as the second optimized optical flow; each optical flow optimization process includes residual processing and upsampling processing.

Preferably, according to the forward time sequence of the lung lobe images at multiple times, the first forward optical flow of adjacent images in the lung motion sequence images is determined, and according to the reverse time sequence of the lung lobe images at multiple times , determining the first reverse optical flow of adjacent images in the lung motion sequence images.

In a second aspect, the present invention provides a lung lobe segmentation device, which is characterized in that it includes:

An acquisition unit, configured to acquire lung images at multiple moments in the breathing process;

A determining unit, configured to determine a lung image to be segmented among the lung images at multiple times, and a lung image other than the image to be segmented is used as a first lung image;

A fusion unit, configured to use at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image, and the at least two first lung images include at least one lung image at a moment before the image to be segmented and/or at least one lung image at a time subsequent to the image to be segmented;

A segmentation unit, configured to segment the fused lung image by using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented.

In a third aspect, the present invention provides a storage medium. When the computer program instructions are executed by a processor, the above method is implemented, including:

Obtain lung images at multiple moments during breathing;

Determining the lung images to be segmented in the lung images at multiple moments, and the lung images other than the images to be segmented as the first lung images;

Using at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image, the at least two first lung images include at least one lung image at a time before the image to be segmented and/or at least one a lung image at a time after the image to be segmented;

The fused lung image is segmented using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented.

The present invention has at least the following beneficial effects:

The present invention provides a lung lobe segmentation method, device and storage medium to solve the problem of poor segmentation effect caused by insufficient characteristic information of the current lung (lobe).

Description of drawings

Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above-mentioned and other objects, features and advantages of the present invention are more clear, in the accompanying drawings:

FIG. 1 is a schematic flowchart of a lung lobe segmentation method according to an embodiment of the present invention.

Detailed ways

The present invention will be described below based on examples, but it should be noted that the present invention is not limited to these examples. In the following detailed description of the invention, some specific details are set forth in detail. However, the present invention can be fully understood by those skilled in the art about the parts that are not described in detail.

In addition, those of ordinary skill in the art should understand that the provided drawings are only for illustrating the objects, features and advantages of the present invention, and the drawings are not actually drawn to scale.

At the same time, unless the context clearly requires, the words "include", "include" and other similar words in the entire specification and claims should be interpreted as an inclusive meaning rather than an exclusive or exhaustive meaning; that is, "include but not limited to the meaning of ".

The execution subject of the lung lobe segmentation method provided by the embodiments of the present disclosure may be any image processing device. For example, the lung lobe segmentation method may be executed by a terminal device or a server, wherein the terminal device may be a user equipment (UserEquipment, UE), a mobile device, a user Terminals, Terminals, Cellular Phones, Cordless Phones, Personal Digital Assistant (PDA), Handheld Devices, Computing Devices, Vehicle Devices, Wearable Devices, etc. The server can be a local server or a cloud server. In some possible implementation manners, the lung lobe segmentation method may be implemented by a processor invoking computer-readable instructions stored in a memory.

FIG. 1 is a schematic flowchart of a lung lobe segmentation method according to an embodiment of the present invention. As shown in Figure 1, a lung lobe segmentation method includes: Step 101: Acquire lung images at multiple moments in the breathing process; Step 102: Determine the lung image to be segmented in the lung images at multiple moments, the image to be segmented Other lung images are used as the first lung image; Step 103: Use at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image, and the at least two first lung images include at least one in the A lung image at a time before the image to be segmented and/or at least one lung image at a time after the image to be segmented; Step 104: Using a preset lung lobe segmentation model to segment the fused lung image to obtain the lung image to be segmented image of the lung lobes. In order to solve the problem of insufficient feature information of the current lung (lobe), resulting in poor segmentation effect.

Step 101: Acquire lung images at multiple moments in the breathing process.

In the embodiment of the present disclosure, the acquisition of lung images at multiple moments in the breathing process may be lung images at multiple moments in the inspiration process, or lung images at multiple moments in the breathing process, or multiple moments in the inspiration and breathing process The lung images at multiple moments; the above-mentioned multi-moment lung lobe images are multi-moment lung images respectively obtained by the same patient at multiple moments during exhalation and/or inhalation. The moment in the embodiments of the present disclosure may be expressed as a time period, that is, the time information of collecting a group of lung images. The specific acquisition process can be carried out according to the guidance of the imaging doctor; for example, during the breathing process, at least one set of lung images can be collected at the moment of deep inhalation, or at least one set of lung images can be collected at the moment of deep exhalation, Collect at least one set of lung images in a calm state, where the calm state is a set of lung images collected after normal exhalation. For another example, in the process from breathing to exhaling, the patient is asked to hold his breath at different moments in the inhalation or exhalation phase, so as to collect multi-moment lung images.

The segmentation result may include position information corresponding to each partition (lung lobe) in the identified lung image. For example, the lung image may include five lung lobe regions, which are the upper right lobe, the middle right lobe, the lower right lobe, the upper left lobe, and the lower left lobe, and the obtained segmentation results may include location information of the five lung lobes in the lung image. The embodiment of the present disclosure can represent the segmentation result by means of mask features, that is to say, the segmentation result obtained by the embodiment of the present disclosure can be a feature expressed in the form of a mask, for example, the embodiment of the present disclosure can be the above-mentioned five lung lobe regions Assign unique corresponding mask values, such as 1, 2, 3, 4, and 5, respectively, and the area formed by each mask value is the position area where the corresponding lung lobe is located. The foregoing mask values are only illustrative, and other mask values may also be configured in other embodiments.

In some possible implementation manners, the embodiments of the present disclosure may obtain lung images at multiple moments by taking CT (Computed Tomography, computerized tomography). The specific method includes: determining the number of scanning layers, layer thickness and interlayer distance of the multi-time lung lobe image; and acquiring the multi-time lung image according to the scanning layer number, layer thickness and interlayer distance. Wherein, the lung image obtained in the embodiment of the present disclosure is composed of multi-layer images, which can be regarded as a three-dimensional image structure.

In some possible implementations, multiple time lung images can be requested from other electronic devices or servers, that is, multiple sets of lung images can be obtained, each set of lung images corresponds to a time, and multiple sets of lung images constitute multiple time points lung image. In addition, in the embodiment of the present disclosure, in order to reduce images with other features, lung parenchyma segmentation processing can be performed on the lung image when the lung image is obtained, the location of the lung region in the lung image is determined, and the location region is used to Subsequent processing of the images is performed as lung images. The lung parenchyma segmentation can be obtained in an existing manner, for example, through a deep learning neural network, or through a lung parenchyma segmentation algorithm, which is not specifically limited in the present disclosure.

Step 102: Determine the lung images to be segmented in the lung images at multiple times, and use the lung images other than the images to be segmented as the first lung images.

In some possible implementations, the lung image at any time among the lung images at multiple times can be determined as the lung image to be segmented, or the input time information can also be received, and the lung image corresponding to the time information can be determined as the lung image to be segmented Lung image.

Alternatively, in the embodiment of the present disclosure, the method of determining the lung image to be segmented in the lung images at multiple times, and the lung image other than the image to be segmented as the first lung image may also include: calculating the lung images respectively The lung volume of each lung image in the lung images at the multiple time points is determined, and the lung image with the largest lung volume is determined as the lung image to be segmented.

That is to say, in the embodiment of the present disclosure, the lung image with the largest lung volume can be determined as the lung image to be segmented, so that the characteristics of the lung lobe can be more fully reflected, and the accuracy of lung lobe segmentation can be improved.

In an embodiment of the present disclosure, the method of determining the lung image to be segmented in the lung images at multiple times, and the lung image other than the image to be segmented as the first lung image further includes: when calculating the multi-time Before the lung volume in the lung image, respectively extract the left lung and the right lung of the multi-moment lung image, respectively calculate the first volume of the left lung and the second volume of the right lung of the multi-moment lung image , calculating lung volumes in the multi-moment lung images according to the first volume and the second volume, respectively. Specifically, the lung volume in the multi-moment lung image is the sum of the respective first volume of the left lung and the second volume of the right lung. Wherein, the extraction of the left and right lungs may be performed by using a lung parenchyma extraction algorithm or a neural network for lung parenchyma segmentation to obtain the left and right lung regions. The volumes of the left and right lungs can be calculated by using the sum of the areas of the left and right lungs extracted from each layer of the lung image respectively. Those skilled in the art can also use other calculation methods, which are not specifically limited in the present disclosure.

Step 103: Using at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image, the at least two first lung images include at least one lung image at a time before the image to be segmented and/or Or at least one lung image at a time after the image to be segmented.

In some possible implementations, at least one group of lung images at the time before the lung image to be segmented and/or at least one group of lung images at the time after the lung image to be segmented can be used to perform supplementary correction and fusion on the features of the image to be segmented, to obtain Fusion of lung images. Alternatively, in the embodiment of the present disclosure, all lung images other than the lung image to be segmented may be used as the first lung image, so that all feature information of the lung image extracted during breathing can be retained.

In an embodiment of the present disclosure, the method of using at least one first lung image to fuse the lung image to be segmented to obtain a fused lung image includes:

Executing a registration operation of the first lung image to the lung image to be segmented to obtain a lung image to be fused; fusing the lung image to be fused with the lung image to be segmented to obtain a fused lung image.

In some embodiments of the present invention, on the one hand, the registration operation is to find the difference between the lung image at the moment before and/or at the moment after the lung image to be segmented and the lung image to be segmented. Correspondingly, the matching process between the lung image to be segmented and the first lung image is completed, and on the other hand, image features of the first lung image at each moment can be further fused through a registration process. Wherein, a registration algorithm may be used to realize the registration between each first lung image and the lung image to be segmented, and register the first lung image to the lung image to be segmented. The registration algorithm can use the elastic registration algorithm or use the VGG network (VGG-net) in deep learning for registration, such as the paper Deformable image registration using convolutional neural networks or the U network (U-net) for registration, such as the paper PulmonaryCT Registration through Supervised Learning with Convolutional Neural Networks. The present invention does not limit the specific registration algorithm.

In other embodiments of the present disclosure, the method of performing the registration operation of the first lung image to the lung image to be segmented to obtain the lung image to be fused includes: from the at least two first lung images Extracting an image at the same position in the image and the image to be segmented to obtain a lung motion sequence image formed by the image extracted at the same position; respectively calculating the lung displacement of adjacent images in the lung motion sequence image, and according to the lung displacement A registration operation of the at least one first lung image to the lung image to be segmented is performed.

In the embodiment of the present disclosure, the same position can be represented as the same number of layers. As described in the above embodiment, each group of lung images can include multiple layers of images, and the lung images selected from the first lung image and each group of lung images to be segmented Images with the same number of layers constitute a group of lung motion sequence images. That is to say, the embodiments of the present disclosure can obtain the lung motion sequence images with the same number of layers as the number of layers, that is, the lung motion sequence images at each position.

In an embodiment of the present disclosure, the method of extracting images from the same position in the at least two first lung images and the image to be segmented, and obtaining a lung motion sequence image formed by the images extracted at the same position includes : Determining the number of layers of the lung images at multiple moments; determining the at least two first lung images and the lung images at the same position of the image to be segmented according to the layers; according to the lung images at the same position Obtain the lung motion sequence images.

In a specific embodiment of the present invention, when acquiring lung images at multiple moments in the respiratory process, the number of scanning layers, layer thicknesses, and interlayer distances of the lung images at multiple moments have been determined; therefore, according to the layers Determine the lung images at the same position of the multiple time lung images, select the lung images at the same position from the multiple time lung images to obtain the lung motion sequence image, for example, the lung image at the first time The position corresponding to the N layer is the same as the position corresponding to the Nth layer of the lung image at the second moment and the Mth moment, which is a lung plane, and the same lung plane at all times is combined to form the lung motion sequence image, and M is An integer greater than 1, used to represent the number of moments or groups, and N represents the value of any layer.

In the case of obtaining multiple lung motion sequence images, the image corresponding to the image to be segmented in the lung motion sequence may be determined, and the rest are images corresponding to the first lung image. The images in the lung motion sequence images are arranged in order of time. Since the image to be segmented has been determined in the foregoing embodiments, the time corresponding to the image to be segmented can also be obtained correspondingly, and the image corresponding to the image to be segmented can be determined in the lung motion sequence image according to this time point, as well as the first lung image corresponding image. Included in the lung motion sequence images is a layer of images in each lung image. For the convenience of description in subsequent embodiments, this layer of images is described as the image to be segmented or the first segmented image. However, it should be pointed out here that the lung The images in the motion sequence images are only images of the lung image to be segmented and corresponding layers in the first lung image.

In the embodiment of the present disclosure, in the case of obtaining the lung motion sequence images, the motion between the first lung image and the lung image to be segmented may be implemented. That is, lung displacements of adjacent images in the lung motion sequence images may be calculated respectively, and a registration operation of the at least one first lung image to the lung image to be segmented is performed according to the lung displacements. Wherein, by determining the lung displacement between adjacent images, the lung displacement between the first lung image and the lung image to be segmented can be determined, and then the registration operation of the first lung image and the lung image to be segmented is performed. The lung displacement may represent the displacement between the image to be segmented and the lung feature points in the lung image to be segmented.

In an embodiment of the present disclosure, the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images includes: respectively determining the first forward optical flow of adjacent images in the lung motion sequence images; A lung displacement of the adjacent image is determined based on the first forward optical flow.

In a specific embodiment of the present invention, optical flow (optical flow) can be used to represent changes between moving images, which refers to the movement speed of patterns in time-varying images. When the lungs are moving, the brightness pattern of its corresponding point on the image is also moving, so the optical flow can be used to represent the changes between images. Since it contains the information of the lung motion, it can be used by the observer to determine the of sports conditions. In the embodiment of the present disclosure, the optical flow evaluation is performed on each adjacent image in the lung motion sequence image, and the optical flow information between the adjacent images can be obtained. Wherein, it is assumed that the times corresponding to the lung images at multiple times are t1, t2, ..., tM, and M represents the number of groups. The Nth lung motion sequence image may respectively include Nth layer images F1N, F2N, .

When performing optical flow estimation, according to the forward sequence of groups 1 to M, the first forward optical flow of two adjacent images in each lung motion sequence image is respectively obtained, for example, the first forward optical flow of F1N to F2N can be obtained Optical flow, the first forward optical flow from F2N to F3N, and so on, to obtain the first forward optical flow from F(M-1) to FMN. Wherein, the first forward optical flow indicates that the moving speed information of each feature point in the adjacent lung image is arranged in the forward order of time. Specifically, the lung motion sequence images can be input into the optical flow estimation model to obtain the first forward optical flow between adjacent images. The optical flow estimation model can be FlowNet2.0, or other optical flow estimation models The flow estimation model, which is not specifically limited in the present disclosure. Alternatively, an optical flow estimation algorithm such as a sparse optical flow estimation algorithm or a dense optical flow estimation algorithm may be used to evaluate the optical flow of adjacent images, which is also not specifically limited in the present disclosure.

In a specific embodiment of the present invention, the method for determining the lung displacement of the adjacent image according to the first forward optical flow includes: using the velocity information of the first forward optical flow and the lung lobe motion sequence Temporal information of adjacent images in the image yields lung displacements of the adjacent images. The scan time and the number of slices are included in the dicom file in the lung image collected by CT, and the time information of adjacent images in the lung motion sequence images can be obtained approximately by dividing the scan time by the slice number.

In the embodiment of the present disclosure, each layer image in the acquired lung image can have corresponding acquisition time information, using the time difference of the acquisition time of two adjacent images in the lung motion sequence image and the first forward optical flow The product of , the lung displacement of two adjacent images within the time difference range can be obtained.

In addition, since the time information of adjacent images in the lung motion sequence images is small, in the embodiment of the present disclosure, the velocity information corresponding to the optical flow may also be approximately equal to the lung displacement.

Wherein, since the image to be segmented and the first lung image are determined in advance, the first forward optical flow of the first lung image and the image to be segmented in the lung motion sequence image and the first forward optical flow of the first lung image and the image to be segmented can be determined in sequence correspondingly. The time information between images, correspondingly, the lung displacement between the first lung image and the lung image to be segmented can be obtained by multiplying the first forward optical flow and the time information.

In an embodiment of the present disclosure, the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images further includes: respectively determining the first reverse optical flow corresponding to the first forward optical flow; The lung displacement of the adjacent image is determined according to the first reverse optical flow and/or the first reverse optical flow.

In an embodiment of the present disclosure, the first forward optical flow of adjacent images in the lung motion sequence images is determined according to the forward time sequence of the multi-time lung images, and the reverse optical flow of the multi-time lung images may be determined. In time order, determine the first reverse optical flow of adjacent images in the lung motion sequence images.

Correspondingly, when performing optical flow estimation, according to the reverse sequence from M to 1 group, the first reverse optical flow of two adjacent images in each lung motion sequence image is respectively obtained, for example, FMN to F(M -1) the first reverse optical flow of N, the first reverse optical flow from F(M-2)N to F(M-1)N, and so on, to obtain the first reverse optical flow from F2N to F1N. Wherein, the first reverse optical flow indicates that the moving speed information of each feature point in the adjacent lung image is arranged in reverse order of time. Similarly, lung motion sequence images can be input into the optical flow estimation model to obtain the first reverse optical flow between adjacent images, or sparse optical flow estimation algorithms, dense optical flow estimation algorithms, etc. The estimation algorithm performs optical flow evaluation on adjacent images, which is also not specifically limited in the present disclosure.

In a specific embodiment of the present disclosure, the method for determining the lung displacement of the adjacent image according to the first reverse optical flow includes: using the velocity information of the first reverse optical flow and the lung lobe motion sequence Temporal information of adjacent images in the image yields lung displacements of the adjacent images. The scan time and the number of slices are included in the dicom file in the lung image collected by CT, and the time information of adjacent images in the lung lobe motion sequence images can be obtained approximately by dividing the scan time by the slice number.

In the embodiment of the present disclosure, each image in the acquired lung images can have corresponding acquisition time information, using the time difference of the acquisition time of two adjacent images in the lung lobe motion sequence image and the first reverse optical flow The product of , the lung displacement of two adjacent images within the time difference range can be obtained. In addition, since the time information of adjacent images in the lung lobe motion sequence images is small, in the embodiment of the present disclosure, the velocity information corresponding to the optical flow may also be approximately equal to the lung lobe displacement.

Wherein, since the image to be segmented and the first lung image are pre-determined, the first reverse optical flow of the first lung image and the image to be segmented in the lung motion sequence image and the first lung image and the image to be segmented can be correspondingly determined in sequence. The time information between images, correspondingly, the lung displacement between the first lung image and the lung image to be segmented can be obtained by multiplying the first reverse optical flow and the time information.

In an embodiment of the present disclosure, the method for separately calculating lung displacements of adjacent images in the lung motion sequence images further includes: respectively performing optical flow on the first forward optical flow and the first reverse optical flow Optimizing processing to obtain a second forward optical flow corresponding to each of the first forward optical flows, and a second reverse optical flow corresponding to each of the first reverse optical flows; respectively according to the second forward optical flow The lung displacement of the adjacent image is determined towards the optical flow and/or the second reverse optical flow.

In a specific embodiment of the present invention, the method for determining the lung displacement of the adjacent image according to the first reverse optical flow and the first reverse optical flow respectively includes: Performing operations on the forward optical flow and the second reverse optical flow to obtain a corrected optical flow; respectively determining the lung displacement of the adjacent image according to the corrected optical flow.

In a specific embodiment of the present invention, calculating the second forward optical flow and the second reverse optical flow respectively to obtain a method for correcting optical flow includes: calculating the second forward optical flow and the second reverse optical flow An addition operation is performed on the second reverse optical flow to obtain a two-way optical flow sum, and then an average value of the two-way optical flow sum is calculated to obtain a corrected optical flow. That is, the average value of the second forward optical flow and the second reverse optical flow is calculated, and corrected optical flow=(second forward optical flow+second reverse optical flow)/2.

In a specific embodiment of the present invention, the optical flow optimization processing is performed on the first forward optical flow and the first reverse optical flow respectively, and the second forward optical flow corresponding to each of the first forward optical flows is obtained. To the optical flow, and the method for the second reverse optical flow corresponding to each of the first reverse optical flows, comprising: connecting each of the first forward optical flows to obtain a first connection optical flow, and connecting each of the first forward optical flows The first reverse optical flow obtains the second connection optical flow; perform N times of optical flow optimization processing on the first connection optical flow and the second connection optical flow respectively, and obtain the first optimized light corresponding to the first connection optical flow flow, and the second optimized optical flow corresponding to the second connection optical flow; according to the first optimized optical flow, the second forward optical flow corresponding to each first forward optical flow is obtained, and according to the second optimized optical flow flow to obtain the second reverse optical flow corresponding to each first reverse optical flow; wherein, N is a positive integer greater than or equal to 1.

Wherein, connecting each of the first forward optical flows to obtain a first connection optical flow, and connecting each of the first reverse optical flows to obtain a second connection optical flow includes: sequentially connecting the adjacent images in the lung motion sequence images The first forward optical flow among the lung motion sequence images is obtained to obtain the first connection optical flow corresponding to the group of lung motion sequence images, and the first reverse optical flow between adjacent images in the lung motion sequence images is sequentially connected to obtain the group of lung motion sequence images. The second connection optical flow corresponding to the motion sequence image. The connection here is splicing in the depth direction.

After the first connection optical flow and the second connection optical flow are obtained, an optical flow optimization process may be performed on the first connection optical flow and the second connection optical flow respectively, and the embodiment of the present disclosure may perform the optical flow optimization process at least once. For example, each optical flow optimization process in the embodiments of the present disclosure may be performed by using an optical flow optimization module, and the optical flow optimization module may be composed of a neural network, or may also use a corresponding optimization algorithm to perform optimization operations. Correspondingly, when performing N times of optical flow optimization processing, it may include N sequentially connected optical flow optimization network modules, the input of the latter optical flow optimization network module is the output of the previous optical flow optimization network module, and the last optical flow optimization network module The output of the optimized network module is the optimization result of the first connection optical flow and the second connection optical flow.

Specifically, when only one optical flow optimization network module is included, the optical flow optimization network module can be used to perform optimization processing on the first connection optical flow to obtain the first optimized optical flow corresponding to the first connection optical flow, and through the optical flow optimization The network module performs optimization processing on the second connection optical flow to obtain a second optimized optical flow corresponding to the second connection optical flow. The optical flow optimization processing may include residual processing and upsampling processing. That is, the optical flow optimization network module may further include a residual unit and an upsampling unit, and perform residual processing on the input first connected optical flow or the second connected optical flow through the residual unit, wherein the residual unit may include multiple volumes Convolution layer, the convolution kernel used by each convolution layer is not specifically limited in the embodiment of the present disclosure, the scale of the first connection optical flow after residual processing by the residual unit becomes smaller, such as reduced to the input connection optical flow A quarter of the scale, which is not specifically limited in the present disclosure, and can be set according to requirements. After the residual processing is performed, upsampling processing can be performed on the first connected optical flow or the second connected optical flow after the residual processing, and the scale of the output first optimized sub-optical flow can be adjusted to the first The scale of the connected optical flow, and the scale of the output second optimized sub-optical flow is adjusted to the scale of the second connected optical flow. And through the optical flow optimization process, the characteristics of each optical flow can be integrated, and the accuracy of the optical flow can be improved at the same time.

In some other embodiments, the optical flow optimization module may also include multiple optical flow optimization network modules, such as N optical flow optimization network modules. Wherein the first optical flow optimization network module can receive the first connection optical flow and the second connection optical flow, and respectively perform the first optical flow optimization process on the first connection optical flow and the second connection optical flow, the first time Optical flow optimization processing includes residual processing and upsampling processing, and the specific process is the same as that of the above-mentioned embodiment, and will not be repeated here. The first optimized sub-optical flow of the first connected optical flow and the first optimized sub-optical flow of the second connected optical flow can be obtained through the first optical flow optimization process.

Similarly, each optical flow optimization network module can be used to perform an optical flow optimization process, that is, the i+1th optical flow optimization network module can be used to perform the first connection optical flow and the second connection optical flow The i optimized sub-optical flow executes the i+1th optical flow optimization process to obtain the i+1th optimized sub-optical flow corresponding to the first connected optical flow, and the i+1th optimized sub-optical flow corresponding to the second connected optical flow stream, where i is a positive integer greater than 1 and less than N. Finally, the Nth optimized sub-optical flow of the first connected optical flow and the Nth optimized sub-optical flow of the second connected optical flow can be obtained through the Nth optimization process performed by the Nth optical flow optimization network module, and the obtained The Nth optimized sub-optical flow of the first connected optical flow is determined as the first optimized optical flow, and the obtained Nth optimized sub-optical flow of the second connected optical flow is determined as the second optimized light flow. In the embodiment of the present disclosure, the optical flow optimization process performed by each optical flow optimization network module may be residual processing and upsampling processing. That is, each optical flow optimization network module can be the same optical flow optimization module.

In the case of obtaining the first optimized optical flow and the second optimized optical flow of each lung motion sequence image, the first optimized optical flow can be used to obtain the second forward optical flow corresponding to each first forward optical flow, and A second reverse optical flow corresponding to each first reverse optical flow is obtained according to the second optimized optical flow.

After N times of optical flow optimization processing, the scale of the first optimized optical flow is the same as the scale of the first connected optical flow, and the first optimized optical flow can be split into M-1 second forwards according to the depth direction Optical flow, the M-1 second forward optical flows correspond to the optimization results of the first forward optical flows. Similarly, after N times of optical flow optimization processing, the scale of the second optimized optical flow is the same as the scale of the second connected optical flow, and the second optimized optical flow can be split into M-1 first Two reverse optical flows, the M-1 second reverse optical flows correspond to the optimization results of the first reverse optical flows.

Through the above-mentioned embodiment, the optimized second forward optical flow between the first forward optical flow between adjacent images of the lung motion sequence images and the second forward optical flow between adjacent images in the lung motion sequence images can be obtained. A second reverse optical flow optimized by reverse optical flow.

In the case of obtaining the second forward optical flow and/or the second reverse optical flow, the second forward optical flow and/or the second reverse optical flow can be used to determine the motion displacement of the lung lobe corresponding to the adjacent image, and then The lung displacement between the lung segmentation image and the first lung image can be obtained. For details, refer to the above-mentioned manner of determining the motion displacement by the first forward optical flow and/or the first reverse optical flow, which will not be repeated here.

Based on the above, the embodiment of the present disclosure can obtain the motion displacement (displacement of lung lobe) of each layer image in the lung image in each time range, and in the case of performing key point detection on each layer image of the lung image, the matching key point can be obtained The motion trajectory in each time range, so that the motion state and motion trajectory of the entire lung in each time range can be obtained.

Through the above embodiment, the lung displacement between the first lung image and the lung image to be segmented can be obtained, and then the registration operation of the first lung image to the lung image to be segmented can be performed according to the lung displacement. Wherein, the lung displacement in the embodiment of the present disclosure may include the displacement value between any pixel in the first lung image and the lung image to be segmented, and the first lung image and the lung displacement can be obtained by adding the first lung image A registration result corresponding to a lung image, that is, the lung image to be fused. Then, the embodiment of the present disclosure can obtain the lung image to be fused corresponding to the registration operation of each first lung image and the lung image to be segmented.

In the case of obtaining the lung image to be fused, the fused lung image can be obtained directly through the addition process between the lung image to be fused and the lung image to be segmented, or different weights can be set for the lung image to be fused, using the setting weights to get the fused lung image.

In an embodiment of the present disclosure, the method of fusing the lung image to be fused and the lung image to be segmented to obtain a fused lung image includes: determining a weight value of the lung image to be fused; value and the lung image to be fused to obtain a weighted lung image; and to fuse the weighted lung image and the lung image to be segmented to obtain the fused lung image.

In some embodiments of the present disclosure, corresponding weight values may be pre-configured for the lung images to be fused. The weight value of each lung image to be fused may be the same or different, for example, the weight value of the lung image to be fused may be 1/k, where k is the number of lung images to be fused. Or the configured weight value can also be determined according to the image quality of the lung image to be fused, for example, the image quality score of each lung image to be fused can be determined by Single Stimulus Continuous Quality Evaluation (SSCQE), and the The score is normalized to the range [0,1] to obtain the weight value of each lung image to be fused. Alternatively, the input lung image to be fused can also be evaluated by an image quality assessment model NIMA (Neural Image Assessment) to obtain corresponding weight values.

Or, in other embodiments of the present disclosure, the method for determining the weight value of the lung image to be fused includes: determining registration points of the lung image to be fused, and points other than registration points are non-registration points, The weight value of the registration point is greater than that of the non-registration point. That is to say, in the embodiment of the present disclosure, the weight value of each pixel in the lung image to be fused can be different, and the weight value of the registration point can be set to be greater than the weight value of the non-registration point, so as to highlight the feature information of the registration point, where , the registration point is the feature point that highlights the lung feature. In a specific embodiment of the present invention, determining the registration point of the lung image to be fused may detect the lung image to be fused and the fused lung image through SIFT (Scale-invariant feature transform). Keypoints (registration points). The weight of the obtained key points can be a value greater than 0.5, and the weight value of non-registered points can be 1-a, or any other positive value smaller than a.

Alternatively, the setting of the weight value may also be realized through a neural network of an attention mechanism. The neural network of the attention mechanism may include at least one layer of convolutional layer, and an attention mechanism module (attention module) connected to the convolutional layer, through which the convolutional layer performs convolution processing on the image to be fused to obtain convolutional features, and The convolution feature is input into the attention module to obtain the attention feature map corresponding to each image to be fused. The attention feature map includes the attention value corresponding to each pixel in the image to be fused. The attention value can be used as the corresponding The weight value of the pixel point, where the pixel point with the attention value greater than 0.5 is the registration point. Those skilled in the art can select an appropriate method to obtain the weight value of the lung image to be fused according to requirements, which is not specifically limited in the present disclosure. In a specific embodiment of the present invention, determining the registration point of the lung image to be fused may detect key points of the lung image to be fused and the fused lung image through SIFT (Scale-invariant feature transform) (registration point)

When the weight of the lung image to be fused is obtained, the weighted lung image can be obtained by multiplying the weight by the lung image to be fused.

In a specific embodiment of the present invention, the method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image includes: adding the weighted lung image to the lung image to be segmented image to obtain the fused lung image.

In a specific embodiment of the present invention, the method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image can also be implemented by using the following embodiments, including: using the first The preset neural network is used to fuse the lung image to be fused with the lung image to be segmented to obtain a fused lung image, comprising: connecting the weighted lung image and the lung image to be segmented to obtain a connected lung image; Perform at least one layer of convolution processing on the connected lung image to obtain a fusion feature of the connected lung image, and the image corresponding to the fusion feature is a fused lung image. The first preset neural network is a network that has been trained in advance and can realize the fusion and extraction of lung feature information, such as residual network, Unet, feature pyramid network, etc., which is not specifically limited in the present disclosure.

Step 104: Using a preset lung lobe segmentation model to segment the fused lung image to obtain a lung lobe image of the lung image to be segmented.

In the embodiment of the present disclosure, the preset lung lobe segmentation model can be a traditional machine learning lung lobe segmentation model, or a progressive dense V-network (PDV-NET) lung lobe segmentation model proposed by Voxel Technology in deep learning in 2018. In the present invention, the fused lung image is the lung image at a moment before and/or at a moment after the lung image to be segmented and all information of the lung image to be segmented, thus ensuring the information of the lung image to be segmented for better segmentation of lung lobes.

Alternatively, in the embodiments of the present disclosure, the preset lung lobe segmentation model may also be implemented by a neural network, which may include at least one of residual networks Resnet, Unet, and Vnet, which is not specifically limited in the present disclosure. The preset lung lobe segmentation model in the embodiment of the present disclosure can be used to realize the segmentation detection of at least one type of lung lobe, and the obtained segmentation result includes the position information of the detected lung lobe, for example, the detected lung lobe can be represented by a preset mask in the lung The location area in the image.

In an embodiment of the present disclosure, the lung lobe segmentation method further includes: there are at least two preset lung lobe segmentation models, and the features of the lung lobe segmentation images obtained by the preset lung lobe segmentation models are fused to obtain fusion features, and the The fusion features are classified and processed to obtain the final lung lobe image. In an embodiment of the present disclosure, the method of fusing features of the lung lobe segmentation image obtained by the preset lung lobe segmentation model to obtain fusion features includes:

Splicing the lung lobe segmentation images obtained by the preset lung lobe segmentation model respectively to obtain splicing features, and inputting the splicing features into a second preset neural network for convolution operation to obtain the fusion features.

Wherein, the two preset segmentation models may be different segmentation models. For example, the first preset segmentation model can be Resnet, and the second preset segmentation model can be Unet, but it is not a specific limitation of the present disclosure. Any two different neural networks that can be used for lung lobe segmentation can be used as the preset lung lobe Split the model. The fused lung image is input to the first preset segmentation model and the second preset segmentation model to obtain the first segmentation result and the second segmentation result respectively. Wherein, the first segmentation result and the second segmentation result may respectively include position information of the detected lung lobe region. Since there may be differences in the segmentation results obtained by performing segmentation processing through different preset segmentation models, the embodiment of the present disclosure can further improve the segmentation accuracy by combining the two segmentation results. Wherein, the position information of the first segmentation result and the second segmentation result may be averaged to obtain the final lung lobe segmentation result.

Alternatively, in some implementations, the first feature map output by the convolutional layer before the first segmentation result is output by the first preset segmentation model and the convolutional layer output before the second segmentation result is output by the second preset segmentation model The second feature map is fused to obtain the fusion feature. Wherein, the first preset segmentation model and the second preset segmentation model may include corresponding feature extraction modules and classification modules respectively, wherein the classification module obtains the final first segmentation result and the second segmentation result, and the feature extraction module may include multiple In the convolutional layer, the feature map output by the last convolutional layer is input to the classification module to obtain the first segmentation result or the second segmentation result. The embodiment of the present disclosure can obtain the first feature map output by the last convolutional layer of the feature extraction module in the first preset segmentation model and the output of the last convolutional layer of the feature extraction module in the second preset segmentation model. The second feature map. The first feature map and the second image of the lung lobe segmentation image obtained by the preset lung lobe segmentation model are fused to obtain a fusion feature, and the fusion feature is classified to obtain a final lung lobe image. Specifically, the first feature map and the second feature map can be concatenated to obtain concatenated features, and the concatenated features can be input to at least one convolutional layer to obtain fusion features. Then, the fusion feature is classified and processed through the classification network to obtain the classification (segmentation) result of the lung lobe to be detected, that is, the segmentation result of the lung lobe corresponding to the lung image to be detected is obtained.

In the embodiments of the present disclosure, static lung data during lung breathing are usually used for clinical analysis, without taking into account the motion information of the lungs, which will definitely affect the analysis accuracy of the lung characteristic data. If the correlation between lung features in different time periods can be combined, the detection accuracy of lung motion data will be improved. The embodiments of the present disclosure can solve the current problem of insufficient feature information of the lung (lobe), resulting in poor segmentation effect.

The present invention also provides a lung lobe segmentation device, comprising: an acquisition unit, configured to acquire lung images at multiple moments in the breathing process; a determination unit, configured to determine a lung image to be segmented at a certain moment in the lung images at multiple moments; The fusion unit is used to fuse the lung image to be segmented by using the lung image at a time before and/or at a time after the lung image to be segmented to obtain a fused lung image; the segmentation unit is used to The fused lung image is segmented using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented. For its specific implementation, refer to the detailed description in a lung lobe segmentation method.

In addition, the present invention provides a storage medium. When the computer program instructions are executed by a processor, the above method is implemented, including: acquiring lung images at multiple moments in the breathing process; determining the lung to be segmented in the lung images at multiple moments images, lung images other than the image to be segmented are used as the first lung image; at least one first lung image is used to fuse the lung image to be segmented to obtain a fused lung image, and the at least two first lung images include at least A lung image at a time before the image to be segmented and/or at least one lung image at a time after the image to be segmented; using a preset lung lobe segmentation model to segment the fused lung image to obtain the lung to be segmented Image of the lung lobes.

The present disclosure can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the present disclosure are implemented by executing computer readable program instructions.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions.

Having described various embodiments of the present disclosure above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

The above-mentioned embodiments are only to express the implementation of the present invention, and the descriptions thereof are more specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be noted that those skilled in the art can make several modifications, equivalent replacements, improvements, etc. without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (27)

1. A lung lobe segmentation method, characterized in that, comprising: Obtain lung images at multiple moments during breathing; determining the lung images to be segmented in the lung images at multiple times, and the lung images other than the lung images to be segmented as the first lung images; Use at least one or at least two first lung images to fuse the lung image to be segmented to obtain a fused lung image; wherein, the at least one or at least two first lung images include at least one of the lung images to be segmented a lung image at a previous moment and/or at least one lung image at a later moment than the lung image to be segmented; The fused lung image is segmented using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented. 2. The method according to claim 1, wherein the method of using the at least one or at least two first lung images to fuse the lung images to be segmented to obtain a fused lung image comprises: performing a registration operation of the at least one or at least two first lung images to the lung image to be segmented to obtain a lung image to be fused; The lung image to be fused and the lung image to be segmented are fused to obtain a fused lung image. 3. The method according to any one of claims 1-2, wherein the method for determining the lung image to be segmented in the multi-moment lung image comprises: The lung volumes of the lung images in the multi-moment lung images are respectively calculated, and the lung image with the largest lung volume is determined as the lung image to be segmented. 4. The method according to claim 2, wherein the method of merging the lung image to be fused and the lung image to be segmented to obtain a fused lung image comprises: Determining the weight value of the lung image to be fused; Obtaining a weighted lung image according to the weight value and the lung image to be fused; The weighted lung image is fused with the lung image to be segmented to obtain the fused lung image. 5. The method according to claim 3, wherein the method for determining the lung image to be segmented in the multi-moment lung image further comprises: Before calculating the lung volumes in the lung images at multiple times, respectively extracting the left lung and the right lung of the lung images at multiple times; respectively calculating the first volume of the left lung in the lung images at multiple times and the second volume of the right lung; calculating the lung volume in the multi-moment lung image according to the first volume and the second volume respectively. 6. The method according to claim 4, wherein the method for determining the weight value of the lung image to be fused comprises: determining the registration point of the lung image to be fused, and determining the weight of the registration point The value is greater than the weight value of non-registration points, and the feature points other than the registration points are non-registration points. 7. The method according to claim 4 or 6, wherein the method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image comprises: Perform summing processing on the lung image and the lung image to be segmented to obtain the fused lung image; or, The method of fusing the weighted lung image and the lung image to be segmented to obtain the fused lung image includes: using a first preset neural network to process the lung image to be fused and the lung image to be segmented Fusion is performed to obtain a fusion lung image. 8. The method according to any one of claims 2 or 4 or 6, wherein the registration operation of the at least one or at least two first lung images to the lung images to be segmented is performed to obtain The method to be fused lung image comprises: Extract images from the same position in the at least one or at least two first lung images and the lung image to be segmented, and obtain a lung motion sequence image formed by the images extracted at the same position; The lung displacements of adjacent images in the lung motion sequence images are respectively calculated, and a registration operation of the at least one or at least two first lung images to the lung image to be segmented is performed according to the lung displacements. 9. The method according to claim 7, wherein the method of performing the registration operation of the at least one or at least two first lung images to the lung image to be segmented to obtain the lung image to be fused, include: Extract images from the same position in the at least one or at least two first lung images and the lung image to be segmented, and obtain a lung motion sequence image formed by the images extracted at the same position; The lung displacements of adjacent images in the lung motion sequence images are respectively calculated, and a registration operation of the at least one or at least two first lung images to the lung image to be segmented is performed according to the lung displacements. 10. The method according to any one of claims 2, 4-6, and 9, wherein the lung lobe segmentation method further comprises: the preset lung lobe segmentation models are at least two, and the preset The features of the lung lobe segmentation image obtained by the lung lobe segmentation model are fused to obtain fusion features, and the fusion features are classified to obtain a final lung lobe image. 11. The method according to claim 3, wherein the lung lobe segmentation method further comprises: the preset lung lobe segmentation model is at least 2, and the lung lobe segmentation image obtained by the preset lung lobe segmentation model is The features are fused to obtain fused features, and the fused features are classified to obtain a final lung lobe image. 12. The method according to claim 7, wherein the lung lobe segmentation method further comprises: the preset lung lobe segmentation model is at least 2, and the lung lobe segmentation image obtained by the preset lung lobe segmentation model is The features are fused to obtain fused features, and the fused features are classified to obtain a final lung lobe image. 13. The method according to claim 8, wherein the lung lobe segmentation method further comprises: the preset lung lobe segmentation model is at least 2, and the lung lobe segmentation image obtained by the preset lung lobe segmentation model is The features are fused to obtain fused features, and the fused features are classified to obtain a final lung lobe image. 14. The method according to claim 8, wherein said extracting images from the same position in said at least one or at least two first lung images and said lung image to be segmented obtains said same position extraction A method of image forming a lung motion sequence image comprising: determining the number of layers of the multi-moment lung image; determining the at least one or at least two first lung images and the lung images at the same position as the lung image to be segmented according to the number of layers; The lung motion sequence image is obtained according to the lung image at the same position. 15. The method according to claim 9, wherein said extracting images from the same position in said at least one or at least two first lung images and said lung image to be segmented obtains said same position extraction A method of image forming a lung motion sequence image comprising: determining the number of layers of the multi-moment lung image; determining the at least one or at least two first lung images and the lung images at the same position as the lung image to be segmented according to the number of layers; The lung motion sequence image is obtained according to the lung image at the same position. 16. The method according to claim 8, wherein the method for calculating the lung displacements of adjacent images in the lung motion sequence images respectively comprises: respectively determining the first forward optical flow of adjacent images in the lung motion sequence images; The lung displacements of the adjacent images are respectively determined according to the first forward optical flow. 17. The method according to claim 9 or 14, wherein the method for calculating the lung displacements of adjacent images in the lung motion sequence images respectively comprises: respectively determining the first forward optical flow of adjacent images in the lung motion sequence images; The lung displacements of the adjacent images are respectively determined according to the first forward optical flow. 18. The method according to claim 15, wherein the method for calculating the lung displacements of adjacent images in the lung motion sequence images respectively comprises: respectively determining the first forward optical flow of adjacent images in the lung motion sequence images; The lung displacements of the adjacent images are respectively determined according to the first forward optical flow. 19. The method according to claim 10, wherein the method of merging the features of the lung lobe segmentation image obtained by the preset lung lobe segmentation model to obtain fusion features comprises: Splicing the lung lobe segmentation images obtained by the preset lung lobe segmentation model respectively to obtain splicing features, and inputting the splicing features into a second preset neural network for convolution operation to obtain the fusion features. 20. The method according to any one of claims 11-13, wherein the method of fusing features of the lung lobe segmentation image obtained by the preset lung lobe segmentation model to obtain fusion features comprises: Splicing the lung lobe segmentation images obtained by the preset lung lobe segmentation model respectively to obtain splicing features, and inputting the splicing features into a second preset neural network for convolution operation to obtain the fusion features. 21. The method according to claim 16 or 18, wherein the method for calculating the lung displacements of adjacent images in the lung motion sequence images respectively, further comprises: respectively determining the first reverse optical flow corresponding to the first forward optical flow; The lung displacement of the adjacent image is determined according to the first forward optical flow and the first reverse optical flow respectively. 22. The method according to claim 17, wherein the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images further comprises: respectively determining the first reverse optical flow corresponding to the first forward optical flow; The lung displacement of the adjacent image is determined according to the first forward optical flow and the first reverse optical flow respectively. 23. The method according to claim 21, wherein the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images further comprises: respectively calculating the first forward optical flow and the second forward optical flow A reverse optical flow performs optical flow optimization processing to obtain a second forward optical flow corresponding to each of the first forward optical flows, and a second reverse optical flow corresponding to each of the first reverse optical flows ; Determine the lung displacement of the adjacent image according to the second forward optical flow and the second reverse optical flow respectively. 24. The method according to claim 22, wherein the method for separately calculating the lung displacements of adjacent images in the lung motion sequence images further comprises: respectively calculating the first forward optical flow and the second forward optical flow A reverse optical flow performs optical flow optimization processing to obtain a second forward optical flow corresponding to each of the first forward optical flows, and a second reverse optical flow corresponding to each of the first reverse optical flows ; Determine the lung displacement of the adjacent image according to the second forward optical flow and the second reverse optical flow respectively. 25. The method according to claim 23 or 24, wherein the method of determining the lung displacement of the adjacent image according to the second forward optical flow and the second reverse optical flow respectively, include: respectively performing operations on the second forward optical flow and the second reverse optical flow to obtain a corrected optical flow; The lung displacements of the adjacent images are respectively determined according to the corrected optical flow. 26. A lung lobe segmentation device, comprising: An acquisition unit, configured to acquire lung images at multiple moments in the breathing process; A determining unit, configured to determine a lung image to be segmented among the lung images at multiple times, and a lung image other than the lung image to be segmented is used as a first lung image; A fusion unit, configured to use at least one or at least two first lung images to fuse the lung image to be segmented to obtain a fused lung image; wherein the at least one or at least two first lung images include at least one a lung image at a time before the lung image to be segmented and/or at least one lung image at a time after the lung image to be segmented; A segmentation unit, configured to segment the fused lung image by using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented. 27. A storage medium, characterized in that, when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 25 is implemented, comprising: Obtain lung images at multiple moments during breathing; determining the lung images to be segmented in the lung images at multiple times, and the lung images other than the lung images to be segmented as the first lung images; Use at least one or at least two first lung images to fuse the lung image to be segmented to obtain a fused lung image; wherein, the at least one or at least two first lung images include at least one of the lung images to be segmented a lung image at a previous moment and/or at least one lung image at a later moment than the lung image to be segmented; The fused lung image is segmented using a preset lung lobe segmentation model to obtain a lung lobe image of the lung image to be segmented.

Images