JP2021086263A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To provide a technology for accurately extracting information on a shape of an area of interest from a medical image.SOLUTION: An image processing apparatus for estimating shape information, which is information on a shape of an area of interest, from an image includes: image acquisition means which acquires a first image to be processed; partial space information acquisition means which acquires two or more pieces of partial space information generated with different datasets; partial space information selection means which selects first partial space information from the two or more pieces of partial space information, on the basis of pixel value information of the first image; and estimation means which estimates shape information of an area of interest in the first image from the pixel value information of the first image, by use of the first partial space information. The dataset includes sample data including pixel value information of a learning image and the shape information of the area of interest.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus, an image processing method, and a program.

In the medical field, diagnosis is performed using images acquired by various image imaging devices (modality) such as an ultrasonic diagnostic imaging device. In this diagnosis, information such as the area, volume, and dimensions of the region of interest reflected in the image is used for the diagnosis, but in order to calculate the area of the region, the contour of the region is used from the image. It is necessary to extract (estimate the contour information which is the information representing the contour shape). However, when the work of extracting this area is performed manually, it is a problem to impose a great deal of labor on the worker. For this reason, various techniques have been proposed for automatic or semi-automatic area extraction techniques from images in order to reduce the labor of workers.

As an example, as in the following prior art, there is a method of using an image and the contour information of the correct answer of the region of interest shown in the image collected for a large number of cases as learning data. In this technique, the contour information of the region of interest reflected in the unknown input image is estimated (that is, the contour is extracted) based on the result of statistical analysis of the training data. Non-Patent Document 1 describes a technique for constructing a statistical model called an Active Appearance Model based on information on pixel values of an image of training data and information on coordinate values of a point cloud representing the outline of a region of interest in the image. Is disclosed. Further, using the evaluation value indicating the similarity between the pixel value information of the unknown input image and the pixel value information of the image in the statistical model, the region of interest reflected in the input image is subjected to iterative processing by the gradient method. A technique for estimating contour information is also disclosed.

Elco Ost, et. al. "Active Appearance Models in Medical Image Processing" Medical Imaging Technology. vol. 27. No3.2009 May.

In the estimation process using the statistical model, a plausible estimation result can be obtained when an image close to the sample data group included in the training data is input, but rarely when an image away from the sample data group is input. Inappropriate estimation results may be output. In general, when generating a statistical model, it is desirable to have as many variations of training data as possible. This is because it is expected that as the variation of the training data increases, the generalization ability of the statistical model will increase and it will be possible to handle more unknown data. However, as the variation of the training data is increased and the generalization ability of the statistical model is increased, the above-mentioned inappropriate estimation result may be output.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of extracting information on the shape of a region of interest from a medical image with higher accuracy.

It should be noted that not only the above-mentioned purpose but also the action and effect derived by each configuration shown in the embodiment for carrying out the invention described later, and the action and effect which cannot be obtained by the conventional technique can be obtained. It can be positioned as one of the other purposes.

The first aspect of the present invention is an image processing device that estimates shape information that is information about the shape of a region of interest from an image, and uses a data set different from that of an image acquisition means that acquires a first image to be processed. A first portion from the information of the two or more subspaces based on the subspace information acquisition means for acquiring the information of the two or more subspaces generated by the method and the pixel value information of the first image. Using the subspace information selection means for selecting spatial information and the information in the first subspace, the shape information of the region of interest in the first image is estimated from the pixel value information of the first image. Provided is an image processing apparatus comprising an estimation means, wherein the data set includes sample data including pixel value information of an image for learning and shape information of a region of interest.

A second aspect of the present invention is an image processing method for estimating shape information which is information about the shape of a region of interest from an image, using a step of acquiring a first image to be processed and a different data set. Based on the step of acquiring the generated information of the two or more subspaces and the pixel value information of the first image, the information of the first subspace is selected from the information of the two or more subspaces. The data set includes a step and a step of estimating the shape information of the region of interest in the first image from the pixel value information of the first image using the information of the first subspace. Provides an image processing method characterized in that it includes sample data including pixel value information of an image for learning and shape information of a region of interest.

The third aspect of the present invention is a program for causing the computer to function as each means of the image processing apparatus according to the first aspect, or causing the computer to execute each step of the image processing method according to the second aspect. To provide, or a computer-readable storage medium in which such a program is stored non-temporarily.

According to the present invention, information regarding the shape of the region of interest can be extracted more accurately from the medical image.

The figure which shows the structure of the function of an image processing apparatus. The flowchart which shows the example of the processing procedure of an image processing apparatus. The figure which shows the example of the ultrasonic image of the heart. The figure which shows the example of the point group which shows the outline of the region of interest in the ultrasonic image of a heart.

Hereinafter, embodiments of the present invention will be described in detail exemplarily with reference to the drawings. However, the components described in this embodiment are merely examples, and the technical scope of the present invention is determined by the scope of claims and is not limited by the following individual embodiments. Absent.

The image processing apparatus according to the embodiment of the present invention provides a function of estimating information regarding the shape of a region of interest (ROI) from an input image. The input image to be processed is a medical image, that is, an image of a subject (human body, etc.) taken or generated for the purpose of medical diagnosis, examination, research, etc., and is typically an imaging image called a modality. This is an image acquired by the system. For example, an ultrasonic image obtained by an ultrasonic diagnostic apparatus, an X-ray CT image obtained by an X-ray CT apparatus, an MRI image obtained by an MRI apparatus, and the like can be processed. The input image may be a two-dimensional image or a three-dimensional image.
Further, an image of one time phase or an image of a plurality of time phases may be used. The region of interest is a portion of the image, such as an anatomical structure (organs, blood vessels, bones, etc.) or a lesion. What is selected as the area of interest can be set arbitrarily. The information regarding the shape of the region of interest (also referred to as "shape information" in the present specification) is information representing the spatial (or geometric) feature of the region of interest in the image. For example, contour information of the region of interest (information representing the contour shape), positions of feature points on the region of interest, relative positions between feature points (intervals, etc.), length / area / volume of the region of interest, shape of the region of interest (shape of the region of interest) A circle, an ellipse, a triangle, etc.) can be exemplified. Not only the primary information obtained directly from the image, but also the secondary information generated by using the primary information (for example, the difference in the position of the feature point, the area ratio between the two time phases, etc.) is also the shape information. include.

The image processing apparatus according to the embodiment of the present invention acquires information on two or more subspaces generated by using different data sets, and from the information on the two or more subspaces, a subspace suitable for an input image. It has one of the features in that the shape information is estimated by selecting the information of. When the training data is divided into multiple data sets, the expressive power and generalization ability of the statistical model (subspace) may be inferior to those when statistical analysis is performed on the entire training data, but it is an inappropriate estimation. The risk of output results is reduced. As the learning data, it is preferable to use data in which each sample data is composed of pixel value information of the image for learning and shape information (correct answer data) of the region of interest in the image for learning. By using such learning data, it is possible to obtain a model in which the correlation between the pixel value information of the image and the shape information of the region of interest is well reflected, and it is possible to improve the estimation accuracy.

In the following, a specific example of the image processing apparatus according to the embodiment of the present invention will be described in detail by taking as an example a case where the region of the left atrium of the heart is extracted from a two-dimensional ultrasonic image.

The image processing apparatus according to the present embodiment has a function of automatically extracting contour information of the left atrium, which is a region of interest, from an input image. The image processing apparatus of the present embodiment selects an appropriate subspace from two or more subspaces (statistical models) constructed from the training data according to the input image to be processed, and uses the selected subspace. The shape information of the region of interest of the input image is estimated.

Hereinafter, the configuration and processing of the image processing apparatus of the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of an image processing system (also referred to as a medical image processing system) including the image processing apparatus of the present embodiment. The image processing system includes an image processing device 10 and a database 22. The image processing device 10 is communicably connected to the database 22 via the network 21. The network 21 includes, for example, a LAN (Local Area Network) and a WAN (Wide Area Network).

The database 22 holds and manages a plurality of images. The image managed in the database 22 includes an image in which the contour information of the region of interest is unknown and an image in which the contour information of the region of interest is known. The former image is subjected to contour extraction processing by the image processing device 10. The latter image is associated with contour information (correct answer data) and is used as learning data. It is desirable that these images are spatially normalized to some extent with respect to the size of the image and the region of interest appearing in the image. Spatial normalization is an operation of aligning the size (dimension per pixel) and position in an image with a certain standard. The information managed in the database 22 may include information on the subspace used in the contour extraction process (for example, information on the estimation matrix). The subspace information may be stored in the internal storage (ROM 32 or storage unit 34) of the image processing device 10 instead of the database 22. The image processing device 10 can acquire the data held in the database 22 via the network 21.

The image processing device 10 includes a communication IF (Interface) 31 (communication unit), a ROM (Read Only Memory) 32, and a RAM (Random Access Memory).
ry) 33, a storage unit 34, an operation unit 35, a display unit 36, and a control unit 37 are provided.

The communication IF 31 (communication unit) is composed of a LAN card or the like, and realizes communication between an external device (for example, a database 22 or the like) and the image processing device 10. The ROM 32 is composed of a non-volatile memory or the like, and stores various programs and various data. The RAM 33 is composed of a volatile memory or the like, and is used as a work memory for temporarily storing a program or data being executed. The storage unit 34 is composed of an HDD (Hard Disk Drive) or the like, and stores various programs and various data. The operation unit 35 is composed of a keyboard, a mouse, a touch panel, and the like, and inputs instructions from a user (for example, a doctor or an examination engineer) to various devices.

The display unit 36 is composed of a display or the like, and displays various information to the user. The control unit 37 is composed of a CPU (Central Processing Unit) or the like, and controls the processing in the image processing device 10 in an integrated manner. The control unit 37 includes an image acquisition unit 51, a subspace information acquisition unit 52, a contour information estimation unit 53, and a display processing unit 54 as its functional configuration. The control unit 37 may include a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), and the like.

The image acquisition unit 51 acquires a first image (an image whose contour information is unknown) to be processed from the database 22. That is, the image acquisition unit 51 corresponds to an example of an image acquisition means for acquiring a first image. The first image is an image of a subject acquired by various modality. The image acquisition unit 51 may acquire the first image directly from the modality. In this case, the image processing device 10 may be mounted in the console of the modality (imaging system). In the present embodiment, an example in which the first image is a two-dimensional ultrasonic image will be described, but other types of images may be used. The method of the present embodiment may be a two-dimensional or higher-dimensional image (a plurality of two-dimensional images, a two-dimensional moving image, a three-dimensional still image, a plurality of three-dimensional images, a three-dimensional moving image, or the like). Applicable. Moreover, it can be applied regardless of the type of modality.

The subspace information acquisition unit 52 acquires learning data from the database 22, and performs statistical analysis two or more times using the learning data. Each of the sample data constituting the learning data is data having a configuration including the pixel value information of the image for learning and the shape information of the region of interest in the image for learning (details will be described later). Then, the subspace information acquisition unit 52 calculates the subspace information (for example, the base information constituting the subspace) representing the distribution of the sample data group from the result of the statistical analysis. At this time, the subspace information acquisition unit 52 divides the learning data into two or more data sets, and generates information on two or more subspaces using different data sets. That is, the subspace information acquisition unit 52 corresponds to an example of the subspace information acquisition means for acquiring the information of two or more subspaces generated by using different data sets of the training data.

The subspace information selection unit 55 is based on the first image acquired by the image acquisition unit 51 and two or more subspace information acquired by the subspace information acquisition unit 52, and the outline of the region of interest from the first image. Select a first subspace information suitable for estimating the information. That is, the subspace information selection unit 55 corresponds to an example of the subspace information selection means for selecting the first subspace information from the information of two or more subspaces based on the pixel value information of the first image. ..

The contour information estimation unit 53 performs an operation using the first subspace information selected by the subspace information selection unit 55 to obtain the contour of the left atrium in the first image from the first image acquired by the image acquisition unit 51. Estimate the information. That is, the contour information estimation unit 53 is an example of an estimation means that estimates the shape information of the region of interest in the first image from the pixel value information of the first image by using the first subspace information.

The display processing unit 54 is in the image display area of the display unit 36 in a display form so that the contour information of the region of interest estimated as the first image can be easily visually recognized based on the result calculated by the contour information estimation unit 53. To display.

Each component of the image processing apparatus 10 functions according to a computer program. For example, the function of each component is realized by the control unit 37 (CPU) reading and executing the computer program stored in the ROM 32 or the storage unit 34 or the like with the RAM 33 as the work area. Note that some or all the functions of the components of the image processing device 10 may be realized by using a dedicated circuit. Further, some functions of the components of the control unit 37 may be realized by using a cloud computer. For example, an arithmetic unit located at a location different from the image processing apparatus 10 is communicably connected to the image processing apparatus 10 via the network 21, and the image processing apparatus 10 and the arithmetic unit transmit and receive data, whereby the image processing apparatus The function of the component of 10 or the control unit 37 may be realized.

Next, an example of processing of the image processing apparatus 10 of FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing an example of a processing procedure of the image processing apparatus 10. In this embodiment, an example will be described in which the left atrium region of the heart is the region of interest. However, this embodiment can also be applied to other sites including the left ventricle, the right ventricle, and the right atrium, or a region in which a plurality of these regions are combined as a region of interest.

(Step S101: Image acquisition / display)
In step S101, when the user instructs the acquisition of an image via the operation unit 35, the image acquisition unit 51 acquires the first image specified by the user from the database 22 and stores it in the RAM 33. At this time, the display processing unit 54 may display the first image in the image display area of the display unit 36. Here, an example of the first image is shown in FIG. FIG. 3 shows an example in which the first image is an image of the apex of the heart in an ultrasonic image of the heart. In the following description, it is assumed that the number of pixels in the x direction constituting the first image is Nx and the number of pixels in the y direction is Ny. That is, the total number of pixels constituting the first image is Nx × Ny.

(Step S102: Acquisition of two or more subspace information)
In step S102, the subspatial information acquisition unit 52 stores the pixel value information (for example, image data) of the image and the contour information of the correct answer of the region of interest in the image for a plurality of images different from the first image in the database 22. Get from. That is, the subspace information acquisition unit 52 acquires the learning data used for calculating the subspace information. It is desirable that the plurality of images used as training data are composed of images taken of different patients in order to enhance the robustness of the statistical model. However, images of the same patient taken at different times may be included. Alternatively, data augmentation (inflated learning data) may be performed by deforming or processing the image.

Subsequently, the subspace information acquisition unit 52 generates data in which the pixel value information and the contour information of the region of interest are concatenated for each of the plurality of images acquired as learning data. Here, the pixel value information of the image and the contour information of the region of interest will be described in detail with reference to FIGS. 3 to 4.

As shown in FIG. 3, the image handled in this embodiment is an image composed of Nx × Ny pixels. The pixel value information of the image in the present embodiment refers to a column vector in which the pixel values of each pixel are arranged in the order of raster scan of the image. That is, assuming that the upper left of the image shown in FIG. 3 is the origin (0,0) and the pixel value at the location of the pixel (x, y) is represented by I (x, y), the pixel value information a of the image is as follows. It is defined as.

Next, FIG. 4 shows an example in which when the region of interest is the left atrium region, the contour information of the region is represented by a point cloud (p1 to p9) of 9 points. In the present embodiment, each of these points is also represented by (x, y) using the coordinate values of the x-axis and the y-axis with the upper left of the image as the origin (0,0). Taking point p1 as an example, the coordinates of point p1 are (x1, y1). The contour information of the region of interest in the present embodiment refers to a column vector in which the x-coordinate values and the y-coordinate values of a plurality of feature points arranged on the contour of the region of interest are arranged. That is, in the example shown in FIG. 4, the contour information b representing the left atrium is defined as follows. Even when the "shape information" is the position of the feature point on the region of interest, define b in a format in which the coordinate values of the feature points are arranged in the same manner as above, such as b = {x1, y1}. Just do it. When the "shape information" is the length (length), area (area), or volume (volume) of the region of interest, each of them is based on those values such as b = definition, b = area, and b = volume. b may be defined.

After acquiring the pixel value information a and the contour information b, the subspace information acquisition unit 52 further connects these column vectors to generate one column vector in which the two pieces of information are concatenated. That is, the subspace information acquisition unit 52 generates a column vector composed of an element corresponding to the pixel value information a and an element corresponding to the contour information b (shape information) of the region of interest. In the examples shown in FIGS. 3 to 4, the following data c is generated.

Here, since the size of the variance differs between the pixel value information of the image and the contour information of the region of interest, weighting may be performed on at least one of the data. At this time, weighting may be performed so that the total variances are equal or have a predetermined balance according to the magnitude of the total variances of the pixel value information and the contour information of the learning data. Alternatively, the data may be weighted using the weight set by the user via the operation unit 35.

The subspace information acquisition unit 52 generates data c in which pixel value information and contour information are concatenated in the above flow for each of the acquired plurality of images (plurality of learning sample data). Then, the subspace information acquisition unit 52 divides the generated data group into two or more data sets, and then performs statistical analysis on each data set. For the statistical analysis, a known method such as principal component analysis (PCA, Principal Component Analysis) may be used, or a method such as Kernel PCA or Weighted PCA may be used. By performing the principal component analysis, it is possible to calculate the average vector, the eigenvector, and the eigenvalues corresponding to each eigenvector for the data c in which the pixel value information and the contour information are integrated. That is, by using the average vector (c bar) of the data c in which the pixel value information and the contour information are concatenated, the eigenvector e, and the coefficient g for each eigenvector e, the point d on the subspace is expressed by the following equation. it can. This point d represents one learning sample data. The coefficient g is the principal component score (principal component score).

Here, L represents the number of eigenvectors used in the calculation, and the specific value thereof may be determined based on the cumulative contribution rate that can be calculated from the eigenvalues. For example, the cumulative contribution rate of 95% may be set in advance as a threshold value, the number of eigenvectors having a cumulative contribution rate of 95% or more may be calculated, and the value may be set as L. Alternatively, the number of eigenvectors set by the user may be set as L.

As the final process of step S102, the subspace information acquisition unit 52 acquires the subspace information corresponding to each data set from the result of the statistical analysis for each data set and stores it in the RAM 33. The subspace information is information that defines a subspace, and includes, for example, information on a plurality of eigenvectors and an average vector that constitute the subspace. As described above, in the present embodiment, two or more subspace information having different bases is acquired based on the learning data.

The subspace information calculation process in this step is data processing independent of the contour extraction (contour estimation) of the first image. Therefore, the calculation process of the subspace information may be performed in advance, and the calculated subspace information may be stored in the storage device (for example, the database 22 or the storage unit 34). In this case, in step S102, the subspace information acquisition unit 52 may read two or more subspace information from the storage device and store it in the RAM 33. By calculating the subspace information in advance, there is an effect that the processing time when performing the estimation processing of the region of interest in the first image can be shortened. The subspace information calculation process may be performed by another device different from the image processing device 10.

Further, in the present embodiment, an example in which data in which the pixel values of the entire image are arranged in the order of raster scan is used as the pixel value information, but other feature amounts related to the pixel values (feature amounts related to the texture of the image, etc.) are used as pixels. It may be used as value information. Further, the data in which the pixel values of the partial image representing a part of the image are arranged may be used as the pixel value information, and the data in which only the pixel values around the outline of the region of interest of the image are arranged is used as the pixel value information. It doesn't matter.

Alternatively, data obtained by arranging the principal component scores obtained by projecting an image onto a subspace obtained by principal component analysis of a plurality of images for learning may be used as pixel value information of the image. .. For example, after calculating the pixel value information a for each of a plurality of images for training, principal component analysis is performed on the data group of the pixel value information a, and the vector of the principal component score of each image is used as a new pixel of the image. It may be used instead of a as the value information a'. When the total number of a plurality of images used as training data is smaller than the number of pixels constituting the image, the number of dimensions of the pixel value information a'is smaller than the number of dimensions of the pixel value information a. Therefore, using the pixel value information a'has an effect of reducing the calculation cost of statistical analysis for the data in which the pixel value information and the contour information are concatenated. Further, the calculation cost may be further reduced by setting a threshold value for the cumulative contribution rate and reducing the number of dimensions of the principal component score (that is, the number of eigenvectors).

On the other hand, as the contour information of the region of interest, the coordinate values of the point cloud representing the contour of the region of interest are used, but other values may be used. In one example, information is obtained by arranging the results of calculating the level set function representing the region of interest (for example, the signed distance value from the contour with the inside of the region of interest as negative and the outside as positive) for each pixel of the image in the order of raster scan. It may be used as contour information. Alternatively, a label image or a mask image that distinguishes between the region of interest and the rest may be used as contour information. Further, similarly to the method of calculating the pixel value information a'from the pixel value information a described above, the value calculated by using the principal component analysis may be used as the new contour information b'for the contour information b. .. That is, the calculation is performed by performing the principal component analysis on the data group of only the contour information b corresponding to a plurality of images for learning and setting the vector of the principal component score of each image as the new contour information b'. Further cost reduction is possible.

Here, a specific example of a method of grouping the training data into two or more data sets will be described. For example, the image acquired from the database 22 and the contour information of the region of interest in the image may be subjected to preset image deformation, and then grouped for each image deformation. That is, two or more data sets may be created by performing data augmentation (inflating of learning data) with respect to the image data relating to the plurality of images and the point cloud representing the outline of the region of interest. More specifically, by generating data obtained by performing multiple operations such as translation, rotation, and enlargement / reduction, and combining data having the same or similar deformation parameters into the same set, two or more data. You may create a set. Alternatively, the grouping may be performed based on the patient information (for example, height, weight, or medical history) of the subject shown in the image. Alternatively, the time phase is different (diastole and systole), the cross-sectional direction is different (four-chamber image and two-chamber image), the examination type and imaging protocol are different, the modality model is different, the hospital is different, and the photographer is different. The training data may be grouped according to the above. Further, the grouping may be performed by combining these a plurality of conditions. In addition to this, grouping may be performed using any classification that affects the shape and image quality of the region of interest.

Here, the effect of dividing the learning data into two or more sets, generating two or more subspace information, and selecting one subspace from them will be described. As shown in equation (4), in the result obtained from the principal component analysis, the subspace is expressed by the linear sum of the mean vector and the eigenvector. Here, considering the principal component analysis of the pixel values in each pixel of the image, in the above-mentioned subspace, an image (an image close to the training data) that is plausible as image data can be obtained in the peripheral portion of the learning data of the analysis. it is conceivable that. However, coordinates away from the training data in the subspace may result in an unnatural image when returned to the original image space by the operation of back projection from the subspace. This is a problem that arises because the relationship between the pixel values of different images is not linear and is modeled by using a linear model. If such a point exists in the subspace, the output may be inappropriate when the estimation process is performed using the subspace.

In general, it is desirable that the learning data to be statistically analyzed has as many variations as possible, such as the shape of the region of interest and where the region of interest appears in the image. This is because as the variation of this data increases, the expressive power of the subspace obtained from the statistical analysis increases, and it becomes possible to deal with more unknown data. However, there is a problem that the higher the expressive power of the subspace, the higher the risk that the subspace contains the above-mentioned unnatural image. In the present embodiment, in view of the above problems, the training data is divided into a plurality of data sets in which relatively close sample data are collected, instead of generating one subspace using all the sample data of the training data. Above, we take the method of generating subspaces for each of those datasets. That is, according to this method, a subspace specialized for a learning sample data group having a certain feature can be generated according to the data classification method. By dividing the training data into multiple data sets, the expressive power of the subspace for one set is reduced compared to the case where one subspace is generated as a whole, but it is unnatural in the subspace. It can be expected to have the effect of reducing the risk of including various images. In the method of the present embodiment, when an unknown input image is given, it is important to select which subspace information to use for estimation. Therefore, in the next step S103, the estimation process is performed. You have selected the appropriate subspace information for this.

(Step S103: Selection of subspace information)
In step S103, the subspace information selection unit 55 is one portion based on the first image acquired by the image acquisition unit 51 and two or more subspace information acquired by the subspace information acquisition unit 52. Spatial information is selected and the result is stored in RAM 33. That is, as subspace information (information on the first subspace) that is more suitable for estimating the contour information of the region of interest in the first image, among two or more subspace information based on the first image. Select one subspace information from.

The process of selecting suitable subspatial information for estimating the contour information of the region of interest from two or more subspatial information includes the pixel value information of the first image and the pixels of the sample data included in each data set. This can be achieved by using the similarity with the value information. That is, the subspatial information selection unit 55 identifies a data set including sample data similar to the first image based on the pixel value information of the image, and the subspatial information generated from the specified data set. You just have to select.

Here, as a specific method of identifying one data set most suitable for the first image from two or more data sets, a method of evaluating the difference between the first image and the image of each sample is used. Can be mentioned. The difference between the two images may be evaluated by, for example, the sum of squared differences (SSD) of the pixel values. For example, assuming that the sample data having the smallest sum of differences squared is the sample data most similar to the first image (called similar sample data), the data set including the similar sample data is used for estimation processing. Identify as the optimal data set. By using the subspace information generated from the data set including the sample data most similar to the first image in this way, it is expected that the accuracy of the estimation process in the subsequent step S104 will be improved. However, in this method, when the total number of sample data is enormous, there is a problem that the calculation cost becomes enormous.

Therefore, in the present embodiment, the subspatial information selection unit 55 performs principal component analysis on the pixel value information of the sample data of each data set, and then the result of the principal component analysis and the pixels of the first image. Value information is used to identify the optimal dataset by the subspace method. According to this method, the calculation cost of the above-mentioned specific processing can be reduced.

The procedure for processing by the subspace method is as follows. First, a subspace is generated by performing principal component analysis on pixel value information for each of two or more datasets. Subsequently, the reconstruction error is calculated by performing the projection / backprojection processing of the first image on the subspace for each of the subspaces. That is, after projecting the pixel value information of the first image into the subspace, the difference squared sum of the pixel values (that is, the sum of the difference squares of the pixel values) between the image obtained from the result of back-projection to the original image space and the first image. Reconstruction error) is calculated. The dataset corresponding to this subspace with the smallest reconstruction error provides a dataset containing sample data similar to the first image, in other words, the subspace that can best represent the first image. It is a data set. By such a method, the data set corresponding to the subspace information most suitable for the processing of the subsequent step S104 can be specified.

Although an example of using the difference squared sum of pixel values to evaluate the similarity between images is shown here, other indexes such as the difference absolute value sum (SAD: Sum of Absolute Difference) may be used. ..

Further, the configuration is not limited to selecting one optimum data set from a plurality of data sets, and may be a configuration in which all data sets satisfying a predetermined condition are selected. For example, all the data sets in which the reconstruction error of the pixel value information is smaller than a predetermined threshold value may be selected. Alternatively, a predetermined number of high-order data sets with a small reconstruction error may be selected. When a plurality of data sets are selected, in step S104, the contour information estimation unit 53 may execute the contour information estimation process using the subspace information of each data set and integrate the results. For example, the average value or the median value of the contour information estimated using each subspace information may be acquired.

(Step S104: Estimating the contour information of the region of interest)
In step S104, the contour information estimation unit 53 estimates the contour information of the region of interest appearing in the first image from the first image acquired in step S101 and the first subspace information selected in step S103. .. The estimation result is stored in the RAM 33.

The method of estimating the contour information does not matter. For example, in Non-Patent Document 1, iterative processing by the gradient method is performed by using an evaluation value indicating consistency between the pixel value information of an unknown input image and the pixel value information of an image in subspatial information (statistical model). A technique for estimating the contour information of the region of interest reflected in the input image based on the above is disclosed. The contour information estimation unit 53 may estimate the contour information by using such an iterative process by the gradient method.

However, when iterative processing is used when estimating contour information, it is necessary to repeat the iterative processing until the change before and after the iterative processing becomes sufficiently small or a certain predetermined number of times is reached for the predefined energy function. , It costs a certain amount of calculation cost. Further, since the number of iterations (timing at which the processing converges) changes depending on the input data, there is also a problem that it is difficult to grasp the calculation time required for the processing in advance due to the variation in the total calculation time.

Therefore, in the present embodiment, when estimating the contour information of the region of interest, by applying a method that does not use iterative processing, it is possible to realize the estimation processing of the contour information with a lower calculation cost.

Here, the problem of estimating the contour information representing a predetermined region of interest (left atrium) when an unknown input image (ultrasound image) is given is summarized. The first subspace information selected in step S103 above is information related to the subspace in which the pixel value information of the ultrasonic image and the information of the point cloud representing the outline of the left atrium reflected in the image are associated with each other. Then, when handling an unknown input image, an ultrasonic image is given, so that the pixel value information a corresponding to the image can be calculated in the same procedure as in step S102. That is, the proposition of this step is to estimate the contour information b corresponding to the input image based on the first subspatial information calculated from the learning data and the pixel value information a related to the input image.

In Non-Patent Document 2 below, information on the pixel value of an unknown image is described by using information on the pixel value of a plurality of images and subspatial information on data obtained by concatenating information representing the posture of an object in the image. Discloses a technique for estimating information representing the posture of an object in the image. In other words, this technique is a technique for interpolating the data of the defective portion from the result of statistically analyzing the learning data without the defect for the data in which a certain defect occurs. This technique is called the Back projection for lost pixels (BPLP) method.

[Non-Patent Document 2] Toshiyuki Amano, et. al. "An application based fast liner pose application" MVA 2009 IAPR Convention on Machine Vision Applications. 2009 May 20-22.

In order to apply the BPLP method, it is necessary to specify at the time of estimation which part of the input data is known information and which part is unknown information (missing part). In the present embodiment, the pixel value information of the first image, which is the input image, is set as known information, and the contour information of the region of interest in the first image is set as unknown information. That is, the problem setting is replaced so that the pixel value information part of the image in Non-Patent Document 2 is the pixel value information a and the information part representing the posture of the object is the contour information b, based on the following method. Solve the proposition of this step.

The contour information estimation unit 53 includes elements corresponding to the pixel value information a of the first image, which is known information, and elements corresponding to the contour information b of the region of interest in the first image, which is unknown information. The vector f including the above is obtained by the following equation. Since the calculation based on the following formula does not include the iterative calculation, it is possible to realize the estimation process of the contour information at a lower calculation cost than the conventional gradient method. Further, since the factors that influence the variation of the calculation time related to this calculation are the number of dimensions of the input pixel value information and the contour information of the estimation target, if the size of the input image is determined, the processing time will be increased. It can be expected that the variation will be small.

Here, f'on the right side represents input data in which a certain defect occurs, and in the present embodiment, as shown in the equation (3), contour information which is unknown information among the elements of the vector f. It is a vector in which the element corresponding to is 0.

E is a matrix representing the subspace defined by the first subspace information selected in step S103. When L eigenvectors are e1, e2, ..., EL, for example, the matrix E is given by E = [e1, e2, ..., EL]. Σ is a square matrix in which the diagonal element corresponding to the pixel value information, which is known information, is 1, and the other elements are 0. In other words, Σ is a matrix in which the element corresponding to the contour information, which is unknown information, is set to 0 in the unit matrix. In the present embodiment, Σ is a square matrix having a side of Nx × Ny + 9 × 2 (the number of dimensions of pixel value information a + the number of dimensions of contour information b), and the last 18 diagonal elements are It is a matrix that is 0 and the other diagonal elements are 1.

In the present embodiment, even if the input image is arbitrary, the portion set as unknown information does not change (always becomes the portion of contour information b). Therefore, at the time of calculation of the partial space information at step S102, it can be computed E a ^{(E T ΣE) -1 E T} ( part of the operation performed by the estimation process) results in advance. Therefore, to calculate the in advance to E ^{(E T? En)} -1 result of ^{E T,} memory device and the calculation result as a partial space information (e.g., a database 22 or the storage unit 34) may be stored in. When the E (E ^T ΣE) ^-1 result of E ^T is saved in the process of step S102, the partial space information acquisition unit 52, by reading the result output the calculated as the partial spatial information, estimated The cost related to the calculation for processing can be reduced. Incidentally, the E ^{(E T? En)} -1 part of the ^{E T} (e.g. ^{(E T? En)} portion of ^-1) storage device the calculation result as a partial space information (e.g., a database 22 or the storage unit 34) The operation of the remaining portion may be performed by the subspace information acquisition unit 52 or the contour information estimation unit 53.

Finally, the contour information estimation unit 53 extracts the information of the portion corresponding to the contour information (in the present embodiment, the last 18 elements) from the estimation result f and stores it in the RAM 33.

When the pixel value information a'is used instead of the pixel value information a in step S102, the pixel value information a'is the portion of I (0,0) to I (Nx, Ny) in the equation (6). It is necessary to replace it with the value calculated by the same method as the calculation method of. More specifically, when the pixel value information a'is the principal component score related to the pixel value information of the training image, the input image, the information of the subspace constructed using only the pixel value information of the training data, and the information of the subspace. Based on the above, the projection process may be performed to calculate the principal component score of the input image. Further, when the estimation process for the value such as the contour information b'is performed as the target of the estimation processing, the contour information of the estimation result is not the xy coordinates on the contour line, so the estimated value is changed to the coordinate value. Is converted and stored in the RAM 33. More specifically, when the contour information b'is the principal component score related to the contour information of the training data, the reverse is made based on the principal component score and the subspatial information constructed using only the contour information of the training data. The projection process may be performed.

The estimated contour information of the region of interest may be used as it is as the contour estimation result, or may be used as a rough extraction result to be given as an initial value to a more precise extraction process. In the latter case, the contour information estimation unit 53 uses the contour information obtained in the above process as an initial value and performs precise extraction of the contour by using a known method for precisely extracting the contour of the object.

(Step S105: Display an image based on the contour information estimation result)
In step S105, the display processing unit 54 displays the first image as an input image and the contour information of the region of interest estimated by the contour information estimation unit 53 in the image display area. At this time, the estimated contour information and the first image may be superimposed and displayed. By superimposing the display, it is possible to easily visually recognize how much the estimated contour information matches the first image. In the present embodiment, since the contour information of the region of interest is a discrete point cloud obtained by sampling the contour of the region, it is displayed after interpolating between adjacent points using a known technique such as spline interpolation. May be good.

When the purpose is to analyze or measure the region of interest, the process of step S105 is not always necessary, and the configuration may be such that only the estimated contour information is saved.

Further, in the above description, the case where the image is a two-dimensional image has been described as an example, but even if the image is a three-dimensional image (volume image), the same processing can be performed.

In this embodiment, an example is shown in which the coordinate values of the point group representing the left atrium are used as the contour information of the region of interest, but in addition to the left atrium, there are two such as the left ventricle, the right ventricle, and the right atrium. The contour information may be a combination of the coordinate values of the points representing the above area. In this case, by performing statistical analysis not only on the coordinate values of the point cloud representing the left atrium region but also on the coordinate values of all the point clouds in the two or more regions in step S102, two or more of them are performed in step S104. The coordinate values of all the point clouds in the area of can be estimated at the same time.

According to the present embodiment, a more accurate contour extraction result is provided to the user by selecting the optimum subspace from a plurality of subspaces according to the first image and estimating the contour information of the region of interest. There is an effect that can be done. Further, according to the present embodiment, by estimating the contour information which is the shape information of the region of interest by the matrix calculation using the matrix E representing the subspace constructed from the learning data, the contour extraction result with high accuracy can be obtained more accurately. There is also an effect that it can be provided to the user at a low calculation cost.

Although the present embodiment has been described above, the present invention is not limited to these, and can be modified or modified within the scope of the claims.

(Modification example 1)
In the above embodiment, an example is shown in which the image to be treated as the input image or the training data is a two-dimensional ultrasonic image, but these images are images in which the temporal data of the two-dimensional images are connected (when three-dimensional). It may be a spatial image). That is, a moving image of a two-dimensional ultrasonic image (two-dimensional ultrasonic image of two or more frames) may be treated as an input image. At this time, a three-dimensional spatiotemporal image may be constructed by using a set of images of a specific frame (temporal phase) instead of a set of images of a plurality of frames that are continuous in time.

Taking a moving image of an ultrasonic image of the heart as an example, an image of a frame of diastole (end diastole: ED (End-Diastole)) and a frame of systole (end-systole: ES (End-System)) It is preferable to use a three-dimensional image composed of images of two time phases of the image. In this case, the method of the above embodiment may be implemented based on the pixel value information of the images of the two frames of the diastole and the systole and the contour information corresponding to each of the images. By performing the same processing as in the above embodiment, it is possible to simultaneously estimate the contour information corresponding to all the frames of the three-dimensional image for the input of the unknown three-dimensional image. Any known method may be used to specify the frame of the diastole or systole of the heart. For example, the user may manually identify the frame from the moving image, or the frame may be automatically specified from the information such as the electrocardiogram corresponding to the moving image.

When a three-dimensional image is composed of a plurality of frames as described above, in order to carry out the method of the first embodiment, the pixel values of all the pixel values of the images of all the frames to be the input images are arranged. You can set it in the information. That is, for example, when a three-dimensional image is composed of images in the diastole period (di) and the systole period (sys), the pixel value information a may be set as follows.

Here, I _di (x, y) represents the pixel value of the image in the diastole period, and I _sys (x, y) represents the pixel value of the image in the systole period. That is, when the input image is an image composed of a set of images of a plurality of frames (time phase), the data obtained by concatenating the pixel value information of the images of all frames may be used as the pixel value information of the input image. ..

Further, regarding the contour information, by using the data obtained by concatenating the information of the point cloud representing the contour information of the attention region corresponding to each of the images of all frames as the contour information of the attention region corresponding to the input image, the above-described embodiment The method can be implemented. That is, the contour information b may be set as follows. Again, those with the di subscript are the coordinates of the points that represent the contour information of the region of interest in the diastolic image, and those with the subscript of sys are the points that represent the contour information of the region of interest in the systolic image. The coordinates of.

At this time, in step S101, the image acquisition unit 51 acquires an image pair of diastole and systole to be processed as the first image. Further, in step S102, the subspace information acquisition unit 52 performs the process described in the first embodiment with respect to a plurality of image pairs of diastole and systole different from the first image. That is, statistical analysis is performed on the data c in which the pixel value information a (formula (8)) of the image pair and the contour information b (formula (9)) of the correct answer of the region of interest in the image pair are concatenated to perform subspace information. To get. Then, in step S103, the contour information estimation unit 53 estimates the contour information of the region of interest that appears in the first image. By performing the same processing as in the above embodiment after setting the pixel value information and the contour information in this way, all the frames of the three-dimensional image can be input for the input of the unknown three-dimensional image. The contour information to be used can be estimated at the same time.

As described in the above embodiment, the principal component analysis is performed on the data of only the pixel value information of the image, the principal component score corresponding to each image is calculated, and the principal component score is used as a new pixel of each image. It may be used as value information. When the pixel value information is composed of the values in pixel units as in the equation (8), the number of dimensions of the data becomes enormous as the number of frames of the three-dimensional image increases, which leads to an increase in calculation cost. Therefore, the calculation cost can be significantly reduced by using the principal component score having a small number of dimensions for the pixel value information. Similarly, for the contour information, the principal component score calculated by using the principal component analysis on the coordinate values of the points representing the region of interest may be used as new contour information.

In this way, to extract images of two or more frames such as diastole and systole from a moving image to form a three-dimensional image, and to estimate the contour information of the images of those two or more frames at the same time. , There are the following two merits. These effects are not only obtained by combining images in diastole and systole, but are obtained by using a combination of multiple images in different time phases.

The first merit is the effect of stabilizing the estimation accuracy of the region of interest. That is, the robustness of the estimation process is improved by using a plurality of frames. In the moving image of the ultrasonic diagnostic image, the image quality (contrast and the degree of noise) of each frame often varies. That is, even if the outline of the region of interest is clearly drawn for a certain frame in the moving image and the visibility is high, the visibility of the outline may be low for another frame. Therefore, when one frame is extracted from a moving image and the area estimation process is performed, a frame with low image quality may be processed, but the area extraction process is performed on the image with low image quality. Then, the extraction accuracy may decrease. In such a case, in order to improve the region extraction accuracy of an image having low image quality, a method of using an image having high image quality that correlates with the image can be considered. As a premise, it is necessary that the input 3D image contains frames with high image quality, but by performing the area extraction process based on multiple frames, the average extraction accuracy is improved and extraction is performed. The effect of reducing the variation in accuracy can be expected. That is, improvement in robustness of estimation processing can be expected.

The second merit is that the area estimation process can be performed in consideration of the relationship between the positions and sizes of the areas of interest between different frames. Taking an image that captures the movement of the heart as an example, there is a correlation between the diastolic image and the systolic image regarding the size of the atria and ventricles and where their anatomical structure appears in the image. Can be imagined. However, if the region of interest is estimated independently for different frames, the processing is performed without considering the relationship between the frames, so the relationship between the frames, such as the magnitude relationship of the region of the estimation result, is lost. There is a possibility that it will be damaged. For example, even if the area of interest in diastole should be larger than the area of interest in systole, the estimated result of diastole may be smaller. .. By extending the above embodiment to handle images of a plurality of frames, a subspace considering the relationship of the region of interest between the different frames can be obtained. Therefore, the estimation result between the different frames is the original between the frames. The output result is such that the relationship is maintained.

In this modified example as well, since the contour information of the region of interest is estimated by matrix calculation using a matrix representing the subspace constructed from the learning data, a highly accurate contour extraction result is provided to the user at a lower calculation cost. it can. Further, the combination constituting the three-dimensional image may be a combination of a past image and a present image as well as a combination of different frames (time phases) in a single inspection. Further, not only a combination of different times but also a combination of two-dimensional images in a plurality of cross-sectional directions (for example, a four-cavity image and a two-cavity image, a long-axis image and a short-axis image) may be used. Further, a combination of a plurality of frames and cross-sectional images in a plurality of directions may be used.

(Modification 2)
In the above embodiment and the first modification, the pixel value information of the input image is set as known information, the contour information of the region of interest corresponding to the input image is set as unknown information, and the unknown information is estimated by the BPLP method. An example is shown. However, the known information and the unknown information may be set differently.

For example, in the above embodiment, when at least a part of the contour information of the region of interest is known in advance in addition to the pixel value information of the input image, the contour information is also set as known information, and other than that. The contour information may be set as unknown information. For example, the annulus position of the region of interest may be detected by using a method such as extraction of known feature points, and the position may be set as known information. Further, the user may manually set the annulus position on the image via the operation unit 35. For example, assuming that the point p1 in the contour information b is the annulus position, (x1, y1) can be given as known information. That is, in step S103, the contour information estimation unit 53 gives the acquired coordinates of the annulus position to the element corresponding to x1 and y1 of the equation (6), and the diagonal element corresponding to x1 and y1 of the equation (7). After changing the value of to 1, the coordinates of p2 to p9 are estimated by the equation (5).

Further, in the first modification, the pixel value information of the three-dimensional image is set as known information, the contour information of the region of interest corresponding to each frame of the three-dimensional image is set as unknown information, and the unknown information is obtained by the BPLP method. An example to estimate is shown. Even in this case, if at least a part of the contour information of the region of interest in a certain frame is known, the contour information may be set as known information. For example, when a three-dimensional image composed of images of diastole and systole is input, and the contour information in the diastole is known in advance, or when the contour information is specified by the user, the contour information is specified. Only the contour information of the systole may be set as unknown information. By increasing the known information in this way, the information that can be used in the estimation process increases, so that the accuracy of the estimation process of the unknown information can be improved.

In another example, the pixel value information of the image is obtained by the BPLP method based on the contour information input by the user via the operation unit 35, the contour information generated by using the subspace information of the contour information of the region of interest, and the like. It may be estimated by. In this case, if the contour information is set as known information and the pixel value information of the image is set as unknown information, the estimation process can be performed by the same procedure as in the above embodiment.

Also in this modified example, since the unknown information is estimated by the matrix operation using the matrix representing the subspace constructed from the learning data, the effect of providing the estimation result to the user at a low calculation cost is maintained.

Although the above is an example of the embodiment, the present invention is not limited to the embodiment shown in the above and the drawings, and can be appropriately modified and implemented without changing the gist thereof.

<Other Embodiments>
Further, the disclosed technology can take an embodiment as a system, an apparatus, a method, a program, a recording medium (storage medium), or the like. Specifically, it may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to a device composed of one device. good.

Needless to say, the object of the present invention is achieved by doing the following. That is, a recording medium (or storage medium) in which a software program code (computer program) that realizes the functions of the above-described embodiment is recorded is supplied to the system or device. Needless to say, the storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the function of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

10 Image processing device 51 Image acquisition unit 52 Subspace information acquisition unit 53 Contour information estimation unit 54 Display processing unit 55 Subspace information selection unit

Claims (16)

An image processing device that estimates shape information, which is information about the shape of a region of interest, from an image.
An image acquisition means for acquiring the first image to be processed, and
A subspace information acquisition means for acquiring information on two or more subspaces generated using different data sets, and a subspace information acquisition means.
A subspace information selection means for selecting the information of the first subspace from the information of the two or more subspaces based on the pixel value information of the first image.
It has an estimation means for estimating the shape information of the region of interest in the first image from the pixel value information of the first image by using the information of the first subspace.
The data set is an image processing apparatus including sample data including pixel value information of an image for learning and shape information of a region of interest. The image processing apparatus according to claim 1, wherein the information in the subspace represents the distribution of sample data included in the data set. In the subspatial information selection means, sample data similar to the first image is obtained based on the similarity between the pixel value information of the first image and the pixel value information of the sample data included in each data set. The image processing apparatus according to claim 1 or 2, wherein the included data set is specified, and the information of the subspace generated from the specified data set is selected as the information of the first subspace. .. The subspace information selection means and each of the first image and the first image based on the reconstruction error when the pixel value information of the first image is projected and back-projected to the subspace related to the pixel value information of each data set. The image processing apparatus according to claim 3, wherein the similarity with a data set is evaluated. The item according to any one of claims 1 to 3, wherein the information of the subspace is information obtained by performing statistical analysis on a plurality of sample data included in the data set. Image processing device. The statistical analysis is a principal component analysis.
The image processing apparatus according to claim 5, wherein the subspace information includes information on an average vector obtained by principal component analysis and information on a plurality of eigenvectors. Any one of claims 1 to 6, wherein the subspace information acquisition means acquires information on the two or more subspaces from a storage device that stores information on the subspaces generated in advance. The image processing apparatus according to the section. The image processing apparatus according to any one of claims 1 to 7, wherein the first image is an image composed of a plurality of sets of images having different time phases. The estimation means uses data obtained by concatenating pixel value information of a plurality of images of different time phases as pixel value information of the first image, and concatenates shape information corresponding to each of the plurality of images of different time phases. The image processing apparatus according to claim 8, wherein the obtained data is used as shape information corresponding to the first image. The first image is an image of the heart.
The image processing apparatus according to any one of claims 1 to 9, wherein the region of interest is an atrium or a ventricle. The image processing apparatus according to claim 10, wherein the first image is an image composed of a set of an image of a diastole of a heart and an image of a systole. The image processing apparatus according to any one of claims 1 to 11, wherein the pixel value information is data in which pixel values of an image are arranged. The pixel value information is characterized by being data obtained by arranging the principal component scores of the image obtained by projecting the image onto the subspace obtained by performing the principal component analysis of a plurality of images for learning. The image processing apparatus according to any one of claims 1 to 12. The image processing apparatus according to any one of claims 1 to 13, wherein the shape information is contour information of the region of interest. An image processing method that estimates shape information, which is information about the shape of a region of interest, from an image.
The step of acquiring the first image to be processed and
Steps to get information on two or more subspaces generated using different datasets,
A step of selecting the information of the first subspace from the information of the two or more subspaces based on the pixel value information of the first image, and
It has a step of estimating the shape information of the region of interest in the first image from the pixel value information of the first image using the information of the first subspace.
An image processing method characterized in that the data set includes sample data composed of pixel value information of an image for learning and shape information of a region of interest. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14.

Images