JP7560273B2 - Medical image diagnostic device and medical image processing device - Google Patents

Description

Embodiments of the present invention relate to a medical image diagnostic device and a medical image processing device.

The medical image diagnostic device sets various image quality parameters, such as gain, dynamic range, gamma curve, and RGB, to predetermined values for the image data. The medical image diagnostic device generates and displays a medical image based on the image quality conditions defined by the various image quality parameters.

Some medical image diagnostic devices have built-in image analysis applications for performing various image analyses on medical images (also called observation images) displayed on a display device. Image analyses include, for example, extracting specific structures contained in a region of interest (ROI) specified by a user on a medical image. Such image analysis including structure extraction can be realized, for example, by applying a trained model trained by deep learning to a medical image. Here, in order to guarantee the accuracy of image analysis using a trained model, it is necessary to train the trained model using a large number of medical images fixed to a predetermined image quality condition as training data. Accordingly, when a medical image to be analyzed (also called analysis image) is analyzed using a trained model trained using the above-mentioned training data, it is necessary to fix the image quality condition of the analysis image to the same image quality condition as the training data.

Therefore, in the above medical image diagnostic device, when image analysis is performed on an observation image (i.e., when the observation image and the analysis image are the same), the observation image is displayed under the same image quality conditions as those used when training the trained model used for image analysis, and the image quality may be significantly different from that to which users such as medical professionals are accustomed. In addition, if a user freely changes the image quality conditions of the observation image to be analyzed, the accuracy of the image analysis by the trained model must be verified under each of the changed image quality conditions. Therefore, in order to reduce the workload required for accuracy verification, the above medical image diagnostic device is often designed so that the user cannot freely change the image quality conditions of the observation image to be analyzed. As a result, for example, the user cannot observe the observation image under the desired image quality conditions, and it is difficult to properly evaluate the results of the image analysis displayed on the observation image.

JP 2009-17991 A JP 2015-8839 A

One of the problems that the embodiments disclosed in this specification and the drawings attempt to solve is to make it possible to display medical images with the image quality that users are accustomed to when performing image analysis of the medical images using a trained model. However, the problems that the embodiments disclosed in this specification and the drawings attempt to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems.

The medical image diagnostic apparatus according to this embodiment includes an acquisition unit, a generation unit, and a display unit. The acquisition unit captures an image of a subject and acquires image data relating to the subject. The generation unit generates a first medical image from the image data based on a first image quality condition that can be set by a user, and generates a second medical image based on a second image quality condition that was used to train a learned model that performs image analysis. The display unit enables the first medical image and the second medical image to be displayed in association with each other.

1 is a block diagram showing the arrangement of an ultrasound diagnostic apparatus according to a first embodiment. FIG. 2 is a diagram showing the operation of the ultrasound diagnostic apparatus according to the first embodiment. 11A and 11B are diagrams showing a display format of an image for observation in a single display mode. 11A and 11B are diagrams showing a display format of an analysis image in a single display mode. 5A and 5B are diagrams showing the display format of an image for observation and an image for analysis in a dual display mode. 6A to 6C are diagrams showing another operation of the ultrasonic diagnostic apparatus according to the first embodiment. 11A and 11B are diagrams showing the display format of an image for observation and an image for analysis in a quad display mode. 13A and 13B are diagrams showing other display formats of the observation image and the analysis image in the quad display mode. FIG. 13 is a block diagram showing the arrangement of a medical image processing apparatus according to a second embodiment.

The medical image diagnostic device and medical image processing device according to this embodiment will be described below with reference to the drawings. In the following embodiments, parts with the same reference numbers perform similar operations, and duplicated descriptions will be omitted as appropriate. Below, an ultrasound diagnostic device will be described as an example of a medical image diagnostic device, but this is not limited to this. For example, the configuration according to this embodiment can also be applied to various medical image diagnostic devices such as an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a nuclear medicine diagnostic device, and an X-ray diagnostic device.

First Embodiment
The configuration of an ultrasound diagnostic apparatus according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an ultrasound diagnostic apparatus 1 according to the first embodiment. As shown in Fig. 1, the ultrasound diagnostic apparatus 1 includes a main unit 10 and an ultrasound probe 30. The main unit 10 is connected to an external device 40 via a network 100. The main unit 10 is also connected to a display device 50 and an input device 60.

The ultrasonic probe 30 has a plurality of ultrasonic transducers (hereinafter simply referred to as elements), a matching layer provided on the elements, and a backing material that prevents the propagation of ultrasonic waves backward from the elements. The ultrasonic probe 30 is detachably connected to the main unit 10. The ultrasonic probe 30 is used by being grounded to the subject P.

The main unit 10 shown in FIG. 1 is a device that generates an ultrasound image based on a reflected wave signal received by an ultrasound probe 30. As shown in FIG. 1, the main unit 10 includes an ultrasound transmission circuit 11, an ultrasound reception circuit 12, a B-mode processing circuit 13, a Doppler processing circuit 14, a three-dimensional processing circuit 15, a display processing circuit 16, an internal storage circuit 17, an image memory 18 (cine memory), an image database 19, an input interface circuit 20, a communication interface circuit 21, and a control circuit 22.

The ultrasonic transmission circuit 11 is a processor that supplies a drive signal to the ultrasonic probe 30. The ultrasonic transmission circuit 11 is realized by, for example, a trigger generation circuit, a delay circuit, and a pulser circuit. The trigger generation circuit repeatedly generates a rate pulse for forming a transmission ultrasonic wave at a predetermined rate frequency. The delay circuit provides each rate pulse generated by the trigger generation circuit with a transmission delay time for each element required to focus the ultrasonic waves generated from the ultrasonic probe 30 into a beam and determine the transmission directivity. The pulser circuit applies a drive signal (drive pulse) to the ultrasonic probe 30 at a timing based on the rate pulse. By changing the transmission delay time provided to each rate pulse by the delay circuit, the transmission direction from the element surface can be adjusted arbitrarily.

The ultrasonic receiving circuit 12 is a processor that performs various processes on the reflected wave signal received by the ultrasonic probe 30 to generate a received signal. The ultrasonic receiving circuit 12 is realized by, for example, an amplifier circuit, an A/D converter, a reception delay circuit, an adder, etc. The amplifier circuit amplifies the reflected wave signal received by the ultrasonic probe 30 for each channel and performs gain correction processing. The A/D converter converts the gain-corrected reflected wave signal into a digital signal. The reception delay circuit gives the digital signal a reception delay time required to determine the reception directivity. The adder adds multiple digital signals that have been given a reception delay time. The addition process of the adder generates a received signal in which the reflection component from the direction corresponding to the reception directivity is emphasized.

The B-mode processing circuit 13 is a processor that generates B-mode data based on the reception signal received from the ultrasonic reception circuit 12. The B-mode processing circuit 13 performs envelope detection processing, logarithmic amplification processing, and the like on the reception signal received from the ultrasonic reception circuit 12, and generates data in which the signal strength is expressed as luminance brightness (hereinafter, B-mode data). The generated B-mode data is stored in a RAW data memory (not shown) as B-mode RAW data on the ultrasonic scan line. The B-mode RAW data may be stored in the internal storage circuit 17 described below.

The Doppler processing circuit 14 is a processor that generates a Doppler waveform and Doppler data based on the reception signal received from the ultrasound reception circuit 12. The Doppler processing circuit 14 extracts a blood flow signal from the reception signal, generates a Doppler waveform from the extracted blood flow signal, and generates data (hereinafter, Doppler data) that extracts information such as average velocity, variance, and power for multiple points from the blood flow signal.

The three-dimensional processing circuit 15 is a processor capable of generating two-dimensional image data or three-dimensional image data (hereinafter also referred to as volume data) based on the data generated by the B-mode processing circuit 13 and the Doppler processing circuit 14. The three-dimensional processing circuit 15 generates two-dimensional image data composed of pixels by performing RAW-pixel conversion.

The three-dimensional processing circuit 15 also generates volume data consisting of voxels in a desired range by performing a raw-voxel conversion, including an interpolation process that takes into account spatial position information, on the B-mode raw data stored in the raw data memory. The three-dimensional processing circuit 15 performs a rendering process on the generated volume data to generate rendered image data. Hereinafter, the B-mode raw data, two-dimensional image data, volume data, and rendered image data are collectively referred to as ultrasound data.

The display processing circuit 16 converts the image data generated by the three-dimensional processing circuit 15 into display data (e.g., video signals) by performing various processes such as corrections related to gain, dynamic range, and gamma curve, and RGB conversion. The display processing circuit 16 generates a medical image based on the display data and displays the generated medical image on the display device 50. The display processing circuit 16 may generate a user interface (GUI: Graphical User Interface) for the operator to input various instructions via the input interface circuit 20 and display the GUI on the display device 50. As the display device 50, for example, a CRT display, a liquid crystal display, an organic EL display, an LED display, a plasma display, or any other display known in the art may be used as appropriate.

The internal storage circuit 17 has, for example, a magnetic or optical recording medium, or a processor-readable recording medium such as a semiconductor memory. The internal storage circuit 17 stores a control program for implementing ultrasonic transmission and reception, a control program for performing image processing, and a control program for performing display processing. The internal storage circuit 17 also stores a group of data such as diagnostic information (for example, subject ID, doctor's findings, etc.), diagnostic protocols, a body mark generation program, and a conversion table that presets the range of color data used for visualization for each diagnostic site. The internal storage circuit 17 may also store an anatomical diagram, for example, an atlas, relating to the structure of organs in a living body.

In this embodiment, the internal storage circuitry 17 stores various trained models for performing various image analyses and image quality conditions at the time of training associated with each trained model. The trained models and image quality conditions may be stored in association with each other in the form of a look-up table (LUT). The image quality conditions are defined by various image quality parameters such as gain, dynamic range, gamma curve, and RGB related to the image data. Note that the image quality parameters are not limited to numerical values and include color settings such as color maps. The internal storage circuitry 17 also stores various programs for realizing various display modes that define the layout when displaying medical images on the display device 50.

The internal storage circuitry 17 also stores the two-dimensional image data, volume data, and rendering image data generated by the three-dimensional processing circuitry 15 in accordance with storage operations input via the input interface circuitry 20. The internal storage circuitry 17 may also store the two-dimensional image data, volume data, and rendering image data generated by the three-dimensional processing circuitry 15, including the order of operations and the time of operations, in accordance with storage operations input via the input interface circuitry 20. The internal storage circuitry 17 can also transfer the stored data to the external device 40 via the communication interface circuitry 21.

The image memory 18 includes, for example, a magnetic or optical recording medium, or a processor-readable recording medium such as a semiconductor memory. The image memory 18 stores image data corresponding to multiple frames immediately before a freeze operation input via the input interface circuit 20. The image data stored in the image memory 18 is, for example, continuously displayed (cine display).

The image database 19 stores image data transferred from the external device 40. For example, the image database 19 receives and stores past medical image data relating to the same subject acquired in a past examination that is saved in the external device 40. The past medical image data includes ultrasound image data, CT image data, MR image data, PET (Positron Emission Tomography)-CT image data, PET-MR image data, and X-ray image data.

The image database 19 may store desired image data by reading image data recorded on a storage medium such as an MO, CD-R, or DVD.

The input interface circuit 20 accepts various instructions from a user via the input device 60. The input device 60 is, for example, a mouse, a keyboard, a panel switch, a slider switch, a trackball, a rotary encoder, a touch command screen (TCS), and an operation panel. The input interface circuit 20 is connected to the control circuit 22 via, for example, a bus, converts an operation instruction input by an operator into an electrical signal, and outputs the electrical signal to the control circuit 22. In this embodiment, the input interface circuit 20 is not limited to a circuit that connects to physical operation components such as a mouse and a keyboard. For example, an example of the input interface circuit 20 includes an electrical signal processing circuit that receives an electrical signal corresponding to an operation instruction input from an external input device provided separately from the ultrasound diagnostic device 1 as a wireless signal and outputs the electrical signal to the control circuit 22. For example, the input interface circuit 20 may be an external input device that can transmit an operation instruction corresponding to an instruction by a gesture of the operator as a wireless signal.

The communication interface circuit 21 is connected to an external device 40 via a network 100, and performs data communication with the external device 40. The external device 40 is, for example, a database of a PACS (Picture Archiving and Communication Systems) which is a system for managing data of various medical images, or a database of an electronic medical record system for managing electronic medical records to which medical images are attached. The external device 40 may also be various medical image diagnostic devices other than the ultrasound diagnostic device 1 according to this embodiment, such as an X-ray CT device, an MRI device, a nuclear medicine diagnostic device, and an X-ray diagnostic device. The communication standard with the external device 40 may be any standard, for example, DICOM (Digital Imaging and COmmunications in Medicine).

The control circuit 22 is, for example, a processor that functions as the core of the ultrasound diagnostic device 1. The control circuit 22 executes a control program stored in the internal storage circuit 17 to realize a function corresponding to the program. Specifically, the control circuit 22 executes an acquisition function 101, a generation function 102, an analysis function 103, a calculation function 104, a decision function 105, and a display function 106.

By executing the acquisition function 101, the control circuit 22 captures an image of the subject and acquires image data relating to the subject.
By executing the generation function 102, the control circuit 22 generates a first medical image from the image data based on first image quality conditions that can be set by the user, and generates a second medical image based on second image quality conditions used to train the learned model that performs image analysis.
By executing the analysis function 103, the control circuit 22 performs image analysis by applying a learned model trained under second image quality conditions to at least one of the first medical image and the second medical image.
By executing the calculation function 104, the control circuit 22 performs calculations on the results of the image analysis.
By executing the determination function 105, the control circuit 22 determines a display mode relating to the format in which the first medical image and the second medical image are displayed from a plurality of display modes.
By executing the display function 106, the control circuit 22 can display the first medical image and the second medical image in association with each other. The display function 106 also displays the result of the image analysis superimposed on the medical image on which the image analysis has been performed. The display function 106 also displays the result of the image analysis on one medical image superimposed on the other medical image on which the image analysis has not been performed.

The acquisition function 101, generation function 102, analysis function 103, calculation function 104, decision function 105, and display function 106 may be incorporated as a control program, or a dedicated hardware circuit capable of executing each function may be incorporated in the control circuit 22 itself or in the main unit 10.

The control circuit 22 may be realized by an Application Specific Integrated Circuit (ASIC) incorporating these dedicated hardware circuits, a Field Programmable Gate Array (FPGA), other Complex Programmable Logic Devices (CPLDs), or Simple Programmable Logic Devices (SPLDs).

The operation of the ultrasound diagnostic device according to the first embodiment will be described with reference to FIG. 2. FIG. 2 is a flow diagram showing an example of the operation of the ultrasound diagnostic device 1 according to the first embodiment. Note that this operation example is an example of the operation for displaying an observation image or an analysis image in the single display mode or dual display mode of FIGS. 3 to 5 described later.

In step S201, the control circuit 22 acquires image data of the subject. Specifically, the control circuit 22 executes the acquisition function 101 to capture an image of the subject and acquire image data related to the subject. The control circuit 22 may acquire image data stored in the image database 19. The acquired image data is two-dimensional image data, volume data, or rendering image data generated by the three-dimensional processing circuit 15, and is stored in the internal storage circuit 17 in a predetermined image format. Image formats include, for example, JPEG, PNG, GIF, SVG, TIFF, DICOM, etc. The image data may be multiple frames acquired in real time, or a single frame.

In step S202, the control circuit 22 generates an image for observation and an image for analysis. Specifically, the control circuit 22 executes the generation function 102 to generate an image for observation from image data based on image quality conditions that can be set by the user, and generates an image for analysis based on image quality conditions used in training a trained model that performs image analysis. The control circuit 22 may, for example, accept an instruction input by the user via an input device 60 such as an operation panel, and set the image quality conditions of the image for observation. The control circuit 22 may also set the image quality conditions of the image for analysis based on the image quality conditions at the time of training associated with the trained model stored in the internal storage circuit 17. In addition, a part of the image quality parameters (e.g., gain) that define the image quality conditions of the image for analysis may be changed by user input. In addition, when the image quality conditions of the image for observation are updated by the user, a new image for observation may be generated based on the updated image quality conditions. The generated image for observation and image for analysis are stored in the internal storage circuit 17 in the same image format as in step S201. At this time, the image quality conditions of the observation image and the image quality conditions of the analysis image are stored in the internal storage circuitry 17 in association with the observation image and the analysis image, respectively.

It is sufficient that at least one observation image and one analysis image are generated. For example, when a user sets multiple different image quality conditions for an observation image, multiple observation images based on the respective image quality conditions may be generated. For an analysis image, multiple analysis images may be generated based on the image quality conditions at the time of training of each trained model, which are associated with multiple different trained models.

Note that in step S202, the control circuit 22 generates an observation image from the acquired image data based on the image quality conditions set by the user, but this is not limited to the above. For example, in step S201, the control circuit 22 may acquire image data for the observation image based on parameters related to ultrasound transmission and reception, etc., set by the user. In addition, the observation image may be generated based on the image data. This makes it possible to adjust the image quality of the observation image, which is difficult to do by adjusting image quality parameters.

In step S203, the control circuit 22 performs image analysis. Specifically, the control circuit 22 executes the analysis function 103 to perform image analysis by applying the trained model based on which the analysis image was generated in step S202 to the analysis image. At this time, the control circuit 22 may apply the trained model only to the region of interest set in the medical image by the user via the input device 60 such as an operation panel. Note that image analysis may also be performed on the observation image. That is, the control circuit 22 may execute the analysis function 103 to apply the trained model to the observation image and perform image analysis. The result of the image analysis is stored in the internal storage circuit 17 in association with the medical image on which the image analysis was performed.

Here, the image quality conditions of the image for analysis are the same as the image quality conditions used to train the trained model, so the accuracy of the results of image analysis of the image for analysis by the trained model is guaranteed. On the other hand, the image quality conditions of the image for observation may differ from the image quality conditions used to train the trained model. However, by applying the trained model to the image for observation as well, the user can compare the results of image analysis of the image for observation with the results of image analysis of the image for analysis.

The type of image analysis may be, but is not limited to, an analysis that extracts specific structures such as lesions, tissues, and organs that appear on medical images. For example, the analysis may be for other purposes on medical images, such as measuring the length of the short and long axes of specific structures, or lesion determination processing such as cancer category determination.

In step S204, the control circuit 22 determines the display mode. Specifically, the control circuit 22 executes the determination function 105 to determine the display mode relating to the format in which the observation image and the analysis image are displayed from among a plurality of display modes. The plurality of display modes include, for example, a single display mode in which the observation image and the display image are switched and displayed, and a dual display mode and a quad display mode in which the observation image and the display image are displayed in parallel. Note that in the single display mode, one medical image is displayed, in the dual display mode, up to two medical images are displayed, and in the quad display mode, up to four medical images are displayed.

The control circuit 22 may determine the display mode by accepting an instruction input by a user via an input device 60 such as an operation panel, or may determine a display mode that is set in advance as an initial setting (default).

In step S205, the control circuit 22 displays the observation image and the analysis image. Specifically, the control circuit 22 executes the display function 106 to enable the observation image and the analysis image to be displayed in association with each other. At this time, the observation image and the analysis image are displayed on the display device 50 based on each layout defined for each display mode. Furthermore, the results of the image analysis in step S203 are displayed superimposed on the observation image and the analysis image. For example, the results of the image analysis may be displayed superimposed on the medical image on which image analysis has been performed. Furthermore, the results of image analysis on one medical image may be displayed superimposed on the other medical image on which image analysis has not been performed.

The display formats of the observation image and the analysis image in the single display mode and the dual display mode as a result of the ultrasound diagnostic device 1 executing the series of operations of steps S201-S205 are shown below in Figs. 3-5. Note that a learned model that extracts a specific structure will be used as an example of the applied learned model. Also, the observation image 310 and the analysis image 320, which are examples of the observation image and the analysis image, are generated under different image quality conditions based on the same image data acquired by ultrasonically capturing the same phantom. Note that in practice, the observation image 310 and the analysis image 320 are displayed with the frames updated in real time.

The display format of an image for observation in the single display mode will be described with reference to Fig. 3. Fig. 3 is a schematic diagram showing an example of a display of an image for observation in the single display mode.
In FIG. 3, an observation image 310 is displayed on a display device 50. Specifically, a detection area 322, which is a result of applying a learned model to a region of interest 321 of an analysis image 320 in FIG. 4, is superimposed as a detection area 312 on an observation image 310 on which analysis has not been performed. Note that the region of interest 311 of the observation image 310 and the region of interest 321 of the analysis image 320 are set to be the same. Here, the detection area 312 is a result of extracting a circular structure, and is represented by a frame surrounding the extracted structure. In addition, as part of the image quality parameters set by the user, for example, a gain "G: 80" and a dynamic range "DR: 70" are displayed in association with the vicinity of the observation image 310. In addition, when the observation image 310 is displayed, the user presses a switching button provided on an input device 60 such as an operation panel, for example, and the ultrasound diagnostic device 1 switches to and displays an analysis image 320 shown in FIG. 4.

The display format of the analysis image in the single display mode will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of the display of the analysis image in the single display mode.
In Fig. 4, an analysis image 320 is displayed on a display device 50. Specifically, a detection region 322, which is a result of applying a trained model to a region of interest 321 of the analysis image 320, is superimposed on the analysis image 320 on which analysis has been performed. In addition, a gain "G: 80" is displayed as part of image quality parameters set by the user, and a dynamic range "DR: 80" is displayed as part of image quality parameters automatically fixed for analysis, in association with the vicinity of the analysis image 320. In addition, when the analysis image 320 is displayed, the user presses a switching button provided on an input device 60 such as an operation panel, for example, so that the ultrasound diagnostic device 1 switches to and displays an observation image 310 shown in Fig. 3.

In this way, in the single display mode, the observation image and the analysis image are switched and displayed, and one of the medical images is displayed on the display device. Therefore, one medical image can be displayed using a relatively large area on the screen. In addition, when the user has doubts about the analysis result, the user can easily check how the analysis image is drawn by switching from the observation image to the analysis image. In addition, the detection area displayed on the analysis image is the analysis result based on the analysis image with image quality conditions suitable for application of the trained model. Therefore, the highly accurate detection area detected on the analysis image can be superimposed on the observation image that is easy for the user to see. Furthermore, if the observation image and the analysis image are displayed at the same position on the screen with the same size, the difference in image quality between the observation image and the analysis image when switching can be easily compared, and the difference in the position and size of the detection area can be easily compared. In addition, by displaying some image quality parameters that specify the image quality conditions of the observation image and the analysis image as numerical values, the difference in image quality can be quantitatively compared.

The display format of the observation image and the analysis image in the dual display mode will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing an example of the display of the observation image and the analysis image in the dual display mode.
In Fig. 5, an image for observation 310 and an image for analysis 320 are displayed on a display device 50. Specifically, a screen configuration including the image for observation 310 shown in Fig. 3 is displayed on the left side, and a screen configuration including the image for analysis 320 shown in Fig. 4 is displayed side by side on the right side. Note that in Fig. 5, the image for observation 310 and the image for analysis 320 are adjacent to each other on one side, and the associated information including the image quality parameters is arranged on the opposite side to the adjacent part. Note that the image for observation 310 and the image for analysis 320 may be swapped left and right, or may be displayed side by side up and down.

Note that in FIG. 5, the detection area 312 of the observation image 310 is set to be hidden, but this is not limited to the above. For example, the area of interest 311 of the observation image 310 may be set to be hidden.

In this way, in the dual display mode, the observation image and the analysis image are displayed in parallel, so that two medical images are displayed on the display device. That is, two medical images can be displayed on the same screen. Furthermore, it is possible to easily check how the analysis image and the analysis image are drawn on the same screen without switching between the observation image and the analysis image. Furthermore, by setting the region of interest and detection region of the observation image to be hidden, the user can easily interpret the observation image. Furthermore, it is possible to easily compare the differences in image quality between the observation image and the analysis image on the same screen, and the differences in the position and size of the detection region. Furthermore, it is possible to quantitatively compare the differences in image quality by displaying some image quality parameters that define the image quality conditions of the observation image and the analysis image as numerical values.

Another operation of the ultrasonic diagnostic apparatus according to the first embodiment will be described with reference to FIG.
6 is a flow chart showing another operation example of the ultrasound diagnostic apparatus according to the first embodiment. Note that this operation example is an operation example for displaying an observation image or an analysis image in the quad display mode shown in FIGS. 7 and 8, which will be described later.

In step S601, the control circuit 22 acquires image data of the subject. The operation of step S601 is similar to the operation of step S201.

In step S602, the control circuit 22 generates an image for observation and an image for analysis. Specifically, the control circuit 22 generates a single image for observation and multiple images for analysis.

In step S603, the control circuit 22 performs image analysis. Specifically, the control circuit 22 performs image analysis on each of the multiple analysis images.

In step S604, the control circuit 22 performs arithmetic processing. Specifically, the control circuit 22 executes the arithmetic function 104 to integrate the results of each image analysis. As the arithmetic processing, for example, the logical product (AND) or logical sum (OR) of the results of each image analysis may be calculated. Also, it may be calculated whether the analysis accuracy of each image analysis result is equal to or greater than a threshold value. The results of the arithmetic processing are stored in the internal memory circuit 17.

In step S605, the control circuit 22 determines the display mode. The operation of step S605 is similar to the operation of step S204.

In step S606, the control circuit 22 displays the image for observation and the image for analysis. Furthermore, the control circuit 22 superimposes the calculated result on the single image for observation. The control circuit 22 may also superimpose the result of the image analysis, whose analysis accuracy is equal to or greater than a threshold, on the single image for observation.

The display format of the observation images and analysis images in the quad display mode will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing an example of the display of the observation images and analysis images in the quad display mode. Note that in the quad display mode shown in FIGS. 7-8, one observation image and three analysis images are generated. The image quality conditions of each analysis image and the trained model applied are different for each analysis image. Note that even if the total number of observation images and analysis images is less than four, they can be displayed in the quad display mode. In this case, it is possible to display nothing in areas where no medical image exists.

7, an observation image 310 is displayed at the top left, an analysis image 320 at the top right, an analysis image 330 at the bottom right, and an analysis image 340 at the bottom left. In each analysis image, a detection area is displayed superimposed as a result of performing different image analyses to extract different structures. Specifically, in the analysis image 320, the result of applying a trained model that detects circular structures to the region of interest 321 is displayed superimposed as a detection area 322. In the analysis image 330, the result of applying a trained model that detects square structures to the region of interest 331 is displayed superimposed as a detection area 332. In the analysis image 340, the result of applying a trained model that detects triangular structures to the region of interest 341 is displayed superimposed as a detection area 342. The logical sum (OR) of the results of these different image analyses is calculated and displayed superimposed on the observation image 310. Note that the region of interest 311 in the image for observation 310, the region of interest 321 in the image for analysis 320, the region of interest 331 in the image for analysis 330, and the region of interest 341 in the image for analysis 340 are all set to be identical.

Note that detection area 322, detection area 332, and detection area 342 may be displayed in different colors or with different frame line shapes (e.g., solid lines, dashed lines, dotted lines), etc., as long as they can be distinguished when superimposed on observation image 310.

Another display format of the observation image and the analysis image in the quad display mode will be described with reference to FIG. 8. FIG. 8 is a schematic diagram showing another display example of the observation image and the analysis image in the quad display mode. Note that, while FIG. 7 applies different trained models that extract different structures, FIG. 8 differs in that different trained models that extract the same structure are applied.

8, an observation image 310 is displayed at the top left, an analysis image 320 at the top right, an analysis image 330 at the bottom right, and an analysis image 340 at the bottom left. In each analysis image, a detection area is displayed superimposed as a result of performing different image analyses to extract the same structure. Specifically, in the analysis image 320, the result of applying a trained model for detecting circular structures to the region of interest 321 is displayed superimposed as a detection area 322. In the analysis image 330, the result of applying a trained model for detecting circular structures to the region of interest 331 is displayed superimposed as a detection area 332. In the analysis image 340, the result of applying a trained model for detecting circular structures to the region of interest 341 is displayed superimposed as a detection area 342. The logical product (AND) of the results of these different image analyses is calculated and displayed superimposed as a detection area 312 on the observation image 310.

The detection area 312 of the observation image 310 is the AND of the results of the three analysis methods, but is not limited to this. For example, the area of overlap of each detection area where the probability of containing a structure in the results of all analysis methods is equal to or greater than a predetermined threshold may be set as the AND area. The threshold may be selected from presets. The method of taking the AND may be softened by including areas where the threshold is exceeded in two of the three analysis methods. The user may also compare the analysis results of the three analysis methods and arbitrarily select from the three analysis methods the analysis method to take the AND.

In this way, in the quad display mode, the observation image and the analysis image are displayed in parallel, so that four medical images are displayed on the display device. That is, four medical images can be displayed on the same screen. Also, it is possible to easily check how the analysis image and the analysis image are drawn on the same screen without switching between the observation image and the analysis image. Furthermore, it is possible to easily compare the difference in image quality between the observation image and the analysis image, and the difference in the position and size of the detection area on the same screen. Also, by displaying some image quality parameters that define the image quality conditions of the observation image and the analysis image as numerical values, it is possible to quantitatively compare the difference in image quality. Furthermore, it is possible to perform calculation processing on the results of different analyses performed on each of the multiple analysis images, and display them superimposed on one observation image. This allows the user to superimpose and display highly accurate calculation results on the observation image that is easy to see.

(Other Modifications)
The display formats of the observation image and the analysis image in each display mode have been described above, but are not limited to these. For example, a gamma curve, RGB, etc. may be displayed as image quality parameters other than gain and dynamic range in association with the observation image and the analysis image. This allows the user to compare the differences in image quality conditions of the observation image and the analysis image together with the medical image on the screen of the display device.

In addition, by displaying color bars corresponding to the observation image and the analysis image, it is possible to compare the differences in image quality conditions related to the color maps of the observation image and the analysis image. In addition, by displaying some image quality parameters that are not displayed on the display device on the operation panel, it is possible to compare the differences in image quality conditions of the observation image and the analysis image due to those image quality parameters on the operation panel.

The region of interest in the image for observation and the region of interest in the image for analysis do not have to be the same, and may be different. In this case, the region of interest in the image for analysis only needs to be set within a range that includes the region of interest in the image for observation. This allows image analysis to be performed on the image for analysis, at least for the region of interest in the image for observation that was set by the user.

Note that each display mode, including the single display mode, dual display mode, and quad display mode, may be switchable, for example, by pressing a switch button provided on an input device such as an operation panel. Of course, a display mode that displays more than four images may also be set.

According to the first embodiment described above, the user can observe the observation image with the image quality conditions and layout desired. Furthermore, by displaying the observation image and the analysis image in association with each other, the image quality of each medical image and the differences in the analysis results can be compared. Furthermore, since the image quality conditions of the analysis image are fixed to one type, it is only necessary to verify the accuracy for that image quality condition, and the workload associated with the accuracy verification can be kept to a minimum. Note that there may be multiple types of image quality conditions for the analysis image as long as the accuracy of the analysis can be verified.

Second Embodiment
The configuration of the medical image processing device according to the second embodiment will be described with reference to FIG. 9. In the ultrasound diagnostic device according to the first embodiment, an observation image and an analysis image are simultaneously generated and displayed based on image data obtained by ultrasonically photographing a subject. The medical image processing device according to the second embodiment differs in that it does not have a mechanism for photographing a subject and obtains image data from an external device. The medical image processing device according to the second embodiment is assumed to be included in an analysis device such as a workstation, but may be mounted on other devices. In other respects, it is the same as the ultrasound diagnostic device 1.

Figure 9 is a block diagram showing an example of the configuration of a medical image processing device 9 according to the second embodiment. The medical image processing device 9 includes a control circuit 90, a memory 91, an input interface circuit 92, a communication interface circuit 93, a display device 94, an input device 95, a network 96, and an external device 97. Each component of the medical image processing device 9 performs the same function as the corresponding component in the ultrasound diagnostic device 1, and redundant explanations will be omitted as appropriate.

The control circuit 90 is, for example, a processor that functions as the core of the medical image processing device 9. The control circuit 90 executes a control program stored in the memory 91 to realize functions corresponding to the program. Specifically, the control circuit 90 executes an acquisition function 901, a generation function 902, an analysis function 903, a calculation function 904, a decision function 905, and a display function 906.

By executing the acquisition function 901, the control circuit 22 acquires image data relating to the subject.
By executing generation function 902 (corresponding to generation function 102), the control circuit 22 generates a first medical image from the image data based on first image quality conditions that can be set by the user, and generates a second medical image based on second image quality conditions used to train the learned model that performs image analysis.
By executing analysis function 903 (corresponding to analysis function 103), the control circuit 22 performs image analysis by applying a learned model trained under the second image quality conditions to at least one of the first medical image and the second medical image.
By executing a calculation function 904 (corresponding to the calculation function 104), the control circuit 22 performs calculation processing on the result of the image analysis.
By executing a decision function 905 (corresponding to the decision function 105), the control circuit 22 decides a display mode relating to the format in which the first medical image and the second medical image are displayed from a plurality of display modes.
By executing a display function 906 (corresponding to the display function 106), the control circuit 22 can display the first medical image and the second medical image in association with each other. The display function 906 also displays the result of the image analysis superimposed on the medical image on which the image analysis has been performed. The display function 906 also displays the result of the image analysis on one medical image superimposed on the other medical image on which the image analysis has not been performed.

Memory 91 (corresponding to internal memory circuit 17) stores each program for implementing each function of control circuit 90. It also stores image data acquired from external device 97 (corresponding to external device 40) via network 96 (corresponding to network 100).

The input interface circuit 92 (corresponding to the input interface circuit 20) accepts various instructions from the user via the input device 95 (corresponding to the input device 60).

The communication interface circuit 93 (corresponding to the communication interface circuit 21) is connected to an external device 97 via a network 96 (corresponding to the network 100) and performs data communication with the external device 97. The external device 97 stores image data.

Display device 94 (corresponding to display device 50) displays medical images.

According to the second embodiment described above, it is possible to simultaneously generate an image for observation and an image for analysis based on image data obtained from an external device. In addition, the image for observation can be observed with the image quality conditions and layout desired by the user. Furthermore, by displaying the image for observation and the image for analysis in correspondence with each other, it is possible to compare the differences in image quality and analysis results of each medical image. Furthermore, since the image quality conditions of the image for analysis are fixed to one type, it is only necessary to verify the accuracy for that image quality condition, and the workload associated with the accuracy verification can be kept to a minimum. Note that there may be multiple types of image quality conditions for the image for analysis as long as the accuracy of the analysis can be verified.

According to at least one of the embodiments described above, when image analysis is performed on medical images using a trained model, it is possible to display the medical images with the image quality that the user is accustomed to.

Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and spirit of the invention.

1: Ultrasound diagnostic device, 9: Medical image processing device, 10: Main unit, 11: Ultrasound transmission circuit, 12: Ultrasound reception circuit, 13: B-mode processing circuit, 14: Doppler processing circuit, 15: Dimensional processing circuit, 16: Display processing circuit, 17: Internal storage circuit, 18: Image memory, 19: Image database, 20, 92: Input interface circuit, 21, 93: Communication interface circuit, 22, 90: Control circuit, 30: Ultrasound probe, 40, 97: External device, 50, 94... display device, 60, 95... input device, 91... memory, 96, 100... network, 101, 901... acquisition function, 102, 902... generation function, 103, 903... analysis function, 104, 904... calculation function, 105, 905... decision function, 106, 906... display function, 310... observation image, 311, 321, 331, 341... region of interest, 312, 322, 332, 342... detection region, 320, 330, 340... analysis image

Claims (11)

an acquisition unit that acquires ultrasound image data relating to a subject by imaging the subject;
A generating unit that generates an observation medical image from the ultrasound image data based on a first image quality condition that can be set by a user, and generates an analysis medical image from the ultrasound image data used to generate the observation medical image based on a second image quality condition corresponding to a trained model that performs image analysis;
An analysis unit that performs image analysis of the medical image for analysis by applying the trained model to the medical image for analysis;
a display unit that displays the medical image for observation and an image in which the result of the image analysis is superimposed on the medical image for analysis;
A medical image diagnostic apparatus comprising: The medical image diagnostic apparatus according to claim 1 , wherein, when the first image quality condition is updated by the user, the generating unit generates a new medical image for observation based on the updated first image quality condition. 3. The medical image diagnostic apparatus according to claim 1, further comprising a determination unit that determines a display mode relating to a format for displaying the medical image for observation and the medical image for analysis from a plurality of display modes. When a first display mode in which the medical image for observation and the medical image for analysis are switched and displayed is determined by the determination unit, the display unit
4. The medical image diagnostic apparatus according to claim 3, wherein the medical image for observation and the medical image for analysis are switched and displayed in response to an image switching instruction from the user. When a second display mode in which the medical image for observation and the medical image for analysis are displayed side by side is determined by the determination unit, the display unit
The medical image diagnostic apparatus according to claim 3 , wherein the medical image for observation and the medical image for analysis are displayed side by side. The medical image diagnostic apparatus according to claim 1 , wherein the display unit displays the result of the image analysis in a superimposed manner on another medical image on which the image analysis has not been performed. A calculation unit that performs calculation processing on the result of the image analysis,
The generating unit generates the single medical image for observation and the multiple medical images for analysis ,
The analysis unit performs image analysis on each of the plurality of medical images for analysis ,
The calculation unit performs a calculation process to integrate the results of each image analysis,
The medical image diagnostic apparatus according to claim 6 , wherein the display section displays a result calculated by the calculation section in a superimposed manner on the single medical image for observation . 8. The medical image diagnostic apparatus according to claim 7, wherein the display unit performs a logical product or a logical sum of the results of the image analyses using the calculation unit, and displays the logical product or the logical sum in a superimposed manner on the single medical image for observation . 8. The medical image diagnostic device according to claim 7, wherein the display unit performs calculations using the calculation unit to determine whether or not the analysis accuracy of each of the image analysis results is equal to or greater than a threshold, and displays the image analysis result that is equal to or greater than the threshold in a superimposed manner on the single medical image for observation . 10. The medical image diagnostic apparatus according to claim 1, wherein the first image quality condition and the second image quality condition are defined by image quality parameters including at least one of a gain, a dynamic range, a gamma curve, and RGB. The medical image diagnostic apparatus according to claim 1 , wherein the first image quality condition and the second image quality condition are stored in association with the medical image for observation and the medical image for analysis , respectively.