JP7226728B2 - Image processing device, image processing method, program - Google Patents

Description

The present invention relates to image processing for images generated by photoacoustic imaging.

2. Description of the Related Art Photoacoustic imaging (also referred to as “photoacoustic imaging”) using a contrast medium is known for examination of blood vessels, lymphatic vessels, and the like. Patent Document 1 describes a photoacoustic image generating apparatus that evaluates a contrast agent used for imaging lymph nodes, lymph vessels, etc., and emits light with a wavelength that is absorbed by the contrast agent and generates a photoacoustic wave. is described.

WO2017/002337

However, in the photoacoustic imaging described in Patent Document 1, the structure of the imaging target inside the subject (for example, the running of blood vessels, lymphatic vessels, etc.) is only visualized, and it is difficult to grasp the state of the structure. It is conceivable that

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing apparatus used in a system that facilitates understanding of the state of a contrast target by photoacoustic imaging.

A first aspect of the present invention is
An image processing apparatus for processing image data generated based on photoacoustic waves generated from within the subject by light irradiation to the subject,
By image analysis of the image data including the lymphatic vessel area in the subject, the lymphatic vessel area is divided into a plurality of divided areas, and the image value of each divided area is determined based on the time change of the image value in each divided area. The image processing apparatus is characterized by comprising state estimating means for estimating the state of the lymphatic vessel by determining the state.
Moreover, the second aspect of the present invention is
An image processing apparatus for processing image data generated based on photoacoustic waves generated from within the subject by light irradiation to the subject,
display control means for displaying a distance between the vein and the lymphatic vessel on a display device by image analysis of the image data including the area of the vein and the lymphatic vessel in the subject;
This image processing apparatus is characterized by:
Moreover, the third aspect of the present invention is
generating image data based on a photoacoustic wave generated from within the subject by irradiating the subject with light;
By image analysis of the image data including the lymphatic vessel area in the subject, the lymphatic vessel area is divided into a plurality of divided areas, and the image value of each divided area is determined based on the time change of the image value in each divided area. estimating the state of the lymphatic vessel by determining the state
This image processing method is characterized by:
Moreover, the fourth aspect of the present invention is
generating image data based on a photoacoustic wave generated from within the subject by irradiating the subject with light;
displaying a distance between the vein and the lymphatic vessel on a display device by image analysis of the image data including the area of the vein and the lymphatic vessel in the subject;
This image processing method is characterized by:

ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus used for the system which makes it easy to grasp|ascertain the state of the structure of imaging object by photoacoustic imaging can be provided.

1 is a block diagram of a system according to an embodiment of the invention; FIG. 1 is a block diagram showing a specific example of an image processing apparatus and its peripheral configuration according to an embodiment of the present invention; FIG. Detailed block diagram of a photoacoustic device according to an embodiment of the present invention Schematic diagram of a probe according to one embodiment of the present invention 1 is a flowchart of an image processing method according to an embodiment of the present invention; FIG. Flow chart of image processing method according to embodiment 1 Contour graph of the calculated value of formula (1) corresponding to the contrast agent when the combination of wavelengths is changed A line graph showing the calculated value of formula (1) corresponding to the contrast agent when the concentration of ICG is changed. Graph showing Moller absorption coefficient spectra of oxyhemoglobin and deoxyhemoglobin FIG. 2 illustrates a GUI according to an embodiment of the invention; FIG. 4 is a diagram illustrating an example of a spectroscopic image of a subject; Diagram exemplifying classification according to the state of lymphatic vessels Diagram exemplifying the classification of lymphatic vessels according to the number of existence, area ratio, and volume ratio Diagram illustrating the classification of lymphatic vessels according to the distance to veins Flow chart of image processing method according to embodiment 2 FIG. 11 illustrates a GUI according to the second embodiment; A diagram showing a display example of classification results of lymphatic vessels according to the second embodiment. A diagram for explaining processing for extracting a region corresponding to a contrast agent. A diagram for explaining processing for extracting a region corresponding to a contrast agent.

Preferred embodiments of the present invention will be described below with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of the components described below should be appropriately changed according to the configuration of the device to which the invention is applied and various conditions. Therefore, it is not intended to limit the scope of the present invention to the following description.

The photoacoustic image obtained by the system according to the present invention reflects the amount and rate of absorption of light energy. A photoacoustic image is an image representing the spatial distribution of at least one object information such as the generated sound pressure (initial sound pressure) of the photoacoustic wave, the light absorption energy density, and the light absorption coefficient. The photoacoustic image may be an image representing a two-dimensional spatial distribution, or an image (volume data) representing a three-dimensional spatial distribution. A system according to this embodiment generates a photoacoustic image by imaging a subject into which a contrast medium has been introduced. Note that the photoacoustic image may be an image representing a two-dimensional spatial distribution or an image representing a three-dimensional spatial distribution in the depth direction from the surface of the subject in order to grasp the three-dimensional structure of the imaging target.

Also, the system according to the present invention can generate a spectroscopic image of the subject using a plurality of photoacoustic images corresponding to a plurality of wavelengths. The spectroscopic image of the present invention is an image generated using photoacoustic signals corresponding to each of a plurality of wavelengths based on photoacoustic waves generated by irradiating a subject with light of a plurality of wavelengths different from each other. be.
Note that the spectroscopic image may be an image showing the concentration of a specific substance in the subject, generated using photoacoustic signals corresponding to each of a plurality of wavelengths. When the light absorption coefficient spectrum of the contrast medium used differs from the light absorption coefficient spectrum of the specific substance, the image value of the contrast medium in the spectral image differs from the image value of the specific substance in the spectral image. Therefore, it is possible to distinguish the region of the contrast agent from the region of the specific substance according to the image value of the spectral image. Note that specific substances include substances that constitute a subject, such as hemoglobin, glucose, collagen, melanin, fat, and water. Also in this case, it is necessary to select a contrast agent having a light absorption spectrum different from the light absorption coefficient spectrum of the specific substance. Also, the spectroscopic image may be calculated by different calculation methods depending on the type of the specific substance.

ここで、Ｉ^λ _１（ｒ）は第１波長λ_１の光照射により発生した光音響波に基づいた計測値であり、Ｉ^λ _２（ｒ）は第２波長λ_２の光照射により発生した光音響波に基づいた計測値である。ε_Ｈｂ ^λ _１は第１波長λ_１に対応するデオキシヘモグロビンのモラー吸収係数［ｍｍ^－１ｍｏｌ^－１］であり、ε_Ｈｂ ^λ _２は第２波長λ_２に対応するデオキシヘモグロビンのモラー吸収係数［ｍｍ^－１ｍｏｌ^－１］である。ε_ＨｂＯ ^λ _１は第１波長λ_１に対応するオキシヘモグロビンのモラー吸収係数［ｍｍ^－１ｍｏｌ^－１］であり、ε_ＨｂＯ ^λ _２は第２波長λ_２に対応するオキシヘモグロビンのモラー吸収係数［ｍｍ^－１ｍｏｌ^－１］である。ｒは位置である。なお、計測値Ｉ^λ _１（ｒ）、Ｉ^λ _２（ｒ）としては、吸収係数μ_ａ ^λ _１（ｒ）、μ_ａ ^λ _２（ｒ）を用いてもよいし、初期音圧Ｐ_０ ^λ _１（ｒ）、Ｐ_０ ^λ _２（ｒ）を用いてもよい。 In the embodiments described below, an image calculated using the oxygen saturation calculation formula (1) will be described as a spectral image. The present inventors have found that the oxygen saturation of blood hemoglobin (which may be an index correlated with oxygen saturation) is calculated based on photoacoustic signals corresponding to each of a plurality of wavelengths. The range of possible values for the oxygen saturation of hemoglobin when substituting the measured value I(r) of the photoacoustic signal obtained with a contrast agent that shows a different trend in the wavelength dependence of the absorption coefficient from that of oxyhemoglobin and deoxyhemoglobin. It has been found that a calculated value Is(r) deviating greatly from is obtained. Therefore, if a spectroscopic image having this calculated value Is(r) as an image value is generated, a hemoglobin region (blood vessel region) and a contrast agent existing region (for example, a lymphatic vessel where the contrast agent is introduced into the subject) can be obtained. In this case, it becomes easy to separate (distinguish) from the area of the lymphatic vessel) on the image.

Here, I ^λ ₁ (r) is a measured value based on a photoacoustic wave generated by light irradiation with a first wavelength λ ₁ , and I ^λ ₂ (r) is a measured value generated by light irradiation with a second wavelength λ ₂ These are measured values based on photoacoustic waves. ε _Hb ^λ ₁ is the Molar absorption coefficient of deoxyhemoglobin corresponding to the first wavelength λ ₁ [mm ⁻¹ mol ⁻¹ ], and ε _Hb ^λ ₂ is the Molar absorption coefficient of deoxyhemoglobin corresponding to the second wavelength λ ₂ [ mm ⁻¹ mol ⁻¹ ]. ε _HbO ^λ ₁ is the Molar absorption coefficient of oxyhemoglobin corresponding to the first wavelength λ ₁ [mm ⁻¹ mol ⁻¹ ], and ε _HbO ^λ ₂ is the Molar absorption coefficient of oxyhemoglobin corresponding to the second wavelength λ ₂ [ mm ⁻¹ mol ⁻¹ ]. r is the position. As the measured values I ^λ ₁ (r) and I ^λ ₂ (r), the absorption coefficients μ _a ^λ ₁ (r) and μ _a ^λ ₂ (r) may be used, or the initial sound pressure P ₀ ^λ ₁ (r) and P ₀ ^λ ₂ (r) may be used.

Substituting the measured value based on the photoacoustic wave generated from the region where hemoglobin exists (blood vessel region) into Equation (1), the calculated value Is(r) is the oxygen saturation of hemoglobin (or index) is obtained. On the other hand, in a subject into which a contrast agent has been introduced, when the measured value based on the acoustic wave generated from the region where the contrast agent exists (for example, the lymphatic region) is substituted into Equation (1), the calculated value Is(r) is obtained as a pseudo A typical concentration distribution of the contrast agent is obtained. Even when calculating the concentration distribution of the contrast medium, the numerical value of the Moller absorption coefficient of hemoglobin may be used as it is in the equation (1). The spectroscopic image Is(r) thus obtained is such that both the hemoglobin-existing region (blood vessel) and the contrast agent-existing region (e.g., lymphatic vessel) inside the subject are separable (distinguishable) from each other. A rendered image.

In the present embodiment, the image value of the spectral image is calculated using the equation (1) for calculating the oxygen saturation. A calculation method other than (1) may be used. As the index and its calculation method, a known one can be used, so a detailed explanation is omitted here.

In addition, the system according to the present invention includes a first photoacoustic image based on the photoacoustic wave generated by the light irradiation of the first wavelength λ ₁ and a second photoacoustic wave generated by the light irradiation of the second wavelength λ ₂ An image showing the ratio of the two photoacoustic images may be used as the spectral image. That is, the ratio of the first photoacoustic image based on the photoacoustic wave generated by the light irradiation of the first wavelength λ ₁ and the second photoacoustic image based on the photoacoustic wave generated by the light irradiation of the second wavelength λ ₂ is The image on which it is based may be the spectral image. Note that the image generated according to the modified expression of formula (1) can also be expressed by the ratio of the first photoacoustic image and the second photoacoustic image, so based on the ratio of the first photoacoustic image and the second photoacoustic image can be said to be an image (spectral image).

Note that the spectral image may be an image representing a two-dimensional spatial distribution or an image representing a three-dimensional spatial distribution in the depth direction from the surface of the subject in order to grasp the three-dimensional structure of the contrast target.
The configuration of the system and the image processing method of this embodiment will be described below.

A system according to this embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of the system according to this embodiment. The system according to this embodiment includes a photoacoustic device 1100 , a storage device 1200 , an image processing device 1300 , a display device 1400 and an input device 1500 . Transmission and reception of data between devices may be performed by wire or wirelessly.

The photoacoustic device 1100 generates a photoacoustic image by imaging a subject into which a contrast medium has been introduced, and outputs the photoacoustic image to the storage device 1200 . The photoacoustic device 1100 is a device that generates characteristic value information corresponding to each of a plurality of positions within a subject using received signals obtained by receiving photoacoustic waves generated by light irradiation. That is, the photoacoustic device 1100 is a device that generates the spatial distribution of characteristic value information derived from photoacoustic waves as medical image data (photoacoustic image).

The storage device 1200 may be a storage medium such as a ROM (Read only memory), a magnetic disk, or a flash memory. The storage device 1200 may also be a storage server via a network such as PACS (Picture Archiving and Communication System).

The image processing device 1300 is a device that processes information such as photoacoustic images stored in the storage device 1200 and incidental information of the photoacoustic images.
A unit that performs the arithmetic function of the image processing apparatus 1300 can be configured by a processor such as a CPU or a GPU (Graphics Processing Unit) and an arithmetic circuit such as an FPGA (Field Programmable Gate Array) chip. These units may be composed not only of a single processor or arithmetic circuit, but also of a plurality of processors or arithmetic circuits.

The unit responsible for the storage function of the image processing apparatus 1300 can be composed of a non-temporary storage medium such as a ROM (Read only memory), a magnetic disk, or a flash memory. Also, the unit responsible for the storage function may be a volatile medium such as RAM (Random Access Memory). Note that the storage medium in which the program is stored is a non-temporary storage medium. It should be noted that the unit that performs the storage function may not only be composed of one storage medium, but may also be composed of a plurality of storage media.

A unit responsible for the control function of the image processing apparatus 1300 is composed of an arithmetic element such as a CPU. A unit responsible for control functions controls the operation of each component of the system. The unit responsible for the control function may receive instruction signals from various operations such as measurement start from the input section and control each configuration of the system. Units responsible for control functions may also read program code stored in computer 150 to control the operation of each component of the system.

The display device 1400 is a display such as a liquid crystal display or an organic EL (Electro Luminescence). In addition, the display device 1400 may display an image or a GUI for operating the device.

As the input device 1500, a user-operable operation console composed of a mouse, a keyboard, and the like can be adopted. Alternatively, the display device 1400 may be configured with a touch panel and used as the input device 1500 .

FIG. 2 shows a specific configuration example of an image processing apparatus 1300 according to this embodiment. The image processing apparatus 1300 according to this embodiment is composed of a CPU 1310 , a GPU 1320 , a RAM 1330 , a ROM 1340 and an external storage device 1350 . A liquid crystal display 1410 as a display device 1400 , a mouse 1510 and a keyboard 1520 as input devices 1500 are connected to the image processing device 1300 . Furthermore, the image processing device 1300
PACS (Picture Archiving and Communication)
System) is connected to an image server 1210 as a storage device 1200 . As a result, image data can be saved on the image server 1210 and image data on the image server 1210 can be displayed on the liquid crystal display 1410 .
Next, a configuration example of the devices included in the system according to this embodiment will be described. FIG. 3 is a schematic block diagram of devices included in the system according to the present embodiment.

A photoacoustic device 1100 according to this embodiment has a driving section 130 , a signal collecting section 140 , a computer 150 , a probe 180 and an introduction section 190 . The probe 180 has a light emitting section 110 and a receiving section 120 . FIG. 4 shows a schematic diagram of the probe 180 according to this embodiment. A measurement target is the subject 100 into which the contrast medium has been introduced by the introduction section 190 . The drive unit 130 drives the light irradiation unit 110 and the reception unit 120 to perform mechanical scanning. The light irradiation unit 110 irradiates the subject 100 with light, and acoustic waves are generated within the subject 100 . Acoustic waves generated by the photoacoustic effect due to light are also called photoacoustic waves. The receiving unit 120 outputs an electric signal (photoacoustic signal) as an analog signal by receiving the photoacoustic wave.

The signal collector 140 converts the analog signal output from the receiver 120 into a digital signal and outputs the digital signal to the computer 150 . The computer 150 stores the digital signal output from the signal collection unit 140 as signal data derived from photoacoustic waves. Signal data is an example of received signal data.

The computer 150 generates a photoacoustic image by performing signal processing on the stored digital signal. Further, the computer 150 outputs the photoacoustic image to the display unit 160 after performing image processing on the obtained photoacoustic image. The display unit 160 displays an image based on the photoacoustic image. The display image is saved in a memory in the computer 150 or in a storage device 1200 such as a data management system connected to the modality via a network, based on a save instruction from the user or the computer 150 .

The computer 150 also drives and controls components included in the photoacoustic device. Also, the display unit 160 may display a GUI or the like in addition to the image generated by the computer 150 . The input unit 170 is configured so that the user can input information. The user can use the input unit 170 to perform operations such as starting and ending measurement and instructing to save the created image.
The details of each configuration of the photoacoustic device 1100 according to this embodiment will be described below.

(Light irradiation unit 110)
The light irradiation unit 110 includes a light source 111 that emits light and an optical system 112 that guides the light emitted from the light source 111 to the subject 100 . The light includes pulsed light such as so-called rectangular waves and triangular waves.

Considering thermal confinement conditions and stress confinement conditions, the pulse width of the light emitted by the light source 111 is preferably 100 ns or less. Also, the wavelength of light may be in the range of about 400 nm to 1600 nm. When imaging blood vessels with high resolution, wavelengths (400 nm or more and 700 nm or less) that are highly absorbed by blood vessels may be used. When imaging a deep part of a living body, light having a wavelength (700 nm or more and 1100 nm or less) that is typically less absorbed by the background tissue (water, fat, etc.) of the living body may be used.

A laser or a light emitting diode can be used as the light source 111 . Moreover, when measuring using light of multiple wavelengths, the light source may be one whose wavelengths can be changed. In the case of irradiating a subject with a plurality of wavelengths, it is also possible to prepare a plurality of light sources that generate light of mutually different wavelengths and alternately irradiate from each light source. Even when a plurality of light sources are used, they are collectively expressed as a light source. Various lasers such as solid lasers, gas lasers, dye lasers, and semiconductor lasers can be used as the laser. For example, a pulsed laser such as an Nd:YAG laser or an alexandrite laser may be used as the light source. Alternatively, a Ti:sa laser or OPO (Optical Parametric Oscillators) laser that uses Nd:YAG laser light as excitation light may be used as a light source. Alternatively, a flash lamp or a light emitting diode may be used as the light source 111 . Alternatively, a microwave source may be used as the light source 111 .

Optical elements such as lenses, mirrors, and optical fibers can be used for the optical system 112 . When the breast or the like is used as the subject 100, the light emitting portion of the optical system may be configured with a diffusion plate or the like for diffusing light in order to irradiate the pulsed light with a wider beam diameter. On the other hand, in the photoacoustic microscope, in order to increase the resolution, the light emitting part of the optical system 112 may be configured with a lens or the like, and the beam may be focused and irradiated.
The light irradiation unit 110 may directly irradiate the subject 100 with light from the light source 111 without the optical system 112 .

(Receiver 120)
The receiver 120 includes a transducer 121 that outputs an electrical signal by receiving acoustic waves, and a support 122 that supports the transducer 121 . Also, the transducer 121 may be a transmitting means for transmitting acoustic waves. A transducer as a receiving means and a transducer as a transmitting means may be a single (common) transducer, or may be configured separately.

A piezoelectric ceramic material typified by PZT (lead zirconate titanate), a polymeric piezoelectric film material typified by PVDF (polyvinylidene fluoride), or the like can be used as a member constituting the transducer 121 . Elements other than piezoelectric elements may also be used. For example, a transducer using a capacitance type transducer (CMUT: Capacitive Micro-machined Ultrasonic Transducers) can be used. Any transducer may be employed as long as it can output an electrical signal by receiving an acoustic wave. Also, the signal obtained by the transducer is a time-resolved signal. That is, the amplitude of the signal obtained by the transducer represents a value based on the sound pressure received by the transducer at each time (for example, a value proportional to the sound pressure).

The frequency components that constitute the photoacoustic wave are typically 100 kHz to 100 MHz, and transducers 121 that can detect these frequencies may be employed.

The support 122 may be made of a metal material or the like having high mechanical strength. In order to allow more irradiation light to enter the subject, the surface of the support 122 on the side of the subject 100 may be mirror-finished or light-scattering. In this embodiment, the support 122 has a hemispherical shell shape and is configured to support a plurality of transducers 121 on the hemispherical shell. In this case, the pointing axes of the transducers 121 arranged on the support 122 are concentrated near the center of curvature of the hemisphere. Then, when an image is formed using the signals output from the plurality of transducers 121, the image quality near the center of curvature is enhanced. Note that the support 122 may have any configuration as long as it can support the transducer 121 . The support 122 may arrange multiple transducers side by side in a flat or curved surface, such as a 1D array, 1.5D array, 1.75D array, or 2D array. A plurality of transducers 121 correspond to a plurality of receiving means.

Further, the support 122 may function as a container for storing the acoustic matching material. That is, the support 122 may be a container for placing the acoustic matching material between the transducer 121 and the subject 100 .

The receiver 120 may also include an amplifier that amplifies the time-series analog signal output from the transducer 121 . Further, the receiving unit 120 may include an A/D converter that converts time-series analog signals output from the transducer 121 into time-series digital signals. That is, the receiver 120 may include a signal collector 140, which will be described later.

A space between the receiving unit 120 and the subject 100 is filled with a medium through which photoacoustic waves can propagate. For this medium, a material that allows propagation of acoustic waves, matches the acoustic characteristics at the interface with the object 100 and the transducer 121, and has the highest possible photoacoustic wave transmittance is adopted. For example, this medium can be water, ultrasonic gel, or the like.

FIG. 4 shows a side view of probe 180 . A probe 180 according to this embodiment has a receiving section 120 in which a plurality of transducers 121 are three-dimensionally arranged on a hemispherical support 122 having an opening. In addition, a light exit part of the optical system 112 is arranged at the bottom of the support 122 .

In this embodiment, as shown in FIG. 4, the shape of the object 100 is held by contacting the holding part 200 .
A space between the receiving unit 120 and the holding unit 200 is filled with a medium through which photoacoustic waves can propagate. For this medium, a material that allows photoacoustic waves to propagate, has matching acoustic characteristics at the interface with the object 100 and the transducer 121, and has a photoacoustic wave transmittance that is as high as possible is used. For example, this medium can be water, ultrasonic gel, or the like.

A holding part 200 as holding means is used to hold the shape of the subject 100 during measurement. By holding the subject 100 with the holding section 200 , the movement of the subject 100 can be suppressed and the position of the subject 100 can be kept within the holding section 200 . Resin materials such as polycarbonate, polyethylene, and polyethylene terephthalate can be used as the material of the holding portion 200 .

The holding portion 200 is attached to the attachment portion 201 . The attachment section 201 may be configured to be exchangeable with a plurality of types of holding sections 200 according to the size of the subject. For example, the mounting portion 201 may be configured to be replaceable with a holding portion having a different radius of curvature, center of curvature, or the like.

(Driving unit 130)
The driving unit 130 is a part that changes the relative position between the subject 100 and the receiving unit 120 . Driving unit 130 includes a motor such as a stepping motor that generates driving force, a driving mechanism that transmits the driving force, and a position sensor that detects position information of receiving unit 120 . A lead screw mechanism, a link mechanism, a gear mechanism, a hydraulic mechanism, or the like can be used as the drive mechanism. As the position sensor, a potentiometer using an encoder, a variable resistor, a linear scale, a magnetic sensor, an infrared sensor, an ultrasonic sensor, or the like can be used.
The driving unit 130 is not limited to changing the relative position between the subject 100 and the receiving unit 120 in the XY directions (two-dimensional), but may change the position one-dimensionally or three-dimensionally.

Note that the driving unit 130 may fix the receiving unit 120 and move the subject 100 as long as the relative positions of the subject 100 and the receiving unit 120 can be changed. When the subject 100 is moved, a configuration in which the subject 100 is moved by moving a holding unit that holds the subject 100 can be considered. Also, both the subject 100 and the receiving unit 120 may be moved.

The driving unit 130 may move the relative position continuously or may move by step-and-repeat. The drive unit 130 may be an electric stage that moves along a programmed trajectory, or may be a manual stage.

In the present embodiment, the drive unit 130 simultaneously drives the light irradiation unit 110 and the reception unit 120 to perform scanning. may
It should be noted that if the probe 180 is of a handheld type provided with a grip portion, the photoacoustic device 1100 does not need to have the driving portion 130 .

(Signal collection unit 140)
The signal collector 140 includes an amplifier that amplifies the electrical signal, which is an analog signal output from the transducer 121, and an A/D converter that converts the analog signal output from the amplifier into a digital signal. A digital signal output from the signal acquisition unit 140 is stored in the computer 150 . The signal acquisition unit 140 is also called a Data Acquisition System (DAS). In this specification, an electric signal is a concept including both analog signals and digital signals. A light detection sensor such as a photodiode may detect light emission from the light irradiation unit 110, and the signal collection unit 140 may start the above processing in synchronization with the detection result as a trigger.

(Computer 150)
A computer 150 as an information processing device is configured with hardware similar to that of the image processing device 1300 . That is, the unit that performs the arithmetic function of the computer 150 can be configured by a processor such as a CPU or GPU (Graphics Processing Unit), and an arithmetic circuit such as an FPGA (Field Programmable Gate Array) chip. These units may be composed not only of a single processor or arithmetic circuit, but also of a plurality of processors or arithmetic circuits.

The unit responsible for the storage function of computer 150 may be a volatile medium such as RAM (Random Access Memory). Note that the storage medium in which the program is stored is a non-temporary storage medium. Note that the unit that performs the storage function of the computer 150 may be configured not only from one storage medium but also from a plurality of storage media.

A unit that performs the control function of the computer 150 is composed of an arithmetic element such as a CPU. The unit responsible for the control function of the computer 150 controls the operation of each component of the photoacoustic device. The unit responsible for the control function of the computer 150 may receive instruction signals from the input unit 170 for various operations such as starting measurement, and control each component of the photoacoustic device. Also, the unit responsible for the control function of the computer 150 reads the program code stored in the unit responsible for the storage function, and controls the operation of each component of the photoacoustic device. That is, the computer 150 can function as a control device of the system according to this embodiment.

Note that the computer 150 and the image processing apparatus 1300 may be configured with the same hardware. A piece of hardware may serve both the functions of the computer 150 and the image processing device 1300 . That is, the computer 150 may serve the functions of the image processing device 1300 . Also, the image processing device 1300 may serve the function of the computer 150 as an information processing device.

(Display unit 160)
The display unit 160 is a display such as a liquid crystal display or an organic EL (Electro Luminescence). The display unit 160 may also display an image or a GUI for operating the device.

Note that the display unit 160 and the display device 1400 may be the same display. That is, one display may have the functions of both the display section 160 and the display device 1400 .

(Input unit 170)
As the input unit 170, an operation console configured by a user-operable mouse, keyboard, or the like can be adopted. Alternatively, the display unit 160 may be configured with a touch panel and used as the input unit 170 .

Note that the input unit 170 and the input device 1500 may be the same device. That is, one device may serve both the functions of the input unit 170 and the input device 1500 .

(Introduction part 190)
The introduction unit 190 is configured to be able to introduce a contrast medium into the subject 100 from the outside of the subject 100 . For example, the introduction part 190 can include a contrast agent container and an injection needle for piercing the subject. However, the introduction part 190 is not limited to this, and various types can be applied as long as the introduction part 190 can introduce the contrast agent into the subject 100 . The introduction part 190 may in this case be, for example, a known injection system or injector. Note that the contrast agent may be introduced into the subject 100 by the computer 150 as a control device controlling the operation of the introduction section 190 . Alternatively, the user may operate the introduction unit 190 to introduce the contrast medium into the subject 100 .

(Subject 100)
Although the subject 100 does not constitute a system, it will be described below. The system according to the present embodiment can be used for purposes such as diagnosing malignant tumors and vascular diseases in humans and animals, monitoring the course of chemotherapy, and the like. Therefore, the subject 100 is assumed to be a living body, specifically, a diagnosis target part such as a human body or animal breast, organs, blood vessel network, head, neck, abdomen, limbs including fingers or toes. be. For example, if the human body is the object of measurement, oxyhemoglobin or deoxyhemoglobin, blood vessels containing many of them, or new blood vessels formed in the vicinity of a tumor may be the object of the light absorber. In addition, plaque on the wall of the carotid artery may be used as a light absorber. In addition, melanin, collagen, lipids, and the like contained in the skin and the like may be used as light absorbers. Furthermore, the contrast medium introduced into the subject 100 can be a light absorber. Contrast agents used for photoacoustic imaging include dyes such as indocyanine green (ICG) and methylene blue (MB), gold particles, mixtures thereof, and externally introduced substances that accumulate or chemically modify them. You may Alternatively, a phantom imitating a living body may be used as the subject 100 .

Each component of the photoacoustic device may be configured as a separate device, or may be configured as an integrated device. Also, at least a part of the photoacoustic device may be configured as one device integrated.

Note that each device constituting the system according to the present embodiment may be configured with separate hardware, or all the devices may be configured with one piece of hardware. The functions of the system according to this embodiment may be configured with any hardware.

Next, the image generation method according to this embodiment will be described using the flowchart shown in FIG.

(S400: Step of determining wavelength of irradiation light)
A computer 150 as wavelength determining means determines the wavelength of the irradiation light based on the information on the contrast agent. In this embodiment, the combination of wavelengths is determined so that the region corresponding to the contrast agent in the spectral image can be easily identified. Note that the computer 150 can acquire information about the contrast agent input by a user such as a doctor using the input unit 170, for example. Further, the computer 150 may store information on a plurality of contrast agents in advance, and acquire information on the default contrast agent from among them.

FIG. 10 shows an example of a GUI displayed on the display unit 160. As shown in FIG. A GUI item 2500 displays examination order information such as a patient ID, an examination ID, and an imaging date and time. The item 2500 may have a display function for displaying inspection order information acquired from an external device such as HIS or RIS, and an input function for allowing the user to input inspection order information using the input unit 170 . A GUI item 2600 displays information about the contrast medium, such as the type of contrast medium and the concentration of the contrast medium. The item 2600 may have a display function for displaying information about the contrast agent acquired from an external device such as HIS or RIS, and an input function for allowing the user to input information about the contrast agent using the input unit 170. good. In the item 2600, it may be possible to input information on the contrast agent such as the type and concentration of the contrast agent by a method such as pull-down from a plurality of options. Note that the GUI shown in FIG. 10 may be displayed on the display device 1400 .

It should be noted that the image processing apparatus 1300 may acquire information on a contrast agent set by default from information on a plurality of contrast agents when an instruction to input information on the contrast agent is not received from the user. In the case of this embodiment, a case where ICG is set as the type of contrast medium and 1.0 mg/mL as the concentration of the contrast medium by default will be described. In this embodiment, the GUI item 2600 displays the type and concentration of the contrast agent set by default, but the information on the contrast agent may not be set by default. In this case, the information about the contrast medium may not be displayed in the item 2600 of the GUI on the initial screen.

Here, the change of the image value corresponding to the contrast medium in the spectral image when the combination of wavelengths is changed will be described. FIG. 7 shows simulation results of image values (oxygen saturation values) corresponding to contrast agents in spectral images for each combination of two wavelengths. The vertical and horizontal axes in FIG. 7 represent the first wavelength and the second wavelength, respectively. FIG. 7 shows contour lines of image values corresponding to the contrast agent in the spectroscopic image. FIGS. 7(a) to 7(d) show contrast agents in spectroscopic images when the concentrations of ICG are 5.04 μg/mL, 50.4 μg/mL, 0.5 mg/mL, and 1.0 mg/mL, respectively. indicates the image value corresponding to . As shown in FIG. 7, depending on the combination of selected wavelengths, the image value corresponding to the contrast agent in the spectral image may be 60% to 100%. As described above, if such a combination of wavelengths is selected, it becomes difficult to distinguish between the blood vessel region and the contrast agent region in the spectroscopic image. Therefore, it is preferable to select a combination of wavelengths such that the image value corresponding to the contrast agent in the spectral image is less than 60% or greater than 100% in the wavelength combinations shown in FIG. Furthermore, in the combination of wavelengths shown in FIG. 7, it is preferable to select a combination of wavelengths such that the image value corresponding to the contrast medium in the spectral image is a negative value.

For example, consider the case where 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength. FIG. 8 shows the relationship between the concentration of ICG and the image value (oxygen saturation value) corresponding to the contrast agent in the spectral image when 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength. graph. According to FIG. 8, when 797 nm is selected as the first wavelength and 835 nm is selected as the second wavelength, contrast enhancement in the spectroscopic image is Image values corresponding to agents are negative. Therefore, according to the spectroscopic image generated by such a combination of wavelengths, since the oxygen saturation value of the blood vessel does not take a negative value in principle, the region of the blood vessel and the region of the contrast medium can be clearly distinguished. be able to.

Hereinafter, a region in which blood vessels exist is referred to as a blood vessel region, and a region in which a contrast agent exists is also referred to as a contrast agent region. A vascular region is a region corresponding to an artery or a vein, and a contrast agent region is a region corresponding to a lymphatic vessel.

Although the wavelength is determined based on the information about the contrast medium, the absorption coefficient of hemoglobin may be taken into consideration in determining the wavelength. FIG. 9 shows spectra of the Molar absorption coefficient of oxyhemoglobin (dashed line) and the Molar absorption coefficient of deoxyhemoglobin (solid line). In the wavelength range shown in FIG. 9, the magnitude relationship between the Molar absorption coefficient of oxyhemoglobin and the Molar absorption coefficient of deoxyhemoglobin is reversed at 797 nm. That is, it can be said that veins can be easily recognized at wavelengths shorter than 797 nm, and arteries can be easily recognized at wavelengths longer than 797 nm. By the way, in the treatment of lymphedema, lymphovenous anastomosis is used to form a bypass between a lymphatic vessel and a vein. For this preoperative examination, it is conceivable to image both veins and lymphatic vessels with accumulated contrast medium by photoacoustic imaging. In this case, by setting at least one of the plurality of wavelengths to a wavelength smaller than 797 nm, veins can be imaged more clearly. Further, it is advantageous for imaging veins to set at least one of the plurality of wavelengths to a wavelength at which the Molar absorption coefficient of deoxyhemoglobin is larger than that of oxyhemoglobin. In addition, when a spectral image is generated from a photoacoustic image corresponding to two wavelengths, both of the two wavelengths are set to wavelengths in which the Molar absorption coefficient of deoxyhemoglobin is larger than the Molar absorption coefficient of oxyhemoglobin. is advantageous. By selecting these wavelengths, it is possible to accurately image both the lymphatic vessel into which the contrast medium has been introduced and the vein in the preoperative examination of the lymphatic venule anastomosis.

By the way, if all of the plurality of wavelengths are wavelengths in which the absorption coefficient of the contrast agent is larger than that of the blood, artifacts derived from the contrast agent reduce the oxygen saturation accuracy of the blood. Therefore, in order to reduce artifacts derived from the contrast agent, at least one of the plurality of wavelengths may be a wavelength at which the absorption coefficient of the contrast agent is smaller than that of blood.

Here, a case of generating a spectral image according to formula (1) has been described, but a spectral image in which the image value corresponding to the contrast agent in the spectral image changes depending on the conditions of the contrast agent and the wavelength of the irradiation light It can also be applied when generating

(S500: Step of irradiating light)
The light irradiation unit 110 sets the wavelength determined in S400 to the light source 111 . Light source 111 emits light of the wavelength determined in S400. The light generated from the light source 111 is applied to the subject 100 as pulsed light through the optical system 112 . Then, the pulsed light is absorbed inside the subject 100, and a photoacoustic wave is generated by the photoacoustic effect. At this time, the introduced contrast medium also absorbs the pulsed light and generates photoacoustic waves. The light irradiation unit 110 may transmit a synchronization signal to the signal collection unit 140 together with transmission of the pulsed light. In addition, the light irradiation unit 110 similarly performs light irradiation for each of the plurality of wavelengths.

The user may use the input unit 170 to specify control parameters such as irradiation conditions of the light irradiation unit 110 (repetition frequency and wavelength of irradiation light, etc.) and the position of the probe 180 . Computer 150 may set control parameters determined based on user instructions. Further, computer 150 may move probe 180 to a specified position by controlling drive unit 130 based on specified control parameters. When imaging at a plurality of positions is designated, the driving section 130 first moves the probe 180 to the first designated position. It should be noted that the drive unit 130 may move the probe 180 to a preprogrammed position when an instruction to start measurement is given.

(S600: Step of receiving photoacoustic waves)
Upon receiving the synchronization signal transmitted from the light irradiation section 110, the signal collection section 140 starts the signal collection operation. That is, the signal collection unit 140 generates an amplified digital electric signal by amplifying and AD converting the analog electric signal derived from the photoacoustic wave output from the receiving unit 120, and outputs it to the computer 150. . Computer 150 stores the signal transmitted from signal collector 140 . When imaging at a plurality of scanning positions is designated, the steps of S500 and S600 are repeatedly executed at the designated scanning positions, and the irradiation of pulsed light and the generation of digital signals derived from acoustic waves are repeated. Note that the computer 150 may acquire and store the position information of the receiving unit 120 at the time of light emission based on the output from the position sensor of the driving unit 130, using the light emission as a trigger.

In this embodiment, an example in which light with a plurality of wavelengths is irradiated in a time-division manner has been described, but the light irradiation method is not limited to this as long as signal data corresponding to each of the plurality of wavelengths can be obtained. . For example, when encoding is performed by light irradiation, there may be timings at which lights of a plurality of wavelengths are irradiated almost simultaneously.

(S700: Step of generating a photoacoustic image)
A computer 150 as photoacoustic image acquisition means generates a photoacoustic image based on the stored signal data. The computer 150 outputs the generated photoacoustic image to the storage device 1200 for storage. In this embodiment, one piece of volume data is generated by image reconstruction using a photoacoustic signal obtained by irradiating the object with light once. Furthermore, time-series three-dimensional volume data is obtained by performing light irradiation a plurality of times and performing image reconstruction for each light irradiation.

Reconstruction algorithms for transforming signal data into a two-dimensional or three-dimensional spatial distribution include analytical reconstruction methods such as backprojection in the time domain and backprojection in the Fourier domain, and model-based methods (repeated calculations). law) can be adopted. For example, backprojection methods in the time domain include universal back-projection (UBP), filtered back-projection (FBP), or delay-and-sum.

In this embodiment, one three-dimensional photoacoustic image (volume data) is generated by image reconstruction using a photoacoustic signal obtained by a single light irradiation of the subject. Furthermore, by performing light irradiation a plurality of times and performing image reconstruction for each light irradiation, time-series three-dimensional image data (time-series volume data) is acquired. Three-dimensional image data obtained by image reconstruction for each light irradiation of multiple times of light irradiation is generically referred to as three-dimensional image data corresponding to multiple times of light irradiation. Since light irradiation is performed a plurality of times in time series, the three-dimensional image data corresponding to the light irradiation a plurality of times forms time-series three-dimensional image data.

The computer 150 generates initial sound pressure distribution information (generated sound pressures at a plurality of positions) as a photoacoustic image by performing reconstruction processing on the signal data. Further, the computer 150 calculates the light fluence distribution inside the subject 100 of the light irradiated to the subject 100, divides the initial sound pressure distribution by the light fluence distribution, and converts the absorption coefficient distribution information into photoacoustic You may acquire it as an image. A known method can be applied as a method of calculating the light fluence distribution. In addition, the computer 150 can generate photoacoustic images corresponding to each of the multiple wavelengths of light. Specifically, the computer 150 can generate a first photoacoustic image corresponding to the first wavelength by performing reconstruction processing on signal data obtained by light irradiation of the first wavelength. Further, the computer 150 can generate a second photoacoustic image corresponding to the second wavelength by performing reconstruction processing on signal data obtained by light irradiation of the second wavelength. Thus, the computer 150 can generate multiple photoacoustic images corresponding to multiple wavelengths of light.

In this embodiment, the computer 150 acquires absorption coefficient distribution information corresponding to each of light of a plurality of wavelengths as a photoacoustic image. Let the absorption coefficient distribution information corresponding to the first wavelength be a first photoacoustic image, and let the absorption coefficient distribution information corresponding to the second wavelength be a second photoacoustic image.

In this embodiment, an example in which the system includes the photoacoustic device 1100 that generates a photoacoustic image has been described, but the present invention can also be applied to a system that does not include the photoacoustic device 1100 . The present invention can be applied to any system as long as the image processing device 1300 as a photoacoustic image acquiring means can acquire a photoacoustic image. For example, the present invention can be applied to a system that does not include the photoacoustic device 1100 but includes the storage device 1200 and the image processing device 1300 . In this case, the image processing device 1300 as a photoacoustic image acquiring means can acquire a photoacoustic image by reading out a specified photoacoustic image from among a group of photoacoustic images stored in advance in the storage device 1200. can.

(S800: Step of generating spectral image)
A computer 150 as spectral image acquisition means generates spectral images based on a plurality of photoacoustic images corresponding to a plurality of wavelengths. The computer 150 outputs the spectroscopic image to the storage device 1200 and stores it in the storage device 1200 . As described above, the computer 150 may generate, as a spectroscopic image, an image showing information corresponding to the concentration of substances that make up the subject, such as glucose concentration, collagen concentration, melanin concentration, and volume fractions of fat and water. good. Further, the computer 150 may generate an image representing the ratio between the first photoacoustic image corresponding to the first wavelength and the second photoacoustic image corresponding to the second wavelength as the spectral image. In this embodiment, an example will be described in which the computer 150 generates an oxygen saturation image as a spectroscopic image according to Equation (1) using the first photoacoustic image and the second photoacoustic image.

Note that the image processing device 1300 as spectral image acquisition means may acquire a spectral image by reading out a specified spectral image from a group of spectral images pre-stored in the storage device 1200 . In addition, the image processing device 1300 as a photoacoustic image acquisition means reads out from among the photoacoustic image groups pre-stored in the storage device 1200, at least one of the plurality of photoacoustic images used to generate the spectral image. You may acquire a photoacoustic image by reading.

Time-series three-dimensional image data corresponding to the multiple times of light irradiation is generated by performing multiple times of light irradiation and subsequent acoustic wave reception and image reconstruction. Photoacoustic image data and spectral image data can be used as the three-dimensional image data. The photoacoustic image data here refers to image data showing the distribution of absorption coefficients, etc., and the spectroscopic image data refers to photoacoustic image data corresponding to each wavelength when the subject is irradiated with light of multiple wavelengths. It refers to image data indicating density and the like generated based on the image data.

(S1100: Step of displaying spectral image)
The image processing device 1300 as a display control unit causes the display device 1400 to display a spectral image based on the information on the contrast agent so that the region corresponding to the contrast agent and the other region can be distinguished. As a rendering method, the maximum intensity projection method (MIP: Maximum
Intensity Projection), volume rendering, and surface rendering can be employed. Here, the setting conditions such as the display area and line-of-sight direction when rendering the three-dimensional image two-dimensionally can be arbitrarily specified according to the observation target.

Here, a case will be described in which 797 nm and 835 nm are set in S400 and a spectral image is generated according to Equation (1) in S800. As shown in FIG. 8, when these two wavelengths are selected, the image value corresponding to the contrast agent in the spectroscopic image generated according to equation (1) is negative regardless of the concentration of the ICG. .

Incidentally, the oxygen saturation in blood vessels (arteries and veins) in the living body falls within a range of approximately 60% to 100% expressed as a percentage. Therefore, for the wavelengths (two wavelengths) of light irradiated to the subject, the oxygen saturation value (calculated value of formula (1)) corresponding to the contrast agent in the spectral image is less than 60%, or more than 100%. It is preferable to set the wavelength so that it becomes large. By doing so, it becomes easy to distinguish between the image corresponding to the artery and vein and the image corresponding to the contrast agent in the spectroscopic image. For example, when using ICG as a contrast agent, two wavelengths of 700 nm or more and less than 820 nm and a wavelength of 820 nm or more and 1020 nm or less are selected, and a spectroscopic image is generated according to formula (1). and blood vessel regions can be distinguished well.

As shown in FIG. 10, the image processing device 1300 causes the GUI to display a color bar 2400 as a color scale indicating the relationship between the image values of the spectral images and the display colors. The image processing apparatus 1300 determines the numerical range of image values to be assigned to the color scale based on information about the contrast agent (for example, information indicating that the type of contrast agent is ICG) and information indicating the wavelength of the irradiation light. may decide. For example, the image processor 1300 may determine a numerical range that includes negative image values corresponding to arterial oxygen saturation, venous oxygen saturation, and contrast agents. The image processing apparatus 1300 may determine the numerical range of -100% to 100% and set the color bar 2400 by assigning -100% to 100% to the color gradation that changes from blue to red. With such a display method, in addition to identifying arteries and veins, regions corresponding to negative contrast agents can also be identified. Further, the image processing apparatus 1300 may display an indicator 2410 indicating the numerical range of image values corresponding to the contrast agent based on the information regarding the contrast agent and the information indicating the wavelength of the irradiation light. Here, in the color bar 2400, an indicator 2410 indicates a negative value area as the numerical range of the image values corresponding to the ICG. By displaying the color scale so that the display color corresponding to the contrast agent can be identified in this manner, the region corresponding to the contrast agent in the spectral image can be easily identified.

The image processing device 1300 as region determining means may determine the region corresponding to the contrast agent in the spectral image based on information about the contrast agent and information indicating the wavelength of the irradiation light. For example, the image processing apparatus 1300 may determine a region having a negative image value in the spectral image as a region corresponding to the contrast medium. Then, the image processing device 1300 may cause the display device 1400 to display the spectral image so that the region corresponding to the contrast agent and the other region can be distinguished. The image processing apparatus 1300 makes the area corresponding to the contrast medium different in display color from the other area, blinks the area corresponding to the contrast medium, and displays an indicator (for example, a frame) indicating the area corresponding to the contrast medium. It is possible to adopt an identification display such as displaying.

Note that it may be possible to switch to a display mode for selectively displaying image values corresponding to the ICG by designating an item 2730 corresponding to the display of the ICG displayed on the GUI shown in FIG. For example, when the user selects the item 2730 corresponding to the display of ICG, the image processing device 1300 selects voxels with negative image values from the spectral image, and selectively renders the selected voxels. Regions of the ICG may be selectively displayed. Similarly, the user may select item 2710 corresponding to the display of arteries and item 2720 corresponding to the display of veins. Based on a user's instruction, the image processing device 1300 selectively selects an image value corresponding to arteries (eg, 90% or more and 100% or less) or an image value corresponding to veins (eg, 60% or more and less than 90%). You may switch to the display mode to display. The numerical ranges of the image values corresponding to arteries and the image values corresponding to veins may be changeable based on user instructions.

At least one of hue, brightness, and saturation is assigned to the image value of the spectral image, and the image obtained by assigning the remaining parameters of hue, brightness, and saturation to the image value of the photoacoustic image is displayed as a spectral image. may For example, an image obtained by assigning hue and saturation to the image value of the spectral image and assigning brightness to the image value of the photoacoustic image may be displayed as the spectral image. At this time, if the image value of the photoacoustic image corresponding to the contrast agent is larger or smaller than the image value of the photoacoustic image corresponding to the blood vessel, assigning the brightness to the image value of the photoacoustic image will give the blood vessel and the contrast agent It may be difficult to see both Therefore, the image value-to-brightness conversion table of the photoacoustic image may be changed according to the image value of the spectral image. For example, if the image values of the spectral image are included in the numerical range of the image values corresponding to the contrast agent, the brightness corresponding to the image values of the photoacoustic image may be made smaller than that corresponding to the blood vessel. That is, if the image values of the photoacoustic image are the same when the contrast agent region and the blood vessel region are compared, the brightness of the contrast agent region may be lower than that of the blood vessel region. Here, the conversion table is a table indicating brightness corresponding to each of a plurality of image values. Further, when the image value of the spectral image is included in the numerical range of the image value corresponding to the contrast agent, the brightness corresponding to the image value of the photoacoustic image may be made higher than that corresponding to the blood vessel. That is, if the image values of the photoacoustic image are the same when the contrast agent region and the blood vessel region are compared, the brightness of the contrast agent region may be made higher than that of the blood vessel region. Further, the numerical range of the image values of the photoacoustic image in which the image values of the photoacoustic image are not converted into brightness may differ depending on the image value of the spectral image.

The conversion table may be changed to one suitable for the type and concentration of the contrast agent and the wavelength of the irradiation light. Therefore, the image processing apparatus 1300 may determine a conversion table for converting the image value of the photoacoustic image into the brightness based on the information about the contrast agent and the information indicating the wavelength of the irradiation light. When the image value of the photoacoustic image corresponding to the contrast agent is estimated to be larger than that corresponding to the blood vessel, the image processing device 1300 sets the brightness corresponding to the image value of the photoacoustic image corresponding to the contrast agent to the blood vessel. It may be smaller than the corresponding one. Conversely, when the image value of the photoacoustic image corresponding to the contrast agent is estimated to be smaller than that corresponding to the blood vessel, the image processing device 1300 sets the brightness corresponding to the image value of the photoacoustic image corresponding to the contrast agent. may be larger than that corresponding to the blood vessel.

The GUI shown in FIG. 10 includes an absorption coefficient image (first photoacoustic image) 2100 corresponding to a wavelength of 797 nm, an absorption coefficient image (second photoacoustic image) 2200 corresponding to a wavelength of 835 nm, and an oxygen saturation image (spectral image) 2300. display. The GUI may indicate which wavelength of light is used to generate each image. Although both the photoacoustic image and the spectral image are displayed in this embodiment, only the spectral image may be displayed. Further, the image processing apparatus 1300 may switch between display of the photoacoustic image and display of the spectral image based on the user's instruction.

Note that the display unit 160 may be capable of displaying moving images. For example, the image processing device 1300 generates at least one of the first photoacoustic image 2100, the second photoacoustic image 2200, and the spectral image 2300 in time series, and generates moving image data based on the generated time-series images. It may be configured to generate and output to the display unit 160 . In view of the fact that the number of times that lymph flows is relatively small, it is also preferable to display as a still image or a time-compressed moving image in order to shorten the user's judgment time. In addition, in moving image display, it is also possible to repeatedly display how lymph flows. The moving image speed may be a predetermined speed defined in advance or a predetermined speed specified by the user.

Also, it is preferable that the frame rate of the moving image is made variable in the display unit 160 capable of displaying the moving image. In order to make the frame rate variable, a window for the user to manually input the frame rate, a slide bar for changing the frame rate, or the like may be added to the GUI of FIG. Here, since the lymph intermittently flows through the lymphatic vessel, only a part of the obtained time-series volume data can be used to confirm the flow of the lymph. Therefore, if the real-time display is performed when confirming the flow of lymph, the efficiency may decrease. Therefore, by making the frame rate of the moving image displayed on the display unit 160 variable, it becomes possible to fast-forward the displayed moving image, so that the user can check the state of the fluid in the lymphatic vessel in a short time. Become.

Moreover, the display unit 160 may be capable of repeatedly displaying moving images within a predetermined time range. At that time, it is also preferable to add a GUI such as a window or a slide bar to FIG. 10 so that the user can specify the range of repeated display. This makes it easier for the user to grasp how the fluid flows in the lymphatic vessels, for example.

The flow of fluid in the lymphatic vessel is displayed on the display unit 160 as flow information in the lymphatic area. The method of displaying flow information in the area of lymphatic vessels is not limited to the above. For example, the image processing device 1300 as a display control means associates the flow information in the lymphatic vessel area with the lymphatic vessel area, and uses at least one of brightness display, color display, graph display, and numerical display, They may be displayed on the same screen of the display device 1400 . In addition, the image processing device 1300 as the display control means may highlight at least one lymphatic vessel region.

(S1200: Step of displaying lymphatic vessel classification results)
In S1200, the image processing device 1300 as a state estimation means analyzes the image data, automatically extracts the area of the lymphatic vessel, and classifies the lymphatic vessel. The image processing device 1300 as display control means causes the display device 1400 to display the classification result of the lymph vessels.

The image processing device 1300 as a state estimating means extracts the lymphatic vessel region in the subject by image analysis of the spectroscopic image generated in S800. In a spectroscopic image, for example, it is possible to distinguish between lymphatic vessels and veins in the subject based on the calculated value of Equation (1). can be done.

The image processing device 1300 as state estimation means classifies the extracted lymphatic vessels by analyzing the spectroscopic image. The image processing apparatus 1300 may, for example, divide a lymphatic vessel into a plurality of divided regions, and classify each divided region by judging and classifying states such as shooting star, contraction, retention, retention, DBF (dermal backflow), and the like. Shooting Star is a healthy condition in which lymph flows like a shooting star. Constriction is a condition in which the width of certain parts of the lymphatic vessels changes, pumping lymph (fluid). Retention is a state in which there is a period of time during which no lymphatic flow is observed. Stagnation is a condition in which little lymph flows.

DBF is a condition in which lymph flows backward toward the skin. DBF also includes conditions of interstitial leakage and lymphangiectasia. Interstitial leak is a condition in which lymph leaks back into the interstitium. Lymphangiectasia is a condition in which refluxing lymph remains in dilated lymphatic capillaries or pre-collecting lymphatic vessels.

The image processing device 1300 is not limited to the state of lymph vessels, and can detect lymph vessels based on the number of lymph vessels existing per unit area, the abundance ratio of lymph vessels per unit area, or the abundance ratio of lymph vessels per unit volume. can be classified. The number of lymph vessels present per unit area, the ratio of lymph vessels per unit area, and the ratio of lymph vessels per unit volume are hereinafter also referred to as the number of lymph vessels present, the area ratio, and the volume ratio. In addition, the image processing apparatus 1300 may classify the lymphatic vessels based on the distance between the lymphatic vessels and the veins or the depth from the subject's skin.

Note that the lymphatic vessel regions may be automatically classified as described above, or may be classified manually. In the case of manual classification, the image processing device 1300 as identifying means can identify a part of the area of the lymphatic vessel and classify the identified area according to the user's instruction.

The image processing device 1300 as display control means causes the display device 1400 to display the classification result of the lymph vessels. The image processing apparatus 1300 may, for example, display the lymphatic vessel region with a hue corresponding to the state of each divided region. In addition, the image processing apparatus 1300 may display the number of existing lymph vessels, the area ratio, or the volume ratio for each unit area in the subject so that the user can check it. The image processing device 1300 may display the distance between lymphatic vessels and veins and the depth of the lymphatic vessels and veins from the skin.

The image processing device 1300 as a storage control unit associates the lymphatic vessel classification results with the analyzed image data and patient information, and stores them in the storage device 1200 . When displaying image data or patient information on the display device 1400, the image processing device 1300 can acquire the classification results of the corresponding lymph vessels from the storage device 1200 and display them together with the image data.

As described above, at least one of the image processing device 1300 and the computer 150 as an information processing device includes spectral image acquisition means, region determination means, photoacoustic image acquisition means, state estimation means, identification means, display control means and It functions as a device having at least one storage control means. Note that each means may be composed of different hardware, or may be composed of one piece of hardware. Moreover, a plurality of means may be configured by one piece of hardware.

In this embodiment, by selecting a wavelength at which the image value corresponding to the contrast agent is a negative value, it is possible to distinguish between the blood vessel and the contrast agent. The image value corresponding to the contrast agent can be any value as long as it can identify . For example, the image processing described in this step can be applied even when the image value of the spectral image (oxygen saturation image) corresponding to the contrast agent is smaller than 60% or larger than 100%. .

(Example 1)
In the first embodiment, the image processing apparatus 1300 automatically detects lymphatic vessels by analyzing image data generated based on received signal data of photoacoustic waves generated from within the subject by irradiating the subject with light. Classify and estimate lymphatic vessel status. The image processing device 1300 causes the display device 1400 to display the classification result. The image processing method according to the first embodiment will be described below with reference to the flowchart shown in FIG.

(S1211: Step of Extracting Lymphatic Region)
The image processing device 1300 as a state estimating means extracts the area of the lymphatic vessel from the image data. Image data for extracting a lymphatic region can be, for example, a spectroscopic image generated using a plurality of photoacoustic images corresponding to a plurality of wavelengths. FIG. 11 is a diagram exemplifying a spectroscopic image of a subject. A method of acquiring the spectral image shown in FIG. 11 will be described later. In the spectroscopic image illustrated in FIG. 11, both the lymphatic vessel A1 and the vein A2 into which the contrast medium has been introduced are imaged. Lymphatic vessels A1 and veins A2 are distinguishably visible by assigning at least one of hue, brightness, and saturation to their respective image values. Therefore, the image processing apparatus 1300 can extract the lymphatic vessel region by image analysis. The image data for extracting the lymphatic vessel region may be a photoacoustic image derived from a single wavelength. Lymphatic vessels can also be imaged in photoacoustic images derived from a single wavelength, and the image processing apparatus 1300 can extract the lymphatic vessel region by image analysis. An example of a lymphatic vessel extraction method using a photoacoustic image derived from a single wavelength will be described. Among the images containing the image data group corresponding to each of the multiple times of light irradiation, the region where the image value changes greatly in the photoacoustic image within a predetermined period reflects the above-described intermittent lymph flow. Therefore, it is possible to consider the region to be the region of the lymphatic vessel. In addition, in the photoacoustic image as a three-dimensional image, the reference value of the image value derived from hemoglobin and contrast agent according to the depth and thickness of the structure is stored in the computer 150 in advance. can be identified.

(S1212: Step of Classifying Lymphatic Vessels)
Lymphatic vessels are classified based on various indices such as the state of lymphatic flow and the distance to veins. By checking these classification results, the user can specify the lymphatic vessels to be anastomosed in anastomotic surgery to connect lymphatic vessels and veins. A method for classifying lymphatic vessels is exemplified below.

[Lymphatic vessel classification method 1]
Using FIG. 12, a method of classifying lymphatic vessels using the state of the lymphatic vessels as an index will be described. Here, an example is shown in which the state of the lymphatic vessel is determined based on the temporal change in luminance value. Lymphatic vessel A1 and vein A2 are shown in FIG. The image processing device 1300 divides the lymphatic vessel A1 into predetermined lengths and extracts divided regions A101, A102, and A103. The segmented regions A101, A102, A103 are approximated, for example, by Hesse matrices, gradient vectors or Hough transforms to determine the major and minor axis directions, respectively.

For example, it can be determined that a segmented region in which a portion with a higher luminance value moves in the longitudinal direction over time is in the Shooting Star state. In addition, it can be determined that a segmented region in which a portion with a higher luminance value narrows or widens in the minor axis direction is in a contracted state. It can be determined that a segmented region having a time period in which the brightness value does not change is in a staying state. A segmented region whose luminance value does not change can be determined to be in a stationary state.

When the segmented region is in the state of DBF, whether it is interstitial leakage or lymphatic dilation can be determined by, for example, the spatial frequency of the image. If the spatial frequency of the image is lower than the threshold, it can be determined that there is a state of interstitial leakage, and if it is higher than the threshold, it can be determined that there is a state of lymphatic dilation.

In this way, lymphatic vessels can be classified according to their condition. The user can judge the health of the lymphatic vessels based on the state of the lymphatic vessels, select the lymphatic vessels to be anastomosed, and determine the anastomosis position.

In this example, the time change of the luminance value in the image is used, but the state of the lymphatic vessels can be determined based on the information corresponding to the image value such as the above-mentioned hue, brightness, and saturation in addition to the luminance value. good too. That is, in this example, it can be said that the state of each divided area is determined based on the time change of the image value in each divided area.

[Lymphatic vessel classification method 2]
With reference to FIG. 13, a method of classifying lymphatic vessels using the number of lymphatic vessels existing per unit area, the area ratio, and the volume ratio as indices will be described. FIG. 13 shows three lymphatic vessels A1a, A1b, and A1c. Each square block shown in FIG. 13 indicates a region corresponding to a unit area. The image processing apparatus 1300 analyzes the image data to calculate the number of lymphatic vessels present and the area ratio to the unit area for each unit area (for example, 2 cm ² ) of the subject. When the image data is an image representing a three-dimensional spatial distribution, the image processing apparatus 1300 can calculate the volume ratio of lymphatic vessels to a unit area (per unit volume).

In the example of FIG. 13, each block is color-coded according to the number of lymphatic vessels present. That is, a block B1 having two lymphatic vessels, a block B2 having one lymphatic vessel, and a block B3 having no lymphatic vessel are shown in different colors. The image processing device 1300 displays each block in different colors according to the area ratio of lymph vessels to a unit area or the volume ratio of lymph vessels to a unit volume, not limited to the number of lymph vessels existing per unit area. good too.

Thus, lymphatic vessels can be classified using the number per unit area, the area ratio, and the volume ratio as indicators. The user can select the lymphatic vessels to be anastomosed and determine the anastomotic position by considering the number of existing lymphatic vessels, area ratio, and volume ratio.

[Lymphatic vessel classification method 3]
Using FIG. 14, a method of classifying lymphatic vessels using the distance between lymphatic vessels and veins as an index will be described. The image processing device 1300 extracts the lymphatic vessel A1 and the vein A2 displayed in the image data and calculates the distance between them. The image processing device 1300 can display the distance between the lymphatic vessel A1 and the vein A2 as shown in FIG. The distance between the lymphatic vessel A1 and the vein A2 may be a distance in an image representing a two-dimensional spatial distribution, or may be a distance in an image representing a three-dimensional spatial distribution. A user may specify the position where the distance between the lymphatic vessel A1 and the vein A2 is displayed. Also, the distance between the lymphatic vessel A1 and the vein A2 may be displayed at predetermined intervals along the lymphatic vessel A1. In this case, the image processing apparatus 1300 may not display the distance at a position where the distance between the lymphatic vessel A1 and the vein A2 exceeds a predetermined threshold.

Furthermore, the image processing apparatus 1300 may highlight the positions (A111 and A112 in FIG. 14) where the lymphatic vessel A1 and the vein A2 intersect in plan view. Further, a position where the calculated distance between the lymphatic vessel A1 and the vein A2 in the three-dimensional image data is short may be highlighted. Also, the lymphatic vessels A1 and veins A2 may be displayed by allocating luminance according to the depth from the skin. In this way, lymphatic vessels can also be classified using the distance from veins and the depth from the skin as indices. The user can select the lymphatic vessel to be anastomosed and determine the anastomosis position based on the distance between the lymphatic vessel and the vein or the depth from the skin. The user can select each index described above according to the position of the region of interest. The image processing device 1300 can cause the display device 1400 to display the lymphatic vessel regions classified by the selected index. In addition, a position where the calculated distance between the lymphatic vessel and the vein is short in the three-dimensional image data may be highlighted.

(S1213: Step of displaying classification results)
The image processing device 1300 as display control means can display each segmented region of the lymphatic vessel in a hue corresponding to the state when the state of the lymphatic vessel is used as an index (lymphatic vessel classification method 1). When the image processing apparatus 1300 classifies the number of lymph vessels existing per unit area, the area ratio, and the volume ratio as indices (lymphatic vessel classification method 2), each block indicating a unit area is classified into the number of existing lymph vessels, the area It may be displayed in a hue according to the value of the ratio or volume ratio. The image processing apparatus 1300 may display the distance between the lymphatic vessel and the vein when the distance between the lymphatic vessel and the vein is used as an index for classification (lymphatic vessel classification method 3). The image processing device 1300 may indicate the distance between the lymphatic vessel and the vein, and may assign at least one of brightness value, hue, brightness, and saturation according to the depth from the skin and display them. At this time, the index assigned to the information about the depth from the skin is preferably an index distinguishable from other information from the viewpoint of visibility. For example, when hue is assigned to information indicating the state of lymphatic vessels, an index other than hue is assigned to information about depth from the skin. In other words, at least one of luminance value, hue, brightness, and saturation is assigned to the image value of the lymphatic vessel region, and luminance values and hues other than those assigned to the lymphatic vessel status are assigned. At least one of , brightness, and saturation is assigned to information about depth from the subject's skin.

In addition, the image processing device 1300 evaluates indicators such as the condition of the lymphatic vessels, the number of existing lymphatic vessels per unit area, the distance from the vein, and the depth from the skin, and highlights the lymphatic vessels and veins suitable for anastomosis. can be A lymphatic vessel suitable for anastomosis is preferably a lymphatic vessel in a healthy state (e.g., Shooting Star state) with flowing lymph, a shorter distance to veins, and a shallower depth from the skin. . The image processing device 1300 identifies lymph vessels that satisfy predetermined conditions such as the condition of the lymph vessels, the number of existing lymph vessels per unit area, the distance from the vein, and the depth from the skin as lymph vessels suitable for anastomosis. can do. Image processing device 1300 may further assess whether a lymphatic vessel is suitable for anastomosis based on conditions in regions upstream and downstream of the lymphatic vessel. By highlighting the lymph vessels suitable for anastomosis, the user can select the lymph vessels more suitable for anastomosis.

(S1214: Step of saving data)
The image processing apparatus 1300 as a storage control unit may associate the lymphatic vessel classification results in S1212 with the analyzed image data and patient information and store them in the storage device 1200 . In this case, the image processing device 1300 can display the lymphatic vessel classification results stored in the storage device 1200 on the display device 1400 together with the image data. The image processing apparatus 1300 can display the lymphatic vessel classification results in a form (for example, FIGS. 13 and 14) according to the index selected by the user. The user can repeatedly check the lymphatic vessel classification results linked to patient information. The patient information may include, in addition to the above-described patient ID, information related to physical and chemical therapy performed on the patient. This makes it easier for the user to grasp changes in the state of lymphatic vessels that accompany physical and chemical therapy. Further, an interface may be added to the GUI shown in FIGS. 10 and 16 so that the user can select which mode to adopt.

(Method of acquiring spectral image)
The acquisition method (first acquisition method) of the spectral image shown in FIG. 11 will be described below.
Since the spectroscopic image can visualize the region where the contrast agent introduced into the body exists, the lymphatic vessels into which the contrast agent has been introduced can be visualized. However, there are cases where the position of the lymphatic vessel cannot be shown correctly with only one image. This is because the flow of lymph fluid is not constant like blood.

Blood is constantly circulated by the beating of the heart, but lymphatic vessels do not have a common organ that acts as a pump. , lymphatic fluid transport takes place. In addition to the contraction of the smooth muscle of the lymphatic vessel wall that occurs once every several tens of seconds to several minutes, the lymph fluid is affected by muscle contraction that occurs with human movement, pressure caused by relaxation, pressure changes caused by breathing, and external pressure. Move due to massage stimulation, etc. Therefore, the movement timing of the lymph fluid is not constant, and becomes an intermittent flow with an irregular feeling such as once every several tens of seconds to several minutes. Even if a spectroscopic image is acquired at the timing when the lymph fluid is not moving, there is not enough contrast agent in the lymphatic vessel, so the lymphatic vessel cannot be visualized or only a part of the lymphatic vessel can be visualized. It is feared that In other words, only one frame image in the moving image can render only the portion of the lymphatic vessel where the contrast medium is present.

Therefore, in the system according to the present embodiment, a plurality of time-series spectral images (a plurality of first image data) are acquired in a predetermined period of time, and based on the acquired plurality of spectral images, lymphatic vessels are identified. Extract the existing regions (ie, the regions through which the contrast agent passes). In this embodiment, the photoacoustic device 1100 acquires a plurality of time-series spectral images and stores them in the storage device 1200 in the processing of steps S500 to S800. It should be noted that the predetermined period is preferably longer than the cycle in which lymph fluid movement occurs (for example, longer than about 40 seconds to 2 minutes).

Step S800 is a step of generating a moving image based on a plurality of spectral images.
By displaying a plurality of spectroscopic images as moving images, the user of the device can observe how the lymph moves. However, since lymph intermittently flows through lymph vessels, only some of the spectral images obtained in time series can be used to confirm the flow of lymph. That is, when spectroscopic images are displayed using only moving images, the user has to keep looking at the screen until the movement of lymph fluid occurs. Furthermore, since the lymph fluid (contrast medium) moves for a short period of time per time, it is difficult for the user to accurately grasp the position of the lymph vessel on the screen.

Therefore, in this embodiment, after executing step S800, the image processing apparatus 1300 generates a still image (second image data) indicating the position of the lymphatic vessel based on a plurality of spectral images. The spectral image of FIG. 11 shows the positions of the lymphatic vessels identified in this way.

Next, after the process of step S800 is completed, the image processing apparatus 1300 acquires a plurality of spectral images (multiple frames of spectral images) stored in the storage device 1200, and displays an image representing a region in which lymphatic vessels exist. Generate.

In this step, first, regions having image values within a predetermined range are extracted from each of a plurality of spectral images obtained in time series. In the example described above, a set of pixels whose image value, which is the calculated value of equation (1), is a negative value is extracted. As a result, as shown in FIG. 18A, regions (indicated by black lines) are extracted for each frame of the moving image, that is, for each spectral image forming the moving image. The extracted regions are the regions where the contrast agent is present in each frame. Note that FIG. 18 exemplifies a two-dimensional image, but if the spectral image is a three-dimensional spectral image, the region may be extracted from within the three-dimensional space.

Then, the regions obtained for each frame are superimposed (synthesized) to generate a region corresponding to the lymphatic vessel. By superimposing the regions shown in FIG. 18(A), a region (reference numeral 1101) corresponding to the lymphatic vessel is obtained as shown in FIG. 18(B).

The image processing device 1300 generates and outputs an image (second image data) representing the position of the lymphatic vessel based on the region thus generated. When generating an image representing the position of the lymphatic vessel, the hue corresponding to the original image value (that is, the image value of the spectral image) may be given, or the image may be highlighted by applying a unique marking. good too. Alternatively, luminance corresponding to the absorption coefficient may be given. The absorption coefficient can be obtained from the photoacoustic image used when generating the spectroscopic image.

The generated image may be output on the same screen as the GUI shown in FIG. 10, or may be output on a different screen. The second image may be a three-dimensional image or a two-dimensional image. Further, an interface for storing the second image data generated as described above in the image server 1210, the storage device 1200, or the like may be added to the GUI shown in FIG. Since the second image data has a smaller amount of data than the first image data, which is a moving image, the position of the lymphatic vessel can be easily grasped even when using a terminal with a low processing capability. can be done.

According to the first spectroscopic image acquisition method, it is possible to provide a user such as a doctor with a still image representing the position of a lymphatic vessel. Lymph fluid (contrast agent) moves periodically, so simply adding (or averaging) a plurality of spectroscopic images cannot accurately present the position of the lymph vessel. On the other hand, in the present embodiment, an area having an image value within a predetermined range is extracted from each frame of the spectral image and synthesized, so information in the time direction is compressed. As a result, the position of the lymphatic vessel can be drawn accurately.

It should be noted that in the illustrated embodiment, an area in which the image value of the spectral image is within a predetermined range is extracted, but an area may be extracted using other conditions as well. For example, a photoacoustic image (an image representing an absorption coefficient) corresponding to the spectral image may be referred to, and regions whose brightness values are below a predetermined threshold may be excluded. This is because even if the image value of the spectral image is within a predetermined range, there is a high possibility that a region with a small absorption coefficient is noise. Also, the luminance value threshold for filtering may be changeable by the user.

Further, in the present embodiment, the wavelengths (two wavelengths) of the irradiation light are selected so that the image value of the pixel corresponding to the blood vessel region becomes positive and the image value of the pixel corresponding to the contrast medium region becomes negative. Any two wavelengths may be selected such that the signs of both image values in the spectral image are opposite.

(Another way to acquire spectroscopic images)
Another method (second acquisition method) for acquiring the spectral image shown in FIG. 11 will be described below.
In the method described above, in the process after step S800, region extraction processing was performed on each frame of spectral images acquired in time series, and a plurality of extracted regions were synthesized. On the other hand, it is conceivable to refer to a plurality of frames of spectral images acquired in time series and directly extract a region that satisfies the conditions within a predetermined period.

In this example, in the process after step S800, a plurality of spectral images included within a predetermined period are selected, and an area (image areas with negative values) are extracted. A region in which the image value falls within a predetermined range within a predetermined period can be said to be a region through which the contrast agent has passed. It should be noted that the predetermined period is preferably longer than the cycle in which lymph fluid movement occurs (for example, longer than about 40 seconds to 2 minutes).

FIG. 19 is a diagram exemplifying the temporal change in the image value of a certain pixel P(x, y) in the spectral image within a predetermined period. The illustrated pixels are subject to extraction because their image values are within a predetermined range.
Thus, contrast agent regions may be extracted based on image values that change within a predetermined period of time. Note that when performing the determination, peak hold of the image value of the photoacoustic image may be performed within a predetermined period.

As a countermeasure against noise, even in the second acquisition method, regions in which the absorption coefficient is below a predetermined value may be excluded in the same manner as in the first acquisition method. That is, a region in which the image value of the spectral image is within a predetermined range and the brightness of the corresponding photoacoustic image exceeds the threshold may be extracted.
Also, as a countermeasure against noise, an area after a certain period of time has passed while the above-described conditions are satisfied may be extracted. Also, the above-mentioned fixed time period may be adjustable by the user.

(Example 2)
In the first embodiment, the image processing apparatus 1300 automatically classifies the lymph vessels and estimates the state of the lymph vessels by image analysis of image data including areas of lymph vessels. On the other hand, in the second embodiment, the user specifies a part of the lymphatic vessel area in the image data including the lymphatic vessel area, and determines the state of the specified area (hereinafter referred to as the region of interest). The image processing device 1300 displays on the display device 1400 an input interface for the user to input information such as the determination result for the region of interest. A user can input data about the region of interest, such as the state of the region of interest and findings about the region of interest, via the input interface. Information input by the user is stored in the storage device 1200 in association with the image data. Information input by the user may be stored in the storage device 1200 in association with the corresponding region of interest. The image processing method according to the second embodiment will be described below with reference to the flowchart shown in FIG.

(S1221: Step of Identifying Lymphatic Region)
The image processing apparatus 1300 as the identifying means first extracts the lymphatic vessel region from the image data, as in the step of S1211 in the first embodiment. The image processing apparatus 1300 identifies a portion of the extracted lymphatic vessel region as a region of interest. The image processing apparatus 1300 can set, for example, a region of a predetermined length including the position designated by the user as the region of interest. By repeating the processing shown in FIG. 15, the image processing apparatus 1300 can divide the region of the lymphatic vessel into a plurality of regions of interest and accept input of information for each region of interest.

The image processing device 1300 may identify the region of interest based on the user's instruction. For example, in the image data displayed on the display device 1400, the user can indicate the position of the region of interest by pointing with a pointing device such as a mouse to the region to be specified among the regions of the lymphatic vessels. For example, the image processing apparatus 1300 may identify a region of a predetermined length including the position designated by the user as the region of interest. Also, the image processing apparatus 1300 may specify the region of interest by allowing the user to specify the positions of the start point and the end point.

A GUI for the user to indicate the position of the region of interest will be described with reference to FIG. An item 3100 displays image data to be analyzed. In the example of FIG. 16, item 3100 displays lymphatic vessel A1 and vein A2. Note that the image data displayed on the item 3100 may be a moving image.

The user indicates with the mouse a position to be specified as the region of interest. In the example of FIG. 16, the location pointed by the user is indicated by arrow 3110 . The image processing apparatus 1300 enlarges and displays a square area centered on the position pointed by the user, that is, an area 3120 surrounded by dotted lines on the item 3200 . The region of lymphatic vessels contained within region 3120 is the identified region of interest. The size of the region of interest (the length of the specified lymphatic vessel) may be specified by the user, or may be determined by the image processing device 1300 to a predetermined size. If the region of interest specified by the user includes a plurality of lymph vessels, the image processing apparatus 1300 may change the region of interest so that only one of the lymph vessels is included. A moving image corresponding to the area 3120 may be displayed on the item 3200 . Furthermore, when the image shown in item 3100 is a moving image, by synchronizing the image with the moving image in area 3120, the user can observe the image at the same time. Item 3300 and item 3400 will be described in step S1222.

(S1222: Step of accepting input of lymphatic vessel classification)
The image processing device 1300 as display control means displays on the display device 1400 an input interface for receiving an input for the region of interest specified in S1221. An item 3300 and an item 3400 illustrated in FIG. 16 correspond to an input interface that accepts input for the region of interest.

Item 3300 is an input interface for inputting the state of the region of interest displayed in item 3200 . Item 3300 includes tabs for "running lymphatics" and "DBF". FIG. 16 shows a state in which the "running lymph vessel" tab is selected. In item 3300, the user can select one of Shooting Star, contraction, dwell, and dwell as the state of the region of interest. Also, when the "DBF" tab is selected, interstitial leak and lymphatic dilation conditions are displayed as options, for example.

Item 3400 is an input interface for inputting a finding for the region of interest displayed in item 3200 . The input interface is not limited to the state of the region of interest and the findings of the region of interest, and may accept input of various types of information such as suitability as an anastomosis position in lymphatic venule anastomosis.

In addition, when a plurality of lymphatic vessels are included in the region of interest, an interface that can specify the condition of the lymphatic vessels entered by the user and the lymphatic vessels to be examined in the item 3100 or item 3200. may be By storing the information of the identified lymphatic vessel together with the status and finding information of the lymphatic vessel, it is possible to easily grasp which lymphatic vessel the evaluation is for when observing later. Become.

(S1223: Step of displaying classification results)
The image processing apparatus 1300 as a display control means can color-code the lymphatic vessels A1 for each region of interest based on the state selected in the item 3300 and display them in the item 3100 as the classification result of the lymphatic vessels A1. .

A display example of the lymphatic vessel classification result according to the second embodiment will be described with reference to FIG. 17 . FIG. 17 shows an example in which classification results are displayed in item 3100 of the GUI shown in FIG. Item 3100 displays lymphatic vessel A1 and vein A2. The example of FIG. 17 shows a state in which a region of interest A121, a region of interest A122, and a region of interest A123 are identified in a lymphatic vessel A1. A region A124 is an unclassified lymphatic vessel region that has not been specified as a region of interest. As indicated by the legend, the region of interest A121 is in the state of Shooting Star, the region of interest A122 is in the state of staying, and the region of interest A123 is in the state of contraction. Note that the unclassified area A124 may be displayed blinking, for example. The image processing apparatus 1300 can prompt the user to instruct the classification of the lymphatic vessels by blinking the unclassified regions. Note that the method of displaying the region specified as the region of interest and the region not specified as the region of interest in different modes is not limited to blinking display. For example, the same effect can be obtained by displaying an unclassified area in a color different from the color given to the classified area, or by displaying a frame indicating the area.

(S1224: Step of saving data)
The image processing apparatus 1300 may associate the lymphatic vessel classification results in S1212 with the analyzed image data and patient information and store them in the storage device 1200 . The image processing device 1300 can display the lymphatic vessel classification results stored in the storage device 1200 on the display device 1400 together with the image data. If the image data is a moving image, the user can confirm his/her classification results while playing back the moving image.

The flow shown in FIG. 15 exemplifies processing for inputting information such as a state or findings for one region of interest. By repeating the flow shown in FIG. 15 for unclassified regions, the lymphatic vessel region is divided into a plurality of regions of interest and classified according to the state of each region. Note that the step of displaying the classification result (S1223) and the step of storing the data (S1224) are executed for each region of interest in the flow shown in FIG. good too.

(Other examples)
The present invention is also realized by executing the following processing. That is, the software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

1100 photoacoustic device 1300 image processing device

Claims (4)

An image processing apparatus for processing image data generated based on photoacoustic waves generated from within the subject by light irradiation to the subject,
By image analysis of the image data including the lymphatic vessel area in the subject, the lymphatic vessel area is divided into a plurality of divided areas, and the image value of each divided area is determined based on the time change of the image value in each divided area. An image processing apparatus, comprising state estimating means for estimating the state of the lymphatic vessel by determining the state. An image processing apparatus for processing image data generated based on photoacoustic waves generated from within the subject by light irradiation to the subject,
An image processing apparatus, comprising display control means for displaying a distance between said vein and said lymphatic vessel on a display device by image analysis of said image data including areas of said vein and said lymphatic vessel in said subject. . generating image data based on a photoacoustic wave generated from within the subject by irradiating the subject with light;
By image analysis of the image data including the lymphatic vessel area in the subject, the lymphatic vessel area is divided into a plurality of divided areas, and the image value of each divided area is determined based on the time change of the image value in each divided area. An image processing method, comprising estimating the state of the lymphatic vessel by determining the state. generating image data based on a photoacoustic wave generated from within the subject by irradiating the subject with light;
An image processing method, comprising displaying a distance between the vein and the lymphatic vessel on a display device by image analysis of the image data including the area of the vein and the lymphatic vessel in the subject.

Images