JP7577734B2 - PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING APPARATUS - Google Patents

Description

The present invention relates to a program, an information processing method and an information processing device.

A catheter system is used in which a diagnostic imaging catheter is inserted into a tubular organ such as a blood vessel to display images for diagnostic imaging (Patent Document 1).

International Publication No. 2017/164071

The orientation of an image obtained by a diagnostic imaging catheter changes depending on the bending state of the diagnostic imaging catheter, etc. In order for doctors and paramedical staff to quickly grasp the condition of the luminal organs, the image may be manually rotated to a specified orientation. However, rotating an image to a specified orientation requires the ability to correctly interpret the image being rendered, which requires extensive training.

In one aspect, the objective is to provide a program or the like that makes it easy to use a catheter system.

The program acquires images generated using an imaging diagnostic catheter inserted into a tubular organ, and each time an image is acquired, calculates the rotation angle required to orient a landmark contained in the acquired image in a predetermined orientation, and if the calculated rotation angle exceeds a predetermined threshold, causes the computer to execute a process of displaying the image in a rotated state so that the extracted landmark is oriented in the predetermined orientation.

In one aspect, a program or the like can be provided to facilitate use of the catheter system.

FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. FIG. 1 is an explanatory diagram for explaining an overview of a catheter for diagnostic imaging. FIG. 1 is an explanatory diagram illustrating a configuration of a catheter system. FIG. 2 is an explanatory diagram illustrating a record layout of a standard image DB. 13 is an example of a screen displayed by the catheter system. 13 is an example of a screen displayed by the catheter system. 13 is an example of a screen displayed by the catheter system. 13 is a modified example of a screen displayed by the catheter system. 11 is a flowchart illustrating a process flow of a program. 10 is a flowchart illustrating a process flow of a rotation angle calculation subroutine. 13 is a flowchart illustrating the flow of processing of a program according to a second embodiment. FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. 13 is a flowchart illustrating a process flow of a rotation angle calculation subroutine according to the third embodiment. FIG. 2 is an explanatory diagram illustrating the record layout of a Merkmal DB. 13 is an example of a screen displayed by the catheter system of embodiment 4. 13 is a flowchart illustrating the processing flow of a program according to a fourth embodiment. 13 is an example of a screen displayed by the catheter system of embodiment 5. 13 is an example of a screen displayed by the catheter system of embodiment 5. 13 is a flowchart illustrating the processing flow of a program according to a fifth embodiment. FIG. 2 is an explanatory diagram illustrating the record layout of a training DB. 23 is a flowchart illustrating the processing flow of a program according to a sixth embodiment. FIG. 13 is a functional block diagram of an information processing device according to a seventh embodiment. FIG. 23 is an explanatory diagram illustrating the configuration of a catheter system according to an eighth embodiment.

[First embodiment]
FIG. 1 is an explanatory diagram for explaining an outline of a catheter system 10. The catheter system 10 includes a catheter 40 for diagnostic imaging, a motor driving unit (MDU) 33, and an information processing device 20. The catheter 40 for diagnostic imaging is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as one unit.

Figure 2 is an explanatory diagram for explaining an overview of the diagnostic imaging catheter 40. Figure 2 shows an example of a diagnostic imaging catheter 40 for IVUS (Intravascular Ultrasound) used to generate ultrasonic tomographic images from inside blood vessels, i.e., for generating ultrasonic tomographic images.

The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 disposed at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 is referred to as the tip side.

A shaft 43 is inserted inside the probe section 41. A sensor 42 is connected to the tip of the shaft 43. The shaft 43 and the sensor 42 can rotate and move forward and backward inside the probe section 41.

The sensor 42 is an ultrasonic transducer that transmits and receives ultrasonic waves. A ring-shaped tip marker 44 is fixed near the tip of the probe portion 41. The material of the tip marker 44 is a material that is opaque to X-rays, such as metal.

The diagnostic imaging catheter 40 may be a catheter for generating optical tomographic images, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generates optical tomographic images using near-infrared light. The sensor 42 of these diagnostic imaging catheters 40 is a transceiver that emits near-infrared light and receives reflected light.

The diagnostic imaging catheter 40 may have both an ultrasonic transducer and a transceiver unit for OCT or OFDI. The diagnostic imaging catheter 40 may have a total of three sensors 42: an ultrasonic transducer, a transceiver unit for OCT, and a transceiver unit for OFDI.

The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and advances and retreats. It may be an electronic radial scanning type diagnostic imaging catheter 40 that uses a sensor 42 in which multiple ultrasonic transducers are arranged in a ring shape.

The diagnostic imaging catheter 40 may have a so-called linear scanning type sensor 42 in which multiple ultrasonic transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may have a so-called two-dimensional array type sensor 42 in which multiple ultrasonic transducers are arranged in a matrix.

Using the imaging diagnostic catheter 40, it is possible to generate cross-sectional images that include not only luminal walls such as blood vessel walls, but also reflectors present inside luminal organs, such as red blood cells, and organs present outside luminal organs, such as the epicardium and heart.

The imaging diagnostic catheter 40 may be a vascular endoscope used to optically observe the inner wall of a tubular organ.

The tubular organ into which the diagnostic imaging catheter 40 is inserted and used is, for example, a blood vessel, pancreatic duct, bile duct, or bronchus. In the following explanation, the diagnostic imaging catheter 40 will be described as an example of a mechanical scanning type IVUS catheter as shown in Figure 2.

Figure 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 has an information processing device 20, an MDU 33, and an imaging diagnostic catheter 40. The information processing device 20 has a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), multi-core CPUs, etc. The control unit 21 is connected to each hardware component that constitutes the information processing device 20 via a bus.

The main memory device 22 is a storage device such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs being executed by the control unit 21.

The auxiliary storage device 23 is a storage device such as an SRAM, a flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores the programs to be executed by the control unit 21, a standard image DB (Database) 65, and various data required for executing the programs. The standard image DB 65 may be stored in an external large-capacity storage device connected to the information processing device 20. The communication unit 24 is an interface that communicates between the information processing device 20 and a network.

The display unit 25 is an interface that connects the display device 31 to the bus. The input unit 26 is an interface that connects the input device 32 to the bus. The catheter control unit 271 controls the MDU 33, controls the sensor 42, and generates images based on signals received from the sensor 42.

The MDU 33 rotates the sensor 42 and shaft 43 inside the probe section 41. The catheter control section 271 generates one image for each rotation of the sensor 42. The generated image is a transverse slice image centered on the probe section 41 and approximately perpendicular to the probe section 41.

The MDU 33 can also move the sensor 42 and shaft 43 back and forth while rotating them inside the probe section 41. By performing a pullback operation in which the sensor 42 is pulled toward the MDU 33 at a constant speed while being rotated, the catheter control section 271 continuously generates multiple transverse slice images approximately perpendicular to the probe section 41 at a predetermined interval.

The function and configuration of the catheter control unit 271 are similar to those of conventional ultrasound diagnostic devices, and therefore detailed description will be omitted. Note that the control unit 21 may also realize the function of the catheter control unit 271.

The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via a HIS (Hospital Information System) or the like.

The information processing device 20 in this embodiment is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, etc. having the functions of an ultrasound diagnostic device.

Figure 4 is an explanatory diagram explaining the record layout of the standard image DB 65. The standard image DB 65 is a database that records the observation target area and the standard image in association with each other. The standard image DB 65 has an area field and a standard image field.

The area to be observed is recorded in the area field. "LAD" in the first line indicates the Left Anterior Descending Coronary Artery. "LCX" and "RCA" similarly indicate areas of the coronary artery. The areas shown in FIG. 4 are examples and are not limited to these. The standard image DB 65 may contain records corresponding to any area into which the imaging diagnostic catheter 40 may be inserted, such as blood vessels in the lower limbs, the bile duct, or bronchial branches.

The standard image field records a standard image generated at the observation site. The standard image is a typical image that a doctor uses for diagnosis using the diagnostic imaging catheter 40, and is rotated to an orientation that is familiar to the doctor. Generally, the standard image includes landmarks that the doctor uses as guides for making decisions.

A landmark is, for example, a side branch structure of a hollow organ into which the diagnostic imaging catheter 40 is inserted, an organ such as the heart that is close to the hollow organ, or a hollow organ that runs parallel to the hollow organ. For example, when the diagnostic imaging catheter 40 is inserted into a coronary artery, doctors and technicians often use a side branch blood vessel branching off from the coronary artery as a landmark and rotate the displayed image so that it appears at the bottom, or use the epicardium and rotate the displayed image so that it appears at the top.

The standard image may be an image artificially generated using technology such as computer graphics. The standard image may be an image generated by combining multiple case images. Standard image DB65 has one record for one observation target area.

Standard image DB 65 may have a field for recording the subject's attributes, such as the subject's age, sex, or underlying disease.Standard image DB 65 may have a field for recording lesions contained in the observation target area, such as the presence or absence of dissociation or the presence or absence of calcification.Standard images corresponding to each attribute or lesion are recorded in the standard image field.

Figures 5 to 7 are examples of screens displayed by the catheter system 10. Figure 5 shows an example of a screen displayed by the control unit 21 on the display device 31 when no correspondence with a standard image has been detected. The screen shown in Figure 5 includes a first image field 51, a site selection button 591, a status notification field 571, and an auto-rotation button 581.

In the first image field 51, an image obtained using the diagnostic imaging catheter 40 is displayed in real time. In the following description, an image displayed in real time is referred to as a real-time image. The area selection button 591 is a pull-down menu button, and the area being observed by the user is selected.

For example, the control unit 21 acquires the area to be observed from an electronic medical record system. The control unit 21 may detect the position of the tip marker 44 from an image from the imaging diagnostic device 37 to determine the area to be observed. The control unit 21 may acquire the area to be observed determined by the imaging diagnostic device 37. The control unit 21 automatically sets the area selection button 591 to a state in which the area to be observed is selected.

The control unit 21 may wait for the user to operate the part selection button 591 instead of automatically setting the part selection button 591. Even if the control unit 21 automatically sets the part selection button 591, the user can operate the part selection button 591 to change the part to be observed. The control unit 21 may accept the user's operation of the part selection button 591 by voice input.

The control unit 21 searches the standard image DB 65 using the observation target area set in the area selection button 591 as a key, and extracts a standard image. The control unit 21 rotates the real-time image to detect the angle at which the image becomes similar to the standard image.

Specifically, the control unit 21 rotates the real-time image by any angle, such as 3 degrees or 5 degrees, and then repeats the process of determining the similarity with the standard image. The control unit 21 may generate an image that has been rotated by one or more sound ray data obtained from the sensor 42, and repeat the process of determining the similarity with the standard image.

Similarity can be evaluated, for example, by the sum of squared differences (SSD), sum of absolute differences (SAD), or normalized cross-correlation (NCC) of pixel values of pixels at the same position in two images. Note that the above methods are merely examples. Methods for evaluating similarity are not limited to these.

After repeating the process until the real-time image rotates once, the control unit 21 extracts the angle with the highest similarity. The control unit 21 determines whether the similarity at that angle exceeds a predetermined threshold value.

If the similarity does not exceed the threshold, the control unit 21 displays "Not Detected" in the status notification field 571 as shown in Figure 5, and sets the auto-rotate button 581 to an unselectable state. If the similarity exceeds the threshold, the control unit 21 sets the auto-rotate button 581 to a selectable state as shown in Figure 6. In Figure 6, the auto-rotate button 581 is set to the "OFF" state. The control unit 21 displays the real-time image in an unrotated state in the first image field 51.

Figure 7 shows an example of an image that the control unit 21 displays on the display device 31 when the user sets the auto-rotation button 581 to the "ON" state. The control unit 21 displays in the first image field 51 a real-time image rotated in an orientation most similar to the standard image. The user can easily grasp the condition of the tubular organ and its surroundings from the displayed real-time image.

The control unit 21 may rotate the real-time image and set the auto-rotation button 581 to the "ON" state at the same time that a similar orientation to the standard image is detected. The control unit 21 may also accept the operation of the auto-rotation button 581 by voice input.

Figure 8 is a modified example of a screen displayed by the catheter system 10. In Figure 8, the second image column 52 is displayed between the site selection button 591 and the auto-rotation button 581. The control unit 21 displays the rotated real-time image in the first image column 51 and the non-rotated real-time image in the second image column 52.

By displaying both the rotated and unrotated real-time images, it is possible to prevent the user from losing sight of the part of the image that they were focusing on when rotating the image.

For example, the image described using Figure 8 may be displayed on a display device 31 located in a place that is easy for the doctor who primarily operates the imaging diagnostic catheter 40 to observe, and the image described using Figure 7 may be displayed on a display device 31 located in a place that is easy for other medical staff to observe.

Figure 9 is a flowchart explaining the flow of the program processing. The control unit 21 acquires the observation target area (step S501). The control unit 21 searches the standard image DB 65 using the observation target area acquired in step S501 as a key, and acquires the extracted standard image (step S502).

The control unit 21 acquires a real-time image from the catheter control unit 271 (step S503). The control unit 21 displays the image described using FIG. 5 (step S504). The control unit 21 starts a rotation angle calculation subroutine (step S505). The rotation angle calculation subroutine is a subroutine that calculates a rotation angle that increases the similarity between the standard image and the real-time image. The processing flow of the rotation angle calculation subroutine will be described later.

The control unit 21 determines whether the similarity between the real-time image and the standard image at the rotation angle calculated by the rotation angle calculation subroutine exceeds a predetermined threshold (step S506). If it is determined that the similarity does not exceed the predetermined threshold, i.e., is not sufficiently similar (NO in step S506), the control unit 21 returns to step S503.

If it is determined that the threshold value has been exceeded (YES in step S506), the control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S505 (step S507). The control unit 21 displays the image described using Figures 6 to 8 on the display device 31 (step S508).

The control unit 21 then updates the image displayed in the first image field 51 in real time. The control unit 21 switches between rotating and not rotating the real-time image displayed in the first image field 51 based on the user's operation of the auto-rotate button 581. When an instruction to end is received from the user, or when the imaging diagnostic catheter 40 is removed from the MDU 33, the control unit 21 ends the processing.

If the user operates the part selection button 591 to select a different part to be observed, the control unit 21 executes the program again after terminating the processing. In step S503 during re-execution, the control unit 21 displays the real-time image at the same rotation angle as immediately before the user operated the part selection button 591. When the user operates the part selection button 591, the image displayed in the first image column 51 is rotated, which can be prevented from confusing the user who is observing the real-time image.

Figure 10 is a flowchart explaining the process flow of the rotation angle calculation subroutine. The rotation angle calculation subroutine is a subroutine that calculates the rotation angle that increases the similarity between the standard image and the real-time image.

The control unit 21 calculates the similarity between the standard image and the real-time image (step S511). As described above, the similarity can be evaluated using any method, such as the sum of squared differences, the sum of absolute differences, or normalized cross-correlation, between pixel values of pixels at the same position in the two images. The control unit 21 may accept a user's selection of the method to be used.

The control unit 21 associates the rotation angle with the similarity and stores them in the auxiliary storage device 23 or the main storage device 22 (step S512). The initial value of the rotation angle is zero. The control unit 21 determines whether the real-time image has rotated once (step S513). If it is determined that the image has not rotated once (NO in step S513), the control unit 21 rotates the image being processed by a predetermined angle (step S514). The control unit 21 returns to step S511.

If it is determined that one rotation has been performed (YES in step S513), the control unit 21 calculates the rotation angle with the highest similarity based on the relationship between the rotation angle and the similarity recorded in step S512 (step S515). Then, the control unit 21 ends the process.

Before calculating the similarity in step S511, the control unit 21 may perform image processing on the standard image and the real-time image, such as edge enhancement or contrast correction, to clearly extract landmarks.

The control unit 21 may calculate the similarity by using an image obtained by averaging images of multiple adjacent frames instead of a real-time image. Since reflectors such as red blood cells flowing inside the tubular organ are no longer depicted, landmarks become clear. By using an image in which landmarks are clearly extracted, the rotation angle at which the similarity between the two images becomes high can be calculated with high accuracy.

The control unit 21 may perform a process of reducing the standard image and the real-time image before calculating the similarity in step S511. Alternatively, the standard image that has been reduced may be recorded in the standard image DB 65. By using the reduced image, the calculation process for calculating the similarity can be speeded up.

If the frame rate of the real-time image is high and the processing of step S505 cannot be completed in time, the control unit 21 may activate step S505, for example, once every two or three frames. The control unit 21 may determine the frame for performing the processing of step S505 to be synchronized with the electrocardiogram.

A part or all of the program may be executed on a large computer connected via a network, on a virtual machine running on a large computer, or on a cloud computing system. A part or all of the program may be executed on multiple personal computers performing distributed processing.

According to this embodiment, a catheter system 10 can be provided that automatically rotates a real-time image to a similar orientation to a standard image. For example, in a catheterization room where multiple medical staff such as doctors and nurses work together to diagnose and treat, each medical staff member can quickly grasp the condition of the tubular organ because the real-time image is displayed in the appropriate orientation. Therefore, catheterization treatment can be performed quickly and appropriately.

According to this embodiment, a catheter system 10 can be provided that does not require skilled staff who can accurately interpret real-time images to manually rotate the real-time images.

Instead of real-time images, video or still images stored in the auxiliary storage device 23 or the like may be used. A catheter system 10 can be provided that can be used for recording medical records after a case is completed. In such a case, the information processing device 20 may be a personal computer, tablet, smartphone, or the like that does not have the function of connecting the MDU 33 and the imaging diagnostic catheter 40.

[Embodiment 2]
This embodiment relates to a catheter system 10 that continuously calculates the rotation angle. Descriptions of parts common to the first embodiment will be omitted.

Figure 11 is a flowchart explaining the processing flow of the program of embodiment 2. The processing from step S501 to step S508 is common to the processing of the program of embodiment 1 explained using Figure 9, so the explanation is omitted.

The control unit 21 acquires a real-time image from the catheter control unit 271 (step S521). The control unit 21 starts a subroutine for calculating the rotation angle (step S522). The subroutine for calculating the rotation angle is the same as the subroutine described using FIG. 10.

The control unit 21 determines whether the real-time image has rotated (step S523). Specifically, the control unit 21 determines that the real-time image has rotated when the difference between the rotation angle set in step S507 and the rotation angle calculated in step S522 is greater than a predetermined threshold. The predetermined threshold may be the angle of one sound ray data, or an angle that can be easily identified by the user visually, such as 30 degrees or 60 degrees. The control unit 21 may accept a threshold setting by the user.

If it is determined that the image has rotated (YES in step S523), the control unit 21 updates the rotation angle set in step S507 to the rotation angle newly calculated in step S522 (step S524). If it is determined that the image has not rotated (NO in step S523), or after step S524 is completed, the control unit 21 determines whether to end the process (step S525).

For example, the control unit 21 determines to end the process when it receives an instruction to end the process from the user, when the diagnostic imaging catheter 40 is removed from the MDU 33, or when the user operates the site selection button 591 to select another site to be observed. If it determines not to end the process (NO in step S525), the control unit 21 returns to step S508. If it determines to end the process (YES in step S525), the control unit 21 ends the process.

According to this embodiment, a catheter system 10 can be provided that automatically tracks the rotation of the real-time image due to the influence of the bending state of the tubular organ, etc., and continues to correct the orientation of the real-time image.

[Embodiment 3]
This embodiment relates to a catheter system 10 that uses a schema, which is a schematic representation of the structure of a hollow organ and the surrounding organs, as a standard image. In this embodiment, after converting a real-time image into an object arrangement image, the image is rotated in a direction that increases the similarity to the schema. Explanations of parts common to the first embodiment will be omitted.

Figure 12 is an explanatory diagram explaining the configuration of the learning model 61. The learning model 61 is a model that accepts a real-time image and outputs an object arrangement image 482 in which the types of multiple objects contained in the real-time image are associated with the ranges of each object and mapped. The learning model 61 is generated by machine learning.

In the object arrangement image 482 shown in Figure 12, thin horizontal hatching indicates the "cross section of the diagnostic imaging catheter 40," diagonal hatching slanting to the right indicates the "interior of the tubular organ," and thick horizontal hatching slanting to the left indicates the "epicardium."

The hatching in FIG. 12 shows that each object is painted in a different color. Painting each object in a different color is one example of a method for displaying each object in a distinctive way. Each object may be displayed in any manner, such as by surrounding its outer edge, to make it distinguishable from other objects.

12, a transverse image is displayed in the portion of the object arrangement image 482 other than the objects listed above. The control unit 21 may classify the entire surface of the object arrangement image 482 into some object, such as "tubular organ wall," "plaque," "calcification," "guidewire," etc., and display it in different colors.

The learning model 61 is, for example, a semantic segmentation model, and includes an input layer, a neural network, and an output layer. The neural network has a U-Net structure that realizes semantic segmentation. The U-Net structure is composed of a multi-layered encoder layer and a multi-layered decoder layer connected to the rear. Semantic segmentation assigns a label indicating the type of object to each pixel that constitutes the input image.

By determining the display method for each pixel according to the label, the control unit 21 can generate an output image in which objects are mapped with different colors according to their type, as shown in the object arrangement image 482 in Figure 4. In the standard image field of the standard image DB 65 in this embodiment, a schema of a standard image painted in the same colors as the object arrangement image is recorded.

The learning model 61 shown in Figure 12 is an example of a model that outputs information about landmarks contained in an image when the image is input. The learning model 61 may be a model that detects landmarks, etc. from an image using Mask R-CNN (Regions with Convolutional Neural Networks).

Figure 13 is a flowchart illustrating the processing flow of the rotation angle calculation subroutine of embodiment 3. The subroutine of Figure 13 is a subroutine that is used in place of the rotation angle calculation subroutine described using Figure 10.

The control unit 21 inputs the real-time image acquired from the catheter control unit 271 into the learning model 61 to acquire an object arrangement image 482 that associates the types of multiple objects contained in the real-time image with the range of each object (step S531).

The control unit 21 calculates the similarity between the standard image and the object arrangement image 482 (step S532). The control unit 21 associates the rotation angle with the similarity and stores it in the auxiliary storage device 23 or the main storage device 22 (step S533). The initial value of the rotation angle is zero.

The control unit 21 determines whether the real-time image has rotated once (step S534). If it is determined that the real-time image has not rotated once (NO in step S534), the control unit 21 rotates the object arrangement image acquired in step S531 by a predetermined angle (step S535). The control unit 21 returns to step S532.

If it is determined that one rotation has been performed (YES in step S534), the control unit 21 calculates the rotation angle with the highest similarity based on the relationship between the rotation angle and the similarity recorded in step S533 (step S536). Then, the control unit 21 ends the process.

According to this embodiment, it is possible to provide a catheter system 10 that can use a schema for a standard image. For example, it is possible to provide a catheter system 10 that can be used even when an actual case image suitable for the standard image cannot be prepared, such as when a new type of diagnostic imaging catheter 40 is used or when a diagnostic imaging catheter 40 is used for a new purpose.

[Fourth embodiment]
This embodiment relates to a catheter system 10 that calculates a rotation angle based on a landmark specified by a user from a real-time image. Description of parts common to the third embodiment will be omitted.

Figure 14 is an explanatory diagram explaining the record layout of the Merkmal DB. The Merkmal DB is a DB that records the observation target area, the name of the Merkmal, and the Merkmal image showing the position where the Merkmal is to be displayed in association with each other. The Merkmal DB has a area field, a Merkmal name field, and a Merkmal image field.

The part being observed is recorded in the part field. The landmark name field records the name of the landmark. The landmark image field records an image showing the extent of the landmark.

The Merkmal DB may record multiple Merkmal names for one observation target area. One Merkmal name may be associated with multiple observation target areas. The Merkmal DB has one record for each combination of an observation target area and a Merkmal name.

A specific example will be given. The first line of Fig. 14 contains a record in which the observation target area is "LAD" and the landmark name is "epicardium". The landmark image field contains a landmark image illustrating the standard range in which the landmark "epicardium" is displayed while the diagnostic imaging catheter 40 is inserted into the "LAD", i.e., the standard position and standard shape in which the "epicardium" is displayed. In Fig. 14, the hatched portion is the standard range in which the landmark is displayed.

The second line in Figure 14 records a record in which the observation target area is "LAD" and the landmark name is "D1 (First Diagonal Branch)." The landmark image field records a landmark image illustrating the standard range in which the landmark "D1" is displayed while the diagnostic imaging catheter 40 is being inserted into the "LAD."

Figure 15 is an example of a screen displayed by the catheter system 10 of embodiment 4. Figure 15 is an example of an image displayed by the control unit 21 on the display device 31 when the user manually specifies a landmark. The screen shown in Figure 15 includes a first image field 51, a site selection button 591, a landmark selection button 592, and an auto-rotation button 581.

The first image field 51 displays a designated image that accepts the user's designation of a landmark. The designated image is a real-time image obtained using the diagnostic imaging catheter 40, or a paused image in which the update of the real-time image is temporarily stopped based on a user operation.

The area selection button 591 is a pull-down menu button, on which the observation area being observed by the user is selected. The landmark selection button 592 is a pull-down menu button, on which the name of the landmark specified by the user is selected.

For example, the control unit 21 obtains the area to be observed from the electronic medical record system. The control unit 21 may detect the position of the sensor 42 from the image being captured by the imaging diagnostic device 37 and determine the area to be observed. The control unit 21 may obtain the area to be observed determined by the imaging diagnostic device 37. The control unit 21 automatically sets the area selection button 591 to a state in which the area to be observed is selected.

The control unit 21 may wait for the user to operate the part selection button 591 instead of automatically setting the part selection button 591. Even if the control unit 21 automatically sets the part selection button 591, the user can operate the part selection button 591 to change the part to be observed. The control unit 21 may accept the user's operation of the part selection button 591 by voice input.

The control unit 21 searches the landmark DB using the observation target area as a key, and extracts the landmark name associated with that area. The control unit 21 sets the extracted landmark name as an option when the pull-down menu of the landmark selection button 592 is opened. The control unit 21 accepts operation of the landmark selection button 592 by the user.

The control unit 21 displays a cursor 68 in the first image field 51. The user operates the cursor 68 using an input device such as a mouse to input a designation frame 69 indicating the range of the landmark. The control unit 21 acquires the range of the landmark designated by the user.

For example, the control unit 21 accepts the specification of the landmark range via a user interface similar to that of so-called paint software. The control unit 21 may display a specification frame 69 of a shape suitable for specifying the landmark range in the first image field 51 based on the observation target area selected via area selection button 591 and the area selection button 592 selected via landmark selection button 592, and accept movement and deformation operations by the user. After completing the specification of the landmark range, the user selects the auto-rotate button 581.

The control unit 21 searches the landmark DB using the observation area set in the area selection button 591 and the landmark name set in the landmark selection button 592 as keys, and extracts the landmark image. The control unit 21 detects the angle at which the range of the designation frame 69 specified by the user is similar to the range of the landmark included in the landmark image.

Specifically, the control unit 21 generates a designated frame image in which the area specified by the designated frame 69 is filled in in the same manner as the landmark image on a white background of the same dimensions as the designated accepted image. The control unit 21 rotates the designated frame image by any angle, and then repeatedly performs a process of determining the similarity with the landmark image. After repeating the process until the designated frame image has rotated once, the control unit 21 extracts the angle with the highest similarity. The control unit 21 rotates the designated accepted image together with the designated frame 69.

Figure 16 is a flowchart explaining the flow of processing of the program of embodiment 4. The control unit 21 acquires the observation target area (step S541). The control unit 21 searches the landmark DB using the observation target area acquired in step S541 as a key, and extracts the landmark name (step S542).

The control unit 21 sets the landmark name extracted in step S542 as an option for the landmark selection button 592. The control unit 21 acquires the landmark name selected by the user (step S543). The control unit 21 searches the landmark DB using the observation target area acquired in step S541 and the landmark name acquired in step S543 as keys, and extracts the landmark image (step S544).

The control unit 21 prompts the user to input a landmark. The user operates the cursor 68 to input a designation frame 69 indicating the range of the landmark depicted in the designation reception image. The control unit 21 acquires the designation frame 69 input by the user (step S545). The control unit 21 generates a designation frame image (step S546).

The control unit 21 starts a rotation angle calculation subroutine (step S547). The rotation angle calculation subroutine is the same as the subroutine described using FIG. 10, and is a subroutine that calculates a rotation angle that increases the similarity between the standard image and the real-time image. In this embodiment, the control unit 21 inputs the milestone acquired in step S544 as the standard image instead of the standard image, and the designated frame image generated in step S546 as the real-time image into the rotation angle calculation subroutine.

The control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S547 (step S548). The control unit 21 rotates the designated reception image being displayed in the first image column 51 (step S549).

When the user instructs the display of a real-time image in the first image column 51, the control unit 21 updates the image displayed in the first image column 51 in real time. The control unit 21 switches between rotating and not rotating the real-time image displayed in the first image column 51 based on the user's operation of the auto-rotate button 581. When an end instruction is received from the user or the imaging diagnostic catheter 40 is removed from the MDU 33, the control unit 21 ends the processing.

The control unit 21 may deform the position and shape of the designation frame 69 so that it follows the real-time image. The method of tracking a designated area or point on a video has been used for a long time, so a description thereof will be omitted.

When the user operates the part selection button 591 to select a different part to be observed, the control unit 21 ends the processing and re-executes the program. When the control unit 21 receives an instruction to display a real-time image during re-execution, it displays the image at the same rotation angle as immediately before the user operated the part selection button 591. This prevents the image displayed in the first image column 51 from rotating when the user operates the part selection button 591, confusing the user who is observing the real-time image.

According to this embodiment, a catheter system 10 can be provided that automatically rotates a real-time image to a similar orientation to a standard image based on a landmark specified by a user through image interpretation. The user can specify an appropriate landmark depending on the planned treatment procedure.

[Modification 1]
In the first modification, the control unit 21 uses the learning model 61 described with reference to Fig. 12 to assist in inputting the designation frame 69. The control unit 21 executes step S545 of the program described with reference to Fig. 16 by replacing it with the following process.

The control unit 21 inputs the designated accepted image into the learning model 61 to obtain an object arrangement image. The control unit 21 extracts the contour line of the object corresponding to the landmark name accepted via the landmark selection button 592. The control unit 21 uses the extracted contour line as the default designated frame 69 and displays it in the first image column 51. The control unit 21 accepts modifications to the designated frame 69 by the user.

According to this modified example, a catheter system 10 can be provided that allows the user to easily specify the designation frame 69 in a short period of time.

[Modification 2]
In the second modification, the rotation angle is automatically calculated without accepting any modification of the designation frame 69 by the user. The control unit 21 executes steps S545 and S546 of the program described with reference to FIG. 16 by replacing them with the following processes.

The control unit 21 inputs the designated acceptance image into the learning model 61 described using FIG. 12 to obtain an object arrangement image. The control unit 21 extracts pixels corresponding to the object corresponding to the landmark name accepted via the landmark selection button 592. The control unit 21 generates a designated frame image by setting the pixels other than the extracted pixels to a background color such as white.

According to this modified example, a catheter system 10 can be provided in which the user does not need to specify a designation frame 69.

[Embodiment 5]
This embodiment relates to a catheter system 10 that extracts a landmark during a pullback operation and rotates an image. Descriptions of parts common to the first embodiment will be omitted.

17 and 18 are examples of screens displayed by the catheter system of embodiment 5. The screen shown in FIG. 17 includes a first image field 51, a second image field 52, a site selection button 591, a landmark selection button 592, and an auto-rotation button 581.

A real-time image is displayed in the first image field 51. A longitudinal tomographic image is displayed in the second image field 52. The right end of the longitudinal tomographic image corresponds to the real-time image, and as the sensor 42 moves toward the MDU 33 by the pullback operation, the longitudinal tomographic image displayed in the second image field 52 extends to the right.

In Figures 17 and 18, the observation area is "SFA (Superficial Femoral Artery)" and the landmark is "DFA (Deep Femoral Artery)."

In the state shown in Figure 17, the landmark has not been detected and the auto-rotate button 581 is set to an unselectable state. Figure 18 shows the state after the landmark has been detected. A detection mark 572 displayed on the edge of the second image column 52 indicates the position where the landmark was detected. After the landmark is detected and the rotation angle is calculated, the control unit 21 sets the auto-rotate button 581 to a selectable state. In Figure 18, the auto-rotate button 581 has been selected and rotated images are displayed in the first image column 51 and the second image column 52.

Figure 19 is a flow chart explaining the flow of processing of the program of the fifth embodiment. The control unit 21 acquires the observation target area (step S551). The control unit 21 searches the standard image DB 65 using the observation target area acquired in step S551 as a key, and acquires the extracted standard image (step S552). The control unit 21 accepts an instruction from the user to start a pullback operation (step S553).

The control unit 21 acquires a real-time image from the catheter control unit 271 (step S554). The control unit 21 displays the image described using FIG. 17 (step S555). The control unit 21 starts a rotation angle calculation subroutine (step S556). The rotation angle calculation subroutine is the same as the subroutine described using FIG. 10.

The control unit 21 determines whether the similarity between the real-time image and the standard image at the rotation angle calculated by the rotation angle calculation subroutine exceeds a predetermined threshold value (step S557). If it is determined that the similarity does not exceed the predetermined threshold value, i.e., is not sufficiently similar (NO in step S557), the control unit 21 determines whether the pullback operation has ended (step S558).

If it is determined that the process has not ended (NO in step S558), the control unit 21 returns to step S554. If it is determined that the process has ended (YES in step S558), the control unit 21 ends the process.

If the predetermined threshold is exceeded, i.e., if it is determined that the images are sufficiently similar (YES in step S557), the control unit 21 sets the rotation angle for rotating the real-time image when the automatic rotation button 581 is selected to the angle calculated in step S557 (step S559). The control unit 21 displays the image described using FIG. 18 on the display device 31 (step S560).

Thereafter, the control unit 21 updates the images displayed in the first image column 51 and the second image column 52 in real time. The control unit 21 switches between rotating and not rotating the images displayed in the first image column 51 and the second image column 52 based on the user's operation of the auto-rotate button 581. When the pullback operation is completed, the control unit 21 ends the processing.

In addition, when the control unit 21 sets the rotation angle in step S560, it may rotate the images displayed in the first image column 51 and the second image column 52.

According to the present embodiment, a catheter system 10 can be provided that detects landmarks during a pullback operation and rotates an image. Even if there are few locations where landmarks are depicted, a catheter system 10 can be provided that automatically rotates an image in a specified direction.

Sixth Embodiment
This embodiment relates to a program for generating a learning model 61. Explanation of parts common to the third embodiment will be omitted.

Figure 20 is an explanatory diagram explaining the record layout of the training DB. The training DB is a database that records inputs and correct labels in association with each other, and is used to train models using machine learning. The training DB has an input data field and a color-coded data field.

The input data field records an input image acquired using the diagnostic imaging catheter 40. The color-coded data field records an image in which a doctor or other expert has colored each object in a different color. That is, the color-coded data field records the object to which each pixel in the input image corresponds. The training DB records a large number of combinations of input images generated using the diagnostic imaging catheter 40 and images colored by an expert or other expert.

Figure 21 is a flowchart explaining the processing flow of the program of embodiment 6. An example will be explained in which machine learning of the learning model 61 is performed using the information processing device 20.

21 may be executed on hardware separate from the information processing device 20, and the learning model 61 after machine learning may be copied to the auxiliary storage device 23 via a network. The learning model 61 trained on one piece of hardware may be used on multiple information processing devices 20.

Prior to executing the program in Figure 21, an untrained model, such as a U-Net structure that realizes semantic segmentation, is prepared. As mentioned above, the U-Net structure is composed of multiple encoder layers followed by multiple decoder layers. The program in Figure 19 adjusts each parameter of the prepared model and performs machine learning.

The control unit 21 acquires a training record to be used for training one epoch from the training DB (step S571). The control unit 21 adjusts the parameters of the model so that when an input image is input to the input layer of the model, a correct image label is output from the output layer (step S572). In acquiring the training record and adjusting the parameters of the model, the program may appropriately have a function of causing the control unit 21 to accept corrections by the user, present the basis for the judgment, re-learn, etc.

The control unit 21 determines whether to end the process (step S573). For example, the control unit 21 determines to end the process when a predetermined number of epochs of learning have been completed. The control unit 21 may obtain test data from the training DB, input the test data to the model being machine-learned, and determine to end the process when an output of a predetermined accuracy is obtained.

If it is determined that the process is not to be terminated (NO in step S573), the control unit 21 returns to step S571. If it is determined that the process is to be terminated (YES in step S573), the control unit 21 records the parameters of the trained model in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 terminates the process. Through the above process, a trained model is generated.

According to this embodiment, a learning model 61 can be generated by machine learning.

[Embodiment 7]
22 is a functional block diagram of an information processing device 20 according to a seventh embodiment. The information processing device 20 includes an acquisition unit 81 and a display unit 82. The acquisition unit 81 acquires an image generated using a diagnostic imaging catheter inserted into a tubular organ. The display unit 82 displays the image acquired by the acquisition unit 81 in a rotated state such that the landmark included in the image is oriented in a predetermined direction.

[Embodiment 8]
23 is an explanatory diagram for explaining the configuration of the catheter system 10 of the eighth embodiment. This embodiment relates to a form in which the catheter system 10 of this embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an image diagnostic catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted.

The catheter control device 27 is an ultrasound diagnostic device for IVUS that controls the MDU 33, the sensor 42, and generates transverse and longitudinal images based on signals received from the sensor 42. The functions and configuration of the catheter control device 27 are similar to those of conventional ultrasound diagnostic devices, and therefore will not be described here.

The catheter system 10 of this embodiment includes a computer 90. The computer 90 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read out the program 97 stored in a semiconductor memory 98, such as a flash memory, implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown), and store it in the auxiliary storage device 23.

The program 97 is installed as a control program for the computer 90, and is loaded into the main memory device 22 and executed. This causes the computer 90 to function as the information processing device 20 described above.

The computer 90 is a general-purpose personal computer, a tablet, a smartphone, a mainframe, a virtual machine running on a mainframe, a cloud computing system, or a quantum computer. The computer 90 may be multiple personal computers performing distributed processing.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.

REFERENCE SIGNS LIST 10 Catheter system 20 Information processing device 21 Control unit 22 Main memory device 23 Auxiliary memory device 24 Communication unit 25 Display unit 26 Input unit 27 Catheter control device 271 Catheter control unit 29 Reading unit 31 Display device 32 Input device 33 MDU
37 Diagnostic imaging device 40 Diagnostic imaging catheter 41 Probe section 42 Sensor 43 Shaft 44 Tip marker 45 Connector section 482 Object arrangement image 51 First image field 52 Second image field 571 Status notification field 572 Detection mark 581 Automatic rotation button 591 Region selection button 592 Merkmal selection button 61 Learning model 65 Standard image DB
68 Cursor 69 Designation frame 81 Acquisition unit 82 Display unit 90 Computer 96 Portable recording medium 97 Program 98 Semiconductor memory

Claims (11)

Acquiring an image generated using an imaging catheter inserted into a hollow organ;
Each time the image is acquired,
Calculating a rotation angle until the landmark included in the acquired image is oriented in a predetermined direction;
A program that causes a computer to execute a process of displaying the image rotated so that the extracted landmark is oriented in a predetermined direction when the calculated rotation angle exceeds a predetermined threshold . The program according to claim 1 , wherein the landmark is extracted by performing image processing on the image. The program according to claim 1, further comprising: inputting the acquired image into a learning model that outputs information about landmarks contained in the image when the image is input; and extracting landmarks based on the information output from the learning model. The program according to claim 1 , further comprising: an input device that receives input of the milestone mark. The program according to claim 1 , wherein the diagnostic imaging catheter is a catheter for generating ultrasonic tomographic images. The program according to claim 1 , wherein the diagnostic imaging catheter is a catheter for generating optical tomographic images. 5. The program according to claim 1, wherein the diagnostic imaging catheter is a catheter equipped with a sensor for generating ultrasonic tomographic images and a sensor for generating optical tomographic images. The landmark includes the epicardium.
The program according to any one of claims 5 to 7 . The program according to claim 1 , wherein the landmark includes a side branch structure of the hollow organ. Acquiring an image generated using an imaging catheter inserted into a hollow organ;
Each time the image is acquired,
Calculating a rotation angle until the landmark included in the acquired image is oriented in a predetermined direction;
When the calculated rotation angle exceeds a predetermined threshold, the image is displayed in a rotated state so that the extracted landmark is oriented in a predetermined direction. an acquisition unit that acquires an image generated using an imaging diagnostic catheter inserted into a hollow organ;
Each time the acquisition unit acquires the image,
A calculation unit that calculates a rotation angle until a landmark included in the acquired image is oriented in a predetermined direction;
and a display unit that displays the image in a rotated state so that the extracted landmark is oriented in a predetermined direction when the rotation angle calculated by the calculation unit exceeds a predetermined threshold .

Images