JP7176041B2 - Medical image processing device and method, endoscope system, processor device, diagnosis support device and program - Google Patents

Description

The present invention relates to a medical image processing apparatus and method, an endoscope system, a processor, a diagnosis support apparatus, and a program, and to an image processing technique for image recognition using a medical image obtained by photographing an observation target.

In the medical field, examinations using endoscope systems are performed. In recent years, systems have been known that recognize lesions contained in medical images by using image analysis techniques. In a general endoscope system, a body cavity is photographed using a camera at the tip of a scope, and the photographed observation image is received by a processor device and displayed on a monitor. At this time, the processor device may perform recognition processing on the received image and notify the user of the recognition result.

For example, if a lesion is imaged, the processor device recognizes whether it is cancerous or non-cancerous from image analysis, and notifies the user of the recognition result. A system having such a function supports decision-making such as diagnostic findings by a doctor or the like.

Japanese Patent Application Laid-Open No. 2004-180932 JP 2015-204960 A Japanese Unexamined Patent Application Publication No. 2014-030488

However, for example, the effects of scope manipulation, fetal movement, and/or debris often cause the image to be blurred or blurred, or the lesion to be hidden by the debris. In addition, even when water is supplied to the observation site using a water supply device to secure a field of view, the image may be blurred or the lesion site may be covered with water due to the influence of the water. Accurate recognition of lesions, etc. by image analysis from such "blurred images", "blurred images", "images containing residues", or "images covered with water". is difficult and may give different recognition results. Such a problem is particularly problematic in a system that performs recognition processing for each image obtained in time series, such as performing recognition processing for each frame of a moving image obtained during an examination, and notifies the user of the recognition results in real time.

In addition to endoscope systems, systems that perform recognition processing from medical images obtained in time series, such as ultrasonic diagnostic equipment, also have the same problem.

The present invention has been made in view of such circumstances. An object of the present invention is to provide a medical image processing apparatus and method, an endoscope system, a processor, a diagnosis support apparatus, and a program that can avoid being provided.

In order to solve the problem, the following aspects of the invention are provided.

A medical image processing apparatus according to aspect 1 includes an image acquisition unit that acquires time-series medical images including a subject image, an image recognition unit that performs image recognition from the medical images, and a recognition result that stores recognition results of the image recognition unit. A storage unit and a change processing unit that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.

According to aspect 1, the recognition result of an image unsuitable for image recognition, such as a blurred image, among time-series medical images is changed to the recognition result of another image stored in the recognition result storage unit. . The recognition result storage unit can store recognition results of images other than images unsuitable for image recognition. This makes it possible to avoid presentation of inaccurate recognition results and present valid recognition results in chronological order.

The time-series medical images may be moving images, or still images captured at specific time intervals, such as continuous shooting or interval shooting.

Aspect 2 is the medical image processing apparatus of Aspect 1, comprising a determination unit that determines whether to save the recognition result of the image recognition unit according to the recognition result of the image recognition unit, according to the determination result of the determination unit, The medical image processing apparatus stores recognition results of an image recognition unit in a recognition result storage unit.

The determination unit determines that the recognition results of images that are unsuitable for image recognition, such as blurred images, are not to be stored in the recognition result storage unit (determines that they are not stored). It is possible to determine whether to store the recognition result in the recognition result storage unit.

Aspect 3 is the medical image processing apparatus according to Aspect 2, wherein the change processing unit stores, in the recognition result storage unit, the recognition result of the image recognition unit for the medical image for which the determination unit has determined that the recognition result is not to be stored. It is a medical image processing device that changes to the recognition result that is being used.

Aspect 4 is the medical image processing apparatus of Aspect 3, wherein the image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition, and the determination unit comprises: A medical image processing apparatus that, when an image is recognized as unsuitable for recognition, determines that the recognition result of the medical image recognized as an unsuitable image is not to be saved.

Aspect 5 is the medical image processing apparatus according to any one of Aspects 1 to 3, wherein the image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition, The change processing unit is a medical image processing device that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit when the image recognition unit recognizes that the image is not suitable for recognition.

Aspect 6 is the medical image processing apparatus according to Aspect 5, wherein the recognition result of the medical image recognized by the image recognition unit as being unsuitable for recognition is not stored in the recognition result storage unit. be.

Aspect 7 is the medical image processing apparatus according to any one of Aspects 4 to 6, wherein the unsuitable image is an image in which the subject is blurred.

Aspect 8 is the medical image processing apparatus according to any one of aspects 4 to 6, wherein the inappropriate image is an image in which the subject is blurred.

Aspect 9 is the medical image processing apparatus according to any one of aspects 4 to 6, wherein the inappropriate image is an image in which the subject is covered with water.

Aspect 10 is the medical image processing apparatus according to any one of aspects 4 to 6, wherein the unsuitable image is an image in which a subject has a residue.

Aspect 11 is the medical image processing apparatus according to any one of aspects 1 to 10, wherein the image recognition performed by the image recognition unit includes a task of detecting a region of interest.

A region of interest may be, for example, a lesion region.

Aspect 12 is the medical image processing apparatus according to any one of aspects 1 to 11, wherein the image recognition performed by the image recognition unit includes image classification.

Aspect 13 is the medical image processing apparatus according to any one of Aspects 1 to 12, wherein the image recognition unit recognizes whether or not the medical image acquired from the image acquisition unit is an image unsuitable for recognition. 1 recognizing unit, and a second recognizing unit that performs at least one of processing of detecting a region of interest from a medical image acquired from the image acquiring unit and image classification of the medical image, is a medical image processing apparatus that does not perform processing in a second recognition unit for a medical image recognized as an image unsuitable for recognition.

Aspect 14 is the medical image processing apparatus according to any one of Aspects 1 to 13, wherein the image recognition unit performs image recognition for each medical image acquired in time series, and outputs a recognition result for each medical image. It is a medical image processing device.

Aspect 15 is the medical image processing apparatus according to any one of Aspects 1 to 14, wherein the recognition result of the medical image in which the recognition result of the image recognition unit is maintained is obtained without undergoing change processing by the change processing unit. It is a medical image processing device stored in a recognition result storage unit.

Aspect 16 is the medical image processing apparatus according to any one of Aspects 1 to 15, wherein the change processing unit changes the recognition result of the image recognition unit to the latest recognition result stored in the recognition result storage unit. It is a medical image processing device.

It is preferable that the recognition result stored in the recognition result storage unit is updated to the latest one each time, and only the most recent recognition result needs to be stored in the recognition result storage unit.

Aspect 17 is the medical image processing apparatus according to any one of Aspects 1 to 16, further comprising a display unit for displaying the recognition result of the image recognition unit, and storing the recognition result in the recognition result storage unit according to the recognition result of the image recognition unit. The medical image processing apparatus displays stored recognition results on a display unit.

Aspect 18 is the medical image processing apparatus according to Aspect 17, wherein, when displaying the recognition result stored in the recognition result storage unit, the display unit indicates that the recognition result is stored in the recognition result storage unit. It is a medical image processing device that displays.

Aspect 19 is the medical image processing apparatus according to Aspect 18, wherein the display indicating that the recognition result is stored in the recognition result storage unit has a different display color than when the recognition result of the image recognition unit is displayed as it is. The medical image processing apparatus includes at least one display form among different color display, symbol display with specific symbols added, character string display with specific character strings added, and blinking display.

Aspect 20 is the medical image processing apparatus according to any one of Aspects 1 to 19, wherein acquisition of medical images in which the recognition result of the image recognition unit is changed to the recognition result stored in the recognition result storage unit continues for a certain period of time. It is a medical image processing apparatus that stops recognition processing of an image recognition unit when

Aspect 21 is the medical image processing apparatus according to any one of Aspects 1 to 20, wherein the medical image is an endoscopic image obtained from an endoscope or an ultrasonic image obtained from an ultrasonic diagnostic apparatus. processing equipment.

A medical image processing method according to aspect 22 includes an image acquisition step of acquiring time-series medical images including a subject image, an image recognition step of performing image recognition from the medical images, and a recognition result storing unit storing recognition results of the image recognition step. and a change processing step of changing the recognition result of the image recognition step to the recognition result saved in the recognition result storage unit according to the recognition result of the image recognition step.

In the medical image processing method of Aspect 22, items similar to the items specified in Aspects 2 to 21 can be appropriately combined. In this case, elements of the processing units and functional units as means for carrying out the processes and operations specified in the medical image processing apparatus can be grasped as the elements of steps (processes) of the corresponding processes and operations. Also, the medical image processing method of aspect 22 may be understood as a method of operating a medical image processing apparatus.

An endoscope system according to aspect 23 is an endoscope system including an electronic endoscope for imaging the inside of a body cavity and a processor device for processing an image signal obtained from the electronic endoscope, wherein the processor device , an image acquisition unit that acquires time-series medical images including subject images captured using an electronic endoscope, an image recognition unit that performs image recognition from the medical images, and a recognition that saves the recognition results of the image recognition unit. A result storage unit and a change processing unit that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.

In the endoscope system of Aspect 23, items similar to the items specified in Aspects 2 to 21 can be appropriately combined.

A processor device according to aspect 24 is a processor device that processes an image signal obtained from an electronic endoscope, and is a processor device that obtains time-series medical images including subject images captured using the electronic endoscope. an image recognition unit that performs image recognition from a medical image; a recognition result storage unit that stores recognition results of the image recognition unit; and a change processing unit that changes the recognition result stored in the storage unit.

In the processor device of aspect 24, the same items as those specified in aspects 2 to 21 can be combined as appropriate.

A diagnosis support apparatus according to aspect 25 includes an image acquisition unit that acquires time-series medical images including a subject image, an image recognition unit that performs image recognition from the medical images, and a recognition result storage that stores recognition results of the image recognition unit. a change processing unit that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit; and a notification that notifies the recognition result of the image recognition unit. and

In the diagnosis support device of aspect 25, items similar to the items specified in aspects 2 to 21 can be appropriately combined. The display unit that displays the recognition result of the image recognition unit can be one form of the notification unit.

The program according to aspect 26 comprises, in a computer, an image acquisition step of acquiring time-series medical images including a subject image; an image recognition step of performing image recognition from the medical images; and a change processing step of changing the recognition result of the image recognition step to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition step.

In the program of aspect 26, items similar to the items specified in aspects 2 to 21 can be appropriately combined. In that case, the elements of the processing unit and functional unit as means for carrying out the processing and operation specified in the medical image processing apparatus can be grasped as program elements that realize the steps (functions) of the corresponding processing and operation. can.

According to the present invention, when images unsuitable for image recognition are included in images obtained in time series, it is possible to avoid providing the user with recognition results obtained from images unsuitable for image recognition. It is possible.

FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a front view showing the distal end face of the rigid distal end portion of the electronic endoscope. FIG. 3 is a block diagram showing the control system of the endoscope system. FIG. 4 is a block diagram showing functions of the medical image processing apparatus according to the first embodiment of the present invention. FIG. 5 is a flow chart showing the operation of the medical image processing apparatus according to the first embodiment. FIG. 6 is a block diagram showing functions of the medical image processing apparatus according to the second embodiment of the present invention. FIG. 7 is a flow chart showing the operation of the medical image processing apparatus according to the second embodiment. FIG. 8 is a diagram showing an example of an endoscopic diagnosis support screen displaying recognition results. FIG. 9 is a diagram showing another example of an endoscopic diagnosis support screen displaying recognition results. FIG. 10 is a block diagram showing a configuration example of a medical information management system.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

<<Configuration example of endoscope system>>
FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. The endoscope system 10 includes an electronic endoscope 12 , a light source device 14 and a processor device 16 . A display device 18 and an input device 19 are connected to the processor device 16 . The electronic endoscope 12 may be called a "scope", an "electronic scope", or simply an "endoscope".

The electronic endoscope 12 of this example is a flexible endoscope. The electronic endoscope 12 includes an insertion section 20 , an operation section 30 and a universal cord 40 . The insertion portion 20 is a portion that is inserted into the body cavity of the person to be examined. The insertion section 20 includes a distal rigid section 22, a bending section 24, and a flexible section 26 in order from the distal end toward the proximal side. An illumination optical system, an objective optical system, an imaging element, and the like are arranged inside the distal end rigid portion 22 . The bending portion 24 has a structure that smoothly bends in four directions of up, down, left, and right from a reference position according to the operation of the angle knob 31 . The proximal side of the flexible portion 26 is called the proximal end portion of the insertion portion 20 .

The operation portion 30 is provided at the proximal end portion of the insertion portion 20 . The operation unit 30 includes various operation members operated by the operator. For example, the operation unit 30 includes an angle knob 31 used for bending operation of the bending portion 24, an air/water supply button 32 for performing an air/water supply operation, and a suction button 33 for performing a suction operation. is provided. By operating the angle knob 31 , the operator can bend the bending portion 24 and freely change the orientation of the distal end rigid portion 22 . The operation unit 30 also includes a mode switching switch 34 used to switch observation modes, a zoom operation unit 35, and a still image photographing instruction unit (not shown) for instructing photographing of a still image of the site to be observed. is provided.

Further, a treatment instrument introduction port 36 is provided in the operation section 30 . The treatment instrument introduction port 36 is an opening for inserting a treatment instrument (not shown) into a treatment instrument insertion passage (not shown) passing through the insertion section 20 . Examples of treatment tools include biopsy forceps, catheters, high-frequency snares, and the like. Treatment tools also include guide tubes, trocar tubes, sliding tubes, and the like. The treatment instrument introduction port 36 may be called a forceps port.

The universal cord 40 is a cord for connecting the electronic endoscope 12 to the light source device 14 and processor device 16 . A cable extending from the insertion portion 20 and a light guide are inserted through the universal cord 40 . The cables extending from the insertion portion 20 include a communication cable used for signal transmission and a power supply cable used for power supply. A connector 42 is provided at one end of the universal cord 40 .

The connector 42 is a composite type connector including a video connector 42A and a light guide connector 42B. One end of a cable is arranged in the video connector 42A. The video connector 42A is detachably connected to the processor device 16. As shown in FIG. One end of a light guide is arranged in the light guide connector 42B. The light guide connector 42B is detachably connected to the light source device 14 . A water supply connector 42C is provided to the light guide connector 42B, and the water supply tank 44 is connected via the water supply connector 42C.

The processor device 16 is electrically connected to the light source device 14 via the connector 42 . The processor device 16 centrally controls the operation of the endoscope system 10 including the light source device 14 . The processor unit 16 supplies power to the electronic endoscope 12 via a cable inserted through the universal cord 40, and controls the driving of the imaging device. The processor unit 16 also receives an imaging signal transmitted from the electronic endoscope 12 via a cable, performs various signal processing on the received imaging signal, and converts it into image data. The image data converted by the processor device 16 is displayed on the display device 18 as an endoscope-captured image (observation image).

FIG. 2 is a front view showing the distal end face 22A of the rigid distal end portion 22 of the electronic endoscope 12. As shown in FIG. An illumination window 50 , an observation window 52 , a forceps outlet 54 , and an air/water supply nozzle 56 are provided on the distal end surface 22</b>A of the distal end rigid portion 22 . Although not shown in FIG. 2, an emission end 122 of the light guide 120 that guides the light from the light source device 14 is arranged behind the illumination window 50 (see FIG. 3). Illumination light is emitted from the illumination window 50 to the observation area. Two illumination windows 50 are arranged at symmetrical positions with the observation window 52 interposed therebetween.

The observation window 52 is a window for taking in the reflected light from the observed area and capturing an image of the observed area. Although not shown in FIG. 2, behind the observation window 52, an objective optical system 60 and an imaging device 62 for taking in image light of the region to be observed in the body cavity are arranged (see FIG. 3). .

The forceps outlet 54 is connected to a forceps channel (not shown) provided in the insertion section 20 and communicates with the treatment instrument introduction port 36 (see FIG. 1) provided in the operation section 30 . The treatment instrument inserted from the treatment instrument introduction port 36 is taken out from the forceps outlet 54 into the body cavity.

The air/water nozzle 56 jets cleansing water or air toward the observation window 52 and/or into the body cavity in response to the operation of the air/water button 32 (see FIG. 1) provided on the operation unit 30. . The cleaning water and air are supplied from an air/water supply device incorporated in the light source device 14 .

FIG. 3 is a block diagram showing the control system of the endoscope system 10. As shown in FIG. The distal rigid portion 22 of the electronic endoscope 12 includes an objective optical system 60, an image sensor 62, an analog front end (AFE) circuit 64, a timing generator (TG) 65, and a CPU. (Central Processing Unit) 66 is provided.

The objective optical system 60 is configured using a zoom lens. A light guiding prism (not shown) is arranged between the objective optical system 60 and the imaging element 62 . The imaging element 62 may be a CMOS (complementary metal oxide semiconductor) type imaging element or a CCD (Charged Coupled Device) type imaging element. Here, an example using a CMOS type image sensor will be described.

Although not shown, the imaging device 62 is a solid-state imaging device of a single-plate color imaging system having a color filter composed of a plurality of color segments. The color filters may be, for example, Bayer array primary color filters including red (R), green (G), and blue (B).

A large number of pixels are arranged in a matrix on the imaging surface of the imaging device 62, and each pixel is provided with a photosensor. Light incident on the imaging surface of the imaging element 62 is accumulated as electric charge in the photosensor of each pixel. The amount of signal charge accumulated in the photosensor of each pixel is sequentially read out as a pixel signal by scanning in the vertical and horizontal directions by a vertical scanning circuit and a horizontal scanning circuit (both not shown), and is read out at a predetermined frame rate. output.

A timing generator 65 generates driving pulses for the imaging element 62 and synchronization pulses for the analog front-end circuit 64 under the control of the CPU 66 . Driving pulses for the imaging device 62 include a vertical scanning pulse, a horizontal scanning pulse, a reset pulse, and the like.

The imaging element 62 is driven by a driving pulse input from the timing generator 65, photoelectrically converts an optical image formed on the imaging surface through the objective optical system 60, and outputs the resultant as an imaging signal.

Analog front end circuitry 64 includes correlated double sampling (CDS) circuitry, automatic gain control (AGC) circuitry, and an A/D converter. The CDS circuit performs correlated double sampling processing on the imaging signal output from the imaging device 62 to remove reset noise and amplifier noise generated in the imaging device 62 .

The AGC circuit amplifies the imaging signal from which noise has been removed by the CDS circuit with a gain (amplification factor) specified by the CPU 66 . The A/D converter converts the imaging signal amplified by the AGC circuit into a digital signal having a predetermined number of bits and outputs the digital signal. The imaging signal digitized and output by the analog front-end circuit 64 is input to the processor unit 16 through the signal line.

Note that the imaging element 62 , the analog front-end circuit 64 and the timing generator 65 can be configured as a monolithic integrated circuit, and each of these circuit elements is included in one imaging chip 68 . The imaging chip 68 mounted on the electronic endoscope 12 of this example is a so-called "CMOS sensor chip" and is mounted on a support substrate (not shown).

The processor device 16 includes a CPU 70 , a ROM (read-only memory) 72 , a RAM (Random Access Memory) 74 , a digital signal processor (DSP) 76 , and a display control circuit 78 .

The CPU 70 controls each part in the processor device 16 and controls the entire endoscope system 10 in an integrated manner. Various programs and control data for controlling the operation of the processor device 16 are stored in the ROM 72 . The RAM 74 temporarily stores programs and data executed by the CPU 70 .

Under the control of the CPU 70, the digital signal processing circuit 76 performs various signal processing such as color interpolation, color separation, color balance adjustment, gamma correction, and image enhancement on the imaging signal input from the analog front end circuit 64. to generate image data. The digital signal processing circuit 76 also performs image recognition processing. The digital signal processing circuit 76 functions as an image processing section. The digital signal processing circuit 76 also has the function of an image recognition section that performs image recognition processing.

Image data output from the digital signal processing circuit 76 is input to the display control circuit 78 . The display control circuit 78 converts the image data input from the digital signal processing circuit 76 into a signal format compatible with the display device 18 and displays it on the screen of the display device 18 .

The display device 18 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The display device 18 can display various types of information such as various setting information necessary for the processing of the processor device 16 or information indicating processing results.

The display device 18 and the input device 19 function as user interfaces. The input device 19 may be, for example, a keyboard, mouse, touch panel, operation buttons, voice input device, or an appropriate combination thereof. A user can use the input device 19 to input various instructions and/or information. The processor device 16 can execute various processes according to instructions and/or information input from the input device 19 .

The light source device 14 generates light for illuminating the body cavity through a light guide 120 inserted into the electronic endoscope 12 . The light source device 14 includes a first light source 100 , a first light source drive circuit 101 , a second light source 102 , a second light source drive circuit 103 , a CPU 104 and a multiplexer 105 . The CPU 104 communicates with the CPU 70 of the processor device 16 and controls the first light source driving circuit 101 and the second light source driving circuit 103 .

The first light source drive circuit 101 causes the first light source 100 to emit light according to instructions from the CPU 104 . The second light source drive circuit 103 causes the second light source 102 to emit light in accordance with instructions from the CPU 104 .

The first light source 100 is, for example, a laser diode that emits a blue laser with a wavelength of 445 nm. The first light source 100 is pulse-driven by a first light source drive circuit 101 to control the amount of light emitted. The second light source 102 is, for example, a laser diode that emits a blue laser with a wavelength of 405 nm. The second light source 102 is pulse-driven by a second light source drive circuit 103 to control the amount of light emitted. Blue light with a wavelength of 405 nm is used for special light observation.

The multiplexing unit 105 multiplexes the emitted lights of the first light source 100 and the second light source 102 and outputs the combined light to the incident end 121 of the light guide 120 .

A phosphor 124 is provided between the exit end 122 of the light guide 120 and the illumination window 50 of the electronic endoscope 12 . The blue laser light that has passed through the light guide 120 irradiates the phosphor 124 to excite the phosphor 124 , and part of the laser light passes through the phosphor 124 and is emitted from the illumination window 50 as blue light.

The phosphor 124 is excited by blue laser light, and emits light (yellow in color) in a wide range of wavelengths from the boundary between blue and green to red. This yellow light and the blue light that passes through the phosphor 124 are mixed to form white light, which illuminates the subject through the illumination window 50 . The blue light that passes through phosphor 124 includes part of the blue light that is emitted from phosphor 124 .

As described above, the phosphor 124 emits yellow light and transmits blue light with a wavelength of 445 nm when irradiated with a blue laser beam with a wavelength of 445 nm, but is irradiated with a blue laser beam with a wavelength of 405 nm. In some cases, it has the property of penetrating most of it.

That is, by controlling the mixing ratio of the blue laser light having a wavelength of 445 nm and the blue laser light having a wavelength of 405 nm, the ratio of the blue light passing through the phosphor 124 and the yellow light emitted from the phosphor 124 can be controlled. is possible.

A color image of the subject is reproduced by receiving reflected light from the subject illuminated with white light using the imaging element 62 .

When observing the inside of a body cavity using the endoscope system 10 configured as described above, the electronic endoscope 12, the light source device 14, the processor device 16, and the display device 18 are turned on. Then, the insertion portion 20 of the electronic endoscope 12 is inserted into the body cavity, and an image of the body cavity captured by the imaging device 62 is displayed on the screen of the display device 18 while illuminating the body cavity with illumination light from the light source device 14 . will be observed in

The endoscope system 10 of this example has a white light observation mode and a narrowband light observation mode. The white light observation mode is a mode in which an observation image with natural colors is displayed on the display device 18 using a captured image obtained by imaging an observation target using white light as illumination light. An image obtained by imaging an observation target in the white light observation mode is called a "white light observation image". The illumination light may also be called "observation light".

The narrow-band light observation mode uses an image signal obtained by imaging the observation target with narrow-band light of a specific wavelength band as illumination light, for example, emphasizing the visualization of blood vessels in a specific depth region of the observation target. In this mode, an image is generated and an image suitable for observing blood vessels is displayed on the display device 18 . An image obtained by imaging an observation target in the narrow-band light observation mode is called a "narrow-band light observation image".

The endoscope system 10 may have a plurality of types of narrow-band light observation modes in which the types of wavelength bands of narrow-band light to be used or their combinations are different.

<<Overview of Diagnosis Support Function in Endoscope System 10>>
The processor unit 16 includes an image recognition unit that recognizes images received from the electronic endoscope 12 . For image recognition processing, for example, a convolutional neural network (CNN) is used. The image recognizer performs the task of image classification, for example, classifying lesions contained within an image into multiple categories, such as according to the NICE classification or the JNET classification. "NICE" is an abbreviation for NBI (Narrow band imaging) International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team."

The NICE classification is a classification by non-magnified NBI, and is classified into Type 1, Type 2 and Type 3 for each of the three items of lesion color (Color), microvessel pattern (Vessels), and surface pattern (Surface pattern). be. Type 1 is a hyperplastic lesion, Type 2 is an adenoma to intramucosal carcinoma, and Type 3 is a diagnostic index for SM (submucosa) deep invasive cancer. The JNET classification is a classification of NBI magnified endoscopy findings for colon tumors. The JNET classification is classified into Type1, Type2A, Type2B, and Type3 for each item of "vessel pattern" and "surface pattern."

The image recognition unit may simply recognize two classifications of "cancerous" and "non-cancerous" in place of or in combination with detailed classification such as the NICE classification.

In addition to recognition processing such as lesion classification, the image recognition unit also recognizes whether an input image is suitable for image recognition or not. "Suitable for image recognition" means that the image is suitable for recognition processing such as classification of lesions, which is the main purpose of recognition. “Unsuitable for image recognition” means that the image is unsuitable for recognition, such as classification of lesions, which is the main purpose. Examples of images that are unsuitable for recognition include a blurred image, a blurred image, an image in which the subject is covered with water, and an image with residue. An image unsuitable for recognition is called an "inappropriate image".

The image recognition unit recognizes, for example, an image suitable for recognition or an image unsuitable for recognition into two categories. perform a task; Alternatively, the image recognition unit may add a classification such as "image unsuitable for recognition" in the recognition process of a classification task such as NICE classification.

The processor device 16 saves the recognition result when the recognition result is other than "image unsuitable for recognition". The next image is then received by the processor unit 16 and image recognition is performed on this newly received image. When the recognition result of the newly received image is determined to be "an image unsuitable for recognition", the processor unit 16 converts this recognition result (a recognition result indicating that the image is "unsuitable for recognition"), Change to the saved recognition result.

The recognition result can be displayed on the display device 18 . Here, if the recognition result is changed to the saved one (previous recognition result that was saved), when displaying the recognition result, display should be made so that it can be seen that it is the previous recognition result. is preferred. As a specific example of "display that shows that it is the previous recognition result", for example, a form in which an icon is displayed next to the information indicating the recognition result, or a form in which the display color of the information indicating the recognition result is changed. can be. The display device 18 is an example of a "display unit". Also, the display device 18 is an example of a “notification unit” that notifies the recognition result of the image recognition unit.

<<Configuration of Medical Image Processing Apparatus According to First Embodiment>>
Processor device 16 is an example of a medical image processing device according to an embodiment of the present invention. FIG. 4 is a block diagram showing functions of the medical image processing apparatus according to the first embodiment of the present invention. The medical image processing apparatus 160 shown in FIG. And prepare.

The image acquisition unit 162 is an interface that acquires time-series images IM1, IM2, IM3, . . . captured using the electronic endoscope 12 . The image acquisition unit 162 may be, for example, a connector terminal to which the video connector 42A is connected, or may be a signal input terminal of the digital signal processing circuit 76 . The image acquisition unit 162 may be a communication network terminal provided in the processor device 16, a media interface terminal for external storage media, a connection terminal for external equipment, or an appropriate combination thereof.

The image recognition unit 164 is a processing unit that performs image recognition from the image acquired via the image acquisition unit 162 . Image recognition section 164 includes a first recognition section 164A and a second recognition section 164B. The first recognition unit 164A recognizes whether the input image is suitable for image recognition or not. The second recognition unit 164B performs image classification processing on the input image according to the NIEC classification, the JNET classification, or the like. Note that the second recognition unit 164B may perform processing for detecting an attention area such as a lesion area instead of or in combination with the image classification processing. The recognition processing performed by the first recognition unit 164A is called "first recognition processing". The recognition processing performed by the second recognition unit 164B is called "second recognition processing". Each of the first recognition unit 164A and the second recognition unit 164B can be configured using, for example, a convolutional neural network (CNN). The image recognition unit 164 is configured using a learned model learned by machine learning.

When the recognition result obtained from the first recognition unit 164A is "an image suitable for recognition", recognition processing is performed by the second recognition unit 164B. The second recognition unit 164B extracts a feature amount from an image and performs at least one of image classification, attention area detection, segmentation, and similarity calculation. For example, the second recognition unit 164B performs image classification according to a predetermined classification method such as NICE classification or JNET classification.

The recognition result storage unit 168 is a storage unit that stores the recognition result of the image recognition unit 164 . The recognition result storage unit 168 may be a storage area of the RAM 74 shown in FIG. The recognition result storage unit 168 may be a storage area of a memory (not shown) other than the RAM 74 . The recognition results stored in the recognition result storage unit 168 are sequentially updated to the latest information. Only the latest (most recent) recognition result should be stored in the recognition result storage unit 168 .

The determination unit 166 determines whether to save the recognition result of the image recognition unit 164 according to the recognition result of the image recognition unit 164 . The determination unit 166 of this example determines whether or not to save the recognition result based on the recognition result from the first recognition unit 164A.

When the recognition result from the first recognition unit 164A is "an image suitable for recognition", the second recognition processing is performed by the second recognition unit 164B. In this case, the determination unit 166 determines to store the recognition result, and the recognition result obtained from the second recognition unit 164B is stored in the recognition result storage unit 168. FIG. Further, the recognition result obtained from the second recognition section 164B is displayed on the display section 174 via the display control section 172. FIG. The display control section 172 corresponds to the display control circuit 78 shown in FIG. The display 174 may be the display device 18 (see FIGS. 1 and 2). Also, the display unit 174 may be a display device different from the display device 18 .

On the other hand, when the recognition result from the first recognition unit 164A is "image unsuitable for recognition", the second recognition processing by the second recognition unit 164B is stopped. That is, the processing in the second recognition unit 164B is not performed for the medical image recognized by the first recognition unit 164A as an image unsuitable for recognition. In this case, the determination unit 166 determines not to save the recognition result (non-save).

The change processing unit 170 changes the recognition result of the image recognition unit 164 to the recognition result stored in the recognition result storage unit 168 according to the recognition result of the image recognition unit 164 . The change processing unit 170 changes the recognition result of the image for which the determination unit 166 has determined that the recognition result should not be saved, to the recognition result saved in the recognition result saving unit 168 .

The recognition result changed by the change processing unit 170 is displayed on the display unit 174 via the display control unit 172 .

The recognition result storage unit 168 stores the recognition result of the image in which the recognition result of the image recognition unit 164 is maintained as it is without undergoing the change processing by the change processing unit 170 . The change processing unit 170 changes the recognition result of the image recognition unit 164 to the latest (most recent) recognition result stored in the recognition result storage unit 168 .

The function of the image recognition section 164 can be realized by the digital signal processing circuit 76 described in FIG. 2, the CPU 70, or a combination thereof. Functions of the determination unit 166 and the change processing unit 170 can be realized by the CPU 70 .

<<Operation of the medical image processing apparatus according to the first embodiment>>
FIG. 5 is a flow chart showing the operation of the medical image processing apparatus according to the first embodiment. The operation of a medical imaging device may be understood as a method of medical image processing, a method of operating a medical imaging device, or a method of operating a processor device.

In step S12, the image acquisition unit 162 receives the input image. The input image here is a medical image including a subject image captured using the electronic endoscope 12, and is one of time-series images sequentially captured in time series. For example, the input image is a one-frame image forming a moving image. Step S12 is an example of the "image acquisition step".

In step S14, the image recognition unit 164 performs first recognition processing on the input image.

In step S16, the CPU 70 determines whether or not the input image is suitable for the second recognition process based on the recognition result of the first recognition process. If the determination result of step S16 is "Yes", that is, if the input image is an image suitable for the second recognition process, the process proceeds to step S18. In step S18, the image recognition unit 164 performs second recognition processing. Steps S14 and S18 are examples of the "image recognition step".

In step S<b>20 , CPU 70 obtains a recognition result from image recognition section 164 .

In step S<b>22 , CPU 70 stores the recognition result obtained in step S<b>20 in recognition result storage unit 168 . Step S22 is an example of a "recognition result saving step".

On the other hand, if the determination result in step S16 is "no", that is, if the input image is an image unsuitable for the second recognition process, the process proceeds to step S24.

In step S<b>24 , CPU 70 acquires the recognition result stored in recognition result storage unit 168 . In step S26, the CPU 70 changes the recognition result of the first recognition process to the recognition result saved in the recognition result saving unit 168 to obtain the recognition result. Steps S24 and S26 are examples of "change processing steps".

The recognition result obtained in step S20 or step S26 is displayed on the display unit 174. FIG.

After step S22 or step S26, the flow chart of FIG. 5 ends. The processing of the flowchart of FIG. 5 is repeated for each input image acquired in time series.

According to such a configuration, when an image that is not suitable for recognition is temporarily included in the images obtained in time series, the most recent recognition result stored in the recognition result storage unit 168 is used to , it is possible to present an appropriate recognition result with a high degree of truth.

As a result, it is possible to prevent a recognition result (erroneous recognition result) having low authenticity from being generated from an image unsuitable for recognition and being provided to the user.

Further, even when the recognition result of an image unsuitable for recognition is changed, when an image suitable for recognition is obtained after that, the processing of steps S18 to S22 is performed, so highly accurate recognition is possible.

<Measures to be taken when an image unsuitable for recognition is continuously acquired for a certain period of time>
Although not shown in the flowchart of FIG. 5, the processor unit 16 stores the recognition result of the image recognition unit 164 in the recognition result storage unit 168 when an image unsuitable for recognition is continuously acquired for a certain period of time. The recognition processing of the image recognition unit 164 may be stopped when the acquisition of the medical image changed to the recognized result continues for a certain period of time. In this case, it is preferable to display a warning message or the like on the display unit 174 to call the user's attention. The display unit 174 is an example of the "notification unit".

In the first embodiment shown in FIG. 4, an example is shown in which the "first recognition process" and the "second recognition process" are performed in stages. Therefore, a configuration may be adopted in which images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

<<Configuration of Medical Image Processing Apparatus According to Second Embodiment>>
In the first embodiment shown in FIG. 4, an example is shown in which the "first recognition process" and the "second recognition process" are performed in stages. Therefore, a configuration may be adopted in which images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

FIG. 6 is a block diagram showing functions of the medical image processing apparatus according to the second embodiment of the present invention. Instead of the configuration shown in FIG. 4, the configuration shown in FIG. 6 can be adopted. 6, elements identical or similar to those in the configuration shown in FIG. 4 are denoted by the same reference numerals. Differences from FIG. 4 will be described.

The image recognition unit 164 of the medical image processing apparatus 160 shown in FIG. 6 is a processing unit that performs image recognition from the image acquired via the image acquisition unit 162 . The image recognition unit 164 is of category classification in which the classification of "image unsuitable for recognition" is added to the recognition processing of the classification task according to the classification method such as NICE classification or JNET classification. The image recognition unit 164 performs image classification processing including classification of “images unsuitable for recognition” in one recognition process from the input image.

Based on the recognition result of the image recognition unit 164, the determination unit 166 determines whether to save the recognition result.

<<Operation of the medical image processing apparatus according to the second embodiment>>
FIG. 7 is a flow chart showing the operation of the medical image processing apparatus according to the second embodiment. In FIG. 7, steps that are the same as or similar to those in the flowchart shown in FIG. 5 are given the same step numbers, and overlapping descriptions are omitted. Differences from FIG. 5 will be described.

The flowchart shown in FIG. 7 includes steps S15 and S17 instead of steps S14, S16 and S18 of FIG.

In step S15, the image recognition unit 164 performs recognition processing on the input image. The recognition processing performed here is, for example, image classification processing in which a category of “image unsuitable for recognition” is added to a classification category such as NICE classification or JNET classification.

In step S17, the CPU 70 determines whether or not the intended recognition has been achieved from the input image. The “targeted recognition” here means recognition of categories other than “images unsuitable for recognition”. If the recognition process in step S15 yields a recognition result indicating that the image is "unsuitable for recognition", the determination result in step S17 is "no". On the other hand, if the recognition process in step S15 obtains a recognition result of a category other than "image unsuitable for recognition", the determination result in step S17 is "Yes".

If the determination result of step S17 is "Yes", the process proceeds to step S20.

In step S<b>20 , the CPU 70 obtains the recognition result of the image recognition section 164 .

In step S<b>22 , CPU 70 stores the recognition result obtained in step S<b>20 in recognition result storage unit 168 .

On the other hand, when the determination result of step S17 is "no determination", it progresses to step S24.

In step S<b>24 , CPU 70 acquires the recognition result stored in recognition result storage unit 168 . In step S26, the CPU 70 changes the recognition result of the image recognition unit 164 to the recognition result saved in the recognition result saving unit 168 to obtain the recognition result.

The recognition result obtained in step S20 or step S26 is displayed on the display unit 174. FIG.

After step S22 or step S26, the flow chart of FIG. 7 ends. The processing of the flowchart in FIG. 7 is repeated for each input image acquired in time series.

<<Display example of recognition result>>
FIG. 8 is an example of an endoscopic diagnosis support screen displayed on the display unit 174. As shown in FIG. A window 300 illustrated in FIG. 8 includes an image display area 301 and a recognition result display area 310 . FIG. 8 shows a display example when an image suitable for recognition has been acquired. In the image display area 301, image contents of a medical image 302 captured using the electronic endoscope 12 are displayed in real time. FIG. 8 shows an example in which a lesion area 303 is included within a medical image 302 . A single image may include a plurality of lesion areas.

In the recognition result display area 310, for example, information indicating whether the recognition result is "cancerous" or "non-cancerous" is displayed. Also, in the recognition result display area 310, for example, the recognition result according to the NIEC classification is displayed.

FIG. 9 is another example of an endoscopic diagnosis support screen displayed on the display unit 174. As shown in FIG. FIG. 9 shows a display example when an image unsuitable for recognition is acquired. A medical image 302 displayed in the image display area 301 of the window 300 shown in FIG. 9 is, for example, a blurred image as an example of an image unsuitable for recognition. When such a blurred image is acquired, the recognition result obtained from the image recognition unit 164 is changed to the recognition result stored in the recognition result storage unit 168 . In the recognition result display area 310, the recognition result after change, that is, the recognition result saved in the recognition result saving unit 168 is displayed.

At this time, an icon 312 is displayed next to the display of the recognition result to inform the user that it is the previous recognition result.

When the image obtained from the electronic endoscope 12 returns to an image suitable for recognition, the screen in FIG. 9 returns to the screen in FIG.

<<Example of display indicating that the recognition result is stored in the recognition result storage unit 168>>
The display indicating that the recognition result is stored in the recognition result storage unit 168 is displayed in a different color or by adding a specific symbol, compared to the case where the recognition result of the image recognition unit 164 is displayed as it is. It may include at least one display form of symbol display, character string display with a specific character string added, and blinking display, and may be any combination thereof. The icon 312 shown in FIG. 9 is an example of symbol display. The different color display is not limited to changing the character color, and includes changing the background color.

<<Advantages of Embodiments of the Present Invention>>
According to the configuration of the embodiment described above, the following effects can be obtained.

(1) When an image that is not suitable for recognition, such as a blurred image, is temporarily included in time-series medical images, the most recent recognition result stored in the recognition result storage unit 168 is used to obtain the true image. It is possible to present highly accurate and appropriate recognition results.

(2) As a result, it is possible to prevent a recognition result (erroneous recognition result) having low authenticity from being generated from an image unsuitable for recognition and being provided to the user.

(3) Even if the recognition result of an image unsuitable for recognition is changed, if an image suitable for recognition is subsequently obtained, the processing of steps S20 to S22 is performed, so highly accurate recognition is possible. is.

(4) According to the embodiments of the present invention, diagnostic support useful for doctors and the like can be provided.

<<Third embodiment: Example of application to a medical information management system>>
The medical image processing apparatus according to the present invention is not limited to being applied to the processor device 16 of the endoscope system 10 illustrated in FIG. 1, and various applications are possible. For example, the medical image processing apparatus can be applied to a medical information management system that manages various medical information including endoscopic images.

FIG. 10 is a block diagram showing a configuration example of a medical information management system. The medical information management system 200 includes an image acquisition terminal 202 , an image storage server 204 , an information management device 210 , a display device 218 and an input device 219 . Each of the image capture terminal 202 , the image storage server 204 and the information management device 210 is connected to an electric communication line 230 . The term "connection" is not limited to wired connections, but also includes the concept of wireless connections.

Telecommunications line 230 may be a local area network or a wide area network. The electric communication line 230 is configured by an appropriate combination of wired and wireless.

The processor unit 16 of the endoscope system 10 is connected to the telecommunications line 230 . Medical images generated by processor device 16 are captured by at least one of image capture terminal 202 , image storage server 204 , and information management device 210 via telecommunications line 230 . For example, medical images generated by processor unit 16 are sent to image capture terminal 202 . Image capture terminal 202 receives medical images from processor unit 16 .

Image capture terminal 202 sends medical images received from processor unit 16 to image storage server 204 . The image capture terminal 202 may also transmit medical images received from the processor device 16 to the information management device 210 .

The image storage server 204 serves as a storage device that stores databases of various medical images. A cloud storage may be used instead of the image storage server 204 . In addition to the medical images, the image storage server 204 may also store image analysis results such as regions of interest (regions of interest) included in the medical images, presence or absence of a target of interest, and image classification results.

Although one endoscope system 10 is shown in FIG. 10 , multiple endoscope systems can be connected to the electrical communication line 230 . In addition, the electric communication line 230 may be connected to other medical image diagnostic apparatuses such as an ultrasonic diagnostic apparatus in addition to the endoscope system. An ultrasound image obtained from an ultrasound diagnostic apparatus is an example of a "medical image."

The information management device 210 is realized by computer hardware and software, for example. A display device 218 and an input device 219 are connected to the information management device 210 . The information management device 210 may include some or all of the functions of the medical image processing device 160 shown in FIG. 4 or FIG. For example, the information management device 210 includes functions of an image acquisition unit 162 , an image recognition unit 164 , a determination unit 166 , a recognition result storage unit 168 , a change processing unit 170 and a display control unit 172 . Display device 218 may function as display unit 174 . The functions of the information management device 210 can be realized by one or more computers, and can also be realized by cloud computing. The information management device 210 may include the functions of the image storage server 204 . Also, the image capture terminal 202 can function as the image acquisition unit 162 . A configuration in which the image capture terminal 202 is omitted is also possible, in which case the image storage server 204 and/or the information management device 210 are configured to capture medical images from the processor device 16 .

According to the medical information management system 200 configured as described above, not only medical images obtained in real time from the electronic endoscope 12 but also moving images stored in the image storage server 204 are played back. 5 or 7 can be performed, and recognition results can be displayed as described with reference to FIGS. 8 and 9. FIG.

The information management device 210 may be installed, for example, in an operating room, an examination room, or a conference room in a hospital, or may be installed in an out-of-hospital medical institution, research institution, or the like. The information management device 210 may be a workstation that supports examination, treatment, diagnosis, etc., or may be a business support device that supports medical work. The work support device may have a function of accumulating clinical information, assisting in the creation of diagnostic documents, assisting in the creation of reports, and the like.

<<Hardware configuration of each processing unit and control unit>>
Various processes such as the image acquisition unit 162, the image recognition unit 164, the determination unit 166, the recognition result storage unit 168, the change processing unit 170, and the display control unit 172 of the medical image processing apparatus 160 described in FIGS. The hardware structure of the executing processing unit is various processors as follows.

The various processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and function as various processing units, and FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. Programmable Logic Devices (PLDs), ASICs (Application Specific Integrated Circuits), and other dedicated electric circuits, which are processors having circuit configurations specially designed to execute specific processing, are included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs or a combination of a CPU and an FPGA. Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with a single processor, first, as represented by a computer such as a client or a server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

<<Modification 1>>
The electronic endoscope is not limited to a flexible endoscope, and may be a rigid endoscope or a capsule endoscope. Further, an apparatus that generates time-series medical images including subject images is not limited to an electronic endoscope, and may be, for example, an ultrasonic diagnostic apparatus.

<<Modification 2>>
The medical image processing apparatus of the present disclosure can be used as a diagnosis support apparatus that supports examination, treatment, diagnosis, or the like by a doctor or the like. The term "diagnostic aid" includes the concepts of diagnostic aid and/or therapeutic aid.

《About the observation light of the endoscope system》
As the observation light, light in various wavelength bands such as white light, light in one or more specific wavelength bands, or a combination thereof is selected according to the purpose of observation. White light is light in a white wavelength band or light in multiple wavelength bands. A "specific wavelength band" is a band narrower than the white wavelength band. Specific examples for specific wavelength bands are shown below.

<First example>
A first example of a specific wavelength band is, for example, the blue band or the green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength of 530 nm or more and 550 nm or less. It has a peak wavelength within the band.

<Second example>
A second example of a specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example is within the wavelength band of 585 nm or more and 615 nm or less or a wavelength of 610 nm or more and 730 nm or less. It has a peak wavelength within the band.

<Third example>
A third example of the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. have The wavelength band of this third example includes a wavelength band of 400 ± 10 nm, a wavelength band of 440 ± 10 nm, a wavelength band of 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is the above 400 ± 10 nm , 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

<Fourth example>
A fourth example of the specific wavelength band is a wavelength band of excitation light that is used for observing fluorescence emitted by a fluorescent substance in vivo (fluorescence observation) and that excites this fluorescent substance, for example, from 390 nm to 470 nm.

<Fifth example>
A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength of 905 nm or more and 970 nm or less. It has a peak wavelength within the band.

《Switching observation light》
A laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof can be used as the light source. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject and the purpose of observation. and/or preferably switch. When switching the wavelength, for example, by rotating a disc-shaped filter (rotary color filter) that is placed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the irradiated light is switched. good too.

The imaging device used in the electronic endoscope is not limited to a color imaging device in which a color filter is provided for each pixel, and may be a monochrome imaging device. When a monochrome imaging device is used, it is possible to sequentially switch the wavelength of illumination light and perform frame sequential (color sequential) imaging. For example, the wavelength of emitted illumination light may be sequentially switched between violet, blue, green, and red, or broadband light (white light) may be emitted and emitted by a rotary color filter (red, green, blue, etc.). The wavelength of illumination light may be switched. Alternatively, one or a plurality of narrow band lights may be applied and the wavelength of the illumination light emitted by the rotary color filter may be switched. The narrowband light may be infrared light having two or more wavelengths different from each other.

<<Generation example of special light image>>
The processor device 16 may generate a special light image having information of a specific wavelength band based on a normal light image obtained by imaging using white light. Note that the generation here includes the concept of “acquisition”. In this case, the processor device 16 functions as a special light image acquisition unit. The processor unit 16 converts signals in specific wavelength bands into red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y), which are normally included in an optical image. ) can be obtained by performing an operation based on the color information.

《Generation example of feature quantity image》
The processor device 16 produces, as medical images, a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band, and a special light image obtained by irradiating light in a specific wavelength band. may be used to generate the feature image. A feature image is a form of medical image.

《About the program that realizes the functions of a medical image processing device on a computer》
A program that causes a computer to realize the functions of the medical image processing apparatus described in the above embodiments is recorded on a computer-readable medium that is a non-temporary information storage medium that is an optical disk, magnetic disk, semiconductor memory, or other tangible object, and this It is possible to provide the program through an information storage medium. Instead of storing the program in a tangible non-temporary information storage medium and providing the program, it is also possible to provide the program signal as a download service using an electric communication line such as the Internet.

It is also possible to provide a part or all of the functions of the medical image processing apparatus described in the above embodiment as an application server, and provide a service of providing processing functions through an electric communication line.

<<Combination of Embodiments and Modifications>>
The components described in the above embodiments and the components described in the modified examples can be used in combination as appropriate, and some of the components can be replaced.

<<Note>>
In addition to the above-described embodiments, modifications, and the like, the present specification includes the disclosure of the configuration inventions described below.

(Appendix 1)
The medical image processing apparatus has a medical image analysis processing unit and a medical image analysis result acquisition unit. A medical image processing apparatus that detects an area and a medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

The medical image analysis processing section may include an image recognition section.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of a target of interest based on the pixel feature amount of the medical image, and the medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit Medical image processing. Device.

(Appendix 3)
The medical image analysis result acquisition unit acquires the analysis result of the medical image from a recording device that records the analysis result. , or both.

(Appendix 4)
A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band.

(Appendix 5)
A medical image processing apparatus in which a medical image is an image obtained by irradiating light of a specific wavelength band, and the specific wavelength band is a narrower band than the wavelength band of white.

(Appendix 6)
A medical image processing device in which the specific wavelength band is a visible blue or green band.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A medical image processing device whose specific wavelength band is the visible red band.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. Medical imaging equipment.

(Appendix 11)
The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ± A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or from 600 nm to 750 nm.

(Appendix 12)
A medical image is an in-vivo image of the inside of a living body, and the in-vivo image is a medical image processing apparatus having information on fluorescence emitted by fluorescent substances in the living body.

(Appendix 13)
Fluorescence is a medical image processing apparatus obtained by irradiating the living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
A medical image processing apparatus in which the medical image is an in vivo image of the inside of a living body, and the specific wavelength band is the wavelength band of infrared light.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating light of a plurality of wavelength bands as light of a white band or light of a white band. A medical image processing apparatus comprising an optical image acquisition unit, wherein the medical image is a special optical image.

(Appendix 17)
A medical image processing apparatus in which a signal in a specific wavelength band is obtained by calculation based on RGB or CMY color information normally included in an optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light of a plurality of wavelength bands as white band light or light of a white band, and a special light image obtained by irradiating light of a specific wavelength band, A medical image processing apparatus comprising a feature amount image generating unit for generating a feature amount image, wherein the medical image is the feature amount image.

(Appendix 19)
The medical image processing apparatus according to any one of appendices 1 to 18, and an endoscope that acquires an image by irradiating at least one of light in a white wavelength band and light in a specific wavelength band , an endoscopic device comprising:

(Appendix 20)
A diagnosis support apparatus comprising the medical image processing apparatus according to any one of appendices 1 to 18.

(Appendix 21)
A medical service support apparatus comprising the medical image processing apparatus according to any one of appendices 1 to 18.

[others]
In the embodiments of the present invention described above, it is possible to appropriately change, add, or delete constituent elements without departing from the gist of the present invention. The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention by those having ordinary knowledge in equivalent and related fields.

10 endoscope system 12 electronic endoscope 14 light source device 16 processor device 18 display device 19 input device 20 insertion section 22 distal end rigid section 22A distal end surface 24 bending section 26 flexible section 30 operating section 31 angle knob 32 air/water supply button 33 Suction button 34 Mode changeover switch 35 Zoom operation unit 36 Treatment instrument introduction port 40 Universal cord 42 Connector 42A Video connector 42B Light guide connector 42C Water supply connector 44 Water supply tank 50 Illumination window 52 Observation window 54 Forceps outlet 56 Air/water supply nozzle 60 Objective Optical system 62 Image sensor 64 Analog front end circuit 65 Timing generator 66 CPU
68 imaging chip 70 CPU
72 ROMs
74 RAM
76 digital signal processing circuit 78 display control circuit 100 first light source 101 first light source drive circuit 102 second light source 103 second light source drive circuit 104 CPU
105 Multiplexer 120 Light guide 121 Incident end 122 Output end 124 Phosphor 160 Medical image processing device 162 Image acquisition unit 164 Image recognition unit 164A First recognition unit 164B Second recognition unit 166 Judging unit 168 Recognition result storage unit 170 Change processing unit 172 Display control unit 174 Display unit 200 Medical information management system 202 Image capture terminal 204 Image storage server 210 Information management device 218 Display device 219 Input device 230 Electric communication line 300 Window 301 Image display area 302 Medical image 303 Lesion area 310 Recognition result display area 312 Icons IM1, IM2, IM3 Images S12 to S26 Steps of processing in the medical image processing apparatus

Claims (27)

A processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor
Acquire time-series medical images including subject images,
performing image recognition from the medical image;
storing a recognition result of the image recognition in the memory;
A medical image processing device for displaying the saved recognition result,
The image recognition processing includes a task of recognizing that the medical image is an image unsuitable for recognition,
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, the processor acquires the first medical image before the first medical image. performing processing for displaying a recognition result of the obtained second medical image together with the first medical image;
Medical imaging equipment. The processor
When the image recognition recognizes that the first medical image is an image unsuitable for recognition, the recognition result of the second medical image is displayed without displaying the recognition result of the first medical image. performing processing for displaying together with the first medical image;
The medical image processing apparatus according to claim 1. The processor
determining whether or not to store the recognition result of the image recognition according to the recognition result of the image recognition, and storing the recognition result of the image recognition in the memory according to the determination result;
The medical image processing apparatus according to claim 1 or 2. The processor
performing processing for displaying the recognition result stored in the memory without displaying the recognition result of the image recognition for the medical image for which the determination that the recognition result is not to be stored is given in the determination; Item 4. The medical image processing apparatus according to item 3. The processor
When the image recognition process recognizes that the image is unsuitable for recognition, determining that the recognition result of the medical image recognized as the unsuitable image is not saved;
The medical image processing apparatus according to claim 3 or 4. Recognition results of the medical image recognized by the processor as being unsuitable for recognition are not stored in the memory.
The medical image processing apparatus according to claim 1. The unsuitable image is an image in which the subject is blurred.
The medical image processing apparatus according to any one of claims 1 to 6. the unsuitable image is an image in which the subject is blurred;
The medical image processing apparatus according to any one of claims 1 to 6. The inappropriate image is an image in which the subject is covered with water.
The medical image processing apparatus according to any one of claims 1 to 6. The unsuitable image is an image with residue on the subject,
The medical image processing apparatus according to any one of claims 1 to 6. The image recognition performed by the processor includes a task of detecting a region of interest.
The medical image processing apparatus according to any one of claims 1 to 10. The image recognition performed by the processor includes image classification.
The medical image processing apparatus according to any one of claims 1 to 11. The image recognition processing includes first recognition processing for recognizing whether or not the acquired medical image is an image unsuitable for recognition;
a second recognition process that performs at least one of detection of a region of interest from the acquired medical image and image classification of the medical image;
including
The processor
Not performing the second recognition process for the medical image recognized as an image unsuitable for recognition by the first recognition process;
The medical image processing apparatus according to any one of claims 1 to 12. The processor
performing the image recognition for each of the medical images acquired in time series;
outputting a recognition result for each of the medical images;
A medical image processing apparatus according to any one of claims 1 to 13. The processor
A process of changing the recognition result of the first medical image to the recognition result of the second medical image when the image recognition recognizes that the first medical image is an image unsuitable for recognition. and
A recognition result of the medical image in which the recognition result of the image recognition is maintained is stored in the memory without undergoing the modification process.
15. The medical image processing apparatus according to any one of claims 1-14. The processor
performing processing for displaying the recognition result of the latest second medical image stored in the memory instead of the recognition result of the first medical image;
16. The medical image processing apparatus according to any one of claims 1-15. A display unit for displaying the recognition result of the image recognition,
17. The medical image according to any one of claims 1 to 16, wherein a recognition result of said second medical image stored in said memory is displayed on said display unit according to a recognition result of said image recognition. processing equipment. The processor
When displaying the recognition result of the second medical image stored in the memory, displaying the recognition result stored in the memory;
17. The medical image processing apparatus according to any one of claims 1-16. The display indicating that the recognition result is stored in the memory includes display in a different color, symbol display with a specific symbol, Including at least one display form of character string display with a specific character string added and blinking display,
The medical image processing apparatus according to claim 18. The processor
stopping the recognition process of the image recognition when the acquisition of the medical image recognized as being unsuitable for recognition by the image recognition continues for a certain period of time;
20. A medical image processing apparatus according to any one of claims 1-19. The processor
performing processing for displaying a message to call the attention of a user when the acquisition of the medical image, which has been recognized as being unsuitable for recognition by the image recognition, continues for a certain period of time;
21. The medical image processing apparatus according to any one of claims 1-20. The medical image processing apparatus according to any one of claims 1 to 21, wherein the medical image is an endoscopic image obtained from an endoscope or an ultrasonic image obtained from an ultrasonic diagnostic apparatus. A medical image processing method executed by a processor, comprising:
Acquiring a time series of medical images including an image to be mapped;
performing image recognition from the medical image;
storing a recognition result of the image recognition in a memory;
displaying the saved recognition results;
the image recognition processing includes a task of recognizing that the acquired medical image is an image unsuitable for recognition;
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, a second medical image acquired earlier than the first medical image display the recognition result for the medical image of the above together with the first medical image;
Medical image processing method. an electronic endoscope for imaging inside body cavities;
and a processor device for processing an image signal obtained from the electronic endoscope,
The processor device
A processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor
Acquiring time-series medical images including subject images captured using the electronic endoscope;
performing image recognition from the medical image;
storing a recognition result of the image recognition in the memory;
performing processing for displaying the saved recognition result;
The image recognition processing includes a task of recognizing that the medical image is an image unsuitable for recognition,
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, the processor acquires the first medical image before the first medical image. performing processing for displaying a recognition result of the obtained second medical image together with the first medical image;
endoscope system. A processor device for processing an image signal obtained from an electronic endoscope,
A processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor
Acquiring time-series medical images including subject images captured using the electronic endoscope;
performing image recognition from the medical image;
storing a recognition result of the image recognition in the memory;
performing processing for displaying the saved recognition result;
The image recognition processing includes a task of recognizing that the medical image is an image unsuitable for recognition,
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, the processor acquires the first medical image before the first medical image. performing processing for displaying a recognition result of the obtained second medical image together with the first medical image;
processor unit. A processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor
Acquire time-series medical images including subject images,
performing image recognition from the medical image;
storing a recognition result of the image recognition in the memory;
A diagnostic support device that performs a process of notifying a recognition result of the image recognition,
The image recognition processing includes a task of recognizing that the medical image is an image unsuitable for recognition,
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, the processor acquires the first medical image before the first medical image. performing processing for displaying a recognition result of the obtained second medical image together with the first medical image;
Diagnostic support device. to the computer,
a process of acquiring time-series medical images including the image to be mapped;
a process of performing image recognition from the medical image;
a process of storing the recognition result of the image recognition in a memory;
A program for executing a process of displaying the saved recognition result,
the image recognition processing includes a task of recognizing that the acquired medical image is an image unsuitable for recognition;
When the first medical image included in the time-series medical images is recognized as an image unsuitable for recognition by the image recognition, a second medical image acquired earlier than the first medical image executing a process for displaying a recognition result of the medical image of together with the first medical image;
program.

Images