CN115336969B

CN115336969B - A multimodal intracavitary imaging system and a multimodal data fusion method

Info

Publication number: CN115336969B
Application number: CN202110516333.XA
Authority: CN
Inventors: 朱锐; 鲁全茂
Original assignee: SHENZHEN VIVOLIGHT MEDICAL DEVICE & TECHNOLOGY CO LTD
Current assignee: SHENZHEN VIVOLIGHT MEDICAL DEVICE & TECHNOLOGY CO LTD
Priority date: 2021-05-12
Filing date: 2021-05-12
Publication date: 2025-08-26
Anticipated expiration: 2041-05-12
Also published as: CN115336969A

Abstract

The present application provides a multimodal intracavitary imaging system and a multimodal data fusion method. The multimodal intracavitary imaging system includes: a first swept-frequency light source, a second swept-frequency light source, a first fiber coupler, a time division multiplexer and an imaging catheter; the first swept-frequency light source is connected to the first fiber coupler; the sample arm of the first fiber coupler is connected to the imaging catheter optical signal through the time division multiplexer; the second swept-frequency light source is connected to the imaging catheter optical signal through the time division multiplexer; wherein, the first swept-frequency light source and the second swept-frequency light source can emit light of different central wavelengths; the imaging catheter can detect and image the intracavitary tissue based on the light emitted by the first swept-frequency light source and the second swept-frequency light source. Based on the multimodal intracavitary imaging system and multimodal data fusion method provided by the present application, the lipid level of lipid plaques in the patient's blood vessels can be accurately assessed.

Description

Multi-mode intracavity imaging system and multi-mode data fusion method

Technical Field

The application relates to the technical field of medical equipment and signal processing, in particular to a multi-mode intracavity imaging system and a multi-mode data fusion method.

Background

Most cases of acute coronary syndromes are rupture of thin cystic fibroaneurysms (i.e., vulnerable plaque), resulting in thrombus formation within the coronary arteries. The vulnerable plaque existing in the coronary artery blood vessel of the patient is timely and accurately identified, and the distribution situation of the vulnerable plaque is analyzed and evaluated, so that a doctor is guided to put forward an effective treatment scheme for the patient.

Optical coherence tomography (optical coherence tomography, OCT) is based on the principle of bottom coherence interferometry to obtain a depth-wise chromatographic capability, by scanning to reconstruct a two-or three-dimensional image of the internal structure of a biological tissue or material whose signal contrast results from the spatial variation of the internal optical reflection or scattering properties of the biological tissue or material. OCT can identify structural features of coronary plaque that cannot be accurately identified by angiography or intravascular ultrasound. Near infrared spectroscopy (Near-infrared spectroscopy, NIRS) is a novel intravascular imaging technique that provides a chemical assessment of the presence of cholesterol esters in the lipid core and distinguishes cholesterol from collagen in tubular plaques by its spectral fingerprint. NIRS can accurately detect lipid-rich plaques in human tissue.

However, the evaluation method based on the single-mode image can only evaluate the vulnerable plaque from the structural or component angle, can not accurately judge the lipid degree of the vulnerable plaque, and is also unfavorable for doctors to propose an adaptive treatment scheme.

Disclosure of Invention

The embodiment of the application provides a lipid plaque evaluation system and a lipid plaque evaluation method based on multi-mode intracavity imaging, which can accurately evaluate the lipid degree of a lipid plaque in a patient.

In a first aspect, an embodiment of the present application provides a multi-modal intracavity imaging system including a first swept source, a second swept source, a first fiber coupler, a time division multiplexer, and an imaging catheter;

The first sweep frequency light source is connected with the first optical fiber coupler, the sample arm of the first optical fiber coupler is connected with the imaging catheter optical signal through the time division multiplexer, and the second sweep frequency light source is connected with the imaging catheter optical signal through the time division multiplexer;

The imaging catheter can detect and image tissues in a cavity according to the light emitted by the first sweep light source and the second sweep light source.

The multimode intracavity imaging system provided by the application is provided with the first sweep light source and the second sweep light source which can emit light with different central wavelengths, the light emitted by the first sweep light source enters the imaging catheter to detect intravascular tissues through the time division multiplexer after being split by the first optical fiber coupler, and the light emitted by the second sweep light source enters the imaging catheter to detect intravascular tissues through the time division multiplexer, so that the first sweep light source can identify the surface microstructure in a blood vessel, and the second sweep light source can evaluate the lipid content of a corresponding area of the inner wall of the blood vessel. Compared with the prior art, when the lipid content is evaluated, light with a central wavelength of a better absorption peak can be adopted for evaluation and identification, so that the reproducibility of identification and imaging of lipid plaques or vulnerable plaques can be improved.

Optionally, the imaging catheter comprises two imaging probes, and one of the two imaging probes is connected with the first sweep light source and the second sweep light source respectively;

The imaging probes respectively connected with the first sweep light source and the second sweep light source can be used for irradiating light emitted by the first sweep light source and the second sweep light source onto the tissue in the cavity and receiving echo signals, and the other one of the two imaging probes is used for receiving the echo signals of the second sweep light source.

Based on the above alternative mode, the first sweep light source and the second sweep light source are respectively irradiated onto the vascular tissue through one imaging probe, and the echo light of the lipid plaque is received through the other imaging probe. Thus, mutual interference between the echo light and the incident light does not occur, so that the accuracy of detection imaging and the degree of reproduction of the evaluation result can be improved.

Optionally, the multi-mode intra-cavity imaging system further comprises a rotary joint connected between the time division multiplexer and the imaging catheter, and echo signals of the two imaging probes can be transmitted through the rotary joint in a beam splitting way.

Optionally, the multi-mode intra-cavity imaging system further includes a first fiber optic circulator connected between the time division multiplexer and the rotary joint, a port of the first fiber optic circulator is connected to the time division multiplexer, two ports of the first fiber optic circulator are connected to a first port of the rotary joint, and a second port of the rotary joint is connected to the imaging catheter;

the echo of the imaging probe connected with the first sweep frequency light source can enter the first optical fiber circulator through two ports of the first optical fiber circulator and is transmitted outwards from three ports of the first optical fiber circulator.

Based on the above-mentioned optional mode, incident light can only propagate along a direction in first optic fibre circulator, and the light that gets into from one port can only be followed two ports and launched, and the light that gets into from two ports can only be followed three ports and launched, can effectively avoid taking place to interfere between the echo light that the light that first sweep frequency light source sent and imaging probe obtained.

Optionally, the multi-mode intracavity imaging system further comprises a second fiber circulator, a collimator, a mirror, a second fiber coupler, and a balanced photodetector;

The reference arm of the first optical fiber coupler is connected with the collimator through the second optical fiber circulator, light emitted by the collimator can be reflected through the mirror surface and is transmitted outwards from the three ports of the second optical fiber circulator through the two ports of the second optical fiber circulator, the three ports of the first optical fiber circulator and the three ports of the second optical fiber circulator are connected with the balance photoelectric detector through the second optical fiber coupler, and the balance photoelectric detector is used for converting an optical signal output by the second optical fiber coupler into an electric signal.

Based on the above optional mode, after the light emitted by the first sweep light source is split by the first optical fiber coupler, the reference light sequentially passes through the second optical fiber circulator and the collimator and is reflected by the mirror surface, and by utilizing the characteristic that the incident light can only propagate along one direction in the second optical fiber circulator, interference between the incident reference light and the reflected light of the mirror surface can be avoided. By arranging the balance photoelectric detector, the influence of receiver noise and electronic circuit noise on detection of weak light signals can be reduced, and the imaging precision can be improved.

Optionally, the multi-mode intracavity imaging system further comprises a control device, wherein the control device is connected with the balance photoelectric detector and is connected with the third port of the rotary joint through the photoelectric detector, and the control device is used for analyzing and displaying and imaging the electric signals output by the balance photoelectric detector and analyzing and displaying and imaging the echoes of the imaging probe. The control device is respectively connected with the first frequency scanning light source and the second frequency scanning light source, and the control device respectively controls the first frequency scanning light source and the second frequency scanning light source to be started alternately.

Based on the above-mentioned optional mode, through controlling means control first sweep frequency light source and second sweep frequency light source luminous in different time quantum, can avoid the mutual interference between the light that first sweep frequency light source and second sweep frequency light source sent. The control device may also process the electrical signals generated by the photodetectors and the balanced photodetectors, respectively, convert them into images or other data and display the images so as to guide the physician in analyzing vulnerable plaque within the blood vessel.

In a second aspect, an embodiment of the present application provides a multi-modality data fusion method, including acquiring a plurality of OCT image sequences and NIRS image sequences of an intra-luminal tissue acquired by the multi-modality intra-luminal imaging system described in the first aspect, the OCT image sequences including a plurality of OCT images, the NIRS image sequences including a plurality of NIRS images in one-to-one correspondence with the plurality of OCT images;

registering each OCT image and an NIRS image corresponding to the OCT image, identifying a lipid plaque area in each OCT image, determining a first determination value of each OCT image according to the areas of the lipid plaque areas of a plurality of OCT images, determining a second determination value of each NIRS image according to the sampling frequency adopted when the NIRS image sequence is acquired and the number of samples of each NIRS image, wherein the number of the samples of each NIRS image is the number that the absorption value is larger than a threshold value, and fusing the first determination value of each OCT image and the second determination value of each NIRS image corresponding to each OCT image to determine the lipid degree indicated by each OCT image.

Based on the multi-mode data fusion method provided by the application, each OCT image and the corresponding NIRS image are registered one by one, then the size and the structure of the plaque in the blood vessel are determined by identifying the lipid plaque area contained in the OCT image, and the first determination value of each OCT image is determined according to the areas of the lipid plaque areas of a plurality of OCT images. Meanwhile, whether the tissue in the blood vessel is lipid is determined according to the absorption value of the tissue in the blood vessel on the spectrum displayed in the NIRS image, so that a second determination value of each NIRS image is determined according to the sampling frequency and the number of samples of each NIRS image. The first decision value and the second decision value are fused, and the lipid level indicated by each OCT image can be determined. Compared with the existing method for evaluating the lipid plaque or vulnerable plaque based on the single-mode image, the method utilizes the complementarity of the OCT image and the NIRS image to fuse the image data of multiple modes, so that the lipid degree of the lipid plaque is evaluated, and the accuracy of the evaluation is improved.

Optionally, the registering each OCT image and the NIRS image corresponding to the OCT image includes:

Inputting the OCT image and the NIRS image corresponding to the OCT image into a trained target detection network respectively, identifying the positions of guide wires in the OCT image and the NIRS image, and adjusting the rotation angle of a pixel matrix of the OCT image relative to a pixel matrix of the NIRS image according to the offset angle of the positions of the guide wires so as to register the OCT image and the NIRS image.

Based on the above optional modes, the offset degree of each OCT image relative to the NIRS image is determined according to the position of the guide wire in each OCT image and the corresponding NIRS image, and the rotation angle of each OCT image relative to the NIRS image is adjusted according to the rotation angle of the guide wire in the two images, so that each OCT image and the corresponding NIRS image are registered one by one, thereby facilitating the subsequent data fusion.

Optionally, the identifying the lipid plaque region in each of the OCT images includes:

Inputting each OCT image into a trained U-Net network respectively to obtain a first lipid segmentation mask of each OCT image;

acquiring a multidimensional feature vector of each OCT image, and inputting the multidimensional feature vector into a trained random forest to obtain a second lipid segmentation mask of each OCT image;

The lipid plaque region in each of the OCT images is determined from the first lipid segmentation mask and the second lipid segmentation mask of each of the OCT images, respectively.

Based on the above optional manner, the semantic segmentation algorithm and the traditional digital image processing method are combined, a plurality of lipid segmentation masks of the OCT image are respectively obtained through a plurality of algorithms, and then the lipid segmentation masks are subjected to AND operation, so that the lipid plaque area in the OCT image is identified, and the accuracy of identifying the lipid plaque area can be improved.

Optionally, the determining the first determination value of each OCT image according to the areas of the lipid plaque areas of the plurality of OCT images includes:

And dividing the area of the lipid plaque area in each OCT image by the maximum value in the areas of the lipid plaque areas in the OCT images respectively to obtain a first determination value of each OCT image.

Optionally, the determining a second determination value of each NIRS image according to a sampling frequency adopted when the NIRS image sequence is acquired and a sample number of each NIRS image includes:

and dividing the sample number of the NIRS images with the sampling frequency adopted when a plurality of NIRS image sequences are acquired for each NIRS image to obtain a second determination value of each NIRS image.

Optionally, the fusing the first determination value of each OCT image with the second determination value of the NIRS image corresponding to each OCT image, to determine the lipid level indicated by each OCT image includes:

For each OCT image, determining a support matrix according to a first decision value of the OCT image and a second decision value of a NIRS image corresponding to the OCT image;

acquiring a weight coefficient according to a feature vector corresponding to the maximum feature value of the support matrix;

the first judgment value of the OCT image and the second judgment value of the NIRS image corresponding to the OCT image are weighted and summed according to the weight coefficient to obtain a fusion value;

Inputting the fusion value into a Kalman filter to obtain an evaluation parameter of each OCT image, wherein the evaluation parameter is used for representing the lipid degree.

In a third aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the method of any of the second aspects above.

In a fourth aspect, embodiments of the present application provide a computer program product for, when run on a terminal device, causing the terminal device to perform the method of any of the second aspects described above.

In a fifth aspect, embodiments of the present application also provide a network device comprising at least one processor, a memory and a computer program stored in the memory and executable on the at least one processor, the processor implementing the method of any one of the second aspects when executing the computer program.

It will be appreciated that the advantages of the third to fifth aspects may be found in the relevant description of the first and second aspects, and are not described here again.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a block diagram of the overall architecture of an intravascular imaging system according to an embodiment of the application;

FIG. 2 is a block diagram illustrating an internal structure of a rotary joint according to an embodiment of the present application;

FIG. 3 is an image of tissue within a lumen of a blood vessel acquired based on the system shown in FIG. 1, in accordance with an embodiment of the present application;

FIG. 4 is a flowchart of a multi-modal data fusion method according to an embodiment of the present application;

FIG. 5 is a diagram of a multi-modal data fusion result provided by an embodiment of the present application.

Reference numerals illustrate 101, a first sweep light source, 102, a second sweep light source, 103, a first optical fiber coupler, 104, a time division multiplexer, 105, an imaging catheter, 1051, an imaging probe, 106, a rotary joint, 1061, a third optical fiber coupler, 107, a first optical fiber circulator, 108, a photoelectric detector, 109, a control device, 1091, a signal processing and displaying device, 1092, a control device, 110, a second optical fiber circulator, 111, a collimator, 112, a mirror, 113, a second optical fiber coupler, 114, a balance photoelectric detector, 200, a vascular wall, 300 and a lipid plaque.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

OCT is based on the principle of low coherence interferometry to obtain a tomographic image in the depth direction, from which a two-or three-dimensional image of the internal structure of a biological tissue or material can be reconstructed by scanning, whose contrast of the signal is due to the spatial variation of the internal optical reflection (scattering) characteristics of the biological tissue or material. It can identify coronary plaque features that cannot be accurately identified by angiography or intravascular ultrasound. While the excellent spatial resolution of OCT can identify many plaque structural features, OCT cannot identify the constituent components of plaque. NIRS is a novel intravascular imaging technique which can provide chemical assessment related to the existence of cholesterol esters in lipid nuclei, and can distinguish cholesterol from collagen in coronary plaques through a unique spectral fingerprint spectrum, and can accurately detect the plaques rich in lipid in human tissues.

The prior art detects plaque based on only a single modality image. Specifically, the structures of different plaques are identified using only OCT images, but specific components of the plaques (e.g., fibers, lipids, calcifications, fibrous lipids, etc.) cannot be accurately distinguished. Or just using NIRS (Near-infrared spectroscopy, NIRS) images to observe plaque distribution in the blood vessel, but not to obtain specific locations of plaque. Therefore, the mixed imaging technology based on OCT and NIRS can realize accurate prediction of the structures and components of the lipid plaques in the blood vessels of patients, and has great positive significance for doctors and patients.

In order to solve the technical problems, the embodiment of the application provides a lipid plaque evaluation system and a lipid plaque evaluation method based on multi-mode intracavity imaging.

The technical scheme of the application is described in detail below with reference to the accompanying drawings. The embodiments described below by referring to the drawings are illustrative and intended to explain the present application and should not be construed as limiting the application.

Aiming at the problems of the OCT/NIRS hybrid imaging technology, the embodiment of the application provides a multi-mode intracavity imaging system, which adopts two sweep-frequency light sources, wherein one sweep-frequency light source emits light with a wavelength length suitable for OCT imaging, the other sweep-frequency light source can emit light with a wavelength length suitable for NIRS to evaluate and analyze lipid plaques, and the two sweep-frequency light sources are connected to an imaging probe through a time division multiplexer, so that NIRS evaluation results of OCT imaging with higher imaging precision and better absorption peak spectrum can be respectively obtained, the reproducibility of lipid content evaluation of a corresponding region of a blood vessel wall can be improved, and an accurate evaluation result is provided for guiding doctors.

Specifically, referring to fig. 1, fig. 1 is an overall block diagram of an imaging system in a chute according to an embodiment of the present application. In one possible implementation, the present application provides a multi-modality intra-cavity imaging system comprising a first swept optical source 101, a second swept optical source 102, a first fiber coupler 103, a time division multiplexer 104, and an imaging catheter 105.

Specifically, in the embodiment of the present application, the first swept light source 101 and the second swept light source 102 may be swept laser light sources, such as superluminescent diodes.

Wherein the first swept optical source 101 may emit light of a wavelength length suitable for OCT imaging, for example, near infrared light having a center wavelength of 1310 nm. The second swept optical source 102 may emit light of a wavelength length suitable for NIRS evaluation imaging, such as near infrared light having a center wavelength of 1200 nm.

Specifically, in the embodiment of the present application, the first fiber coupler 103 may be a 1x2 type fiber coupler, and the spectral ratio thereof may be 90/10, where 90 spectroscopy may be used as sample light, to-be-detected samples (for example, vascular tissues) are detected, and 10 spectroscopy may be used as reference light, mixed with the detected echo, and used as a reference.

In the embodiment of the application, a first sweep light source 101 is connected with a first optical fiber coupler 103, a sample arm of the first optical fiber coupler 103 is connected with an imaging catheter 105 through a time division multiplexer 104 in an optical signal manner, and a second sweep light source 102 is connected with the imaging catheter 105 through the time division multiplexer 104 in an optical signal manner.

Specifically, the tail fiber of the first sweep frequency light source 101 is connected to the first optical fiber coupler 103, and the sample arm of the first optical fiber coupler 103 may specifically be that after the 90-beam splitting arm is connected to the time division multiplexer 104, the time division multiplexer 104 is connected to the imaging catheter 105. In this way, the light emitted by the first swept optical source 101 may be directed through the first fiber coupler 103, the time division multiplexer 104, and the imaging catheter 105 onto vascular tissue, and the cavity may be imaged according to the echo signal.

It should be noted that, in the embodiment of the present application, the tail fiber of the second swept optical source 102 is connected to the imaging catheter 105 through the time division multiplexer 104, so that the light emitted by the second swept optical source 102 can be irradiated onto the vascular tissue through the time division multiplexer 104 and the imaging catheter 105, and the lipid plaque in the blood vessel is evaluated according to the echo signal.

It will be appreciated that the time division multiplexer 104 may simultaneously transmit two or more optical wavelength signals through different optical channels in the same optical fiber, so that the first swept optical source 101 and the second swept optical source 102 may propagate in the same optical fiber to the imaging catheter 105 after passing through the time division multiplexer 104.

According to the embodiment of the application, the first sweep light source 101 and the second sweep light source 102 capable of emitting light with different central wavelengths are arranged, the light emitted by the first sweep light source 101 is split by the first optical fiber coupler 103, then sample light enters the imaging catheter 105 through the time division multiplexer 104 to detect intravascular tissues, and the light emitted by the second sweep light source 102 enters the imaging catheter 105 through the time division multiplexer 104 to detect intravascular tissues, so that the first sweep light source 101 can be used for independently identifying the surface microstructure in a blood vessel, and the second sweep light source 102 can be used for independently evaluating the lipid content of a corresponding area of the inner wall of the blood vessel. Compared with the prior art, when the lipid content is evaluated, light with a central wavelength of a better absorption peak can be adopted for evaluation and identification, so that the reproducibility of identification and imaging of lipid plaques or vulnerable plaques can be improved.

In one embodiment, as shown in fig. 1, the imaging catheter 105 includes two imaging probes 1051, one of the two imaging probes 1051 being connected to a first swept optical source 101 and a second swept optical source 102,

The imaging probes 1051 respectively connected with the first scanning light source 101 and the second scanning light source 102 can be used for irradiating light emitted by the first scanning light source 101 and the second scanning light source 102 onto the tissue in the cavity and receiving the echo signals, and the other one of the two imaging probes 1051 is used for receiving the echo signals of the second scanning light source 102.

Specifically, referring to fig. 1, in an embodiment of the present application, an optical fiber coupler may be disposed in the imaging catheter 105, and the optical fiber coupler may be of a type 1×2, and two ports of the optical fiber coupler may be respectively connected to two imaging probes 1051, so as to irradiate light emitted by the first scanning light source 101 and the second scanning light source 102 from the imaging probes 1051 onto the vascular tissue 200 and the lipid plaque 300.

After the near infrared light impinges on the vascular tissue 200 or the lipid plaque 300, it is reflected or scattered by the vascular tissue 200 or the lipid plaque 300 and the two imaging probes 1051 may receive, detect or detect the reflected or scattered back echo light.

Specifically, referring to fig. 1, in an embodiment of the present application, an imaging probe 1051 located at the lower side shown in fig. 1 may be connected to the first swept optical source 101 and the second swept optical source 102, respectively. That is, the imaging probe 1051 may irradiate light emitted from the first and second swept light sources 101 and 102 onto the blood vessel tissue 200 or the lipid plaque 300, respectively, and receive echo light reflected or scattered by the light of the first and second swept light sources 101 and 102 through the blood vessel tissue 200 or the lipid plaque 300, respectively, in a time period.

It should be noted that, in the embodiment of the present application, only the lower imaging probe 1051 shown in fig. 1 is taken as an example, and those skilled in the art can understand that, in some alternative examples, another imaging probe 1051 may be connected to the first scanning light source 101 and the second scanning light source 102, respectively. The specific location of the imaging probe 1051 is not limited by the embodiments of the present application.

Based on the above-described embodiments, by controlling the first swept optical source 101 and the second swept optical source 102 to emit light in different periods of time, mutual interference between the light emitted by the first swept optical source 101 and the second swept optical source 102 can be avoided. And the light emitted by the second swept optical source 102 is directed onto the vascular tissue 200 or the lipid plaque 300 by one of the imaging probes 1051, and the echo light of the lipid plaque 300 is received by the other imaging probe 1051. Thus, mutual interference between the echo light and the incident light does not occur, and therefore, the accuracy of detection imaging and the degree of reproduction of the evaluation result can be improved.

In another possible implementation, the multi-mode intracavity imaging system provided by the embodiments of the present application further includes a rotary joint 106 connected between the time division multiplexer 104 and the imaging catheter 105, and the echo signals of the two imaging probes 1051 can be transmitted by the optical paths of the rotary joint 106.

By way of example and not limitation, the rotary joint 106 may be a dual light path slip ring.

In one embodiment, fig. 2 is a block diagram of the internal structure of a rotary joint according to an embodiment of the present application. In the embodiment of the present application, a third optical fiber coupler 1061 is disposed in the dual-optical-path slip ring, a tail fiber of the time division multiplexer 104 is connected to a first port (specifically, an a port in fig. 2) of the rotary joint 106, a second port (specifically, a B port in fig. 2) of the rotary joint 106 is connected to the imaging catheter 105, and a third port (specifically, a C port in fig. 2) of the rotary joint 106 is connected to the control device 109 through the photodetector 108, where the control device 109 is used to analyze and display an echo of the imaging probe 1051.

Illustratively, the control apparatus 109 includes a signal processing and display device 1091. The signal processing and display device 1091 may be a liquid crystal display or a display. A processor capable of analyzing and processing the echo signals, such as a central processing unit (Central Processing Unit, CPU) or a micro control unit (Microcontroller Unit, MCU), may be built in the liquid crystal display and the display. Of course, the processor may be another type of processor, which is not listed in the embodiment of the present application.

In this way, the signal processing and display device 1091 in the control apparatus 109 can process the echo signal into an image or other data that can be recognized and read by the doctor and display the image or other data, so as to facilitate the analysis of the doctor.

In the embodiment of the application, the rotary joint is arranged, so that the echo light of the imaging catheter 105 can be transmitted in a split light path, the mutual interference of the echo light of OCT imaging and NIRS evaluation can be avoided, and the accuracy and the reproducibility of intravascular imaging are improved.

It will be appreciated that in order to avoid interference between the echo light and the light emitted by the first swept optical source 101 or the second swept optical source 102. Referring to fig. 1, in the embodiment of the present application, a first fiber circulator 107 is connected between the time division multiplexer 104 and the rotary joint 106, one port of the first fiber circulator 107 is connected to the time division multiplexer 104, and two ports of the first fiber circulator 107 are connected to the rotary joint 106;

echoes of the imaging probe 1051 connected to the first swept optical source 101 can enter the first fiber optic circulator 107 through two ports of the first fiber optic circulator 107 and be transmitted out of three ports of the first fiber optic circulator 107.

In this way, the incident light can only propagate along one direction in the first optical fiber circulator 107, that is, the light entering from one port can only exit from two ports, the light entering from two ports can only exit from three ports, and interference between the light emitted by the first sweep light source 101 and the echo light of the imaging probe 1051 can be effectively avoided.

It should be noted that, in the embodiment of the present application, two ports of the first fiber circulator 107 are connected to a first port of the rotary connector 106.

Alternatively, in the embodiment of the present application, the reference arm of the first fiber coupler 103 is connected to the collimator 111 through the second fiber circulator 110, and the light emitted from the collimator 111 may be reflected by the mirror 112 and transmitted from the three ports of the second fiber circulator 110 through the two ports of the second fiber circulator 110.

Specifically, the coherence wavelength length of the OCT optical system can be adapted by adjusting the position of the mirror 112.

Optionally, the three ports of the first fiber optic circulator 107 and the three ports of the second fiber optic circulator 110 are connected to a balanced photodetector 114 through a second fiber optic coupler 113, and the balanced photodetector 114 is configured to convert the optical signal output by the second fiber optic coupler 113 into an electrical signal.

The second fiber coupler 113 may be a 2x2 type coupler with a 50/50 spectral ratio.

Namely, two beams of echo light of the first sweep frequency light source 101 and two beams of echo light reflected by the mirror surface generate coherent interference in the second optical fiber coupler 113, then the two beams of beat signals with the phase difference pi/2 are divided into two beams of beat signals, the two beams of beat signals enter the balanced photoelectric detector 114, and the balanced photoelectric detector 114 converts the optical signals into electric signals so as to facilitate subsequent analysis and processing.

According to the embodiment of the application, the balanced photoelectric detector is arranged, so that the influence of receiver noise and electronic circuit noise on the detection of weak light signals can be remarkably eliminated, and the OCT imaging precision can be improved.

Specifically, the balanced photodetector 114 is connected to a signal processing and displaying device 1091 in the control apparatus 109, where the signal processing and displaying device 1091 is configured to analyze and display an electrical signal output by the balanced photodetector 114.

In other possible implementations, the control apparatus 109 further includes a control device 1092, where the control device 1092 is connected to the first swept light source 101 and the second swept light source 102, and the control device 1092 can control the first swept light source 101 and the second swept light source 102 to be alternately started.

Specifically, in the embodiment of the present application, the first swept light source 101 and the second swept light source 102 may be operated by the control device 1092 in an electronically controlled manner in time intervals, that is, to drive OCT imaging and NIRS evaluation, respectively.

By way of example, the control device 1092 may be a computer, notebook, tablet, personal digital computer, or the like having a controller.

The following describes in detail the optical path transmission situation of the multi-mode intracavity imaging system provided by the embodiment of the present application:

After the light emitted by the first sweep light source 101 passes through the first optical fiber coupler 103, the light beam is divided into two parts, one part is used as a sample signal (light splitting 90), the other part is used as a reference signal (light splitting 10), the sample signal enters one port of the first optical fiber circulator 107 through the time division multiplexer 104, the light signal exits into the rotary joint 106 through two ports of the first optical fiber circulator 107, the third optical fiber coupler 1061 inside the rotary joint 106 guides the signal light into the imaging catheter 105, the signal light strikes a target tissue (a blood vessel wall 200 or a lipid plaque 300 and the like) through the imaging probe 1051 of the imaging catheter 105, the echo signal is coupled into the optical fiber through the imaging probe 1051, enters two ports of the first optical fiber circulator 107 through the first port of the rotary joint 106, and exits into the second optical fiber coupler 113 from three ports of the first optical fiber circulator 107.

The reference signal is connected to one port of the second fiber optic circulator 110, enters the fiber collimator 111 from two ports of the second fiber optic circulator 110 and exits as space light, and then strikes the mirror 111, and the mirror 111 echoes are collected and coupled into the fiber through the fiber collimator 111. The echo is emitted from the three ports through the two ports of the second fiber circulator 110 and enters the second fiber coupler 113, and after two beams of light are coherently interfered in the second fiber coupler 113, the two beams of beat signals with the phase deviation pi/2 are split into two beams of beat signals and are led into the balance photoelectric detector 114.

The light emitted from the second sweep frequency light source 102 is connected into the time division multiplexer 104 through a tail fiber, and enters one port of the first optical fiber circulator 107, and enters the first port of the rotary joint 106 from two ports of the first optical fiber circulator 107, the light beam enters the imaging catheter 105 from the second port of the rotary joint 106, and strikes target tissue (a blood vessel wall 200 or a lipid plaque 300 and the like) after passing through the imaging probe 1051 of the imaging catheter 105, the information of the lipid plaque 300 is loaded by the signal, and then is coupled into the optical fiber through the imaging probe 1051 of the imaging catheter 105, and the optical signal is output from the third port of the second port of the rotary joint 106 into the photoelectric detector 108.

In order that the lipid level of the intravascular lipid plaques of the patient can be accurately assessed. The application also provides a multi-mode data fusion method. The control device 1092 of the multi-mode intracavity imaging system illustrated in fig. 1 described above controls the first swept light source 101 and the second swept light source 102 to alternately activate the detection of the intracavity tissue. The imaging probe 1051 irradiates light emitted by the first sweep light source 101 and the second sweep light source 102 onto the tissue in the cavity, and transmits the echo signals to the signal processing and displaying device 1091, and the signal processing and displaying device 1091 can analyze and display the echo signals for imaging. After the control device 109 acquires the image, the acquired image data can be fused by the multi-mode data fusion method provided by the application, so that the lipid level of the intra-cavity tissue can be accurately estimated.

In the embodiment of the present application, if the first swept light source 101 emits light with a wavelength length suitable for OCT imaging. The second swept optical source 102 emits light of a wavelength length suitable for NIRS imaging. The control device 1092 controls the first scanning light source 101 and the second scanning light source 102 to alternately start the detection of the intra-cavity tissue, and the signal processing and displaying device 1091 can acquire the OCT image of the intra-cavity tissue and the NIRS image corresponding to the OCT image. Fig. 3 is an image of an intra-luminal tissue obtained based on the system shown in fig. 1, wherein fig. 3 (a) is an OCT image obtained by the signal processing and display device 1091, which can display the structure of the lipid plaques in the intra-luminal tissue. Fig. 3 (b) is a combined image including an OCT image and a NIRS image, and as can be seen from fig. 3 (b), the pixel value of the pixel point of the NIRS image in the axial direction represents the absorption value of the lipid plaque in the OCT image for the light wave in the axial direction.

The flow chart of the multi-mode data fusion method provided by the application is shown in fig. 4. The method comprises the following steps:

S401, acquiring an OCT image sequence and an NIRS image sequence of an intracavity tissue acquired by the multi-modality intracavity imaging system illustrated in fig. 1, the OCT image sequence including a plurality of OCT images, and the NIRS image sequence including a plurality of NIRS images in one-to-one correspondence with the plurality of OCT images.

It will be appreciated that the control device 109 in the multi-modality intra-cavity imaging system may control the first swept optical source 101 and the second swept optical source 102, respectively, to alternately activate the detection of different locations of intra-cavity tissue. The imaging probe 1051 irradiates light emitted from the first swept light source 101 onto the tissue in the cavity and receives an echo signal to transmit the echo signal to the control device 109. The control device 109 analyzes the echo signal to obtain an OCT image. The imaging probe 1051 can illuminate light emitted by the second swept light source 102 onto tissue within the cavity, and another imaging probe 1051 receives an echo signal to transmit the echo signal to the control device 109. The control device 109 analyzes the echo signal to obtain a plurality of NIRS images corresponding to the plurality of OCT images one by one.

It should be noted that the image sequence may be a plurality of images in the same vessel lumen acquired by the multi-mode intra-lumen imaging system. Or may be images of successive multiple frames in video acquired by a multi-modality intra-cavity imaging system.

Wherein, different axial directions in NIRS images show different near infrared attenuation degrees. As shown in FIG. 3 (b), the attenuation degree of the near infrared light by the tissue in the NIRS image is reflected by a normalized numerical value between 0 and 1 in each axial direction, and finally the attenuation spectrum of the cross section in the cavity is presented as a parameter of the lipid distribution condition of the cross section in the reaction cavity.

S402, registering each OCT image and the NIRS image corresponding to the OCT image.

It should be noted that, when the system shown in fig. 1 is used to collect the OCT image and the NIRS image, a certain included angle exists between the OCT data collecting probe and the NIRS data collecting probe, and the OCT image and the NIRS image with completely aligned angles cannot be directly obtained for subsequent data fusion analysis. Therefore, before data fusion is performed on each OCT image and the NIRS image corresponding to each OCT image, each OCT image needs to be registered with its corresponding NIRS image one by one for subsequent operations. The specific implementation mode is as follows:

In one possible implementation, for each OCT image and NIRS image corresponding to the OCT image, the direction and angle of relative rotation of the OCT image and NIRS image may be determined by identifying the locations of the guide wires in the OCT image and NIRS image, thereby registering the OCT image and its corresponding NIRS image.

In one embodiment, for each OCT image, the guide wire in the OCT image can be identified by a target detection model. The object detection model may be a YOLO network, for example. The YOLO network includes 24 convolutional layers and 2 fully-connected layers. The input OCT image is first segmented, i.e. the input image is divided into 7*7 cells of size 224 x 224, each 36 x 36. The YOLO network may label a preset number of bounding boxes in the cells containing the guidewire. In identifying the guidewire, the parameters that need to be determined include bounding box (x, y, w, h), confidence and class probability (class 1). Where (x, y) is the center coordinate of the bounding box and (w, h) represents the width and height of the bounding box. Confidence is defined asIf the cell has an Object, pr (Object) =1, otherwise Pr (Object) =0; It indicates the proportion of the overlap area of the prediction frame and the real frame to the sum of the areas of the prediction frame and the real frame.

And (3) aiming at the NIRS images corresponding to each OCT image one by one, calculating the maximum continuous area with the axial numerical value of zero in the NIRS images to obtain the position of the guide wire in the NIRS images. It will be appreciated that the guide wire is typically a metallic material. In the NIRS image, the numerical value of the position of the guide wire in the axial direction is zero, so that the maximum continuous area with the numerical value of zero in the axial direction in the NIRS image is the position of the guide wire.

After the positions of the guide wires in each OCT image and the NIRS image corresponding to the OCT image are respectively obtained based on the above embodiments, the rotation angle of the pixel matrix of the OCT image relative to the pixel matrix of the NIRS image is rotated according to the offset angle of the guide wire positions in the two images, so that the OCT image and the NIRS image are registered.

S403, identifying lipid plaque areas in each OCT image, and determining a first determination value of each OCT image according to the areas of the lipid plaque areas of the plurality of OCT images.

In one possible implementation, the present application combines a semantic segmentation method with a traditional digital image processing method based on pixel texture features, statistics, etc., for accurately identifying lipid plaque regions in each OCT image. The specific implementation mode is as follows:

Step one, inputting each OCT image into a trained U-Net network respectively to obtain a first lipid segmentation mask of each OCT image.

Specifically, the U-Net network includes an encoder and a decoder. Wherein the encoder comprises a plurality of convolutional layer modules, each comprising two convolutional layers, one RELU active layer and one max pooling layer, for extracting features and downsampling. The decoder includes a plurality of deconvolution modules, each deconvolution module including a deconvolution layer and a RELU activation layer. The final network output is a feature map of the two channels, representing the two categories, background and foreground. For the OCT image in the embodiment of the present application, the foreground is a lipid plaque area, and the background is a non-lipid plaque area. The feature maps of the two channels are then input into a softmax function to output a lipid-segmentation mask image.

In the embodiment of the application, each OCT image is respectively input into a U-Net network, and the U-Net network can identify whether each pixel point in the OCT image belongs to a lipid plaque area, so that a first lipid plaque mask of each OCT image is obtained.

Wherein the size of the first lipid plaque mask is the same as the size of the corresponding OCT image. The pixel values in the first lipid-segmentation mask and the second lipid-segmentation mask represent the decision result of the pixel point at the same position in the OCT image, and the decision result is whether the pixel point is in the lipid plaque region. Illustratively, the pixel value of the pixel point in the first lipid patch mask is 0 or 1. If the pixel value of the pixel point in the first lipid patch mask is 0, the pixel point at the same position in the OCT image is not in the lipid patch region, and if the pixel value of the pixel point in the first lipid segmentation mask is 1, the pixel point at the same position in the OCT image is in the lipid region.

And step two, acquiring a multidimensional feature vector of each OCT image, and inputting the multidimensional feature vector into a trained random forest to obtain a second lipid segmentation mask of each OCT image.

In this embodiment, the second lipid segmentation mask of each OCT image may be obtained by performing feature extraction on each OCT image by a conventional digital image processing method, inputting the multi-dimensional feature vector into a trained random forest, and determining the class of each pixel.

In one example, the multi-dimensional feature vector of the OCT image can be obtained based on a feature extraction method of a gray level co-occurrence matrix (e.g., ASM energy (Angular Second Moment), inertia, contrast, entropy, auto-correlation, etc.). Illustratively, assume that a multi-dimensional feature vector of an OCT image is acquired based on three features of ASM energy, inertia, and contrast in a gray level co-occurrence matrix. For each pixel in the OCT image, the ASM energy value a ₁, the inertia value a ₂, and the contrast value a ₃ of the pixel can be calculated according to the correlation between the pixel itself and its neighboring pixels. Therefore, each pixel point will extract a feature vector a= [ a ₁,a₂,a₃ ] containing the three features. Assuming that the size of one OCT image is mxn, the size of the multi-dimensional feature vector of the OCT image is mxn×3. The multi-dimensional feature vector of the OCT image characterizes the edge texture information of the OCT image.

In another example, a multi-dimensional feature vector of the OCT image can be obtained based on a feature extraction method of the transform domain and the filter. Illustratively, the OCT image may be converted to a transform domain by a linear transformation method (e.g., discrete cosine transform, local fourier transform, walsh hadamard transform, wavelet transform, etc.) to obtain a plurality of feature images, and then the plurality of feature images may be processed by a filter (e.g., bandpass filter, notch filter, etc.) to obtain a multi-dimensional feature vector of the OCT image.

And inputting the obtained multi-dimensional feature vector of the OCT image into a trained random forest, wherein the random forest can classify each pixel point in the OCT image according to the multi-dimensional feature vector of the OCT image, and finally a second lipid segmentation mask is obtained.

Based on the above embodiment, the lipid plaque area in the OCT image is smoother, and the boundary of the lipid plaque is more blurred, and the texture features of the area with repeated and smoother local sequences in the image can be effectively extracted based on the first-order, second-order or higher-order statistical properties of gray scales in the pixels and the neighborhood thereof, autocorrelation functions and other features, so that the classification accuracy can be improved, and the mask accuracy is improved.

And thirdly, determining lipid plaque areas in each OCT image according to the first lipid segmentation mask and the second lipid segmentation mask of each OCT image.

In one embodiment, for each OCT image, if the pixel value of the pixel point in the first lipid segmentation mask and the pixel value of the pixel point at the same position in the second lipid segmentation mask indicate that the pixel point belongs to the lipid patch region, then the pixel point at the same position in the OCT image is indicated as the lipid patch region. That is, the first lipid-division mask and the second lipid-division mask are and-operated, and for each pixel point in the OCT image, it is determined that the pixel point is a lipid patch only when the determination result based on the U-Net network is the same as the determination result based on the conventional digital image processing method.

It should be noted that, in the embodiment of the present application, a plurality of lipid segmentation masks of the OCT image may be obtained by two or more methods, and then the plurality of lipid segmentation masks may be used for performing and operation, so as to identify the lipid plaque region in the OCT image.

And step four, acquiring the areas of the lipid plaque areas in the plurality of OCT images, and dividing the areas of the lipid plaque areas in each OCT image by the maximum value in the areas of the lipid plaque areas in the plurality of OCT images respectively to obtain a first determination value of each OCT image.

In one embodiment, the number of pixels belonging to a lipid plaque in each OCT image is counted in the sequence of OCT images. The lipid plaque area of each OCT image is the total number of pixels belonging to the lipid plaque in the OCT image multiplied by the pixel resolution of the OCT image. It is assumed that H OCT images are included in the OCT image sequence, wherein the lipid plaque area of the H OCT image is the largest, which is S _h. The first decision value x ₁ for any one OCT image in the OCT image sequence can be expressed as equation (1):

Where k represents the kth OCT image in the sequence of OCT images, S _k represents the lipid plaque area of the kth OCT image. The first decision value of the whole OCT image sequence can be expressed as

S404, determining a second determination value of each NIRS image according to the sampling frequency adopted when the NIRS image sequence is acquired and the number of samples of each NIRS image, wherein the number of samples of the NIRS image is the number of absorption values larger than a threshold value.

In one embodiment, for each NIRS image, the value of the NIRS image in each axial direction is expressed as the absorption value of the wavelength by the luminal tissue at that location. And obtaining the maximum pixel value in the NIRS image, and dividing each pixel value in the NIRS image by the maximum pixel value to obtain the normalized NIRS image. The pixel values in the normalized NIRS image lie between 0 and 1, where "1" and "0" correspond to the maximum and minimum values of intra-luminal tissue absorption, respectively. The tissue may be considered to be a lipid when the normalized absorption value in a certain axial direction is greater than 0.6. Assuming that the sampling frequency of each NIRS image in the NIRS image sequence is the same, the sampling frequency is N (i.e., N samples are collected in the entire axial direction of each NIRS image), and the number N of samples with an absorption value greater than 0.6 in the N samples is counted, the second determination value x ₂ of the NIRS image may be expressed as:

wherein, N < = N, k represents the kth NIRS image in the NIRS image sequence; The decision value of the entire NIRS image sequence can be expressed as The number of NIRS images in the NIRS image sequence is the same as the number of OCT images in the OCT image sequence.

S405, fusing the first judging value of each OCT image and the second judging value of the NIRS image corresponding to each OCT image, and determining the lipid degree indicated by each OCT image.

In order to accurately obtain the distribution situation of the lipid plaque in the cavity from the three-dimensional angle, data fusion is required to be carried out on each OCT image and the NIRS image corresponding to each OCT image one by one, so that the distribution situation of the lipid plaque in the longitudinal direction of the blood vessel is obtained, and the lipid degree of the lipid plaque is estimated.

In one possible implementation, for each OCT image, a support matrix is determined from a first decision value of the OCT image and a second decision value of the NIRS image corresponding to the OCT image. And acquiring a weight coefficient according to the feature vector corresponding to the maximum feature value of the support matrix. And then, carrying out weighted summation on the first judging value of the OCT image and the second judging value of the NIRS image corresponding to the OCT image according to the weight coefficient to obtain a fusion value. And inputting the fusion value into a Kalman filter to obtain an evaluation parameter of each OCT image, wherein the evaluation parameter is used for representing the lipid degree. The specific implementation method is as follows:

First, for each OCT image, a support degree matrix is determined from a first determination value x ₁ of the OCT image and a second determination value x ₂ of the NIRS image corresponding to the OCT image, and the support degree matrix R may be expressed as:

d_ij＝|x_i-x_j| (5)

in the above formula, i=1, 2, and j=1, 2. In the fusion, we need to find the weight coefficient of each data in the two determination values Expressed in matrix form asWherein, the V= [ V ₁ v₂]^T represents a eigenvector corresponding to the maximum eigenvalue λ of the matrix R, then the weight coefficient of the i-th data x _i may be represented as formula (6):

the normalized value of the OCT image and the NIRS determination value are subjected to weighted fusion, and the fused result (fusion value) is as follows:

finally, taking into account the consistent correlation between each image in the image sequence and its preceding and following image data, the above-mentioned fusion values are input into a kalman filter, resulting in an evaluation parameter x (k) of the lipid plaque of each image, which represents the lipid level of the lipid plaque:

It can be appreciated that by evaluating the lipid plaques of each OCT image in the long axis direction of the OCT image sequence, the continuous lipid evaluation results in the long axis direction of the blood vessel can be obtained. Fig. 5 is a graph of a multi-modal data fusion result provided by an embodiment of the present application, wherein a waveform diagram of the lipid level in a lumen can be formed according to the evaluation result of the lipid, and the peak in the waveform diagram represents the highest lipid level of a lipid plaque in a vessel lumen at a corresponding position. The evaluation result based on the multi-mode lipid plaque provided by the application is more accurate than that of the traditional method.

In one embodiment, the embodiment of the application provides a training method of a YOLO network, and the specific training process is as follows:

step one, a training sample containing a guide wire is obtained.

Specifically, the training samples include an OCT image sample and an NIRS image sample, the OCT image sample including the guide wire is manually selected, and the guide wire in the OCT image sample is manually labeled. To obtain more training samples, the number of OCT image samples needs to be increased in a data expansion manner.

Illustratively, the number of OCT image samples labeled with a guidewire is expanded by rotating, adjusting contrast, and/or adding noise (e.g., random noise, pretzel noise, etc.) to the OCT image samples. A sufficient number of training samples containing the guide wire are finally obtained.

The same angle of rotation is required for each OCT image sample to the corresponding NIRS image sample for subsequent registration.

And step two, inputting the training samples into an initial YOLO network for iterative training.

During training, the weights of the positioning error and the classification error should be unequal, taking into account that the guide wire occupies a relatively small portion of the pixels of the whole image. Therefore, in choosing the loss function, a weighted loss function more suitable for small object recognition should be chosen. Illustratively, increasing the loss weight of the bounding box coordinate predictions reduces the loss weight of confidence predictions for bounding boxes that do not contain targets. Specifically, the loss function of the YOLO network is equation (8):

In the above-mentioned method, the step of, And inputting the training sample into an initial YOLO network for iterative training, and when the loss function meets the preset requirement, indicating that the model is converged, namely the initial YOLO network is trained, and obtaining the trained YOLO network.

In another embodiment, the application provides a training method of a U-net network model, which comprises the following specific processes:

step one, an OCT image sample labeled with a lipid plaque region is obtained.

In one possible implementation, OCT image samples containing lipid plaques are manually selected from the OCT pullback data. And manually labeling the lipid plaque region in the OCT image sample. To obtain more training samples, the number of OCT image samples needs to be increased in a data expansion manner. Illustratively, the number of OCT image samples is expanded by rotating, adjusting contrast, and/or adding noise (e.g., random noise, pretzel noise, etc.) to the labeled OCT image samples.

And step two, inputting the OCT image sample into an initial U-net network for network training.

Specifically, the OCT image sample is input to an encoder of a U-net network, and characteristic information of a salient region of the OCT image sample is extracted to obtain a multi-channel characteristic image. And then expanding the size of the characteristic image through a decoder, and finally outputting the characteristic image of 2 channels. The 2 feature maps are used as input of a softmax function, and the softmax category with relatively large probability is calculated. And finally, performing back propagation training through a cross entropy loss function. When the loss function reaches a preset condition, the representation model is converged, namely the initial U-net network is trained, and a trained U-net network model is obtained.

Based on the multi-mode data fusion method provided by the application, the size and the structure of the plaque in the blood vessel are determined by identifying the lipid plaque area contained in the OCT images, and the first judgment value of each OCT image is determined according to the areas of the lipid plaque areas of the plurality of OCT images. Meanwhile, whether the tissue in the blood vessel is lipid is determined according to the absorption value of the tissue in the blood vessel on the spectrum displayed in the NIRS image, so that a second determination value of each NIRS image is determined according to the sampling frequency and the number of samples of each NIRS image. The first decision value and the second decision value are fused, and the lipid level indicated by each OCT image can be determined. Compared with the existing method for evaluating the lipid plaque or vulnerable plaque based on the single-mode image, the method utilizes the complementarity of the OCT image and the NIRS image to fuse the image data of multiple modes, so that the lipid degree of the lipid plaque is evaluated, and the accuracy of the evaluation is improved.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.

The embodiment of the application also provides network equipment, which comprises at least one processor, a memory and a computer program stored in the memory and capable of running on the at least one processor, wherein the steps in the multi-mode data fusion method in the embodiment are realized when the processor executes the computer program.

The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can realize the steps in the multi-mode data fusion method in the embodiment when being executed by a processor.

Embodiments of the present application provide a computer program product enabling a cleaning robot to perform the steps of the multi-modal data fusion method described in the above embodiments when the computer program product is run on the cleaning robot.

References to "one embodiment" or "an example" or the like described in this disclosure mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present disclosure. Thus, appearances of the phrases "in one embodiment," "in one example," "in one possible implementation," "in another possible implementation," and the like in various places throughout this specification are not necessarily all referring to the same embodiment, but mean "one or more, but not all, embodiments" unless specifically indicated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

In the description of the present application, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

The embodiments are only used to illustrate the technical scheme of the present application, but not to limit the technical scheme, and although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical scheme described in the foregoing embodiments may be modified or some or all technical features may be equivalently replaced, and the modification or replacement does not deviate the essence of the corresponding technical scheme from the scope of the technical scheme of the embodiments of the present application.

Claims

1. The multi-mode intracavity imaging system is characterized by comprising a first sweep frequency light source (101), a second sweep frequency light source (102), a first optical fiber coupler (103), a time division multiplexer (104), an imaging catheter (105) and a control device (109);

The first sweep frequency light source (101) is connected with the first optical fiber coupler (103), a sample arm of the first optical fiber coupler (103) is connected with an imaging catheter (105) through the time division multiplexer (104) in an optical signal manner, and the second sweep frequency light source (102) is connected with the imaging catheter (105) through the time division multiplexer (104) in an optical signal manner;

The imaging catheter (105) can detect and image the tissue in the cavity according to the light emitted by the first scanning light source (101) and the second scanning light source (102);

The control device (109) is configured to receive echo signals corresponding to the first swept light source (101) and the second swept light source (102) through the imaging catheter (105), analyze the echo signals corresponding to the first swept light source (101) to obtain a plurality of OCT images, and analyze the echo signals corresponding to the second swept light source (102) to obtain a plurality of NIRS images, where the plurality of NIRS images are in one-to-one correspondence with the plurality of OCT images;

the control device (109) is further used for registering each OCT image and an NIRS image corresponding to the OCT image, identifying lipid plaque areas in each OCT image, determining a first determination value of each OCT image according to the areas of the lipid plaque areas of a plurality of OCT images, determining a second determination value of each NIRS image according to the sampling frequency adopted when the NIRS images are acquired and the number of samples of each NIRS image, wherein the number of the samples of the NIRS image is the number that the absorption value is larger than a threshold value, and fusing the first determination value of each OCT image and the second determination value of the NIRS image corresponding to each OCT image to determine the lipid degree indicated by each OCT image.

2. The multi-modality intra-cavity imaging system of claim 1, wherein the imaging catheter (105) includes two imaging probes (1051), one of the two imaging probes (1051) being connected to the first swept light source (101) and the second swept light source (102), respectively;

The imaging probe (1051) respectively connected with the first scanning light source (101) and the second scanning light source (102) can be used for irradiating light emitted by the first scanning light source (101) and the second scanning light source (102) onto the tissue in the cavity and receiving echo signals, and the other one of the two imaging probes (1051) is used for receiving the echo signals of the second scanning light source (102).

3. The multi-modality intra-cavity imaging system of claim 2, further comprising a rotary joint (106) connected between the time division multiplexer (104) and the imaging catheter (105), echo signals of two of the imaging probes (1051) being transmittable by way of the rotary joint (106) spectroscopy.

4. A multi-modal intra-cavity imaging system as set forth in claim 3 further comprising a first fiber optic circulator (107) connected between the time division multiplexer (104) and the swivel (106), one port of the first fiber optic circulator (107) being connected to the time division multiplexer (104), two ports of the first fiber optic circulator (107) being connected to a first port of the swivel (106), a second port of the swivel (106) being connected to the imaging conduit (105);

An echo of an imaging probe (1051) connected to the first swept light source (101) may enter the first fiber optic circulator (107) through two ports of the first fiber optic circulator (107) and be transmitted out of three ports of the first fiber optic circulator (107).

5. The multi-modality intra-cavity imaging system of claim 4, further comprising a second fiber optic circulator (110), a collimator (111), a mirror (112), a second fiber optic coupler (113), and a balanced photodetector (114);

The reference arm of the first optical fiber coupler (103) is connected with the collimator (111) through the second optical fiber circulator (110), and light emitted by the collimator (111) can be reflected by the mirror surface (112) and transmitted outwards from the three ports of the second optical fiber circulator (110) through the two ports of the second optical fiber circulator (110);

The three ports of the first optical fiber circulator (107) and the three ports of the second optical fiber circulator (110) are connected with the balance photoelectric detector (114) through the second optical fiber coupler (113), and the balance photoelectric detector (114) is used for converting an optical signal output by the second optical fiber coupler (113) into an electric signal.

6. The multi-modality intra-cavity imaging system according to claim 5, wherein the control device (109) is connected to the balanced photodetector (114) and to the third port of the swivel (106) through a photodetector (108), the control device (109) being configured to analyze and display imaging the electrical signals output by the balanced photodetector (114) and to analyze and display imaging the echoes of the imaging probe (1051);

The control device (109) is respectively connected with the first frequency-sweeping light source (101) and the second frequency-sweeping light source (102), and the control device (109) respectively controls the first frequency-sweeping light source (101) and the second frequency-sweeping light source (102) to be started alternately.

7. A method of multimodal data fusion, comprising:

Acquiring an OCT image sequence and an NIRS image sequence of an endoluminal tissue acquired by the multi-modality endoluminal imaging system of any one of claims 1-6, the OCT image sequence comprising a plurality of OCT images, the NIRS image sequence comprising a plurality of NIRS images in one-to-one correspondence with a plurality of the OCT images;

Registering each of the OCT images with a NIRS image corresponding to the OCT image;

Identifying a lipid plaque region in each of the OCT images, determining a first decision value for each of the OCT images from the areas of the lipid plaque regions of the plurality of OCT images;

Determining a second determination value of each NIRS image according to sampling frequency adopted when the NIRS image sequence is acquired and the number of samples of each NIRS image, wherein the number of samples of each NIRS image is the number that the absorption value is larger than a threshold value;

And fusing the first judging value of each OCT image with the second judging value of the NIRS image corresponding to each OCT image, and determining the lipid degree indicated by each OCT image.

8. The method of claim 7, wherein said registering each of said OCT images with a NIRS image corresponding to said OCT image comprises:

And respectively identifying the positions of guide wires in the OCT image and the NIRS image corresponding to the OCT image, and adjusting the rotation angle of a pixel matrix of the OCT image relative to the pixel matrix of the NIRS image according to the offset angle of the guide wire positions so as to register the OCT image and the NIRS image.

9. The method of claim 7, wherein the identifying lipid plaque regions in each of the OCT images comprises:

10. The method of claim 7, wherein determining the first determination value for each OCT image from the areas of the lipid plaque regions of the plurality of OCT images comprises:

Acquiring areas of lipid plaque areas in a plurality of the OCT images;

and dividing the area of the lipid plaque area in each OCT image by the maximum value in the areas of the lipid plaque areas in a plurality of OCT images respectively to obtain a first determination value of each OCT image.

11. The method of claim 7, wherein said determining a second determination value for each of said NIRS images based on a sampling frequency employed in acquiring said sequence of NIRS images and a number of samples for each of said NIRS images comprises:

And dividing the sample number of the NIRS images with the sampling frequency adopted when the NIRS image sequence is acquired for each NIRS image to obtain a second judgment value of each NIRS image.

12. The method of any one of claims 7 to 11, wherein said fusing the first decision value of each of the OCT images with the second decision value of the NIRS image corresponding to each of the OCT images, determining the lipid level indicated by each of the OCT images, comprises: