CN114782537A - Human carotid artery localization method and device based on 3D vision - Google Patents

Abstract

The present disclosure provides a method and device for locating human carotid artery based on 3D vision, which relates to the field of smart medical technology, including acquiring ultrasound image data; detecting ultrasound image data using deep learning, and generating human body specific position feature point parameters; Point parameters, the radial basis function is used to determine the projection position of the carotid artery; the position of the carotid artery in the ultrasound image is projected to the neck surface to guide the autonomous ultrasound scan. Identify features from face and neck images captured by 3D cameras, and model the projected position of the carotid artery based on these features, which can be used as an initial planned path for robotic autonomous ultrasound scans to guide automated carotid artery autonomy Ultrasound scan.

Description

Human carotid artery localization method and device based on 3D vision

technical field

The present disclosure relates to the field of smart medical technology, and in particular, to a method and device for locating human carotid arteries based on 3D vision.

Background technique

In a medical imaging system such as a magnetic resonance imaging (MR) system or a computed tomography (CT) system, sometimes a 3D camera needs to be used together to collect auxiliary information such as patient's body position information. The camera of a 3D camera usually consists of a two-dimensional color (RGB) camera and a depth camera.

When installing a 3D camera, it is usually necessary to install the 3D camera directly above the hospital bed to detect the bed and the patient on the bed, so as to obtain the relevant status and information of the patient to the greatest extent.

SUMMARY OF THE INVENTION

The purpose of the present disclosure is to propose a 3D vision-based human carotid artery positioning method and device, and a 3D vision-based human carotid artery positioning method for robotic ultrasound scanning of carotid arteries.

A first aspect of the present disclosure provides a 3D vision-based human carotid artery positioning method, including:

Obtain ultrasound image data;

Use deep learning to detect ultrasound image data and generate specific position feature point parameters of the human body;

According to the specific position feature point parameters of the human body, the radial basis function is used to determine the projection position of the carotid artery;

The location of the carotid artery in the ultrasound image is projected onto the surface of the neck to guide the autonomous ultrasound scan.

A second aspect of the present disclosure provides a 3D vision-based human carotid artery positioning device, including:

an acquisition module to acquire ultrasound image data;

The generation module uses deep learning to detect ultrasonic image data, and generates specific position feature point parameters of the human body;

The determining module adopts the radial basis function to determine the projection position of the carotid artery according to the parameter of the specific position feature point of the human body;

Guidance module that projects the position of the carotid artery in the ultrasound image onto the neck surface to guide the autonomous ultrasound scan.

A third aspect of the present disclosure provides an electronic device, including: a memory and one or more processors;

the memory for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the 3D vision-based human carotid artery localization method as provided in any of the embodiments.

A fourth aspect of the present disclosure provides a storage medium containing computer-executable instructions, where the computer-executable instructions implement the 3D vision-based human carotid artery localization method provided in any of the embodiments by a computer processor.

The present disclosure provides a method and device for locating human carotid artery based on 3D vision, identifying features from face and neck images captured by a 3D camera, and modeling the projection position of the carotid artery according to these features, where the projection position can be Used as the initial planning path for robotic autonomous ultrasound scans to guide automated carotid autonomous ultrasound scans.

Description of drawings

1 is a flowchart of a method for locating a human carotid artery based on 3D vision in an embodiment of the present disclosure;

2 is another flowchart of a method for locating human carotid artery based on 3D vision in an embodiment of the present disclosure;

3 is another flowchart of a method for locating a human carotid artery based on 3D vision according to an embodiment of the present disclosure;

Fig. 4 is the characteristic point schematic diagram in Fig. 1;

5 is a schematic diagram of a 3D vision-based human carotid artery positioning device in an embodiment of the present disclosure;

6 is a schematic diagram of a 3D vision-based human carotid artery positioning device in an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

Since the 3D camera cannot directly perceive the arteries and blood vessels, it is necessary to identify features from the face and neck images captured by the 3D camera, and model the projection position of the carotid artery based on these features, which can be used as a robotic autonomous ultrasound scan The initial planning path to guide automated carotid autonomous ultrasound scanning. The method consists of two stages: feature recognition and projection location modeling.

As shown in FIG. 1 , an embodiment of the present disclosure provides a 3D vision-based human carotid artery positioning method, including:

S101, acquiring ultrasound image data;

The ultrasound image data includes: aligned color map data and depth map data;

The acquiring ultrasound image data includes: taking pictures with a 3D camera to acquire aligned color map data and depth map data.

In the embodiment of the present disclosure, a fixedly installed 3D camera is used to capture an RGB-D color map and a depth map of a complete human face and neck region, and the projection position of the carotid artery on the neck surface is calculated from the RGB-D color map and the depth map.

S102. Use deep learning to detect ultrasonic image data, and generate a human body specific position feature point parameter. In this embodiment of the present disclosure, the human body specific position includes: at least one of the contour of the left mandible, the contour of the right mandible, the contour of the left neck, and the contour of the right neck A sort of.

As shown in Figures 2 and 4, deep learning is used to detect ultrasonic image data, and the parameters for generating specific position feature points of the human body include:

S201, using deep learning to detect the feature points of the contour of the left lower jaw, the contour of the right lower jaw, the contour of the left neck and the contour of the right neck, and generate a corresponding fitting curve;

S202, performing uniform sampling on each curve after fitting according to the set sampling number;

S203, according to the sampling number, calculate the characteristic point parameters, for i from 0 to N-1, calculate the corresponding sampling point coordinates according to the following formula, and complete the sampling; the characteristic point parameters include: sampling step size and sampling point coordinates;

x _i =x ₀ +i·t

y _i =L(x _i )

S204, using the Lagrangian interpolation method according to the feature point parameters, and performing curve fitting on the extracted feature points to generate a corresponding fitting curve; or,

S205. If one of the four curves of the left mandible contour, the right mandible contour, the left neck contour and the right neck contour fails to be successfully extracted in the above step S201, the coordinates of the sampling points on the failed curve are all recorded as Default value. For example, if the head has a large angle deflection, it will lead to the failure to extract the feature points on some curves, and the coordinates of the sampling points on the curves that fail to extract are all recorded as (-1,-1).

The Lagrangian interpolation method is used to perform curve fitting on the extracted feature points, and the following formula is used to generate the corresponding fitting curve:

Among them, L(x) represents the analytical expression of the fitting curve, li (x) is an item in the expression L(x), (x _i , y _i ) is the coordinate of the feature point _{, and xi is} the abscissa , y _i is the ordinate, k is the feature

number of points.

S103, using the radial basis function to determine the projection position of the carotid artery according to the specific position feature point parameters of the human body;

S104 , project the position of the carotid artery in the ultrasound image onto the neck surface to guide the autonomous ultrasound scan.

The following formula is used to guide the autonomous ultrasound scan according to the projection of the carotid artery onto the neck surface:

Among them, u, v are the horizontal and vertical coordinates of the image projection position point, D is the depth map data, where f _x , f _y are the focal length of the 3D camera in the x or y direction, c _x , _cy are the optical axis for the projection plane. The offset of the coordinate center.

As shown in Figure 3, the use of the radial basis function to determine the projection position of the carotid artery according to the feature point parameters of the specific position of the human body includes:

S301. Calculate the probability distribution of the projection position of the carotid artery on the color map according to the feature point parameters of the specific position of the human body;

和λ_i是径向基函数的方差和系数，为待训练的参数。Among them, F represents the set of feature points, n is the number of feature points, μ _i is the center of the radial basis function, and the coordinates of the feature points are determined.

and λ _i are the variances and coefficients of the radial basis function, the parameters to be trained.

S302. According to the probability distribution of the projection position, (or use the maximum likelihood estimation model to pre-calculate and store the parameters of the model), specifically, pre-collect and label the training data {X _i , F _i }, including different deflections Attitude, F _i represents the extracted feature point set _, Xi is the corresponding carotid artery image projection location point set. Estimate the model parameters λ and σ;

S303. According to the parameters of the specific position feature points of the human body and the parameters of the pre-built maximum likelihood estimation model, extract the region and the preset region whose probability reaches the set threshold, so as to obtain the projection of the carotid artery on the color map Location.

As can be seen from the above, the present disclosure provides a 3D vision-based human carotid artery positioning method, which models the position of the carotid artery based on the features of the face and neck, overcomes the defect that the 3D camera cannot directly perceive the carotid artery, and avoids autonomous The process of manually planning the path and searching before the ultrasound scan realizes the automation of the whole process.

As shown in FIGS. 5 and 6 , the 3D vision-based human carotid artery positioning apparatus 600 in the embodiment of the present disclosure includes:

an acquisition module 601, configured to acquire ultrasound image data;

A generating module 602 is used to detect ultrasonic image data by using deep learning, and generate a parameter of a specific position feature point of the human body;

The determining module 603 is used for determining the projection position of the carotid artery by using the radial basis function according to the feature point parameters of the specific position of the human body;

A guidance module 604 for projecting the position of the carotid artery in the ultrasound image onto the surface of the neck to guide the autonomous ultrasound scan.

The ultrasound image data includes: aligned color map data and depth map data;

The acquiring module 601 is configured to use a 3D camera to take pictures to acquire aligned color map data and depth map data.

The specific position of the human body includes: at least one of the contour of the left lower jaw, the contour of the right lower jaw, the contour of the left neck and the contour of the right neck;

The generating module 602 is configured to detect the feature points of the left mandible contour, the right mandible contour, the left neck contour and the right neck contour by using deep learning, and generate a corresponding fitting curve;

According to the set sampling number, each fitted curve is uniformly sampled;

According to the sampling number, the characteristic point parameters are calculated to complete the sampling; the characteristic point parameters include: sampling step size and sampling point coordinates;

According to the feature point parameters, a Lagrangian interpolation method is used to perform curve fitting on the extracted feature points to generate a corresponding fitting curve.

The Lagrangian interpolation method is used to perform curve fitting on the extracted feature points, and the following formula is used to generate the corresponding fitting curve:

Among them, L(x) represents the analytical expression of the fitted curve, li(x) is an item in the expression L(x), (x _i , y _i ) is the coordinate of the feature point, x is the abscissa, y =L(x) is the ordinate, and k is the number of feature points.

The determining module 603 is used to calculate the probability distribution of the projection position of the carotid artery on the color map according to the parameter of the specific position feature point of the human body;

According to the projection position probability distribution, the parameters of the pre-built maximum likelihood estimation model are obtained by calculation;

According to the parameters of the specific position feature points of the human body and the parameters of the pre-built maximum likelihood estimation model, the region and the preset region whose probability reaches the set threshold are extracted to obtain the projection position of the carotid artery on the color map.

The determination module 603 is configured to record all the coordinates of the sampling points on the curve that fails to extract as the preset value if the extracted probability does not reach the area of the set threshold value and/or is not in the preset area.

The following formula is used to guide the autonomous ultrasound scan according to the projection of the carotid artery onto the neck surface:

Among them, u, v are the horizontal and vertical coordinates of the image projection position point, D is the depth map data, where f _x , f _y are the focal length of the 3D camera in the x or y direction, c _x , _cy are the optical axis for the projection plane. The offset of the coordinate center.

The 3D vision-based human carotid artery positioning apparatus provided by the embodiments of the present disclosure can execute the 3D vision-based human carotid artery positioning method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.

FIG. 7 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG. 7 , the electronic device of this embodiment includes: a processor 701 , a memory 702 , and a computer program 703 stored in the memory 702 and executable on the processor 701 . When the processor 701 executes the computer program 703, the steps in each of the foregoing method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules/units in the foregoing device embodiments are implemented.

Illustratively, the computer program 703 may be divided into one or more modules/units, which are stored in the memory 702 and executed by the processor 701 to complete the present disclosure. One or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 703 in the electronic device.

The electronic device may be a desktop computer, a notebook, a palmtop computer, a cloud server and other electronic devices. The electronic device may include, but is not limited to, the processor 701 and the memory 702 . Those skilled in the art can understand that FIG. 7 is only an example of an electronic device, and does not constitute a limitation to the electronic device. It may include more or less components than the one shown in the figure, or combine some components, or different components, such as , the electronic device may also include an input and output device, a network access device, a bus, and the like.

The processor 701 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-available processors Field-Programmable GateArray (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 702 may be an internal storage unit of the electronic device, eg, a hard disk or memory of the electronic device. The memory 702 can also be an external storage device of the electronic device, for example, a pluggable hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card) equipped on the electronic device )Wait. Further, the memory 702 may also include both an internal storage unit of the electronic device and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device. The memory 702 may also be used to temporarily store data that has been or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated to different functional units, Module completion means dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this disclosure.

In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus/electronic device and method may be implemented in other manners. For example, the apparatus/electronic device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer-readable storage medium. Based on this understanding, the present disclosure can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can be processed When the device is executed, the steps of the foregoing method embodiments may be implemented. A computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), random access memory Memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer-readable media may be modified as appropriate in accordance with the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not be Including electrical carrier signals and telecommunication signals.

The above embodiments are only used to illustrate the technical solutions of the present disclosure, but not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the present disclosure. within the scope of protection.

Claims (9)

1. a human carotid artery positioning method based on 3D vision, is characterized in that, comprises: Obtain ultrasound image data; Use deep learning to detect ultrasound image data and generate specific position feature point parameters of the human body; According to the specific position feature point parameters of the human body, the radial basis function is used to determine the projection position of the carotid artery; The location of the carotid artery in the ultrasound image is projected onto the surface of the neck to guide the autonomous ultrasound scan. 2. The method of claim 1, wherein the ultrasound image data comprises: aligned color map data and depth map data; The acquiring ultrasound image data includes: taking pictures with a 3D camera to acquire aligned color map data and depth map data. 3. The method according to claim 1, wherein the specific position of the human body comprises: at least one of the contour of the left lower jaw, the contour of the right lower jaw, the contour of the left neck and the contour of the right neck; Using deep learning to detect ultrasound image data, and generate human body-specific location feature point parameters include: Use deep learning to detect the feature points of the left mandible contour, right mandible contour, left neck contour and right neck contour, and generate the corresponding fitting curve; According to the set sampling number, each fitted curve is uniformly sampled; According to the sampling number, the characteristic point parameters are calculated to complete the sampling; the characteristic point parameters include: sampling step size and sampling point coordinates; According to the feature point parameters, the Lagrangian interpolation method is used to perform curve fitting on the extracted feature points to generate a corresponding fitting curve; or, If one of the four curves of the left mandible contour, the right mandible contour, the left neck contour and the right neck contour cannot be successfully extracted, all the coordinates of the sampling points on the failed curve will be recorded as preset values. 4. method according to claim 3, is characterized in that, described adopting Lagrangian interpolation method, the characteristic point that is extracted carries out curve fitting, and generates corresponding fitting curve and adopts following formula:

Among them, L(x) represents the analytical expression of the fitting curve, li (x) is an item in the expression L(x), (x _i , y _i ) is the coordinate of the feature point _{, and xi is} the abscissa , y _i is the ordinate, k is the number of feature points. 5. The method according to claim 3, characterized in that, determining the projection position of the carotid artery by using the radial basis function according to the feature point parameters of the specific position of the human body comprises: Calculate the projection position probability distribution of the carotid artery on the color map according to the specific position feature point parameters of the human body; According to the projection position probability distribution, the parameters of the pre-built maximum likelihood estimation model are obtained by calculation; According to the parameters of the specific position feature points of the human body and the parameters of the pre-built maximum likelihood estimation model, the region and the preset region whose probability reaches the set threshold are extracted to obtain the projection position of the carotid artery on the color map. 6. The method according to claim 3, wherein the projection of the carotid artery to the neck surface to guide the autonomous ultrasound scan adopts the following formula:

Among them, u, v are the horizontal and vertical coordinates of the image projection position point, D is the depth map data, where f _x , f _y are the focal length of the 3D camera in the x or y direction, c _x , _cy are the optical axis for the projection plane. The offset of the coordinate center. 7. A human carotid artery positioning device based on 3D vision is characterized in that, comprising: an acquisition module for acquiring ultrasound image data; The generation module is used to detect ultrasonic image data using deep learning, and generate the parameters of specific position feature points of the human body; The determining module is used to determine the projection position of the carotid artery by using the radial basis function according to the characteristic point parameters of the specific position of the human body; Guidance module for projecting the position of the carotid artery in the ultrasound image onto the neck surface to guide autonomous ultrasound scans. 8. An electronic device, comprising: a memory and one or more processors; the memory for storing one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6. 9. A storage medium containing computer-executable instructions, wherein the computer-executable instructions are implemented by a computer processor to implement the method according to any one of claims 1-6.

Images