CN118490357B - Target positioning method and device for surgical navigation and surgical navigation system - Google Patents

Abstract

The present application provides a target positioning method, device and surgical navigation system for surgical navigation, the method comprising acquiring surgical images based on a visual camera, identifying key points of target objects in the surgical images based on a pre-trained key point recognition model; acquiring image coordinates and depth information of key points in the surgical images, converting image coordinates into world coordinates based on the depth information; determining the actual position of the target object based on the world coordinates of the key points; and tracking the movement of the target object in real time based on the actual position of the target object. The method provided in the present application does not require the use of markers, avoids the adverse effects on measurement accuracy caused by marker contamination and occlusion, and the constraints of marker binding on surgical instruments, saving costs; and by identifying the precise position of the object through the key points of the object, it can reduce the amount of calculation for object recognition and positioning, improve the accuracy and efficiency of positioning, and improve the overall performance of the surgical navigation system.

Description

Target positioning method, device and surgical navigation system for surgical navigation

Technical Field

The present application relates to the field of surgical navigation technology, and in particular to a target positioning method, device and surgical navigation system for surgical navigation.

Background Art

With the development of science and technology, surgical navigation robots have become an important tool in the medical field, providing doctors with higher surgical precision and operational stability, and playing an important role in high-difficulty operations.

Surgical navigation robots can be widely used in a variety of surgical scenarios. In various types of surgeries such as neurosurgery, orthopedics, otolaryngology, and ophthalmology, surgical navigation robots can assist doctors in precise positioning and navigation, improving the accuracy and safety of the surgery. For example, in brain tumor resection surgery, surgical navigation robots can accurately locate and navigate based on preoperative imaging data, helping doctors to accurately and quickly find the location of the tumor and reduce damage to normal tissues around the tumor. In hip replacement surgery, surgical navigation robots can help doctors accurately determine the position and angle of the implant, improving the accuracy and success rate of the surgery. For another example, in otolaryngology surgery, surgical navigation robots can help doctors accurately determine the location and range of lesions, improving the accuracy of surgery and treatment effects.

In surgical navigation robots, the navigation system is the core of their work. However, due to its high price, cumbersome operation and harsh environmental requirements, its popularization and promotion are greatly restricted, affecting its development and application.

Summary of the invention

To solve at least one of the above problems, the present application provides a target positioning method, device and surgical navigation system for surgical navigation.

In one aspect, a target positioning method for surgical navigation is provided, the method comprising:

Acquire a surgical image based on a visual camera, and identify key points of a target object in the surgical image based on a pre-trained key point recognition model;

Acquire the image coordinates and depth information of the key point in the surgical image, and convert the image coordinates into world coordinates based on the depth information;

Determine the actual position of the target object based on the world coordinates of the key point;

The movement of the target object is tracked in real time based on the actual position of the target object.

In some embodiments, converting the image coordinates to world coordinates based on the depth information includes:

Converting the image coordinates to camera coordinates based on the depth information;

The camera coordinates are converted to the world coordinates based on the extrinsic parameters of the camera.

In some embodiments, converting the image coordinates into camera coordinates based on the depth information includes:

Converting the image coordinates into the camera coordinates based on the depth information and the camera coordinate calculation formula;

The camera coordinate calculation formula includes:

_Among them, ( i , j ) are the image coordinates _of the target pixel, ( Xc , Yc , Zc ₎ are the camera coordinates of the target pixel, fx =F/dx, fy =F/dy, _F is the camera focal length in the form of image coordinate system, dx is the horizontal physical size of a single pixel, dy is _the vertical physical size of a single pixel, u0 is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, _v0 is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and is the depth value of the target pixel _.

In some embodiments, the external parameters of the camera include a rotation matrix and a translation vector, and converting the camera coordinates to the world coordinates based on the external parameters of the camera includes:

Converting the camera coordinates to the world coordinates based on the rotation matrix, the translation vector and the world coordinate calculation formula;

The world coordinate calculation formula includes:

_Among them, ( Xc , Yc , Zc ) are the camera coordinates of the target pixel, ( Xw _, Yw , _Zw ) are the world coordinates of the _target pixel _, R is the rotation matrix, _and T is the translation vector.

In some embodiments, obtaining the image coordinates and depth information of the key point in the surgical image includes:

Acquiring the resolution of the surgical image;

Converting the resolution of the surgical image to a pre-stored image resolution with depth information;

The image coordinates and depth information of the key points are acquired based on the surgical image after resolution conversion.

In another aspect, a target positioning device for surgical navigation is provided, the device comprising:

An image acquisition module, used to acquire a surgical image based on a visual camera, and to identify key points of a target object in the surgical image based on a pre-trained key point recognition model;

A coordinate conversion module, used for obtaining the image coordinates and depth information of the key point in the surgical image, and converting the image coordinates into world coordinates;

A target positioning module, used to determine the actual position of the target object based on the world coordinates of the key point;

The movement control module is used to track the movement of the target object in real time based on the actual position of the target object.

On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the processor can load and execute at least one instruction, at least one program, a code set or an instruction set to implement the target positioning method according to the above-mentioned embodiment of the present application.

On the other hand, a surgical navigation system is provided, comprising a target instrument, a depth camera, a display, and a target positioning device or a computer device as described above.

In some embodiments, the surgical navigation system further includes a navigation robotic arm trolley and a robotic arm, wherein the robotic arm is used to drive the target instrument to move, and the depth camera, the display, and the robotic arm may be disposed on the navigation robotic arm trolley.

On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, code set or instruction set is stored. The processor can load and execute at least one instruction, at least one program, code set or instruction set to implement the target positioning method according to the above-mentioned embodiment of the present application.

On the other hand, a computer program product or a computer program is provided, which includes computer instructions. When the computer instructions are executed by a processor, the target positioning method according to the above-mentioned embodiment of the present application is implemented.

The embodiment of the present application provides a target positioning method, device and surgical navigation system for surgical navigation, the method comprising acquiring a surgical image based on a visual camera, identifying the key points of a target object in the surgical image based on a pre-trained key point recognition model; acquiring the image coordinates and depth information of the key points in the surgical image, converting the image coordinates into world coordinates based on the depth information; determining the actual position of the target object based on the world coordinates of the key points; and tracking the movement of the target object in real time based on the actual position of the target object. On the one hand, the method can remove the markers frequently used in the existing surgical navigation method, that is, it is not necessary to use them in the surgical navigation method according to the embodiment of the present application, but only realize surgical navigation through navigation equipment, remove the constraints of the marker binding on the surgical instrument, thereby avoiding the adverse effects on the measurement accuracy due to the contamination and occlusion of the marker, and also saving the cost of the marker and the sterilization of the marker; on the other hand, the method provided in the embodiment of the present application identifies the precise position of the object through the key points of the object, which not only reduces the calculation amount of object recognition and positioning, improves the efficiency of positioning, but also improves the accuracy of positioning, thereby greatly improving the overall performance of surgical navigation.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram showing a process flow of implementing a target positioning method for surgical navigation provided by the prior art;

FIG2 shows a schematic structural diagram of a surgical navigation system provided by the prior art;

FIG3 shows a schematic diagram of an implementation flow of a target positioning method for surgical navigation according to an exemplary embodiment of the present application;

FIG4 shows a schematic diagram of coordinate system transformation of a target positioning method for surgical navigation according to an exemplary embodiment of the present application;

FIG5 is a schematic diagram showing an implementation flow of a target positioning method for surgical navigation according to an exemplary embodiment of the present application;

FIG6 shows another schematic diagram of an implementation flow of a target positioning method for surgical navigation according to an exemplary embodiment of the present application;

FIG7 shows a structural diagram of a target positioning device for surgical navigation according to an exemplary embodiment of the present application;

FIG8 shows a schematic structural diagram of a surgical navigation system according to an exemplary embodiment of the present application.

DETAILED DESCRIPTION

In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

Current surgical navigation systems generally use optical or electromagnetic navigation devices. Due to their own physical properties, these two navigation systems have their own advantages and disadvantages in surgical navigation applications.

The advantages of optical navigation are that it can achieve sub-millimeter accuracy, a large measurement range, wireless tracking, and high-quality measurement data. On the other hand, although optical navigation has stable performance in most medical environments, it has requirements for ambient light sources, and there must be a line of sight between the navigation device and the object, which cannot be blocked. The reflectivity of contaminated markers in optical navigation will reduce the accuracy of navigation, and the navigation object must be rigid for marker fixation.

The advantage of electromagnetic navigation is that there is no line of sight restriction, which allows for in vivo tool tracking. The electromagnetic navigation sensor can be embedded in flexible or rigid tools, or can be set at the tip of a surgical needle. The electromagnetic navigation sensor is small in size and can be integrated into various tools with minimal changes to the tool shape. However, the tracking range of electromagnetic navigation is smaller than the measurement range of optical navigation, and the accuracy is sometimes weaker than optical navigation. When there is interference from conductive objects or ferromagnetic metals, the quality of the measurement data may be distorted, and the electromagnetic navigation sensor must be wired.

In optical navigation and electromagnetic navigation, optical navigation is more widely used due to its advantages such as large measurement range and wireless tracking. The optical navigation system is mainly composed of navigation equipment and markers. The navigation equipment can include two categories, one is a sensor that emits and receives specific light such as infrared light, and the other is an ordinary camera. The marker type corresponds to the navigation device type, one is a sphere with a special light coating that reflects infrared light, and the other is an image of a specific pattern such as a black and white checkerboard. The marker is used in pairs with the navigation device, and the precise capture and tracking of the marker is achieved through a specific algorithm. Specifically, the marker usually needs to be bound to the surgical instrument to enable the navigation device to track the surgical instrument. The marker needs to be sterile, including disposable or reusable sterilization, and the sterilization requirements are relatively strict and the cost is also high.

FIG. 1 shows a schematic diagram of an implementation flow of a target positioning method for surgical navigation in the prior art.

1 , the implementation process of optical surgical navigation may include: binding an object to a marker point, a navigation device identifying the marker point, and obtaining the precise position of the object according to the coordinates of the marker point.

FIG. 2 shows a schematic diagram of a surgical navigation system in the prior art.

Referring to FIG2 , the optical surgical navigation system may include a robotic arm trolley and a navigation trolley, wherein the robotic arm trolley is provided with a robotic arm, and a navigation target object is bound with a marker; the navigation trolley is provided with a navigation camera; and the robotic arm trolley is also provided with a display. The optical surgical navigation system can realize navigation during the surgical process through the cooperation of the navigation camera and the marker.

The target positioning method for surgical navigation provided by this application not only avoids the occlusion problem that may be caused by markers, but also removes the constraints on surgical instruments, making surgical operations more free and flexible. More importantly, this method also supports the positioning of flexible target objects, further broadening its application scope in the field of surgical navigation. In addition, by abandoning the dependence on markers, this method also effectively reduces the system cost and brings a more economical and efficient solution to the medical field.

Embodiment 1

FIG3 shows a schematic diagram of an implementation flow of a target positioning method for surgical navigation provided by an embodiment of the present invention.

3 , the target positioning method for surgical navigation provided by an embodiment of the present invention may include steps 101 to 104 .

Step 101: Acquire a surgical image based on a visual camera, and identify key points of a target object in the surgical image based on a pre-trained key point recognition model.

The embodiments of the present application use a visual camera (such as a common depth camera, etc.) and artificial intelligence image processing technology to achieve precise positioning during the surgical process. The visual camera is used to collect images within the surgical field, which is the range of vision during surgery.

In a specific example, the visual cameras include various types such as monocular cameras and multi-camera cameras, and these cameras are equipped with at least one RGB camera to realize RGB image acquisition.

This application is based on artificial intelligence image processing technology to identify objects or human body parts. Artificial intelligence image processing technology includes deep learning image recognition algorithms, including, for example, convolutional neural networks, recurrent neural networks, generative adversarial networks, etc. Among them, convolutional neural network algorithms are more typical and widely used.

In some embodiments, the method further comprises:

Acquire an image of the target object as a training set of a key point recognition model; in the training set, mark at least one key point on the image of the target object, and the number of images of the target object in the training set is greater than a first number;

The key point recognition model is trained based on the training set.

Optionally, the first quantity is 50 sheets.

In a specific example, the key points of the object can be obtained through the pre-trained YOLOV8 model. The ablation needle at the front end of the robotic arm is regarded as the object to be tested, and the labelme tool is used to annotate the various posture data of the ablation needle, that is, the key points of the ablation needle dataset are annotated through labelme. The ablation needle dataset can include 50 photos of ablation needles in various postures, and the key points of the ablation needle include the needle head, center point, needle tail, etc. The exported data is converted into a format compatible with YOLOv8, and the YOLOv8 pose model is trained to recognize and estimate the ablation needle posture.

In a specific example, since the input requirement of the YOLOv8 model is a 640*640 resolution image, the 1280*720 image output by the Intel RealSense camera needs to be downsampled to a 640*360 resolution, and then adjusted to a 640*640 resolution by filling gray above and below. The processed image is input into the YOLOv8 model to obtain the position information of the object to be detected, such as xy coordinates. The parameters x and y are the pixel coordinates of the key point on the image, and optionally, the origin of the coordinates is the upper left corner of the image.

Step 102: Obtain image coordinates and depth information of the key point in the surgical image, and convert the image coordinates into world coordinates based on the depth information.

FIG. 4 shows a schematic diagram of a coordinate system conversion relationship in a method according to an embodiment of the present application.

Referring to Figure 4, the world coordinates (Xw _{, Yw} , _Zw ) can be converted to the camera coordinates ( _Xc , _Yc , _Zc ) with the camera as the reference system through rigid body transformation, the camera coordinates can be converted to image coordinates (x, y) through transmission transformation, and the image coordinates can be converted to pixel coordinates (u, v) through secondary transformation. The above process can be reversely transformed, that is, based on the same principle, pixel coordinates can be converted to world coordinates.

Step 102 is to convert the coordinates of the key points of the object into world coordinates with the depth camera as the coordinate origin. The specific steps of coordinate conversion are as follows.

In some embodiments, obtaining the image coordinates and depth information of the key point in the surgical image includes:

Acquiring the resolution of the surgical image;

Converting the resolution of the surgical image to a pre-stored image resolution with depth information;

The image coordinates and depth information of the key points are acquired based on the surgical image after resolution conversion.

In a specific example, pixel depth information can be obtained through the camera manufacturer's own API. The key point coordinates are the pixel coordinates in the 640*640 resolution image. In order to match the original image with the image with depth information, its resolution needs to be expanded to 1280*1280 resolution. Assuming that the needle coordinates are A ₀ (x0, y0), the expanded coordinates are A ₁ (x1, y1), and the expansion formula is x1=x0*2, y1=y0*2. The expanded coordinates A ₁ are still the pixel coordinates in the image, which are consistent with the pixel coordinates of the image with depth information.

Furthermore, obtaining the world coordinates of a certain pixel point according to the image with depth information requires the coordinates of the pixel point in the camera coordinate system, the camera internal parameters and the camera external parameters.

The step 102 converts the image coordinates into world coordinates, including:

Converting the image coordinates to camera coordinates based on the depth information;

The camera coordinates are converted to the world coordinates based on the extrinsic parameters of the camera.

In some embodiments, converting the image coordinates into camera coordinates based on the depth information includes:

Converting the image coordinates into the camera coordinates based on the depth information and the camera coordinate calculation formula;

The camera coordinate calculation formula includes:

_Among them, ( i , j ) are the image coordinates _of the target pixel, ( Xc , Yc , Zc ₎ are the camera coordinates of the target pixel, fx =F/dx, fy =F/dy, _F is the camera focal length in the form of image coordinate system, dx is the horizontal physical size of a single pixel, dy is _the vertical physical size of a single pixel, u0 is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, _v0 is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and is the depth value of the target pixel _.

In a specific example, to obtain the three-dimensional coordinates of pixel point B ₁ (i, j), the coordinate system of the pixel point needs to be converted to the camera coordinate system. The depth value of pixel point B ₁ is D (i, j), and the intrinsic parameters obtained by camera calibration are ( f _x , f _y , u ₀ , v ₀ ). dx and dy describe the actual width and height of a pixel. u ₀ is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and v ₀ is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, which is half of the horizontal resolution and vertical resolution of the image. Since the camera refers to the pixel coordinate system, the focal length of the camera should be converted to the form of the pixel coordinate system.

In some embodiments, the external parameters of the camera include a rotation matrix and a translation vector, and converting the camera coordinates to the world coordinates based on the external parameters of the camera includes:

Converting the camera coordinates to the world coordinates based on the rotation matrix, the translation vector and the world coordinate calculation formula;

The world coordinate calculation formula includes:

_Among them, ( Xc , Yc , Zc ) are the camera coordinates of the target pixel, ( Xw _, Yw , _Zw ) are the world coordinates of the _target pixel _, R is the rotation matrix, _and T is the translation vector.

The camera external parameters are used to realize the conversion between the camera coordinate system and the world coordinate system. Specifically, the rotation matrix includes , the translation vector includes .

Step 103: Determine the actual position of the target object based on the world coordinates of the key points.

The method provided by the embodiment of the present invention further includes: controlling the movement of the target object based on the actual position of the target object.

Step 104: Track the movement of the target object in real time based on the actual position of the target object.

In some embodiments, the ablation needle at the front end of the robotic arm is regarded as the object to be measured. The key points of the ablation needle are the needle head, the center point, and the needle tail. The unique position of the ablation needle in space ( _Xw , _Yw , _Zw ) can be determined through the spatial positions of the key points of the ablation needle. The ablation needle is moved to the specified position by moving the robotic arm based on the position coordinates to realize surgical navigation operation.

In some embodiments, the target object can be clamped and moved by a robotic arm, or can be picked up by a doctor and moved based on the current position and target position of the target object under the guidance of the target positioning device for surgical navigation provided in the present application.

The method provided by the embodiment of the present invention further includes: displaying the actual position of the target object.

Furthermore, the method provided by the embodiment of the present invention may also include displaying the actual position and the target position of the target object to provide surgical navigation support for doctors.

The surgical navigation system provided by the embodiment of the present invention can be applied to, for example, neurosurgery navigation, oral surgery navigation, percutaneous puncture navigation system, orthopedic navigation system, etc.

According to the embodiment of the present application, based on different application scenarios, the front end of the robotic arm can be configured with different surgical instruments to meet different surgical requirements.

FIG5 shows a schematic flow diagram of a target positioning method for surgical navigation using a surgical navigation system target positioning method based on artificial intelligence. Referring to FIG5, in some embodiments, the surgical navigation implementation process may include acquiring an image of the surgical field through a depth camera, recognizing the image based on an artificial intelligence algorithm, and then obtaining the precise position of the object based on the image with depth information.

Fig. 6 shows another implementation flow diagram of the target positioning method for surgical navigation based on artificial intelligence provided by an embodiment of the present application. Referring to Fig. 6, in some embodiments, the surgical navigation implementation process may include collecting surgical field images by a depth camera, obtaining key points of the object based on an artificial intelligence algorithm, obtaining the precise position of the object based on the key points of the object, and finally the system controls the robotic arm to reach the specified position.

Referring to Figures 5 and 6, the method according to the embodiment of the present application uses an artificial intelligence algorithm to obtain the key points of the object, and then determines the precise position of the object based on the position of the key points, which can effectively reduce the computational complexity of target object position identification and improve positioning accuracy.

The target positioning method proposed in the optical surgical navigation system of the embodiment of the present invention significantly optimizes the surgical process. This innovation abandons the reliance on traditional markers and can achieve high-precision surgical navigation with only a single depth camera. This method not only effectively avoids the risk of reduced measurement accuracy due to contamination or occlusion of markers, but also completely eliminates the constraints of markers on surgical instruments, greatly improving the flexibility and execution efficiency of the operation. More importantly, the method also has excellent flexible target recognition and marking capabilities, further broadening the application scenarios of surgical navigation. At the same time, since additional markers and their related sterilization steps are no longer required, the method significantly reduces the overall cost of the operation and brings substantial economic benefits to medical institutions and patients. The target positioning method in the embodiment of the present application determines the precise position of an object by accurately identifying its key points. This strategy not only reduces the amount of calculation in the object recognition and positioning process, but also improves the accuracy and efficiency of positioning, thereby comprehensively improving the overall performance of the surgical navigation system.

Embodiment 2

FIG. 7 shows a schematic structural diagram of a target positioning device for surgical navigation according to an embodiment of the present application.

Referring to FIG. 7 , a target positioning device for surgical navigation according to an embodiment of the present application may include:

An image acquisition module 201 is used to acquire a surgical image based on a visual camera, and to identify key points of a target object in the surgical image based on a pre-trained key point recognition model;

A coordinate conversion module 202, configured to obtain the image coordinates and depth information of the key point in the surgical image, and convert the image coordinates into world coordinates based on the depth information;

A target positioning module 203 is used to determine the actual position of the target object based on the world coordinates of the key points;

The movement control module 204 is used to track the movement of the target object in real time based on the actual position of the target object.

In some embodiments, the coordinate conversion module 202 is specifically used for:

Converting the image coordinates to camera coordinates based on the depth information;

The camera coordinates are converted to the world coordinates based on the extrinsic parameters of the camera.

In some embodiments, the coordinate conversion module 202 is specifically used for:

Converting the image coordinates into the camera coordinates based on the depth information and the camera coordinate calculation formula;

The camera coordinate calculation formula includes:

_Among them, ( i , j ) are the image coordinates _of the target pixel, ( Xc , Yc , Zc ₎ are the camera coordinates of the target pixel, fx =F/dx, fy =F/dy, _F is the camera focal length in the form of image coordinate system, dx is the horizontal physical size of a single pixel, dy is _the vertical physical size of a single pixel, u0 is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, _v0 is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and is the depth value of the target pixel _.

In some embodiments, the external parameters of the camera include a rotation matrix and a translation vector, and the coordinate conversion module 202 is specifically used for:

Converting the camera coordinates to the world coordinates based on the rotation matrix, the translation vector and the world coordinate calculation formula;

The world coordinate calculation formula includes:

_Among them, ( Xc , Yc , Zc ) are the camera coordinates of the target pixel, ( Xw _, Yw , _Zw ) are the world coordinates of the _target pixel _, R is the rotation matrix, _and T is the translation vector.

In some embodiments, the coordinate conversion module 202 is specifically used for:

Acquiring the resolution of the surgical image;

Converting the resolution of the surgical image to a pre-stored image resolution with depth information;

The image coordinates and depth information of the key points are acquired based on the converted surgical image.

In some embodiments, the apparatus provided in the present application may further include a movement control module for controlling the movement of the target object based on the actual position of the target object.

In summary, the target positioning method proposed in the optical surgical navigation system in the embodiment of the present application significantly optimizes the surgical process. This innovation abandons the reliance on traditional markers and can achieve high-precision surgical navigation with only a single depth camera. This method not only effectively avoids the risk of reduced measurement accuracy due to contamination or occlusion of markers, but also completely eliminates the constraints of markers on surgical instruments, greatly improving the flexibility and execution efficiency of the operation. More importantly, the method also has excellent flexible target recognition and marking capabilities, further broadening the application scenarios of surgical navigation. At the same time, since additional markers and their related sterilization steps are no longer required, the method significantly reduces the overall cost of surgery and brings substantial economic benefits to medical institutions and patients. The target positioning method in the embodiment of the present application determines the precise position of an object by accurately identifying its key points. This strategy not only reduces the amount of calculation in the object recognition and positioning process, but also improves the accuracy and efficiency of positioning, thereby comprehensively improving the overall performance of the surgical navigation system.

Embodiment 3

On the basis of the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

Embodiment 4:

An embodiment of the present application also provides a surgical navigation system, including a target instrument, a depth camera, a display, and the target positioning device or computer equipment as described above.

Optionally, in some embodiments, the surgical navigation system further includes a navigation robotic arm trolley and a robotic arm, and the depth camera and the display are disposed on the navigation robotic arm trolley.

Optionally, the target device is disposed at the end of the mechanical arm. The mechanical arm is used to drive the target device to move.

FIG8 shows a schematic diagram of the structure of a surgical navigation system provided in an embodiment of the present application.

In a specific example, the target instrument may include, but is not limited to, a puncture needle.

In addition, the surgical navigation system according to the embodiment of the present application may include, for example, a depth camera, a robotic arm, and a navigation robotic arm trolley with the above-mentioned target positioning device or the above-mentioned computer equipment built in. The depth camera is fixed to the trolley by a movable telescopic bracket, and the depth camera and the robotic arm can be set on the same trolley. The position of the depth camera can be adjusted by the bracket. During surgery, the depth camera is placed obliquely above the surgical field, about 1 meter away and does not affect the doctor's surgical operation. The end of the robotic arm is equipped with a puncture needle clamping device, which can clamp various puncture needles, ablation needles, etc. Furthermore, based on different surgical requirements, the end of the robotic arm can be equipped with different clamping devices and different surgical instruments.

The display can be placed directly on the navigation robot arm trolley, or can be set on the navigation robot arm trolley through various brackets such as telescopic brackets, so as to facilitate the doctor's viewing and use.

The surgical navigation system provided in the present application can also integrate a depth camera, a display, and a robotic arm into a navigation robotic arm trolley, thereby reducing the space occupied in the operating room and facilitating use by doctors.

Optionally, the depth camera can be a RealSense D435i camera from Intel, which can output images with a resolution of 1280*720 and a frame rate of 30FPS. The robotic arm can be a UR5 series robotic arm from Universal Robots, which has a load capacity of 10 catties and a working radius of 85 cm. The RealSense D435i camera and the UR5 robotic arm are placed on the same vehicle, and the RealSense D435i camera can be fixed by a separate movable telescopic bracket.

The depth camera can obtain the surgical field image. The key points of the object to be identified are obtained through deep learning models or algorithms such as YOLOv8, and the spatial position information of the object is determined based on the key point coordinates and their depth information.

According to the surgical navigation system of the embodiment of the present application, the ablation needle at the end of the robotic arm can be regarded as an object to be identified, and the image obtained by the depth camera is processed by a deep learning algorithm to obtain the spatial position of the ablation needle. The spatial position of the puncture needle is obtained in real time through the video image to achieve real-time position tracking. Furthermore, the ablation needle is moved to the needle insertion position of the puncture plan by controlling the robotic arm, and the puncture is performed in real time. The needle insertion position is set by the doctor, including the needle insertion point and the needle insertion angle, and the needle insertion point is a world coordinate point; optionally, the needle insertion angle is the angle between the puncture needle and the x-axis or y-axis in the horizontal direction and the angle between the puncture needle and the z-axis in the vertical direction.

Embodiment 5

An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, which is loaded and executed by a processor to implement the above-mentioned target positioning method for surgical navigation.

Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk, etc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

Embodiment 6

The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the target positioning method for surgical navigation described in any of the above embodiments.

The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the pros and cons of the implementation.

It can be understood by those skilled in the art that all or part of the steps of implementing the above-mentioned embodiments can be completed by hardware, or can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased. It can be clearly understood by those skilled in the art that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit, and the above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the above-mentioned method embodiment, and will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form.

The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims (9)

1. A target positioning method for surgical navigation, characterized in that the method comprises: Acquire a surgical image based on a visual camera, and identify key points of a target object in the surgical image based on a pre-trained key point recognition model; Acquire the image coordinates and depth information of the key point in the surgical image, convert the image coordinates into camera coordinates based on the depth information, and convert the camera coordinates into world coordinates based on external parameters of the camera; Determine the actual position of the target object based on the world coordinates of the key point; Tracking the movement of the target object in real time based on the actual position of the target object; The converting the image coordinates into camera coordinates based on the depth information comprises: Converting the image coordinates into the camera coordinates based on the depth information and the camera coordinate calculation formula; The camera coordinate calculation formula includes: Among them, (i, j) are the image coordinates of the target pixel, (Xc, Yc, Zc) are the camera coordinates of the target pixel, fx=F/dx, fy=F/dy, F is the camera focal length in the form of image coordinate system, dx is the horizontal physical size of a single pixel, dy is the vertical physical size of a single pixel, u0 is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, v0 is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and is the depth value of the target pixel. 2. The method according to claim 1, characterized in that the external parameters of the camera include a rotation matrix and a translation vector, and the converting the camera coordinates into the world coordinates based on the external parameters of the camera comprises: Converting the camera coordinates to the world coordinates based on the rotation matrix, the translation vector and the world coordinate calculation formula; The world coordinate calculation formula includes: Among them, (Xc, Yc, Zc) is the camera coordinates of the target pixel, (Xw, Yw, Zw) is the world coordinates of the target pixel, R is the rotation matrix, and T is the translation vector. 3. The method according to any one of claims 1 to 2, characterized in that the step of obtaining the image coordinates and depth information of the key point in the surgical image comprises: Acquiring the resolution of the surgical image; Converting the resolution of the surgical image to a pre-stored image resolution with depth information; The image coordinates and depth information of the key points are acquired based on the surgical image after resolution conversion. 4. A target positioning device for surgical navigation, characterized in that the device comprises: An image acquisition module, used to acquire a surgical image based on a visual camera, and to identify key points of a target object in the surgical image based on a pre-trained key point recognition model; A coordinate conversion module, used for obtaining the image coordinates and depth information of the key point in the surgical image, converting the image coordinates into camera coordinates based on the depth information, and converting the camera coordinates into world coordinates based on the external parameters of the camera; The converting the image coordinates into camera coordinates based on the depth information comprises: Converting the image coordinates into the camera coordinates based on the depth information and the camera coordinate calculation formula; The camera coordinate calculation formula includes: Wherein, (i, j) is the image coordinate of the target pixel, (Xc, Yc, Zc) is the camera coordinate of the target pixel, fx=F/dx, fy=F/dy, F is the focal length of the camera in the form of the image coordinate system, dx is the horizontal physical size of a single pixel, dy is the vertical physical size of a single pixel, u0 is the horizontal distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, v0 is the vertical distance from the pixel coordinate of the center of the image to the origin of the pixel coordinate system, and is the depth value of the target pixel; A target positioning module, used to determine the actual position of the target object based on the world coordinates of the key point; The movement control module is used to track the movement of the target object in real time based on the actual position of the target object. 5. A computer device, characterized in that the computer device includes a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the target positioning method as described in any one of claims 1 to 3. 6. A surgical navigation system, characterized in that it comprises a target instrument, a depth camera, a display, and the target positioning device as described in claim 4 or the computer device as described in claim 5. 7. The surgical navigation system according to claim 6 is characterized in that the system also includes a navigation robotic arm trolley and a robotic arm, the robotic arm is used to drive the target instrument to move, and the depth camera, the display and the robotic arm can be arranged on the navigation robotic arm trolley. 8. A computer-readable storage medium, characterized in that the readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the target positioning method as described in any one of claims 1 to 3. 9. A computer program product, comprising computer instructions, characterized in that when the computer instructions are executed by a processor, the target positioning method according to any one of claims 1 to 3 is implemented.