CN114041875B - An integrated surgical positioning and navigation system - Google Patents

Abstract

The invention discloses an integrated operation positioning navigation system which comprises three parts, namely imaging equipment, a pitching self-adaptive device and a marker, wherein the pitching self-adaptive device is connected with the imaging equipment and is carried on an operation robot, the marker is respectively arranged on the surface of an end effector and the surface above a focus of a patient, the precise movement direction of each multi-degree-of-freedom mechanical arm is calibrated and determined, the coordinate system of the imaging equipment and the coordinate system of the robot are unified, the movement navigation of the multi-degree-of-freedom mechanical arm at the far end of the focus is carried out based on the detection of the focus marker and the measurement position of the imaging equipment, the relative space geometrical relation is calculated based on the detection of the focus marker and the effector, and the movement navigation of the multi-degree-of-freedom mechanical arm at the near end of the focus is carried out. The invention has compact structure and simple flow, and improves the intelligent, automatic and integrated degree of the operation robot; meanwhile, by tracking the surgical instrument and the focus in real time, the method has good robustness for the position movement of the patient.

Description

An integrated surgical positioning and navigation system

technical field

The invention relates to the technical field of medical equipment, in particular to an integrated surgical positioning and navigation system.

Background technique

Robot-assisted surgery is to replace doctors with surgical robots to complete some traditional surgical tasks. The robot can be operated by a doctor or trained by the doctor's experience. Robot-assisted surgery combines the experience of doctors, and has the advantages of high precision, stable work, no hand trembling, simplified working steps, avoiding radiation damage, minimizing surgical damage, and reducing patient pain. The precise work of the surgical robot needs to ensure the exact location of the lesion and surgical instruments during the operation, so as to plan and guide the robot's action execution.

The current medical imaging technology can complete the inspection and observation of patient lesions. Common imaging techniques include CT scanning (Computer Tomography, computerized tomography), MRI (Magnetic Resonance Imaging, magnetic resonance imaging), ultrasound imaging and other techniques. Among them, CT and MRI have high resolution and can obtain fine three-dimensional images, but lack real-time navigation capabilities, and the shooting process is complicated; CT radiation may cause damage to patients. Ultrasound imaging can obtain pictures of the lesion area in real time, but the image resolution is low. These imaging technologies can realize the imaging of tissues and organs near the lesion, but do not have the imaging and navigation of the remote robot of the lesion. Therefore, the completion of robot-assisted surgery requires not only medical imaging technology, but also a navigation system with positioning capabilities on the robot side, tracking lesions and surgical instruments, and combining with medical imaging to complete the entire surgical operation.

The current common surgical navigation systems can be roughly divided into two categories: optical navigation and electromagnetic navigation, according to different spatial positioning principles. Optical navigation is to shoot the target object through multiple optical imaging devices, or calculate the time of light emission and return from the target object, and then calculate the spatial coordinates of the marker points through geometric relations.

Patent CN 110025891A proposes a binocular-based surgical vision navigation device, which uses the principle of binocular parallax to measure the position of surgical instruments and markers of patients to achieve tracking.

Patent CN 113229937A proposes a structured light imaging device for surgical navigation, which uses structured light scanning to obtain real-time three-dimensional point cloud data, locates instruments and lesions, and completes navigation tasks.

Electromagnetic navigation generally uses space coils arranged in different directions to establish a three-dimensional magnetic field space, and detects and calculates the spatial parameters of the probe through the magnetoelectric sensitive probe.

Patent CN113069206A proposes a calibration method for an augmented reality surgical navigation system based on electromagnetic positioning. The invention is used for rapid calibration of the surgical site and improves the accuracy of virtual-real fusion.

In addition, there is also work on the combination of optical navigation and electromagnetic navigation. Patent CN110537983A proposes an opto-magnetic integrated puncture surgery navigation platform, which uses electromagnetic surgery navigation technology and optical surgery navigation technology simultaneously to track the tip of the puncture needle and perform three-dimensional positioning of the puncture needle. reconstruction.

However, the above-mentioned navigation devices mainly adopt an independent design, and are generally erected separately from the surgical robot. The operation process is relatively complicated and occupies a large space. In addition, the existing clinical navigation devices are expensive, which is not conducive to popularization.

Contents of the invention

The purpose of the present invention is to propose an integrated surgical positioning and navigation system. Based on depth information, the system adopts the integrated design of the surgical positioning and navigation device and the mechanical arm, and can track the position of the patient's lesion in real time during the operation. Information such as the position and posture of the robot can provide accurate real-time positioning and navigation for the surgical robot.

The surgical positioning and navigation system of the present invention is divided into three parts: imaging equipment, pitch self-adaptive device and markers. The imaging device is connected with the pitch self-adaptive device and mounted on a multi-degree-of-freedom mechanical arm. The markers are respectively set above the patient's lesion The point set on the body surface (the number of points is M) and the point set on the shell of the end effector (the number of points is E). The imaging equipment captures depth and RGB image information, and obtains information such as the position of the patient's lesion, the coordinates and attitude of the instrument at the end of the robot's manipulator through processing, and the pitch adaptive device automatically adjusts the shooting angle of the imaging equipment according to the lesion position information, which can realize the focus of the whole operation. Tracking and navigation of surgical instruments. The pitch adaptive device adjusts the shooting direction of the imaging device to keep the lesion marker in the field of view; the imaging device recognizes and measures the marker, and determines the spatial position of the lesion relative to the robot by processing the geometric relationship. The imaging device can obtain depth information and RGB image information at the same time. The depth information is matched with the RGB image information. The image is used to identify the marker and obtain its center position. The depth information is used to obtain the coordinates of the center position in the camera coordinate system. End effector markers are distinguished from focal markers. The relative geometric relationship between the lesion markers, the lesion, and the puncture path has been calculated through medical images such as CT before operation. The coordinates of the lesion marker point set are used to calculate the coordinates of the lesion and the puncture path in the robot coordinate system, or the geometric relationship of the end effector relative to the lesion. The pitch adaptive device automatically adjusts the orientation of the imaging device according to the coordinates of the lesion to ensure that the lesion marker is always in the imaging field of view. Specifically, the imaging device may adopt a binocular stereo vision camera, a structured light stereo vision camera or a laser radar plus an RGB camera. The pitch adaptive device is composed of a motor drive module, a rotary encoder, a shaft coupling and a camera fixing bracket. The motor drive module includes a motor and a driver, and the camera fixing bracket includes a set of fixing brackets for connecting the motor, imaging device, encoder and mechanical arm. During work, the driver drives the motor to control the rotation of the imaging device, and the rotation of the imaging device is measured and read in real time by the rotary encoder, which is used for closed-loop control of attitude adjustment. The motor controls the imaging device in real time according to the spatial position coordinates of the lesion markers measured by the imaging device Keep the lesion marker in the field of view continuously.

In the present invention, the imaging device can be composed of a set of lidar and an RGB camera. Lidar remote sensing technology is based on the principle of TOF laser ranging. It uses laser transmitters and receivers to measure the flight time of the laser from the imaging device to the target object to obtain the distance, and uses the micro-MEMS to control the mirror to scan the space scene to obtain the full range. The depth information of the object points in the field of view is used to construct a spherical coordinate system centered on the radar to achieve high-resolution real-time 3D imaging. The RGB camera corresponds to the lidar camera field of view. The present invention uses an RGB camera to acquire surgical images, identifies the positions of marker point sets on the patient's body surface and end effectors in the RGB surgical images, and the laser radar acquires the coordinates of the positions as the spatial coordinates of the marker point sets. By processing the coordinate information of the marker point set, the spatial position and posture of the lesion and surgical instruments are obtained.

The pitch adaptive device includes motors, encoders, couplings and camera brackets. The camera bracket is connected with a multi-degree-of-freedom mechanical arm, a motor, an imaging device and an encoder, and the imaging device is respectively connected with the motor and the encoder through a coupling. The motor drives the imaging device to rotate, and the encoder measures the rotation angle of the imaging device. During the operation, the imaging equipment obtains the position of the lesion in real time, and the pitch adaptive device controls the rotation of the imaging equipment, so that the lesion marker point set is always kept in the center of the field of view. Integrated surgical positioning and navigation means that the imaging equipment is connected to the multi-degree-of-freedom robotic arm instead of being placed separately. The purpose of the pitch adaptive device is to ensure that the imaging equipment keeps recording and measuring the position of the lesion when it moves with the robotic arm.

The markers are divided into two types: focus markers and end-effector markers, both of which are composed of several pattern blocks with fixed shapes. The lesion markers are a group of stickers, which are dispersedly pasted by the doctor on the body surface above the lesion in the patient's chest cavity before the operation. The number is M, and M can be 4 or other values. The end effector marker is a group of marking point patterns on the surface of the end effector shell, the number is E, and E can be 12 or other values.

The multi-degree-of-freedom robotic arm includes a chassis, an XYZ three-dimensional mobile platform, a three-axis gimbal module, and a closed-loop control module for the robotic arm. The bottom of the chassis is equipped with universal wheels with horizontally adjustable center of gravity, which can realize the overall movement and fixed-point anchoring of the piercing robot. The XYZ three-dimensional mobile platform is braked by a ball screw, which has high precision and strong stability, and is used to move the surgical end effector to a specified spatial position. The three-axis gimbal module includes a yaw adjustment unit, a roll adjustment unit and a pitch adjustment unit, which are used to adjust the end effector to a specified attitude. The purpose of the multi-degree-of-freedom robotic arm is to realize the full-dimensional position and attitude adjustment of the end effector in the surgical area.

The working principle of an integrated surgical positioning and navigation system proposed by the present invention includes three parts: calibration navigation coordinate system, remote navigation and proximal navigation. During the operation, the multi-degree-of-freedom robotic arm drives the end effector to move toward the lesion from far to near. For the integrated surgical positioning and navigation system, the multi-degree-of-freedom robotic arm and the end effector are far away from the lesion in the early stage of the operation. When the lesion marker point set is within the field of view of the imaging device, the end effector cannot be guaranteed to be within the field of view of the imaging device at the same time. This stage is handled by remote navigation. In the later stage of the operation, the end effector moves toward the lesion, and when the lesion marker and the end effector marker are both in the field of view of the imaging device, proximal navigation can be used for processing. The purpose of this design is to ensure that the integrated surgical positioning and navigation system completes precise positioning and navigation at all stages of surgery.

The purpose of calibrating the navigation coordinate system is to determine the precise motion direction of each degree of freedom of the multi-degree-of-freedom manipulator, and to unify the imaging system coordinate system and the robot coordinate system. First, establish the world coordinate system of the manipulator, that is, the robot coordinate system, determine the position of the coordinate origin of the coordinate system, and measure the spatial position of the rotation center of the imaging device and the origin of the imaging device. Secondly, the movement of the three-dimensional platform of the multi-degree-of-freedom manipulator is controlled separately, and the movement direction of each degree of freedom of the three-dimensional mobile platform of the multi-degree-of-freedom manipulator is determined by measuring the moving direction of a fixed position marker in the imaging device. Finally, the end effector is controlled to rotate multiple times within the field of view of the imaging device, and the movement direction of each degree of freedom of the three-axis gimbal of the multi-degree-of-freedom manipulator is determined by measuring the rotation direction of the marked point.

Remote navigation means that in the early stage of surgery, firstly, when the multi-degree-of-freedom robotic arm is in the reset state, the navigation module is activated, the angle of the imaging device is adjusted, and the patient's lesion marker is placed in the center of the field of view of the imaging device. Next, the focus marker point set is identified through the RGB image, and the imaging device detects the spatial position of the focus marker point set to obtain its specific position in the coordinate system of the imaging system. Then, through the coordinate transformation relationship, the position of the lesion marker in the coordinate system of the imaging system is transformed into the coordinates of the robot coordinate system. Then, by registering the coordinates of the lesion marker point set obtained by the imaging equipment with the coordinates of the lesion marker point set obtained by the three-dimensional reconstruction of the CT image, the corresponding transformation matrix is obtained, and the coordinates of the lesion and the puncture path in the robot coordinate system are obtained by using this matrix. The lower position realizes the remote positioning of the lesion position and puncture path in the robot coordinate system.

Proximal navigation refers to controlling the robotic arm to move the end effector to the proximal end of the lesion at the later stage of the operation. At this time, the end effector marker and the lesion marker appear in the field of view of the imaging device at the same time. At this time, first, the imaging device recognizes and detects the two types of markers, and obtains the corresponding position information, from which the relative relationship between the end effector marker and the lesion marker, that is, the relative position of the end effector and the lesion marker can be obtained. Then, the lesion is projected into the coordinate system of the imaging system through point registration and matrix transformation, and the point cloud information is processed by Kalman filtering to eliminate noise, so that the positional relationship of the lesion relative to the end effector can be obtained, and the instrument on the end effector The tip point is used as the origin to establish the instrument tip coordinate system, and the direction of the coordinate system is consistent with the direction of the robot coordinate system, so as to realize the proximal positioning of the lesion position and the puncture path in the instrument tip coordinate system.

The integrated surgical positioning and navigation system proposed by the present invention has a compact structure, avoids the complexity of the split surgical navigation system, and improves ease of use; at the same time, it simplifies the operating conditions of the surgical navigation system, enables the surgical robot to work independently, and increases the usage scenarios; in addition , the present invention proposes that laser radar and RGB image fusion information can be used to locate the target, which can accurately guide surgical instruments and greatly reduce the cost; by tracking surgical instruments and lesions in real time, it has good robustness to patient position movement; the present invention is a non-invasive The contact guidance scheme improves the operation efficiency and intelligence level.

Description of drawings

In order to make the purpose, technical scheme and beneficial effect of the present invention clearer, the present invention provides the following drawings for illustration:

Fig. 1 is a structure diagram of the whole machine mounted on a multi-degree-of-freedom surgical robot of the surgical positioning and navigation system of the present invention.

Fig. 2 is a structure diagram of a multi-degree-of-freedom manipulator base and a Z-axis moving platform.

Fig. 3 is a structural diagram of the XY-axis moving platform of the multi-degree-of-freedom manipulator device.

Fig. 4 is a structural diagram of the robotic arm platform and the surgical end effector.

Fig. 5 is a structural diagram of the surgical positioning and navigation system of the present invention.

Fig. 6 is a working principle diagram of the surgical positioning and navigation system of the present invention.

Fig. 7 is a schematic diagram of remote navigation in the present invention.

Fig. 8 is a schematic diagram of near-end navigation in the present invention.

Fig. 9 is a schematic diagram of the coordinate system of the present invention.

Specific implementation method

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

An example of the present invention, as shown in Figure 1, is an integrated surgical positioning and navigation surgical robot. The surgical robot consists of a multi-degree-of-freedom mechanical arm device 1 , a surgical positioning and navigation system 2 , and a surgical end effector device 3 . The multi-degree-of-freedom robotic arm device 1 includes a chassis unit 4 , a Z-axis moving platform 5 , an X-axis moving platform 6 , a Y-axis moving platform 7 , a roll adjustment unit 8 , a yaw adjustment unit 9 , and a pitch adjustment unit 10 . The multi-degree-of-freedom manipulator 1 is constructed by connecting the above units one by one.

Chassis unit 4 shown in Fig. 2 is mainly made up of car body aluminum frame 11 and universal fast anchoring wheel set 12, and universal quick anchoring wheel set 12 is the universal wheel of four additional supporting plates, and universal wheel 18 moves in place Finally, the supporting disc 19 can be rotated, and connected with the universal wheel frame 20 by threads, and the rotation can reduce the height, support the ground, and keep the surgical robot position. The universal fast anchoring wheel is fixed to the bottom plate, and the chassis is fixed to the aluminum frame of the car body, so that the chassis can move in all directions. After reaching the target position, it can quickly fix the position and adjust the level of the machine.

The sliding part of the Z-axis mobile platform 5 is fixed on two parallel aluminum tubes on the chassis aluminum frame 11 by two square guide rails in parallel. As shown in FIG. 3 , the sliding parts of the X-axis mobile platform 6 and the Y-axis mobile platform 7 are respectively fixed on two parallel aluminum tubes on the aluminum frame of the platform by two square guide rails 28 in parallel. The slide table of the Z-axis mobile platform 5 is connected with the X-axis mobile platform 6, and the slide table of the X-axis mobile platform 6 is connected with the slide table of the Y-axis mobile platform 7 to form an XYZ three-dimensional mobile platform. The braking parts of the three-dimensional mobile platform are all equipped with SFU-1605 ball screw rods 22, 23, 24 with a lead of 4mm and driven by stepping motors 21, 25, 26. (If used with 57 stepper motors, the needle insertion speed of 6cm/s can be realized, and the controllable precision is less than 0.1mm). The ball screw rods of the three-dimensional mobile platform are all connected with the encoder 27 through shaft couplings 29 and 30, and the closed-loop control slide table moves. The XYZ three-dimensional mobile platform can realize the positioning of the target puncture position of the end effector device 3 above the CT machine bed.

As shown in FIG. 4 , while the surgical end effector device 3 is positioned above the CT machine, the angle of the puncture needle actuator needs to be adjusted according to the lesion position obtained from the CT image and the puncture needle path. Therefore, the three adjustment units of the roll adjustment unit 8 , the yaw adjustment unit 9 , and the pitch adjustment unit 10 are used to adjust the optimal needle insertion angle of the end effector. The roll adjustment unit 8, the yaw adjustment unit 9, and the pitch adjustment unit 10 all use the slewing bearing 31 to build the rotation center, and are fixedly connected with the bearing box 39. , 37) drive the worm, and the three adjustment unit slides 34,35,36 are all connected with the encoder (such as 40) (wherein, 34 is the roll adjustment unit slide, 35 is the yaw adjustment unit slide, and 36 is the pitch Adjusting unit slide table) for closed-loop control of rotation. The base of the roll adjusting unit 8 is connected with the Y-axis moving platform. The slide table of the roll adjustment unit 8 is connected to the base of the yaw adjustment unit 9, and the slide table of the yaw adjustment unit 9 is connected to the base of the pitch adjustment unit 10 to form the platform of the surgical end effector device. The surgical end effector device 3 It is connected with the slide platform 36 of the pitch adjustment unit on the cloud platform. The spatial three-dimensional posture adjustment of the surgical end effector device 3 can be realized through the parallel calling of the three adjustment units. The end effector marker 41 is attached to the end effector housing.

As shown in Figure 5, the surgical positioning and navigation system 2 is connected to the lower middle section of the Y-axis mobile platform 7. The surgical positioning and navigation system mainly includes two parts, one is the imaging device used to obtain depth information and image information, and the other is the imaging device. Attitude adjustment module. An imaging device solution adopted in this example is a laser radar camera 13 combining a laser radar and an RGB camera, and the attitude adjustment module is specifically a pitch adjustment device 14 . The laser radar camera 13 uses TOF laser ranging technology to obtain depth information of space objects, uses RGB camera to obtain 2D images, and the depth information of space objects and 2D images are matched and fused. The present invention determines the coordinates of the target in the camera coordinate system in the visual field by processing the depth information of the space object and the 2D image. The pitch adjustment device 14 is connected with the Y-axis mobile platform 7 by the camera fixed bracket 15, the stepper motor 16 and the encoder 17, and the stepper motor 16 and the encoder 17 are respectively connected with the lidar camera 13 by a shaft coupling (such as 38), In order to realize the pitch adjustment of the imaging field of view of the camera.

Specifically, the imaging device may also adopt a binocular stereo vision camera. The imaging of the binocular stereo vision camera is based on the principle of parallax and uses the camera to take surgical images containing markers from different positions. The marker pattern can be segmented from the image, and the positional deviation between the corresponding marker points in the left and right camera images can be calculated. , to obtain the three-dimensional geometric information of the marker. By processing the three-dimensional coordinate information of the marker point set, the spatial position and posture of the lesion, the puncture path, and the surgical instrument can be obtained, as well as the spatial position of the puncture path.

The use of structured light stereo vision cameras for imaging equipment is also a feasible method. The basic principle of structured light stereo vision cameras is to project light with certain structural characteristics onto the object to be photographed (i.e., the chest cavity of the patient and the end of the robot). device), and then the camera will record the projected object. Establish a spherical coordinate system with the camera as the origin as the imaging system coordinate system. For the light projected on the target object, different image phase information will be returned due to the different depth of the surface of the object, and then the phase change is converted into Depth information to obtain three-dimensional position information. The position of the marker will be obtained by segmenting the image captured by the camera, and the position information of the segmented marker in space will be obtained through phase calculation. Similarly, by processing the three-dimensional position information of the marker point set, the spatial position and posture of the lesion and surgical instruments, as well as the spatial position of the puncture path can also be obtained.

Figure 6 shows an example of the working principle of the surgical positioning and navigation system, which includes three parts: calibration of the navigation coordinate system, distal navigation and proximal navigation. Before the surgical positioning and navigation system and the surgical robot are put into use, the navigation coordinate system is first calibrated 108 . The purpose of calibration is to determine the direction of the surgical robot’s motion coordinate system and the coordinates and rotation direction of the lidar pitch point. When the XYZ three-axis of the multi-degree-of-freedom robotic arm device 1 is in the initial position, that is, the zero state, set the center point of the top of the Z-axis mobile platform is the origin of robot coordinate system 103 The direction of the robot motion coordinate system is and /> (/> and /> Respectively represent the movement direction of the XYZ three-dimensional mobile platform and the roll, yaw, and pitch three-axis gimbal), and the coordinate Q of the laser radar pitch rotation point is /> The lidar acquires depth information of objects within the field of view, and projects it in the coordinate system of the imaging system (ie, the spherical coordinate system of the lidar) 102 . Use a fixed position block C with markers to calibrate /> The coordinates of the block marker point C _h in the imaging system coordinate system are /> h is the serial number of the marker point, the number is G, G=10 (G is 10 as an example, and other values can also be taken), the coordinate A _f of the marker point on the surgical end effector A is /> f is the serial number of the marking point, the number is E, and E=12 (E takes 12 as an example, and can also take other values). When calibrating the moving direction of the XYZ three-axis, the block mark point C _h is used as the reference object, and the relative moving direction of the marker when calculating the moving single axis is the direction of the axis. Move the X-axis, Y-axis or Z-axis successively to calculate the average coordinate difference of the block mark point C _h respectively> (Δr _ch , Δθ _ch , Δφ _ch respectively represent the coordinate difference of the object mark point at the two moments before and after the movement of the mobile platform), and converted to the Cartesian coordinate system, respectively get /> as /> When calibrating the three-axis movement direction of roll, pitch, and yaw, the marked point A _f on the surgical end effector is used as a reference object, and the single axis is rotated successively to calculate the plane where the track of the marked point is located in the space. The three planes represent the gimbal Three directions of motion. In addition, the measurement obtains the coordinate Q of the pitch rotation point of the lidar. Before the operation, an image 116 of the patient's lungs will be obtained through a CT machine, and image segmentation 105 and three-dimensional reconstruction 106 are performed on the CT image to obtain the lesion marker points in the CT coordinate system 101, the position of the lesion point cloud, and the puncture path confirmed by the doctor 107 ( The starting point of the puncture path is located on the chest surface, and the end point is located in the center of the lesion).

During the operation, the robotic arm is first initialized 112 to complete the self-check of the device. The multi-degree-of-freedom mechanical arm device drives the surgical end effector device to move from far to near to the lesion. In the early stage of the operation, the multi-degree-of-freedom robotic arm device and the surgical end effector device are far away from the lesion, which can only ensure that the lesion marker is within the field of view of the imaging device 109. At this time, the remote navigation 113 is used, as shown in Figure 7, the imaging device recognizes The lesion marker point set 203, through the transformation of the imaging system coordinate system 102 and the robot coordinate system 103, projects the position of the lesion marker point set to the robot coordinate system, and then the lesion marker point set under the robot coordinate system and the CT coordinate system Perform point cloud registration 201 to obtain a transformation matrix and convert the lesion point 202 and the starting point of the puncture path 206 to the robot coordinate system 103, thereby realizing the navigation of the robot arm movement. During work, the lidar moves a distance D on the X axis, the distance L between the actual signal receiving end of the lidar camera and the coordinate Q of the lidar pitch rotation point is L, and the pitch adjustment angle is θ. At this time, the coordinates of the actual receiving point of the lidar camera are For remote navigation, the laser radar observes the marker point set 203 around the disease focus, and returns the coordinates 204 of the focus marker point set as /> u is the serial number of the lesion marker, the number is M, M=4, (M takes 4 as an example, and other values can also be taken), and P _u is a point on the coordinate system of the imaging system centered on R. Convert the point P _u on the coordinate system of the imaging system to the midpoint of the Cartesian coordinate system with R as the origin /> That is, the coordinates of the lesion marker point set in the robot coordinate system should be R and /> The sum of the corresponding xyz components is /> The coordinates 205 of each marker point obtained from the CT image are /> Registration of point sets 201, 104 is performed on Mu _{and Nu} .

When the surgical end effector device moves to the proximal end of the lesion, as shown in FIG. 8 , proximal navigation 114 is performed, and the laser radar camera is used to simultaneously observe (110) the end effector device and two types of markers 304, 203 near the lesion. , and obtain its coordinate system information at the same time, so that the positional relationship between the end effector marker point set A _f 302 and the surrounding markers 303 can be obtained. Matching the lesion marker point set 301 in the coordinate system of the imaging system and the CT coordinate system converted to the spherical coordinate system, that is, for the point cloud 303 and point cloud /> 305 (N _u ^* is the projection of _Nu in spherical coordinates) matches 301, 104. Substituting the optimal rotation matrix and translation matrix of the completed point cloud registration, the positional relationship between the lesion and the surgical end effector device is obtained, that is, the solution of the lesion coordinate B ₀ without pitch and translation errors is completed. Considering that there is a systematic error in the establishment of the coordinates of the end effector on the surgical end effector device due to manual operation, the position of the lesion will no longer be restored to the robot coordinate system 103 at this stage, and the distance between the end effector and the coordinates of the end effector will be measured by laser radar. The relative positional relationship between the lesions controls the movement of the end effector in three-dimensional space to achieve the puncture of the lesions. The coordinate system is the instrument tip coordinate system established at a fixed point T′ on the instrument tip. The direction of the coordinate system is the same as that of the robot coordinate system. The same direction. During the puncture and needle insertion process of the end effector, the lidar will always track the two types of markers, and use the above algorithm to solve the tracking of the lesion position coordinates and calculate the starting point coordinates of the puncture path. Among them, there is noise in the measurement of the marker points, and the optimal position value is estimated by using the Kalman filter. Finally, the vector relationship between the end effector and the lesion is converted into the movement amount of each degree of freedom of the mechanical arm, and the mechanical arm is adjusted to move to the needle insertion position 115 . During the entire needle insertion process, the imaging device always tracks the two types of markers to realize real-time confirmation and correction of the coordinates of the lesion and the coordinates of the starting point of the puncture path111.

The calculation method for calculating the actual direction of motion of each degree of freedom of the multi-degree-of-freedom manipulator in the calibration navigation coordinate system is specifically: moving the end effector with a single degree of freedom multiple times, measuring and tracking the marker through the imaging device, and calculating the moving direction of the marker, as The actual movement direction of each degree of freedom of the robot arm is used to construct the actual robot coordinate system.

The method for calculating the coordinates of the lesion in the robot coordinate system by the remote navigation is specifically: using a CNN convolutional neural network to process the RGB image of the imaging device, segmenting the pattern of the lesion marker, extracting the center point of the contour calculation, and obtaining the point set of the lesion marker in the imaging After the coordinates in the system coordinate system (that is, the spherical coordinate system of the depth camera), the coordinates of the lesion in the imaging system coordinate system are calculated through the relative geometric relationship between the lesion marker and the lesion, and through the transformation matrix between the imaging system coordinate system and the robot coordinate system, Project the lesion coordinates to the robot coordinate system.

The calculation method of the relative spatial position relationship between the surgical instrument and the lesion by the proximal navigation is specifically as follows: segmenting and detecting the lesion marker and the end effector marker, establishing the instrument tip coordinate system with the tip of the surgical instrument as the origin, and the direction of the coordinate axis and the mechanical The direction of motion of the arm degrees of freedom is the same. Transform the coordinates of the lesion in the coordinate system of the imaging system to the coordinate system of the instrument tip, and use this coordinate system to guide the multi-degree-of-freedom robotic arm to move towards the lesion.

A specific implementation scheme of point cloud registration shown in FIG. 9 , taking the point set registration 301 performed by P _u and _Nu ^* in FIG. 8 as an example. The idea of registration is to firstly construct local geometric features according to the two point set data P _u and N _u ^* to be registered, and then perform point cloud data relocation 401 according to the local geometric features, mainly using iterative algorithm 402 to P _u and N u * _u ^* The two point set data are registered. The alignment registration transformation of two point sets should minimize the following objective function Among them, S is the rotation matrix, T is the translation matrix, M is the number of lesion marker points, ||·|| represents the norm calculation), which is the rotation parameter and translation parameter between the point cloud data to be registered and the reference point cloud data found , so that the data of the two point sets satisfy an optimal matching 403 under a certain measurement criterion. After obtaining the optimal matching rotation matrix S and translation matrix T, perform rotation and translation operations on the lesion center coordinate V ₀ obtained in the CT image, and obtain the lesion coordinate 404B ₀ =V ₀ S+T in the robot coordinate system , similarly the coordinates of the starting point of the puncture path can be calculated. After the coordinates of the focus and the starting point of the puncture path are known in the instrument tip coordinate system, the end effector is controlled to move to the focus of the disease. In addition, the point cloud registration 201 in FIG. 7 adopts the same method, and the difference is that the registration point cloud is a point set _Mu and a point set _Nu 205 .

An integrated surgical positioning and navigation device disclosed in the present invention includes three parts: an imaging device, a pitch adaptive device and a marker. The pitch adaptive device is connected to the imaging equipment and mounted on the surgical robot. Markers were placed on the surface of the end effector and the body surface above the patient's lesion, respectively. Calibration determines the precise motion direction of each degree of freedom of the multi-degree-of-freedom robotic arm, and unifies the coordinate system of the imaging device and the robot coordinate system. Based on the detection of the lesion markers and the measurement position of the imaging equipment, the motion navigation of the multi-degree-of-freedom manipulator at the distal end of the lesion is performed. Based on the detection of the lesion markers and actuator markers, their relative spatial geometric relationship is calculated, and the motion navigation of the multi-degree-of-freedom manipulator at the proximal end of the lesion is performed. The integrated positioning and navigation system realizes the real-time and precise navigation of the whole operation, with compact structure and simple process, which improves the intelligence, automation and integration of surgical robots.

The embodiments described above are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts all fall within the protection scope of the present invention. The present invention aims to use an integrated surgical positioning and navigation system to complete precise navigation of surgical instruments and operations during surgery.

Claims (4)

1. The integrated surgical positioning navigation system is characterized by comprising imaging equipment, a pitching self-adaptive device and a marker, wherein the imaging equipment is connected with the pitching self-adaptive device and is carried on a multi-degree-of-freedom mechanical arm, the marker is a patient focus marker point set which is respectively arranged on the surface above a patient focus and a surgical end effector marker point set on a surgical end effector shell, the imaging equipment captures depth and RGB image information, the position of the patient focus, the coordinates and posture information of a multi-degree-of-freedom mechanical arm end instrument are obtained through processing, and the pitching self-adaptive device automatically adjusts the shooting angle of the imaging equipment according to focus position information, so that focus tracking and surgical instrument navigation in the whole surgical process are realized; the imaging equipment and the pitching self-adaptive device form a navigation device, the navigation device is fixedly connected with the multi-degree-of-freedom mechanical arm to form an integrated operation robot, and the navigation device is not separated from the multi-degree-of-freedom mechanical arm to erect;

the imaging device consists of a group of laser radars and an RGB camera, the laser radar remote sensing technology is based on the principle of TOF laser ranging, the laser transmitter and the receiver are utilized to measure the flight time of laser from the imaging device to a target object so as to acquire the distance, a micro-electromechanical system is adopted to control a reflecting mirror to scan a space scene, the depth information of an object point in a full field of view is acquired, a spherical coordinate system taking the radar as a center is constructed, high-resolution real-time three-dimensional imaging is realized, the RGB camera corresponds to the field of view of the laser radar camera, a 2D operation image is acquired by utilizing the RGB camera, and the depth information of the space object and the 2D image are matched and fused; identifying the positions of a patient focus and a marker point set of an operation end effector in an RGB operation image, and acquiring coordinates of the positions as space coordinates of the marker point set by a laser radar; the method comprises the steps of obtaining the space positions and the postures of a focus and a surgical instrument and the space positions of puncture paths by processing coordinate information of marker point sets;

The pitching self-adaptive device comprises a motor, an encoder, a coupler and a camera bracket, wherein the camera bracket is connected with the multi-degree-of-freedom mechanical arm, the motor, imaging equipment and the encoder, and the imaging equipment is respectively connected with the motor and the encoder through the coupler; the motor drives the imaging equipment to rotate, and the encoder measures the rotation angle of the imaging equipment to realize a rotation control closed loop; in the operation process, the imaging device acquires focus positions in real time, and the pitching self-adaptive device controls the imaging device to rotate, so that focus marker point sets of patients always keep a centered position in a visual field; the integrated operation positioning navigation means that the imaging equipment is directly connected to a fixed position of the mechanical arm with multiple degrees of freedom to work instead of being separately placed, and the pitching self-adaptive device aims to ensure that the imaging equipment always keeps shooting and position measurement of a focus when moving along with the mechanical arm;

the markers are divided into two types, namely a patient focus marker and an operation end effector marker, the shape of the marker is fixed, the patient focus marker is a group of stickers, a doctor is dispersedly stuck on the body surface above a focus in the chest of the patient before operation, the number is M, and the M is an integer larger than or equal to 1; the surgical end effector marker is a group of marker point patterns on the surface of the surgical end effector shell, the number of the marker point patterns is E, and the E is an integer greater than or equal to 1;

The multi-degree-of-freedom mechanical arm comprises a chassis, an XYZ three-dimensional moving platform, a triaxial holder module and a mechanical arm closed-loop control module, and the multi-degree-of-freedom mechanical arm device is formed by connecting and building the modules one by one; the chassis unit (4) consists of a vehicle body aluminum frame (11) and a universal quick anchoring wheel set (12), the universal quick anchoring wheel set (12) is a universal wheel with four additional supporting plates, after the universal wheel (18) moves in place, the supporting plate (19) is rotated to be connected with the universal wheel frame (20) through threads, the universal quick anchoring wheel is rotated to be used for lowering the height and supporting the ground, the chassis is fixedly connected with the vehicle body aluminum frame through the universal quick anchoring wheel, so that the chassis can move universally, the quick anchoring position can be achieved after the chassis reaches a target position, and the horizontal condition of the machine can be adjusted; the XYZ three-dimensional moving platform is braked by using a ball screw, has high precision and strong stability, and is used for realizing the positioning of the target puncture position of the surgical end effector (3) above a sickbed of the CT machine; the sliding part of the Z-axis moving platform (5) is formed by connecting two square guide rails in parallel on two parallel aluminum pipes on a chassis aluminum frame (11), the sliding parts of the X-axis moving platform (6) and the Y-axis moving platform (7) are respectively formed by connecting two square guide rails (28) in parallel on two parallel aluminum pipes on the chassis aluminum frame, the sliding table of the Z-axis moving platform (5) and the X-axis moving platform (6), the sliding table of the X-axis moving platform (6) and the sliding table of the Y-axis moving platform (7) are connected to form an XYZ three-dimensional moving platform, the braking parts of the XYZ three-dimensional moving platform are respectively provided with SFU-1605 ball screws (22), (23) and (24), the lead is 4mm, the ball screws of the XYZ three-dimensional moving platform are driven by stepping motors (21), (25) and (26), and the ball screws of the XYZ three-dimensional moving platform are connected with an encoder (27) through couplings (29), (30), and the sliding platform is controlled to move in a closed loop;

The surgical end effector (3) is positioned above the CT machine, the angle of the puncture needle insertion effector is required to be adjusted according to a focus path and a puncture needle insertion path which are obtained by CT images, and the optimal needle insertion angle of the surgical end effector is adjusted by utilizing a three-axis holder module consisting of three adjusting units, namely a rolling adjusting unit (8), a yaw adjusting unit (9) and a pitching adjusting unit (10); the device comprises a rolling adjusting unit (8), a yaw adjusting unit (9), a pitching adjusting unit (10) and a pitching adjusting unit, wherein a rotating center is built by adopting a slewing bearing (31), the pitching adjusting unit is fixedly connected with a bearing box (39), a braking part adopts a worm wheel and a worm (32), stepping motors (33) and (37) drive the worm, three adjusting unit sliding tables (34), 35) and 36) are connected with an encoder (40), and the rolling adjusting unit sliding table (34), the yaw adjusting unit sliding table (35) and the pitching adjusting unit sliding table (36) are used for realizing closed-loop control of rotation; the base of the rolling adjusting unit (8) is connected with the Y-axis moving platform, the sliding table of the rolling adjusting unit (8) is connected with the base of the yaw adjusting unit (9), the sliding table of the yaw adjusting unit (9) is connected with the base of the pitch adjusting unit (10) to form a cloud deck of the surgical end effector device, the surgical end effector (3) is connected with the sliding table (34) of the pitch adjusting unit on the cloud deck, and the three-dimensional space posture adjustment of the surgical end effector (3) is realized through the parallel calling of the three adjusting units; the multi-degree-of-freedom mechanical arm aims to realize the full-dimensional position and posture adjustment of the surgical end effector (3) in the surgical area;

The operating principle of the operation positioning navigation system comprises a calibration navigation system, a remote navigation and a near navigation; the calibration navigation system is used for measuring the relative spatial distance between the rotation center of the imaging device and the origin of the coordinate system of the surgical robot and the conversion matrix before operation, measuring the relative spatial distance between the marker of the surgical end effector and the tip of the surgical instrument, and calculating the actual motion direction of each degree of freedom of the mechanical arm; the positioning navigation in the operation adopts a staged navigation method, and the navigation of the movement of the surgical instrument is completed by utilizing the combination of the distal navigation and the proximal navigation: when the tail end of the mechanical arm is far away from the focus, namely the surgical end effector is not in the field of view of the imaging device, enabling remote navigation, only identifying and tracking focus markers, and calculating coordinates of the focus and the puncture path in a surgical robot coordinate system; when the tail end of the mechanical arm is close to a focus, namely the surgical end effector enters the field of view of the imaging device, a near-end navigation module is started, the surgical end effector and a focus marker are identified and tracked simultaneously, the relative spatial position relation between the surgical instrument and the focus is calculated, and the position coordinates of the puncture path are calculated.

2. The integrated surgical positioning and navigation system of claim 1, wherein the surgical positioning and navigation system

The working principle of the system specifically comprises:

the surgical positioning navigation comprises three parts of calibrating a navigation coordinate system, namely a far-end navigation part and a near-end navigation part, wherein before the surgical positioning navigation system and the surgical robot are put into use, firstly, the navigation coordinate system is calibrated (108), the purpose of calibration is to determine the direction of the surgical robot coordinate system, the pitch point coordinate and the rotation direction of the laser radar, and when the XYZ three axes of the multi-degree-of-freedom mechanical arm device (1) are all in an initial position, namely in a zero state, the center point at the top of the Z-axis moving platform is set as the origin of the surgical robot coordinate system (103)The direction of the surgical robot coordinate system is +.>And->Wherein->And->Respectively representing the motion direction of the XYZ three-dimensional moving platform and the rolling, yawing and pitching three-axis cradle head, and the coordinate Q of the pitching rotation point of the laser radar is +.>The laser radar acquires depth information of an object in the field of view and projects it in the imaging system coordinate system (102), calibrated using a fixed position object block C with a marker>Object block mark point C _h In the imaging system coordinate system the coordinates are +.>h is the number of the marking point, h=1, 2, …, G, the coordinates a of the marking point on the surgical end effector a _f Is->f is the serial number of the marking point, f=1, 2, …, E, and the object block is adopted to mark the point C when the XYZ triaxial moving direction is calibrated _h As a reference object, calculating the relative movement direction of the moving uniaxial mark object, namely the axial direction, gradually moving the X axis, the Y axis or the Z axis, and respectively calculating the mark point C of the object block _h Average coordinate difference->Wherein Deltar _ch 、Δθ _ch 、Δφ _ch Marking points C of the object blocks at two moments before and after the movement of the mobile platform are respectively shown _h And converted to rectangular coordinate system to obtain respectivelyAs->When the three-axis movement direction of rolling, pitching and yawing is marked, a marking point A on an operation end effector is adopted _f As a reference object, sequentially rotating a single shaft, calculating a plane in which a mark point track is positioned in a space, wherein three planes represent three directions of motion of a cradle head, measuring to obtain a pitching rotation point coordinate Q of a laser radar, acquiring a lung image (116) of a patient before an operation through a CT machine, and obtaining focus mark object points and focus point cloud positions under a CT coordinate system (101) and a puncture path (107) confirmed by a doctor through image segmentation (105) and three-dimensional reconstruction (106) of the CT image;

during the execution of the operation, the mechanical arm is initialized (112) to finish the self-inspection of the equipment, and the multi-degree-of-freedom mechanical arm device drives the operation end effector to move from far to near to the focus; in the preoperative period, the multi-degree-of-freedom mechanical arm device and the surgical end effector are far away from a focus, only a focus marker can be ensured to be in the field of view of imaging equipment (109), at the moment, a remote navigation (113) is adopted, the imaging equipment is used for identifying a focus marker point set (203) of a patient, the focus marker point set position of the patient is projected to the surgical robot coordinate system through the conversion of the imaging system coordinate system (102) and the surgical robot coordinate system (103), then the point cloud registration (201) is carried out on the focus marker point set under the surgical robot coordinate system and the CT coordinate system, a transformation matrix is obtained, focus points (202) are converted to the surgical robot coordinate system (103), and therefore the navigation on the movement of the mechanical arm is realized; in the process, the laser radar moves by a distance D on the X axis, the distance L between the actual signal receiving end of the laser radar camera and the coordinate Q of the pitching rotation point of the laser radar, the pitching adjustment angle is theta, and at the moment, the actual receiving point coordinate of the laser radar camera is Performing remote navigation, observing a marker point set (203) around a lesion of a patient by using a laser radar, and returning coordinates (204) of the marker point set of the lesion to beu is the focus of infection markObject number, u=1, 2, …, M, P _u Is a point on the imaging system coordinate system with R as the center, and points P on the imaging system coordinate system _u Rectangular coordinate system midpoint converted into R as origin>That is, the coordinates of the focus marker point set under the coordinate system of the surgical robot are R and P _u ^* The sum of the corresponding xyz components isThe coordinates (205) of each marker point obtained from the CT image are +.>For M _u And N _u Registering (201) or (104) the point set;

when the surgical end effector moves to the near end of a lesion of a patient, near-end navigation (114) is performed, and two types of markers (304) and (203) near the surgical end effector and the lesion are simultaneously observed (110) by using a laser radar camera, and coordinate system information of the two types of markers is simultaneously acquired, so that a marker point set A of the surgical end effector is obtained _f (302) Matching focus marker point sets (301) in the imaging system coordinate system and in the imaging system coordinate system after the CT coordinate system is converted into the imaging system coordinate system with the position relation of focus surrounding markers (303), namely point clouds(303) And (3) point cloud->(305) Matching (301) or (104), where N _u ^* Is N _u Projection under an imaging system coordinate system; substituting the optimal rotation matrix and translation matrix with the completed point cloud registration to obtain the position relationship between the focus and the operation end effector, namely completing the focus coordinate B without pitching translation error ₀ Is solved; considering that systematic errors exist in the establishment of end effector coordinates on a surgical end effector due to manual operation, lesions are no longer consideredThe position is restored to a surgical robot coordinate system (103), the relative position relation between the surgical end effector and the focus is measured through a laser radar, the surgical end effector is controlled to move in a three-dimensional space to realize the puncture of the focus, the coordinate system of the instrument tip established by a fixed point T' on the instrument tip is consistent with the coordinate system of the surgical robot in direction, the laser radar always tracks two types of markers in the process of puncturing the surgical end effector into a needle, the method is utilized to realize the implementation of solving and tracking of the focus position coordinates, and the starting point coordinates of a puncture path are calculated, wherein noise exists in the marker point set measurement, the optimal position value is estimated by utilizing a Kalman filtering mode, finally, the vector relation between the surgical end effector and the focus is converted into the motion quantity of each mechanical arm, the mechanical arm is adjusted, the mechanical arm is moved to a needle-entering position (115) is prepared, and the imaging equipment always tracks the two types of markers in the whole needle-entering process, so that the focus coordinates and the starting point coordinates of the puncture path are confirmed and corrected in real time (111).

3. The integrated surgical positioning and navigation system of claim 2, wherein P is _u And N _u ^* Performing point set

The working principle of an implementation method of point cloud registration for registration (301) is as follows: according to two point set data P to be registered _u And N _u ^* Firstly, constructing local geometric features, then, repositioning point cloud data (401) according to the local geometric features, and mainly utilizing an iterative algorithm (402) to perform P _u And N _u ^* Registering the two point set data, and enabling the following objective function to be minimum through the alignment registration conversion of the two point setsThe rotation parameters and translation parameters between the point cloud data to be aligned and the reference point cloud data are found, so that the two point set data meet the optimal matching under a certain measurement criterion (403), wherein S is a rotation matrix, T is a translation matrix, and the expression norm is calculated; in the process of finding the rotation matrix S and translation of the optimal matchAfter matrix T, focus center coordinate V obtained from CT image ₀ Performing rotation and translation operations to obtain lesion coordinates (404) B in the surgical robot coordinate system ₀ ＝V ₀ S+T; after knowing the coordinates of the lesion in the surgical robot coordinate system, the surgical end effector is controlled to move to the lesion, and in addition, the point cloud registration (201) adopts the same method, except that the registration point cloud is a point set M _u And point set N _u (205)。

4. The integrated surgical positioning and navigation system according to claim 3, wherein the imaging device is replaced by a binocular stereoscopic camera or a structured light stereoscopic camera, wherein imaging of the binocular stereoscopic camera is based on parallax principle and uses the camera to shoot a surgical image containing a marker from different positions, the marker pattern is segmented from the image, three-dimensional geometric information of the marker is obtained by calculating position deviation between corresponding marker points of left and right camera images, and spatial positions and attitudes of a lesion, a puncture path and a surgical instrument and spatial positions of the puncture path can be obtained by processing three-dimensional coordinate information of a set of marker points;

in addition, the basic principle of the structured light stereoscopic vision camera is: the method comprises the steps of projecting light rays with certain structural characteristics onto a shot object through projection equipment, shooting the shot object by a camera, establishing a spherical coordinate system with the camera as an origin as an imaging system coordinate system, returning different image phase information according to different depths of the surface of the shot object for the light rays projected onto a target object, and converting the phase change into depth information through calculation, so that three-dimensional position information is obtained; the position of the marker is obtained by dividing an image shot by a camera, and the position information of the divided marker in space is obtained by phase calculation; likewise, by processing the three-dimensional position information of the marker point set, the spatial position and posture of the lesion and the surgical instrument, and the spatial position of the puncture path can be obtained, wherein the photographed object is the chest of the patient and the surgical end effector.