WO2025019373A1

WO2025019373A1 - Autonomous navigation of a continuum robot

Info

Publication number: WO2025019373A1
Application number: PCT/US2024/037924
Authority: WO
Inventors: Franklin King; Fumitaro Masaki; Nobuhiko Hata; Takahisa Kato; Lampros Athanasiou; Brian NINNI
Original assignee: Canon U.S.A., Inc.; The Brigham And Women's Hospital Inc.
Priority date: 2023-07-14
Filing date: 2024-07-12
Publication date: 2025-01-23
Also published as: WO2025019378A1; WO2025019377A1

Abstract

Examples of such autonomous navigation, movement detection, and/or control include, but are not limited to, autonomous navigation of one or more portions of a continuum robot towards a particular target, movement detection of the continuum robot, Follow-The-Leader smoothing, and/or state change(s) for a continuum robot. Examples of applications include imaging, evaluating, and diagnosing biological objects, such as, but not limited to, for gastro-intestinal, cardio, bronchial, and/or ophthalmic applications, and being obtained via one or more optical instruments, such as, but not limited to, optical probes, catheters, endoscopes, and bronchoscopes. Techniques provided herein also improve processing and imaging efficiency while achieving images that are more precise, and also achieve devices, systems, methods, and storage mediums that reduce mental and physical burden and improve ease of use.

Description

Autonomous Navigation of a Continuum Robot

BACKGROUND

Cross-Reference to Related Applications

[0001] This application claims priority to U.S. Patent Application Serial No. 63/513,794, filed July 14, 2023, U.S. Patent Application Serial No. 63/513,803, filed July 14, 2023, U.S. Patent Application Serial No. 63/587,637, filed Oct. 3, 2023, and to U.S. Patent Application Serial No. 63/603,523 filed Nov. 28, 2023, the disclosure of each of which are incorporated by reference herein in their entirety.

Field of the Disclosure

[0002] The present disclosure generally relates to a continuum robot system and more particularly to a continuum robot that is a steerable catheter than can be navigated autonomously as well as methods and mediums for autonomous navigation.

Description of the Related Art

[0003] Endoscopy, bronchoscopy, catheterization, and other medical procedures facilitate the ability to look inside a body. During such a procedure, a flexible medical tool may be inserted into a patient’s body, and an instrument may be passed through the tool to examine or treat an area inside the body. For example, a bronchoscope is an endoscopic instrument to view inside the airways of a patient. Catheters and other medical tools may be inserted through a tool channel in the bronchoscope to provide a pathway to a target area in the patient for diagnosis, planning, medical procedure(s), treatment, etc.

[0004] Robotic bronchoscopes, robotic endoscopes, or other robotic imaging devices may be equipped with a tool channel or a camera and biopsy tools, and such devices (or users of such devices) may insert/ retract the camera and biopsy tools to exchange such components. The robotic bronchoscopes, endoscopes, or other imaging devices may be used in association with a display system and a control system.

[0005] An imaging device, such as a camera, may be placed in the bronchoscope, the endoscope, or other imaging device/system to capture images inside the patient and to help control and move the bronchoscope, the endoscope, or the other type of imaging device, and a display or monitor may be used to view the captured images. An endoscopic camera that may be used for control may be positioned at a distal part of a catheter or probe (e.g., at a tip section).

[0006] The display system may display, on the monitor, an image or images captured by the camera, and the display system may have a display coordinate used for displaying the captured image or images. In addition, the control system may control a moving direction of the tool channel or the camera. For example, the tool channel or the camera may be bent according to a control by the control system. The control system may have an operational controller (such as, but not limited to, a joystick, a gamepad, a controller, an input device, etc.), and physicians may rotate or otherwise move the camera, probe, catheter, etc. to control same. However, such control methods or systems are limited in effectiveness. Indeed, while information obtained from an endoscopic camera at a distal end or tip section may help decide which way to move the distal end or tip section, such information does not provide details on how the other bending sections or portions of the bronchoscope, endoscope, or other type of imaging device may move to best assist the navigation.

[0007] At least one application is looking inside the body relates to lung cancer, which is the most common cause of cancer-related deaths in the United States. It is also a commonly diagnosed malignancy, second only to breast cancer in women and prostate cancer in men. Early diagnosis of lung cancer is shown to improve patient outcomes, particularly in peripheral pulmonary nodules (PPNs). During a procedure, such as a transbronchial biopsy, targeting lung lesions or nodules may be challenging. Lately, Electromagnetically Navigated Bronchoscopy (ENB) is increasingly applied in the transbronchial biopsy of PPNs due to its excellent safety profile, with fewer pneumothoraxes, chest tubes, significant hemorrhage episodes, and respiratory failure episodes than a CT-guided biopsy strategy (see e.g., as discussed in C. R. Dalek, et al., J Bronchology Interv Pulmonol, vol. 19, no. 4, pp. 294-303, Oct. 2012, doi: 10.1097/LBR.0B013E318272157D, which is incorporated by reference herein in its entirety). However, ENB has lower diagnostic accuracy or value due to dynamic deformation of the tracheobronchial tree by bronchoscope maneuvers (see e.g., as discussed in T. Whelan, et al., International Journal of Robotics Research, vol. 35, no. 14, pp. 1697-1716, Dec. 2016, doi: 10.1177/0278364916669237, which is incorporated by reference herein in its entirety) and module motion due to the breathing motion of the lung (see e.g., as discussed in A. Chen, et al., Chest, vol. 147, no. 5, pp. 1275-1281, May 2015, doi: 10.1378/CHEST.14-1425, which is incorporated by reference herein in its entirety). Robotic-assisted biopsy has emerged as a minimally invasive and precise approach for obtaining tissue samples from suspicious pulmonary lesions in lung cancer diagnosis. However, the reliance on human operators to guide a robotic system introduces potential variability in sampling accuracy and operator-dependent outcomes. Such operators may introduce human error, reduce efficiency of using a robotic system, have a steeper learning curve to using a robotic system, and affect surgeries as a result.

[0008] Vision-based tracking (VNB), as opposed to ENB, has been proposed to address the aforementioned issue of CT-to-body divergence (see e.g., as discussed in D. J. Mirota, et al., Annu Rev Boeng, vol. 13, pp. 297-319, Jul. 2011, doi: 10.1146/ANNUREV- BIOENG-071910-124757, which is incorporated by reference herein in its entirety). Visionbased tracking in VNB does not require an electromagnetic tracking sensor to localize the bronchoscope in CT; rather, VNB directly localizes the bronchoscope using the camera view, conceptually removing the chance of CT-to-body divergence. [0009] Depth estimation was proposed as an alternative method of VNB to further reduce the CT-to-body divergence and overcome the intensity-based image registration drawbacks (see e.g., as discussed in M. Shen, S. et al., Int J Comput Assist Radiol Surg, vol. to, no. 6, pp. 801-813, Jun. 2015, doi: 10.1007/S11548-015-1197-Y, which is incorporated by reference herein in its entirety).

[0010] Alternatively, autonomous navigation in robotic guided bronchoscopy is a relative new concept. Sganga et al. (as discussed in J. Sganga, et al., “Autonomous Driving in the Lung using Deep Learning for Localization,” Jul. 2019, Accessed: Jun. 28, 2023. [Online]. Available: https://arxiv.org/abs/1907.08136vi, which is incorporated by reference herein in its entirety) proposed the first attempt to autonomously navigate through the lung airways havi ng as primary focus to improve the intraoperative registration between CT and live images and then attempt to autonomously navigate to the target. The Sganga, et al. method was limited to only 4 airways. However, the Sganga, et al. method requires a great co-registration between the live image and the pre-operative CT scan which can be detrimentally affected by the same drawbacks as VNB.

[0011] Other efforts have been made not to autonomously navigate through the airways but to automatically control the catheter tensioning system. Jaeger, et al. (as discussed in H. A. Jaeger et al., IEEE Trans Biomed Eng, vol. 64, no. 8, pp. 1972-1979, Aug. 2017, doi: 10.1109/TBME.2016.2623383, which is incorporated by reference herein in its entirety) proposed such a method where Jaeger, et al. incorporated a custom tendon-driven catheter design with Electro-magnetic (EM) sensors controlled with an electromechanical drive train. However, the system needed heavy user interaction as the clinician uses a computer interfaced joystick to manipulate the catheter. A semi-automatic navigation of the biopsy needle during bronchoscopy was proposed by Kuntz, et al. (as discussed in A. Kuntz et al., “Autonomous Medical Needle Steering In Vivo,” Nov. 2022, Accessed: Jun. 28, 2023. [Online]. Available: https://arxiv.org/abs/2211.02597v i, which is incorporated by reference herein in its entirety). The method uses pre-operative CT scans (3D) and EMC sensors to co-register cloud points of the nodule and live guidance. Nevertheless, the method cannot be used to navigate the bronchoscopic catheter into the airways, which is a critical step for reaching the nodules. Moreover, similar methods (as discussed in S. Chen, et al., Int J Comput Assist Radiol Surg, vol. 17, no. 2, pp. 295-303, Feb. 2022, doi: 10.1007/S11548-021-02519-6/FIGURES/8, which is incorporated by reference herein in its entirety) allow automatic localization of the needle in real-time w ithout the need of an automatic needle navigation.

[0012] As such, there is a need for devices, systems, methods, and/or storage mediums that provide the feature(s) or details on how the other bending sections or portions of such imaging devices, imaging systems, etc. (e.g., endoscopic devices, bronchoscopes, other types of imaging devices/systems, etc.) may move to best assist navigation and/or state or state(s) for same, to keep track of a path of a tip of the imaging devices, imaging systems, etc., and there is a need for a more appropriate navigation of a device (such as, but not limited to, a bronchoscopic catheter being navigated to reach a nodule).

[0013] Accordingly, it would be desirable to provide at least one imaging, optical, or control device, system, method, and storage medium for controlling one or more endoscopic or imaging devices or systems, for example, by implementing automatic (e.g., robotic) or manual control of each portion or section of the at least one imaging, optical, or control device, system, method, and storage medium to keep track of and to match the state or state(s) of a first portion or section in a case where each portion or section reaches or approaches a same or similar, or approximately same or similar, state or state(s) and to provide a more appropriate navigation of a device (such as, but not limited to, a bronchoscopic catheter being navigated to reach a nodule).

SUMMARY [0014] To solve the problems discussed above, an autonomous navigation robot including 1) perception step, 2) planning step, 3) control step is described. For the planning step, a method and system is provided user-interface for user to instruct commands, and reflects the commands to a plan. Since the level of control for a user as compared to the more automatic navigation of the continuum robot, is not always clear, it can be counterintuitive for users to instruct the system for the intended autonomously navigated route or make any changes or modifications effectively within the autonomous navigation. Thus, there is provided a system and method that determines the target paths among the various paths based on user instruction, pre-operative instructions, and the autonomous system.

[0015] For the control step, combining the information from the perception step and the planning step and defining the criteria to decide when to move forward into the lumen and when to continue to bend or optimize tip direction for future movement is described.

[0016] Accordingly, it is a broad object of the present disclosure an autonomous navigation robot system having a continuum robot, a camera at the distal end of the continuum robot, one or more actuators to steer and move the continuum robot, (or alternatively actuators for bending motions and one or more motors for linear motion), and a controller. The controller is configured to perform three steps: 1) perception step, 2) planning step, and 3) control step, where each of these three steps may be performed using the images from the camera and without the need for registering the continuum robot w ith an external image. The continuum robot may be a steerable catheter, such as a bronchoscope.

[0017] It is also an object of the present disclosure to provide an autonomous navigation robot system, comprising: a steerable catheter; a camera at the distal end of the steerable catheter; one or more actuators to steer and move the steerable catheter; a user input device, a display; and a controller. The controller is configured to: detect one or more lumens, show, on the display, the detected one or more lumen and/or an indicator thereof, select a target path by inputting instructions as to the target path, and show, on the display, information of the selected target.

[0018] It is a further object of the present disclosure to provide an autonomous navigation robot system, comprising: a continuum robot; a camera at the distal end of the continuum robot; one or more actuators and/or motors to bend the distal end of the continuum robot and to move the continuum robot forward; and a controller. The controller is configured to: receive an image from the camera, define a position point in the image; determine a target point in the image based on a target path; command the one or more actuators and/or motors. If the distance between the position point and the target point is less than a threshold value, the command is to move the continuum robot forward, and if the distance between the position point and the target point is more than the threshold value, the command is to bend the distal end of the continuum robot tow ards the target point.

[0019] In some embodiments, the position point is the center of the image received from the camera and/or the target point is a center of a circle that indicates a lumen as the target path. The lumen may be in an airway. The autonomous navigation robot system may be designed to repeat the process steps until a target is reached or until the user stops the automated process. Thus, the controller may receive additional image(a), defining a position point, and commanding the one or more actuators until a predetermined insertion depth is reached. A predetermined insertion depth may be used to define the end point for the automated process. Thus, the controller may determine whether the continuum robot has reached a predetermined insertion depth, and stop the movement and/or bending when the predetermined insertion depth is reached.

[0020] The threshold value may be adjustable (e.g., between 10 - 50 percent of the diagonal length of the image or between 20 and 40 percent of the diagonal length of the image, or between 25 and 35 of the diagonal length of the image), and may be adjusted to require increasingly accurate bending before moving forward as the continuum robot progresses through a lumen. The speed of bending and/or the speed of forward movement and/or the frame rate of images from the camera may be adjustable. A user input device, that, when activated, stops the controller from moving or bending towards the target point without further user input may be provided.

[0021] It is a further object of the present disclosure to provide an information processing apparatus to control a continuum robot with at least one memory storing instructions; and at least one processor that executes the instructions stored in the memory. The instruction cause the information processing apparatus to perform: receiving an image, determining a target point in the image based on a target path; determining whether or not a distance from the position point to the target point in the image is more or less than a threshold value. In n a case where the distance is less than the threshold value, the processor controls the continuum robot to advance, and in a case where the distance is more than the threshold value, the processor controls the continuum robot to bend so that the distance become less.

[0022] It is yet another object of the present disclosure to provide a non-transitory computer-readable storage medium storing at least one program for causing a computer to execute a method for controlling a continuum robot, the method comprising: receiving an image, determining a target point in the image based on a target path; determining whether or not a distance from the position point to the target point in the image is more or less than a threshold value. In a case where the distance is less than the threshold value, the continuum robot is caused to advance, and in a case where the distance is more than the threshold value, the processor controls the continuum robot to bend so that the distance become less.

[0023] In some embodiments, tw o or three of the perception, planning, and control steps are performed, or there is provided an autonomous navigation robot system configured to perform two or three of the perception, planning, and control steps. The autonomous navigation robot system of these embodiments comprises a steerable catheter; a camera at the distal end of the steerable catheter; one or more actuators to steer and move the steerable catheter; a user input device; and a controller. The controller is configured to:

• in a perception step: receive camera view; identify path candidates in the camera view by processing the camera view; and determine paths among path candidates with computation,

• in a planning step: determine target paths among the path candidates based on concurrent user instruction from user input device and/or pre-operative instruction,

• in a control step: compute the commands to actuator based on the target paths and the camera view and/or the current posture of the steerable catheter and command the actuator to move the steerable catheter, wherein the actuators bend and move the steerable catheter automatically.

[0024] Further features of the present disclosure will become apparent from the follow ing description of exemplary embodiments with reference to the attached raw ings, w here like structure is indicated with like reference numerals.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] For the purposes of illustrating various aspects of the disclosure, wherein like numerals indicate like elements, there are show n in the drawings simplified forms that may be employed, it being understood, however, that the disclosure is not limited by or to the precise arrangements and instrumentalities shown. To assist those of ordinary skill in the relevant art in making and using the subject matter hereof, reference is made to the appended drawings and figures.

[0026] FIG. 1 illustrates at least one embodiment of an imaging, continuum robot, or endoscopic apparatus or system in accordance with one or more aspects of the present disclosure.

[0027] FIG. 2 is a schematic diagram showing at least one embodiment of an imaging, steerable catheter, or continuum robot apparatus or system in accordance with one or more aspects of the present disclosure;

[0028] FIG. 3(a) illustrate at least one embodiment example of a continuum robot and/or medical device that may be used with one or more technique(s), including autonomous navigation technique(s), in accordance with one or more aspects of the present disclosure. Detail A illustrates one guide ring of the steerable catheter.

[0029] FIGS. 3(b) - 3(c) illustrate one or more principles of catheter or continuum robot tip manipulation by actuating one or more bending segments of a continuum robot or steerable catheter 104 of FIG. 3(a) in accordance with one or more aspects of the present disclosure.

[0030] FIG. 4 is a schematic diagram showing at least one embodiment of an imaging, continuum robot, steerable catheter, or endoscopic apparatus or system in accordance with one or more aspects of the present disclosure.

[0031] FIG. 5 is a schematic diagram showing at least one embodiment of a console or computer that may be used w ith one or more autonomous navigation technique(s) in accordance w ith one or more aspects of the present disclosure.

[0032] FIG. 6 is a flowchart of at least one embodiment of a method for planning an operation of at least one embodiment of a continuum robot or steerable catheter apparatus or system in accordance with one or more aspects of the present disclosure. [0033] FIG. 7 is a flowchart of at least one embodiment of a method for performing autonomous navigation, movement detection, and/or control for a continuum robot or steerable catheter in accordance with one or more aspects of the present disclosure.

[0034] FIG. 8(a) shows images of at least one embodiment of an application example of autonomous navigation technique(s) and movement detection for a camera view (left), a depth map (center), and a thresholded image (right) in accordance w ith one or more aspects of the present disclosure.

[0035] FIG. 8(b) shows images of one embodiment showing a camera view’ (left), a semi-transparent color coded depth map overlaid onto a camera view (center) and a thresholded image (right).

[0036] FIG. 9 is an exemplary image having two airways and indicators of circle fit and target path in accordance with one or more aspects of the present disclosure.

[0037] FIG. 10 is an exemplary image having two airways and indicators of circle fit and target path in accordance with one or more aspects of the present disclosure.

[0038] FIG. 11 is a diagram showing two lumens and the threshold.

[0039] FIG. 12 is a flow chart of at least one embodiment of a method for controlling the steerable catheter in accordance with one or more aspects of the present disclosure.

[0040] FIG. 13 is a flowchart of at least one embodiment of a method for controlling the steerable catheter, including speed setting, in accordance w ith one or more aspects of the present disclosure.

[0041] FIG. 14 is a diagram indicating bending speed and moving speed in accordance with one or more aspects of the present disclosure.

[0042] FIG. 15 is a flowchart of at least one embodiment of a method for controlling the steerable catheter including setting the threshold, in accordance with one or more aspects of the present disclosure.

[0043] FIG. 16 is a diagram of an airw ay with an indication of two thresholds, in accordance with one or more aspects of the present disclosure. [0044] FIG. 17 is a flowchart of at least one embodiment of a method for controlling the steerable catheter, including blood detection, in accordance with one or more aspects of the present disclosure.

[0045] FIG. 18 shows at least one embodiment a control software or a User Interface that may be used with one or more robots, robotic catheters, robotic bronchoscopes, methods, and/or other features in accordance with one or more aspects of the present disclosure.

[0046] FIGS. 19(a) - 19(b) illustrate at least one embodiment of a bronchoscopic image with detected airways and an estimated depth map (or depth estimation) with or using detected airways, respectively, in one or more bronchoscopic images in accordance with one or more aspects of the present disclosure.

[0047] FIGS. 20(a) - 20(b) illustrate at least one embodiment of a pipeline that may be used for a bronchoscope, apparatus, device, or system (or used with one or more methods or storage mediums), and a related camera view employing voice recognition, respectively, of the present disclosure in accordance with one or more aspects of the present disclosure.

[0048] FIGS. 21(a) - 21(c) illustrate a navigation screen for a clinical target location in or at a lesion reached by autonomous driving, a robotic bronchoscope in a phantom having reached the location corresponding to the location of the lesion in an ex vivo setup, and breathing cycle information (FIG. 21(c) using EM sensors, respectively, in accordance with one or more aspects of the present disclosure.

[0049] FIGS. 22(a) - 22(c) illustrate views of at least one embodiment of a navigation algorithm performing at various branching points in a phantom where FIG. 22(a) shows a path on w hich the target location (dot) was not reached (e.g., the algorithm may not have traversed the last bifurcation where an airway on the right was not detected), where FIG. 22(b) shows a path on which the target location (dot) was successfully reached, and where FIG. 22(c) shows a path on which the target location was also successful reached in accordance with one or more aspects of the present disclosure.

[0050] FIGS. 23(a) - 23(b) illustrate graphs showing success at branching point(s) with respect to Local Curvature (LC) and Plane Rotation (PR), respectively, for all data combined in one or more embodiments in accordance with one or more aspects of the present disclosure.

[0051] FIGS. 24(a) - 24(c) illustrate one or more impacts of breathing motion on a performance of the one or more navigation algorithm(s) w here FIG. 24(a) show s a path on which the target location (ex vivo #1 LLL) was reached with and w ithout breathing motion (BM), w here FIG. 24(b) show s a path on which the target location (ex vivo #1 RLL) was not reached without BM but was reached with BM, and where FIG. 24(c) show-s a path on which the target location (ex vivo #1 RML) was reached without BM was not reached with BM in accordance with one or more aspects of the present disclosure.

[0052] FIGS. 25(a) - 25(b) illustrate the box plots for time for the operator or the autonomous navigation to bend the robotic catheter in one or more embodiments and for the maximum force for the operator or the autonomous navigation at each bifurcation point in one or more embodiments in accordance with one or more aspects of the present disclosure.

[0053] FIGS. 26(a) - 26(d) illustrate one or more examples of depth estimation failure and artifact robustness that may be observed in one or more embodiments in accordance with one or more aspects of the present disclosure.

[0054] FIGS. 27(a) - 27(b) illustrate graphs for the dependency of the time for a bending command and the force at each bifurcation point, respectively, on the airway generation of a lung in accordance with one or more aspects of the present disclosure.

DESCRIPTION OF THE EMBODIMENTS [0055] Various exemplary embodiments, features, and aspects of the disclosure will be described below with reference to the drawings.

< Rob otic Catheter System >

[0056] An embodiment of a robotic catheter system too is described in reference to FIG. 1 through FIG. 4. FIG. 1 illustrates a simplified representation of a medical environment, such as an operating room, where a robotic catheter system too can be used. FIG. 2 illustrates a functional block diagram of the robotic catheter system too. FIGS. 3(a) - 3(c) represents the catheter and bending. FIGS. 4 - 5 illustrates a logical block diagram of the robotic catheter system too. In this example, the system too includes a system console 102 (computer cart) operatively connected to a steerable catheter 104 via a robotic platform 106. The robotic platform 106 includes one or more than one robotic arm 108 and a linear translation stage 110.

[0057] In FIG. 1, A user 112 (e.g., a physician) controls the robotic catheter system too via a user interface unit (operation unit) to perform an intraluminal procedure on a patient 114 positioned on an operating table 116. The user interface may include at least one of a main display 118 (a first user interface unit), a secondary display 120 (a second user interface unit), and a handheld controller 124 (a third user interface unit). The main display 118 may include, for example, a large display screen attached to the system console 102 or mounted on a wall of the operating room and may be, for example, designed as part of the robotic catheter system too or be part of the operating room equipment. Optionally, there is a secondary display 120 that is a compact (portable) display deuce configured to be removably attached to the robotic platform 106. Examples of the secondary display 120 include a portable tablet computer or a mobile communication device (a cellphone).

[0058] The steerable catheter 104 is actuated via an actuator unit 122. The actuator unit 103 is removably attached to the linear translation stage 110 of the robotic platform 106. The handheld controller 124 may include a gamepad-like controller with a joystick having shift levers and/or push buttons. It may be a one-handed controller or a two- handed controller. In one embodiment, the actuator unit 122 is enclosed in a housing having a shape of a catheter handle. One or more access ports 126 are provided in or around the catheter handle. The access port 126 is used for inserting and/or withdrawing end effector tools and/or fluids when performing an interventional procedure of the patient 114. [0059] The system console 102 includes a system controller 128, a display controller

130, and the main display 118. The main display 118 may include a conventional display device such as a liquid crystal display (LCD), an OLED display, a QLED display or the like. The main display 118 provides a graphic interface unit (GUI) configured to display one or more views. These views include live view image 132, an intraoperative image 134, and a preoperative image 136, and other procedural information 138. Other views that may be displayed include a model view, a navigational information view, and/or a composite view. The live image view 132 may be an image from a camera at the tip of the catheter. This view may also include, for example, information about the perception and navigation of the catheter 104. The preoperative image 136 may include pre-acquired 3D or 2D medical images of the patient acquired by conventional imaging modalities such as computer tomography (CT), magnetic resonance imaging (MRI), or ultrasound imaging. The intraoperative image 134 may include images used for image guided procedure such images may be acquired by fluoroscopy or CT imaging modalities. Intraoperative image 134 may be augmented, combined, or correlated with information obtained from a sensor, camera image, or catheter data.

[0060] In the various embodiments where a catheter tip tracking sensor 140 is used, the sensor may be located at the distal end of the catheter. The catheter tip tracking sensor 140 may be, for example, an electromagnetic (EM) sensor. If an EM sensor is used, a catheter tip position detector 142 is included in the robotic catheter system too; this catheter tip position detector would include an EM field generator operatively connected to the system controller 128. Suitable electromagnetic sensors for use with a steerable catheter are well-known and described, for example, in U.S. Pat. No.: 6,201,387 and international publication W02020194212A1.

[0061] Similar to FIG. 1, the diagram of FIG. 2 illustrates the robotic catheter system too includes the system controller 128 operatively connected to the display controller 130, which is connected to the display unit 118, and to the hand held control 124. The system controller 128 is also connected to the actuator unit 122 via the robotic platform 106, which includes the linear translation stage 110. The actuator unit 122 includes a plurality of motors 144 that control the plurality of drive wires 160. These drive wires travel through the steerable catheter 104. One or more access ports 126 may be located on the catheter. The catheter includes a proximal section 148 located between the actuator and the proximal bending section 152 where they actuate the proximal bending section. Three of the six drive wires 160 continue through the distal bending section 156 where they actuate this section and allow for a range of movement. This figure is shown with two bendable sections (152 and 156). Other embodiments as described herein can have three bendable sections (see FIG. 3). In some embodiments, a single bending section may be provided, or alternatively, four or more bendable sections may be present in the catheter.

[0062] FIG. 3A shows an exemplary embodiment of a steerable catheter 104. The steerable catheter 104 includes a non-steerable proximal section 148, a steerable distal section 150, and a catheter tip 158. The proximal section 148 and distal bendable section 150 (including 152, 154 and 156) are joined to each other by a plurality of drive wires 160 arranged along the wall of the catheter. The proximal section 148 is configured w ith thru- holes or grooves or conduits to pass drive wires 160 from the distal section 150 to the actuator unit 122. The distal section 150 is comprised of a plurality of bending segments including at least a distal segment 156, a middle segment 154, and a proximal segment 152. Each bending segment is bent by actuation of at least some of the plurality of drive wires 160 (driving members). The posture of the catheter may be supported by non-illustrated supporting wires (support members) also arranged along the wall of the catheter (see U.S. Pat. Pub. US2021/0308423). The proximal ends of drive wires 160 are connected to individual actuators or motors 144 of the actuator unit 122, while the distal ends of the drive wares 160 are selectively anchored to anchor members in the different bending segments of the distal bendable section 150.

[0063] Each bending segment is formed by a plurality of ring-shaped components (rings) with thru-holes, grooves, or conduits along the wall of the rings. The ring-shaped components are defined as wire-guiding members 162 or anchor members 164 depending on their function within the catheter. Anchor members 164 are ring-shaped components onto which the distal end of one or more drive wires 160 are attached. Wire-guiding members 162 are ring-shaped components through which some drive wires 160 slide through (without being attached thereto).

[0064] Detail “A” in FIG. 3(a) illustrates an exemplary embodiment of a ringshaped component (a wire-guiding member 162 or an anchor member 164). Each ringshaped component includes a central opening which forms the tool channel 168, and plural conduits 166 (grooves, sub-channels, or thru-holes) arranged lengthwise equidistant from the central opening along the annular wall of each ring-shaped component. Inside the ringshaped component, an inner cover such as is described in U.S. Pat. Pub US2021/0369085 and US2022/0126060, may be included to provide a smooth inner channel and provide protection. The non-steerable proximal section 148 is a flexible tubular shaft and can be made of extruded polymer material. The tubular shaft of the proximal section 148 also has a central opening or tool channel 168 and plural conduits 166 along the wall of the shaft surrounding the tool channel 168. An outer sheath may cover the tubular shaft and the steerable section 150. In this manner, at least one tool channel 168 formed inside the steerable catheter 104 provides passage for an imaging device and/or end effector tools from the insertion port 126 to the distal end of the steerable catheter 104.

[0065] The actuator unit 122 includes one or more servo motors or piezoelectric actuators. The actuator unit 122 bends one or more of the bending segments of the catheter by applying a pushing and/or pulling force to the drive wires 160. As shown in FIG. 3(a), each of the three bendable segments of the steerable catheter 104 has a plurality of drive wires 160. If each bendable segment is actuated by three drive wires 160, the steerable catheter 104 has nine driving wires arranged along the wall of the catheter. Each bendable segment of the catheter is bent by the actuator unit 122 by pushing or pulling at least one of these nine drive wires 160. Force is applied to each individual drive wire in order to manipulate/steer the catheter to a desired pose. The actuator unit 122 assembled with steerable catheter 104 is mounted on the linear translation stage 110. Linear translation stage 110 includes a slider and a linear motor. In other words, the linear translation stage 110 is motorized, and can be controlled by the system controller 128 to insert and remove the steerable catheter 104 to/from the patient’s bodily lumen.

[0066] An imaging device 170 that can be inserted through the tool channel 168 includes an endoscope camera (videoscope) along with illumination optics (e.g., optical fibers or LEDs). The illumination optics provides light to irradiate the lumen and/or a lesion target which is a region of interest within the patient. End effector tools refer endoscopic surgical tools including clamps, graspers, scissors, staplers, ablation or biopsy needles, and other similar tools, which serve to manipulate body parts (organs or tumorous tissue) during examination or surgery. The imaging device 170 may be w hat is commonly known as a chip-on-tip camera and may be color or black-and-white.

[0067] In some embodiments, a tracking sensor 140 (e.g., an EM tracking sensor) is attached to the catheter tip 158. In this embodiment, steerable catheter 104 and the tracking sensor 140 can be tracked by the tip position detector 142. Specifically, the tip position detector 142 detects a position of the tracking sensor 140, and outputs the detected positional information to the system controller 100. The system controller 128, receives the positional information from the tip position detector 142, and continuously records and displays the position of the steerable catheter 104 with respect to the patient’s coordinate system. The system controller 128 controls the actuator unit 122 and the linear translation stage no in accordance with the manipulation commands input by the user 112 via one or more of the user interface units (the handheld controller 124, a GUI at the main display 118 or touchscreen buttons at the secondary display 120).

[0068] FIG. 3(b) and FIG. 3(c) show exemplary⁷ catheter tip manipulations by actuating one or more bending segments of the steerable catheter 104. As illustrated in FIG. 3(B), manipulating only the most distal segment 156 of the steerable section changes the position and orientation of the catheter tip 158. On the other hand, manipulating one or more bending segments (152 or 154) other than the most distal segment affects only the position of catheter tip 158, but does not affect the orientation of the catheter tip. In FIG. 3(b), actuation of distal segment 155 changes the catheter tip from a position Pi having orientation 01, to a position P2 hay ing orientation 02, to position P3 having orientation O3, to position P4 having orientation O4, etc. In FIG. 3(c), actuation of the middle segment 154 changes the position of catheter tip 158 from a position Pl having orientation 01 to a position P2 and position P3 having the same orientation 01. Here, it should be appreciated by those skilled in the art that exemplary catheter tip manipulations shown in FIG. 3(b) and FIG. 3(c) can be performed during catheter navigation (i.e., while inserting the catheter through tortuous anatomies). In the present disclosure, the exemplary catheter tip manipulations shown in FIG.3(b) and FIG.3(c) apply namely to the targeting mode applied after the catheter tip has been navigated to a predetermined distance (a targeting distance) from the target.

[0069] FIG. 4 illustrates the system controller 128 executes software programs and controls the display controller 130 to display a navigation screen (e.g., a live view image 132) on the main display 118 and/or the secondary display 120. The display controller 130 may include a graphics processing unit (GPU) or a video display controller (VDC).

[0070] FIG. 5 illustrates components of the system controller 128 and/or the display controller 130. The system controller 128 and the display controller 130 can be configured separately. Alternatively, the system controller 128 and the display controller 102 can be configured as one device. In either case, the system controller 128 and the display controller 130 comprise substantially the same components. The system controller 128 may be a computer, where the computer or other system may also include a database and/or another type of memory⁷ as well as one or more input devices (e.g., a mouse, a keyboard, a speaker etc.,) that may be connected through an operations interface and/or output devices. The system controller 128 may comprise a processor. Specifically, the system controller 128 and display controller 130 may include a central processing unit (CPU 182) comprised of one or more processors (microprocessors), a random access memory (RAM 184) module, an input/output (I/O 186) interface, a read only memory (ROM 180), and data storage memory (e.g., a hard disk drive (HDD 188) or solid state drive (SSD). The system controller 128 or computer may also include a GPU, a solid state drive (SSD), an operational interface and/or a networking interface.

[0071] The ROM 180 and/or HDD 188 store the operating system (OS) software, and software programs necessary for executing the functions of the robotic catheter system 100 as a whole. The RAM 184 is used as a workspace memory. The CPU 182 executes the software programs developed in the RAM 184. The I/O 186 inputs, for example, positional information to the display controller 130, and outputs information for displaying the navigation screen to the one or more displays (main display 118 and/or secondary display 120). In the embodiments descried below, the navigation screen is a graphical user interface (GUI) generated by a software program but, it may also be generated by firmware, or a combination of software and firmware.

[0072] The system controller 128 may control the steerable catheter 104 based on any known kinematic algorithms applicable to continuum or snake-like catheter robots. For example, the system controller controls the steerable catheter 104 based on an algorithm known as follow the leader (FTL) algorithm. By applying the FTL algorithm, the most distal segment 156 of the steerable section 150 is actively controlled with forward kinematic values, while the middle segment 154 and the proximal segment 152 (following sections) of the steerable catheter 104 move at a first position in the same way as the distal section moved at the first position or a second position near the first position.

[0073] The display controller 130 acquires position information of the steerable catheter 104 from system controller 102. Alternatively, the display controller 130 may acquire the position information directly from the tip position detector 142. The steerable catheter 104 may be a single-use or limited-use catheter device. In other words, the steerable catheter 104 can be attachable to, and detachable from, the actuator unit 122 to be disposable.

[0074] During a procedure, the display controller 130 can generate and outputs a live- view image or other view(s) or a navigation screen to the main display 118 and/or the secondary display 120. This view can optionally be registered with a 3D model of a patient’s anatomy (a branching structure) and the position information of at least a portion of the catheter (e.g., position of the catheter tip 158) by executing pre-programmed software routines. Upon completing navigation to a desired target, one or more end effector tools can be inserted through the access port 126 at the proximal end of the catheter, and such tools can be guided through the tool channel 168 of the catheter body to perform an intraluminal procedure from the distal end of the catheter.

[0075] The tool may be a medical tool such as an endoscope camera, forceps, a needle or other biopsy or ablation tools. In one embodiment, the tool may be described as an operation tool or working tool. The working tool is inserted or removed through the working tool access port 126. In the embodiments below, an embodiment of using a steerable catheter to guide a tool to a target is explained. The tool may include an endoscope camera or an end effector tool, which can be guided through a steerable catheter under the same principles. In a procedure there is usually a planning procedure, a registration procedure, a targeting procedure, and an operation procedure.

< Autonomous navigation function > [0076] The system controller 128 includes an autonomous navigation mode. During the autonomous navigation mode, the user do not need to control the bending and translational insertion position of steerable catheter 104. The autonomous navigation mode comprises 1) perception step, 2) planning step and 3) control step. In the perception step, system controller 128 receives endoscope view and analyses the endoscope view to find addressable airways from the current position/orientation of steerable catheter 104. At end of this analysis, the system controller 128 percepts these addressable airways as paths in the endoscope view.

[0077] The autonomous navigation mode can use a novel supervised-autonomous driving approach(es) that integrate a novel depth-based airway tracking method(s) and a robotic bronchoscope. The present disclosure provides extensively developed and validated autonomous navigation approaches for both advancing and centering continuum robots, such as, but not limited to, for robotic bronchoscopy. The inventors represent, to the best of the inventors’ knowledge, that the feature(s) of the present disclosure provide the initial autonomous navigation technique(s) applicable in continuum robots, bronchoscopy, etc. that require no retraining and have undergone full validation in vitro, ex vivo, and in vivo. For example, one or more features of the present disclosure incorporate unsupervised depth estimation from an image (e.g., a bronchoscopic image), coupled with a continuum robot (e.g., a robotic bronchoscope), and functions without any a priori knowledge of the patient’s anatomy, which is a significant advancement. Rooted in the detection of airways within the estimated depth map (e.g., an estimated bronchoscopic depth map), one or more methods of the present disclosure constitutes and provides one or more foundational perception algorithms guiding the movements of the robot, continuum robot, or robotic bronchoscope. By simultaneously handling the tasks of advancing and centering the robot, probe, catheter, robotic bronchoscope, etc. in a target (e.g., in a lung or airw ay), the method(s) of the present disclosure may assist physicians in concentrating on the clinical decision-making to reach the target, which achieves or provides enhancements to the efficacy of such imaging, bronchoscopy, etc.

[0078] One or more devices, systems, methods, and storage mediums for performing control or navigation, including of a multi-section continuum robot and/or for viewing, imaging, and/ or characterizing tissue and/ or lesions, or an object or sample, using one or more imaging techniques (e.g., robotic bronchoscope imaging, bronchoscope imaging, etc.) or modalities (such as, but not limited to, computed tomography (CT), Magnetic Resonance Imaging (MRI), any other techniques or modalities used in imaging (e.g., Optical Coherence Tomography (OCT), Near infrared fluorescence (NIRF), Near etc.) are disclosed herein. Several embodiments of the present disclosure, which may be carried out by the one or more embodiments of an apparatus, system, method, and/or computer- readable storage medium of the present disclosure are described diagrammatically and visually in the figures included herewith.

[0079] The planning step is a step to determine a target path, which is the destination for the steerable catheter 104. While there are a couple of different approaches to select one of the paths as the target path, this invention uniquely include means to reflect user instructions concurrently for the decision of target path among the precepted paths. Once the system determines the target paths with this concurrent user instructions, the target path is sent to the next step, a control step.

[0080] The control step is a step to control the steerable catheter 104 and linear translation stage 110 to navigate the steerable catheter 104 to the target path. This step is also an automatic step. The system controller 128 uses an information relating to the real time endoscope view, the target path and an internal design & status information on the robotic catheter system 100.

[0081] Through these three steps, the robotic catheter system too can navigate steerable catheter 104 autonomously by reflecting the user’s intention efficiently. [0082] FIG. 8(a) signifies one of the design examples of this invention. The realtime endoscope view 800 are displayed in main display 118 (as a user output device) in system console 102 (800). The user can see the airways in the real-time endoscope view 1 through main display 118. This real-time endoscope view 800 is also sent to system controller 128. In the perception step, system controller 128 processes real-time endoscope view 800 and identifies path candidates by using image processing algorithms. Among these path candidates, the system controller 128 select the paths 2 with the designed computation processes, then displays the paths 2 wit h a circle w ith the real-time endoscope view 800.

[0083] In planning step, the system controller 128 provides for interaction from the user, such as a cursor, so that the user can indicate the target path by moving the cursor with joystick 124. When the cursor locates within the area of the path to be selected (one of the two circles), the system controller 128 recognizes the path w ith the cursor as the target path (FIG. 8(a)).

[0084] In further design example, the system controller 128 can pause the motion of the actuator unit 122 and linear translation stage 110 during the user is moving the cursor 3 so that the user can select the target path with the minimal change of the real-time endoscope view 1 and paths 2 since the system does not move.

< Perception >

[0085] One or more of the features discussed herein may be used for perception and planning procedures, including using one or more models for artificial intelligence applications. As an example of one or more embodiments, FIG. 6 is a flowchart show ing steps of at least one planning procedure of an operation of the continuum robot/catheter device 104. One or more of the processors discussed herein may execute the steps shown in FIG. 6, and these steps may be performed by executing a software program read from a storage medium, including, but not limited to, the ROMno or HDD 150, by CPU 120 or by any other processor discussed herein. One or more methods of planning using the continuum robot/catheter device 104 may include one or more of the following steps: (i) In step s6oi, one or more images such, as CT or MRI images, may be acquired; (ii) In step S602, a three dimensional model of a branching structure (for example, an airway model of lungs or a model of an object, specimen or other portion of a body) may be generated based on the acquired one or more images; (iii) In step S603, a target on the branching structure may be determined (e.g., based on a user instruction, based on preset or stored information, etc.); (iv) In step S604, a route of the continuum robot/ catheter device 104 to reach the target (e.g., on the branching structure) may be determined (e.g., based on a user instruction, based on preset or stored information, based on a combination of user instruction and stored or preset information, etc.); (v) In step S605, the generated model (e.g., the generated two-dimensional or three-dimensional model) and the decided route on the model may be stored (e.g., in the RAM 130 or HDD or data storage 150, in any other storage medium discussed herein, in any other storage medium known to those skilled in the art, etc.). In this way, a model (e.g., a 2D or 3D model) of a branching structure may be generated, and a target and a route on the model may be determined and stored before the operation of the continuum robot 104 is started.

[0086] In one or more of the embodiments below', embodiments of using a catheter device/continuum robot 104 are explained, such as, but not limited to features for performing autonomous navigation, movement detection, and/or control technique(s).

[0087] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform an autonomous navigation mode. During the autonomous navigation mode, the user does not need to control the bending and translational insertion position of the steerable catheter 104. The autonomous navigation mode may include or comprise: (1) a perception step, (2) a planning step, and (3) a control step. In the perception step, the system controller 102 may receive an endoscope view (or imaging data) and may analyze the endoscope view- (or imaging data) to find addressable airways from the current position/orientation of the steerable catheter 104. At an end of this analysis, the system controller 102 identifies or perceives these addressable airways as paths in the endoscope view (or imaging data).

[0088] The planning step is a step to determine a target path, which is the destination for the steerable catheter 104. While there are a couple of different approaches to select one of the paths as the target path, the present disclosure uniquely includes means to reflect user instructions concurrently for the decision of a target path among the identified or perceived paths. Once the system 1000 determines the target paths while considering concurrent user instructions, the target path is sent to the next step, i.e., the control step.

[0089] The control step is a step to control the steerable catheter 104 and the linear translation stage 122 (or any other portion of the robotic platform 108) to navigate the steerable catheter 104 to the target path, pose, state, etc. This step may also be performed as an automatic step. The system controller 102 operates to use information relating to the real time endoscope view (e.g., the view 134), the target path, and an internal design & status information on the robotic catheter system 1000.

[0090] Through these three steps, the robotic catheter system 1000 may navigate the steerable catheter 104 autonomously, which achieves reflecting the user’s intention efficiently.

[0091] As shown in FIG. 1, the real-time endoscope view 134 may be displayed in a main display 101-1 (as a user input/output device) in the system 1000. The user may see the airways in the real-time endoscope view 134 through the main display 101-1. This realtime endoscope view 134 may also be sent to the system controller 102. In the perception step, the system controller 102 may process the real-time endoscope view 134 and may identify path candidates by using image processing algorithms. Among these path candidates, the system controller 102 may select the paths with the designed computation processes, and then may display the paths w ith a circle, octagon, or other geometric shape with the real-time endoscope view 134 as discussed further below for FIGS. 7-8.

[0092] In planning step, the system controller 102 may provide a cursor so that the user may indicate the target path by moving the cursor with the joystick 105. When the cursor is disposed or is located within the area of the path, the system controller 102 operates to recognize the path with the cursor as the target path.

[0093] In a further embodiment example, the system controller 102 may can pause the motion of the actuator unit 103 and the linear translation stage 122 while the user is moving the cursor so that the user may select the target path with a minimal change of the real-time endoscope view 134 and paths since the system 1000 would not move in such a scenario. Additionally or alternatively, the features of the present disclosure may be performed using artificial intelligence, including the autonomous driving mode. For example, deep learning may be used for performing autonomous driving using deep learning for localization. Any features of the present disclosure may be used with artificial intelligence features discussed in J. Sganga, D. Eng, C. Graetzel, and D. B. Camarillo, “Autonomous Driving in the Lung using Deep Learning for Localization,” Jul. 2019, Available: https://arxiv.org/abs/1907.08136vi, the disclosure of which is incorporated by reference herein in its entirety.

[0094] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform a depth map mode. A depth map may be generated or obtained from one or more images (e.g., bronchoscopic images, CT images, images of another imaging modality, etc.). A depth of each image may be identified or evaluated to generate the depth map or maps. The generated depth map or maps may be used to perform autonomous navigation, movement detection, and/or control of a continuum robot, a steerable catheter, an imaging device or system, etc. as discussed herein. The depth map may be generated as described in PCT/US2024/025546, herein incorporated by reference in its entirety.

[0095] In one or more embodiments, thresholding may be applied to the generated depth map or maps, or to the depth map mode, to evaluate accuracy for navigation purposes. For example, while not limited to only this type of a threshold, a threshold may be set for an acceptable distance between the ground truth (and/or a target camera location, a predetermined camera location, an actual camera location, etc.) and an estimated camera location for a catheter or continuum robot (e.g., the catheter or continuum robot 104). By way of a further example, the threshold may defined such that the distance between the ground truth (and/or a target camera location, a predetermined camera location, an actual camera location, etc.) and an estimated camera location is equal to or less than, or less than, a set or predetermined distance of one or more of the following: 5 mm, 10 mm, about 5 mm, about 10 mm, any other distance set by a user of the device (depending on a particular application). In one or more embodiments, the predetermined distance may be less than 5 mm or less than about 5 mm. Any other type of thresholding may be applied to the depth mapping to improve and/or confirm the accuracy of the depth map(s).

[0096] Additionally or alternatively, thresholding may be applied to segment the one or more images to help identify or find one or more objects and to ultimately help define one or more targets used for the autonomous navigation, movement detection, and/ or control features of the present disclosure. For example, a depth map or maps may be created or generated using one or more images (e.g., CT images, bronchoscopic images, images of another imaging modality, vessel images, etc.), and then, by applying a threshold to the depth map, the objects in the one or more images may be segmented (e.g., a lung may be segmented, one or more airways may be segmented, etc.). In one or more embodiments, the segmented portions of the one or more images (e.g., the one or more segmented airways, the segmented portions of a lung, etc.) may define one or more navigation targets for a next automatic robotic movement, navigation, and/or control. Examples of segmented airways are discussed further below w ith respect to FIGS. 8(a) and 8(b). In one or more embodiments, one or more of the automated methods that may be used to apply thresholding may include one or more of the follow ing: a watershed method (such as, but not limited to, watershed method(s) discussed in L. J. Belaid and W. Mourou, 2011, vol. 28, no. 2, p. 10, 2011, doi: io.5566/ias.v28.p93-iO2, which is incorporated by reference herein in its entirety), a k-means method (such as, but not limited to, k-means method(s) discussed in T. Kan ungo et al., IEEE Trans Pattern Anal Mach Intell, vol. 24, no. 7, pp. 881-892, 2002, doi: Doi 10. iiO9/Tpami.2OO2.1017616, which is incorporated by reference herein in its entirety), an automatic threshold method (such as, but not limited to, automatic threshold method(s) discussed in N. Otsu, IEEE Trans Syst Man Cybern, vol. 9, no. 1, pp. 62-66, 1979, which is incorporated by reference herein in its entirety) using a sharp slope method (such as, but not limited to, sharp slope method(s) discussed in U.S. Pat. Pub. No. 2023/0115191 Al, published on April 13, 2023, which is incorporated by reference herein in its entirety) and/or any combination of the subject methods. In one or more embodiments, peak detection may include any of the techniques discussed herein, including, but not limited to, the techniques discussed in at least “8 Peak detection,” Data Handling in Science and Technology, vol. 21, no. C, pp. 183-190, Jan. 1998, doi: 10.1016/80922-3487(98)80027-0, which is incorporated by reference herein in its entirety.

[0097] In one or more embodiments, the depth map(s) may be obtained, and/or the quality of the obtained depth map(s) may be evaluated, using artificial intelligence structure, such as, but not limited, convolutional neural networks, generative adversarial networks (GANs), neural networks, any other Al structure or feature(s) discussed herein, any other Al network structure(s) known to those skilled in the art, etc. For example, a generator of a generative adversarial network may operate to generate an image(s) that is/ are so similar to ground truth image(s) that a discriminator of the generative adversarial network is not able to distinguish between the generated image(s) and the ground truth image(s). The generative adversarial network may include one or more generators and one or more discriminators. Each generator of the generative adversarial network may operate to estimate depth of each image (e.g., a CT image, a bronchoscopic image, etc.), and each discriminator of the generative adversarial network may operate to determine whether the estimated depth of each image (e.g., a CT image, a bronchoscopic image, etc.) is estimated (or fake) or ground truth (or real). In one or more embodiments, an Al network, such as, but not limited to, a GAN or a consistent GAN (cGAN), may receive an image or images as an input and may obtain or create a depth map for each image or images. In one or more embodiments, an Al network may evaluate obtained one or more images (e.g., a CT image, a bronchoscopic image, etc.), one or more virtual images, and one or more ground truth depth maps to generate depth map(s) for the one or more images and/or evaluate the generated depth map(s). A Three Cycle-Consistent Generative Adversarial Network (3CGAN) may be used to obtain the depth map(s) and/or evaluate the quality of the depth map(s), and an unsupervised learning method (designed and trained in an unsupervised procedure) may be employed on the depth map(s) and the one or more images (e.g., a CT image or images, a bronchoscopic image or images, any other obtained image or images, etc.). Any feature or features of obtaining a depth map or performing a depth map mode of the present disclosure may be used with any of the depth map or depth estimation features as discussed in A. Banach, F. King, F. Masaki, H. Tsukada, and N. Hata, “Visually Navigated Bronchoscopy using three cycle-Consistent generative adversarial network for depth estimation,” Med Image Anal, vol. 73, p. 102164, Oct. 2021, doi: 10.1016/J.MEDIA.2021.102164, the disclosure of which is incorporated by reference herein in its entirety.

[0098] In one or more embodiments, the system controller 102 (or any other controller, processor, computer, etc. discussed herein) may operate to perform a computation of one or more lumen e.g., a lumen computation mode) and/or one or more of the following: a one or more set or predetermined geometric shapes or one or more circles, rectangles, squares, ovals, octagons, and/or triangles fit/blob process, a peak detection, and/or a deepest point analysis. In one or more embodiments, the computation of one or more lumen may include a one or more set or predetermined geometric shapes or one or more circles, rectangles, squares, ovals, octagons, and/or triangles fit/blob process, a peak detection, and/or a deepest point analysis.

[0099] The problem of fitting a circle to a binary object is equivalent to the problem of fitting a circle to a set of points. In our case the set of points is the boundary points of the binary object. Given a set of points (xl, yl), (%2,y2), ( 3,y3), . . . , (xn,yr) a circle (x - a)² + (y - b)² = c² can be fit to the points by summing the squares of the distances from the points to the circle: SS(a, b, c)

(xi - a)² + (yi - b)² . However, a circle/blob fit is not limited thereto (as discussed herein, any one or more set or predetermined geometric shapes or one or more circles, rectangles, squares, ovals, octagons, and/ or triangles (or other shape(s)) may be used). Indeed, there are several other variations that can be applied as described in D. Umbach and K. N. Jones, "A few methods for fitting circles to data," in IEEE Transactions on Instrumentation and Measurement, vol. 52, no. 6, pp. 1881-1885, Dec. 2003, doi: 10.1109/TIM.2003.820472. Blob fitting can be achieved on the binary objects by calculating their circularity as t Area/ perimeter)² and then defining the circle radius.

[00100] Peak detection is performed in a 1-D signal and is defined as the extreme value of the signal. Similarly, 2-D image peak detection is defined as the highest value of the 2-D matrix. Herein, depth map is the 2-D matrix, and its peak is the highest value of the depth math which actually correspond to the deepest point. However, since there might be more than one airway which are represented by different depth value concentrations along the depth map image, more than one peaks exist. The depth map produces an image which predicts the depth of the airways, therefore for each airway there is a concentration of non-zero pixels around a deepest point that the GANs predicted. By applying peak detection to all the non-zero concentrations of the 2-D depth map the peak of each concentration is detected; each peak corresponds to an airway.

[00101] One or more features discussed herein may be used for performing autonomous navigation, movement detection, and/or control technique(s) for a steerable catheter, continuum robot, imaging device or system, etc. as discussed herein. FIG. 7 is a flowchart showing steps of at least one procedure for performing autonomous navigation, movement detection, and/or control technique(s) for a continuum robot/catheter device (e.g., such as continuum robot/catheter device 104). One or more of the processors discussed herein, one or more Al networks discussed herein, and/or a combination thereof may execute the steps shown in FIG. 7, and these steps may be performed by executing a software program read from a storage medium, including, but not limited to, the ROM 110 or HDD 150, by CPU 120 or by any other processor discussed herein.

[00102] While not limited thereto, one or more methods of performing autonomous navigation, movement detection, and/or control technique(s) for a catheter or probe of a continuum robot device or system may include one or more of the following steps: (i) in step S700, one or more images (e.g., one or more camera images, one or more CT images (or images of another imaging modality), one or more bronchoscopic images, etc.) are obtained; (ii) in step S701, a target detection method is selected (automatically or manually) (e.g., target detection (td) = 1 for the peak detection method or mode, td = 2 for the thresholding method or mode, td =3 for the deepest point method or mode, etc. - the target detection methods shown in FIG. 7 are illustrative, are not limited thereto, and may be exchanged or substitute or used along with any combination of detection methods discussed in the present disclosure or known to those skilled in the art); (iii) in step S703, based on the td value, the method continues to perform the selected target detection method and proceeds to step S704 for the peak detection method or mode, to step S706 for the thresholding method or mode, or to step S711 for the deepest point method or mode; (iv) in a case where td = 1, the peak detection method or mode is performed in step S704, a target or targets are set to be the detected peak or peaks in step S705 and a counter (cn) is set to 2 (cn=2), and a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = 0) and the counter = 2, then td is set to a value of 3 and the process returns to the depth map step S702 and proceeds to step S711 for the deepest point method or mode; (v) in a case where td = 2, the thresholding method or mode is performed in step S706 to identify one or more objects, the counter is set to be equal to 1 (cn = 1), binarization is performed in step S707 to process the image data (e.g., the image data may be converted from color to black and white images, the image data may be split into data sets, etc.), fitting a circle in or on each object is performed in step S708 (also referred to as a blob fit or blob detection method), a target or targets is/are set to be at a predetermined or set location (e.g., a center) of the circle for each object of the one or more objects in step S709, a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = o) and the counter = 1, then td is set to a value of 1 and the process returns to the depth map step S702 and proceeds to step S704 for the peak detection method or mode in step S704, a target or targets is/are identified as the detected peak or peaks in step S705, and a number of targets is evaluated in step S710; and (vi) in a case where the number of targets evaluated in step S710 is 1 or more, then the process proceeds to step S712 where the continuum robot or steerable catheter (or other imaging device or system) (e.g., the continuum robot or steerable catheter 104) is moved to the target or targets.

[00103] In one or more embodiments, the steps S701 through S712 of FIG. 7 may be performed again for an obtained or received next image or images to evaluate the next movement, pose, position, orientation, or state for the autonomous navigation, movement detection, and/or control of the continuum robot or steerable catheter (or imaging device or system) 104. In step S702, the method may estimate (automatically or manually) the depth map or maps (e.g., a 2D or 3D depth map or maps) of one or more images. The one or more depth maps may be estimated or determined using any technique discussed herein, including, but not limited to, artificial intelligence. For example, any Al network, including, but not limited to a neural network, a convolutional neural network, a generative adversarial network, any other Al network or structure discussed herein or known to those skilled in the art, etc., may be used to estimate or determine the depth map or maps (e.g., automatically). The use of a target detection method value (e.g., td = i, td = 2, td = 3) and/or a counter (cn = 1 or cn = 2) is illustrative, and the autonomous navigation, movement detection, and/or control technique(s) of the present disclosure are not limited thereto. For example, in one or more embodiments, a counter may not be used and/or a target detection method value may not be used such that at least one embodiment may iteratively perform a target detection method of a plurality of target detection methods and move on and use the next target detection method of the plurality of the target detection methods until a target or targets is/are found. Alternatively or additionally, even in a case where a target or targets has/have been found already using a particular target detection method, one or more embodiments may continue to use one or more of the other target detection methods (or any combination of the plurality of target detection methods or modes) to confirm and/or evaluate the accuracy and/or results of the target detection method or mode used to find the already-identified one or more targets. In other words, the identified one or more targets may be double checked, triple checked, etc. In one or more embodiments, the deepest point method or mode of step S711 may be used as a backup to identify a target or targets in a case where other target detection methods do not find any targets (# targets = 0). Additionally or alternatively, one or more steps of FIG. 7, such as, but not limited to step S707 for binarization, may be omitted in one or more embodiments.

[00104] In one or more embodiments, a non-transitory computer-readable storage medium may store at least one program for causing a computer to execute a method for performing autonomous navigation, movement detection, and/or control of a continuum robot or catheter, the method comprising one or more of the following steps: (i) in step S700, one or more images (e.g., one or more camera images, one or more CT images (or images of another imaging modality), one or more bronchoscopic images, etc.) are obtained; (ii) in step S701, a target detection method is selected (automatically or manually) (e.g., target detection (td) = t for the peak detection method or mode, td = 2 for the thresholding method or mode, td =3 for the deepest point method or mode, etc. - the target detection methods shown in FIG. 7 are illustrative, are not limited thereto, and may be exchanged or substitute or used along with any combination of detection methods discussed in the present disclosure or known to those skilled in the art); (iii) in step S703, based on the td value, the method continues to perform the selected target detection method and proceeds to step S704 for the peak detection method or mode, to step S706 for the thresholding method or mode, or to step S711 for the deepest point method or mode; (iv) in a case where td = 1, the peak detection method or mode is performed in step S704, a target or targets are set to be the detected peak or peaks in step S705 and a counter (cn) is set to 2 (cn=2), and a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = 0) and the counter = 2, then td is set to a value of 3 and the process returns to the depth map step S702 and proceeds to step S711 for the deepest point method or mode; (v) in a case where td = 2, the thresholding method or mode is performed in step S706 to identify one or more objects, the counter is set to be equal to 1 (cn = 1), binarization is performed in step S707 to process the image data (e.g., the image data may be converted from color to black and white images, the image data may be split into data sets, etc.), fitting a circle in or on each object is performed in step S708 (also referred to as a blob fit or blob detection method), a target or targets is/are set to be at a predetermined or set location (e.g., a center) of the circle for each object of the one or more objects in step S709, a number of targets is evaluated in step S710 such that, in a case where no targets are found (# targets = o) and the counter = 1, then td is set to a value of 1 and the process returns to the depth map step S702 and proceeds to step S704 for the peak detection method or mode in step S704, a target or targets is/ are identified as the detected peak or peaks in step S705, and a number of targets is evaluated in step S710; and (vi) in a case where the number of targets evaluated in step S710 is 1 or more, then the process proceeds to step S712 where the continuum robot or steerable catheter (or other imaging device or system) (e.g., the continuum robot or steerable catheter 104) is moved to the target or targets.

[00105] In one or more embodiments, the steps S701 through S712 of FIG. 7 may be performed again for an obtained or received next image or images to evaluate the next movement, pose, position, orientation, or state for the autonomous navigation, movement detection, and/or control of the continuum robot or steerable catheter (or imaging device or system) 104. In step S702, the method may estimate (automatically or manually) the depth map or maps (e.g., a 2D or 3D depth map or maps) of one or more images. The one or more depth maps may be estimated or determined using any technique discussed herein, including, but not limited to, artificial intelligence. For example, any Al network, including, but not limited to a neural network, a convolutional neural network, a generative adversarial network, any other Al network or structure discussed herein or known to those skilled in the art, etc., may be used to estimate or determine the depth map or maps (e.g., automatically). The use of a target detection method value (e.g., td = 1, td = 2, td = 3) and/or a counter (cn = 1 or cn = 2) is illustrative, and the autonomous navigation, movement detection, and/or control technique(s) of the present disclosure are not limited thereto. For example, in one or more embodiments, a counter may not be used and/or a target detection method value may not be used such that at least one embodiment may iteratively perform a target detection method of a plurality of target detection methods and move on and use the next target detection method of the plurality of the target detection methods until a target or targets is/ are found.

[00106] Alternatively or additionally, even in a case where a target or targets has/have been found already using a particular target detection method, one or more embodiments may continue to use one or more of the other target detection methods (or any combination of the plurality of target detection methods or modes) to confirm and/or evaluate the accuracy and/or results of the target detection method or mode used to find the already-identified one or more targets. In other words, the identified one or more targets may be double checked, triple checked, etc. In one or more embodiments, the deepest point method or mode of step S711 may be used as a backup to identify a target or targets in a case where other target detection methods do not find any targets (# targets = o). Additionally or alternatively, one or more steps of FIG. 7, such as, but not limited to step S707 for binarization, may be omitted in one or more embodiments. For example, if segmentation is done using three categories, such as airways, background and edges of the image, then instead of a binary image, the image has three colors.

[00107] FIG. 8(a) shows images of at least one embodiment of an application example of autonomous navigation and/or control technique(s) and movement detection for a camera view 800 (left), a depth map 801 (center), and a thresholded image 802 (right) in accordance with one or more aspects of the present disclosure. A depth map may be created using the bronchoscopic images and then, by applying a threshold to the depth map, the airways may be segmented. The segmented airways shown in thresholded image 802 may define the navigation targets (shown in the octagons of image 802) of the next automatic robotic movement.

[00108] Fig. 8(b) shows images of show ing a camera view (left), a semi-transparent depth map (that may be color coded) overlaid onto a camera view (center) and a thresholded image (right).

[00109] In one or more embodiments, the continuum robot or steerable catheter 104 may follow the target(s) (which a user may change by dragging and dropping the target(s) (e.g., a user may drag and drop an identifier for the target, the user may drag and drop a cross or an x element representing the location for the target, etc.) in one or more embodiments), and the continuum robot or steerable catheter 104 may move forward and rotate on its own while targeting a predetermined location (e.g., a center) of the target(s) of the airway. In one or more embodiments, the depth map (see e.g., in image 801) may be processed with any combination of blob/circle fit, peak detection, and/or deepest point methods or modes to detect the airways that are segmented. As aforementioned, the detected airways may define the navigation targets of the next automatic robotic movement. In a case where a cross or identifier is used for the target(s), the continuum robot or steerable catheter 104 may move in a direction of the airway with its center closer to the cross or identifier. The continuum robot or steerable catheter 104 may move forward and may rotate in an autonomous fashion targeting the center of the airway (or any other designated or set point or area of the airway) in one or more embodiments.

[00110] A circle fit algorithm is discussed herein for one or more embodiments. The circle shape provides an advantage in that it has a low computational burden, and the lumen within a lung may be substantially circular. However, as discussed herein, other geometric shapes may be used or preferred in a number of embodiments. For example, as may be seen in the camera view 800 in FIG. 8(a), the lumen are more oval than circular, so an oval geometric shape may be used or preferred. The apparatuses, systems, methods, and/or other features of the present disclosure may be optimized to other geometries as well, depending on the particular application (s) embodied or desired. For example, one or more airways may be deformed due to one or more reasons or conditions (e.g., environmental changes, patient diagnosis, structural specifics for one or more lungs or other objects or targets, etc.). In addition, while the circle fit may be used for the planning shown in FIG. 8(a), this figure shows an octagon defining the fitting of the lumen in the images. Such a difference may help with clarifying the different information being provided in the display. In a case where an indicator of the geometric fit (e.g., a circle fit) may be shown in a display, it may have the same geometry’ as used in the fitting algorithm, or it may have a different geometry, such as the octagon shown in FIG. 8(a).

[00111] Additionally, a study was conducted to introduce and evaluate new and non- obvious techniques for achieving autonomous advancement of a multi-section continuum robot within lung airways, driven by depth map perception. By harnessing depth maps as a fundamental perception modality, one or more embodiments of the studied system aims to enhance the robot’s ability to navigate and manipulate within the intricate and complex anatomical structure of the lungs (or any other targeted anatomy, object, or sample). The utilization of depth maps enables the robot to accurately perceive its environment, facilitating precise localization, mapping, and obstacle avoidance. This, in turn, helps safer and more effective robot-assisted interventions in pulmonary procedures. Experimental results highlight the feasibility and potential of the depth map-driven approach, showcasing its ability to advance the field of minimally invasive lung surgeries (or other minimally invasive surgical procedures, imaging procedures, etc.).

[00112] As aforementioned, continuum robots are flexible systems used in transbronchial biopsy, offering enhanced precision and dexterity. Training these robots is challenging due to their nonlinear behavior, necessitating advanced control algorithms and extensive data collection. Autonomous advancements are crucial for improving their maneuverability.

[00113] Sganga, et al. introduced deep learning approaches for localizing a bronchoscope using real-time bronchoscopic video as discussed in J. Sganga, D. Eng, C. Graetzel, and D. Camarillo, “Offsetnet: Deep learning for localization in the lung using rendered images,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 5046-5052, the disclosure of which is incorporated by reference herein in its entirety. Zou, et al. proposed a method for accurately detecting the lumen center in bronchoscopy images as discussed in Y. Zou, B. Guan, J. Zhao, S. Wang, X. Sun, and J. Li, “Robotic-assisted automatic orientation and insertion for bronchoscopy based on image guidance,” IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598, 2022, the disclosure of which is incorporated by reference herein in its entirety. However, there are drawbacks to the techniques discussed in the Sganga, et al. and Zou, et al. publications.

[00114] This study of the present disclosure aimed to develop and validate the autonomous advancement of a robotic bronchoscope using depth map perception. The approach involves generating depth maps and employing automated lumen detection to enhance the robot’s accuracy and efficiency. Additionally, an early feasibility study evaluated the performance of autonomous advancement in lung phantoms derived from CT scans of lung cancer subjects.

[00115] Bronchoscopic operations were conducted using a snake robot developed in the researchers’ lab (some of the features of which are discussed in F. Masaki, F. King, T. Kato, H. Tsukada, Y. Colson, and N. Hata, “Technical validation of multi-section robotic bronchoscope with first person view control for transb ronchial biopsies of peripheral lung,” IEEE Transactions on Biomedical Engineering, vol. 68, no. 12, pp. 3534-3542, 2021, which is incorporated by reference herein in its entirety), equipped with a bronchoscopic camera (OVM6946 OmniVision, CA). The captured bronchoscopic images were transmitted to a control workstation, where depth maps were created using a method involving a Three Cycle-Consistent Generative Adversarial Network (3CGAN) (see e.g., a 3cGAN as discussed in A. Banach, F. King, F. Masaki, H. Tsukada, and N. Hata, “Visually navigated bronchoscopy using three cycle-consistent generative adversarial network for depth estimation,” Medical Image Analysis, vol. 73, p. 102164, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841521002103, the disclosure of which is incorporated by reference herein in its entirety). A combination of thresholding and blob detection algorithms, methods, or modes was used to detect the airway path, along w ith peak detection for missed airways.

[00116] A control vector was computed from the chosen point of advancement (identified centroid or deepest point) to the center of the depth map image. This control vector represents the direction of movement on the 2D plane of original RGB and depth map images. A software-emulated joystick/gamepad was used in place of the physical interface to control the snake robot (also referred to herein as a continuum robot, steerable catheter, imaging device or system, etc.). The magnitude of the control vector was calculated, and if magnitude fell below a threshold, the robot advanced. If the magnitude exceeded the threshold, the joystick was tilted to initiate bending. This process was repeated using a new image from the Snake Robot interface.

[00117] During each trial of the autonomous robotic advancement, the robotic bronchoscope was initially positioned in front of the carina within the trachea. The program was initiated by the operator or user, who possessed familiarity with the software, initiating the robot’s movement. The operator’s or user’s sole task was to drag and drop a green cross within the bronchoscopic image to indicate the desired direction. Visual assessment was used to determine whether autonomous advancement to the intended airway was successfully achieved at each branching point. Summary statistics were generated to evaluate the success rate based on the order of branching generations and lobe segments.

[00118] One or more of the aforementioned features may be used with a continuum robot and related features as disclosed in U.S. Pat. U.S. Pat. 11,882,365, filed on February 18, 2021, the disclosure of which is incorporated by reference herein in its entirety. For example, FIGS. 9 to 11 illustrate features of at least one embodiment of a continuum robot apparatus 10 configuration to implement automatic correction of a direction to which a tool channel or a camera moves or is bent in a case where a displayed image is rotated. The continuum robot apparatus 10 enables to keep a correspondence between a direction on a monitor (top, bottom, right or left of the monitor) and a direction the tool channel or the camera moves on the monitor according to a particular directional command (up, down, turn right or turn left) even if the displayed image is rotated. The continuum robot apparatus 10 also may be used with any of the autonomous navigation, movement detection, and/or control features of the present disclosure.

< Planning >

[00119] FIGS. 9 and 10 signify a design example of the planning step. In this design example, the system controller 128 highlights the two paths found in the perception step as path 1 (902) and path 2 (904). The display also includes a cursor 906 shown as a crosshair. Other indicators that highlight the path or paths found in the perception step may alternatively be provided, such as a ring or circle or any other indication of path.

In FIG. 9, the cursor 906 has been moved into path 2 (904), having a white circular feature. This can be done, for example by user instruction to select this path is selected as the target path. For example, the user can use the handheld controller (e.g., a joystick) 124 to select or change the target path by clicking on or inside a lumen or on or inside an indication of the lumen /path candidate. In other embodiments, a touchscreen or a voice input is used. The system controller may show, on the display, an indication of which lumen was selected by the user. This may be done in one of a number of ways, such as by highlighting the selected lumen, including a feature such as a circle, crosshair, or similar indicator on the chosen lumen, including a feature on each of the lumen and havi ng a a color change (black versus white circles, or any other color scheme or other indicator, such as red and green, black and green, bolded, highlighted, dashed) or other visual indicator to indicate the detected lumen.

[00120] In the example shown in FIG. 9, the system controller 128 change the path 1 or path 2 based on the user instruction. FIG. 9 is a selection of the right-most lumen. FIG. 10 is a selection of the left-most lumen. In FIG. 10, the selection is indicated by a bolder circle for path 1/904 the target path (e.g., path 1, 904) and a dashed circle for the unselected path 902. Concurrent user instruction for the target path among the paths allows reflecting user’s intention to the robotic catheter system during autonomous navigation effectively. Another optional feature is also shown in FIG. 10, where the cursor 906 has changed from a crosshair to a circle. In the workflow leading up to the image shown in FIG. 10, the user moved the cursor 906, which is optimized for the user to view both the cursor and the underlying image, until it was touching or inside the selected path 904, at which time, the cursor 906 is changed to a less obvious indicator, shown here as a small circle.

[00121] While the user may select or change the target path at any point in the planning step, as the autonomous system move into the control step, the user can optionally adjust the target path as described herein, or the prior target path can be used until the predetermined insertion depth is reached. The need for additional selection of the target path can depend on the branching of the lumen network, where, for example, the user provides target path information as each branch within the airway becomes visible in the camera image.

[00122] While color is used in this example to indicate the selection of the target path, other colors or other indicators may be used as well, such as a bolder indicator, a flashing indicator, removal of the indicator around the non-selected path lumen, etc.

[00123] By having the user output device and displaying symbols for the paths and user instruction GUI with endoscope view in the user output device, user can form their intention intuitively and accurately with the visual information in one place. Especially, the dedicated user instruction GUI allows the user to select the target path immediately even when the paths are more than two.

[00124] By having the user output device and displaying symbols for the paths and differentiating the sy mbol for the target path from the other paths with endoscope view in the user output device, the user can form intention intuitively and accurately with the visual information in one place. Particularly, the differentiating the symbol for the target path achieve the user instruction with the minimal sy mbols without the dedicated user instruction GUI and allows the user to learn/understand how to read symbols with the minimal effort.

[00125] As shown in embodiments herein, circles (or ovals) are used as the sy mbol of the paths on the 2D interface, and the cursor provides a symbol for input of user instruction to the GUI. These GUIs have minimal obstacles for the endoscope view in the user output device. Also, selecting object with the cursor are very familiar maneuver from common computer operation, the user can easily learn how to use it. [00126] By pausing the actuator and the linear translation stage during user’s instruction, the system can reduce the risk where the user miss the target path in their interaction. Also, this gives users to think and judge the target path among the paths without pressurizing the user to make decisions in the short time.

[00127] By allow ing the user to add a new target path if the user cannot find the target path among the existing paths and find the target path in the endoscope view, the plan generated by the system becomes more accurate with minimal effort of the user.

[00128] In some embodiments, the user input device is a voice input device. Since the autonomous system is driving the steerable catheter, full directional control of the catheter is unnecessary for autonomous driving. It can also be unwanted. A limited library of commands that the user can provide gives full control to the user to select which lumen is the correct one for the next navigation step, but prevents the user from, for example, trying to keep the steerable catheter in the center of the lumen as they would do with a manual catheter since this can be accomplished through the autonomous function. The limited library also simplifies the system.

[00129] Effective voice commands can be in the form of a limited library' in the system and are, for the one system as provided herewith are (1) start, (2) stop, (3) center, (4) up, (5) down, (6) right, (7) left, and (8) back. Other systems may have more or fewer commands. Some systems will have a different selection of commands, depending on the use for the steerable catheter. In this embodiment, voice commands are classified as one of the effective commands and acted on as such. Verbalizations that are not classified as one of the effective commands are ignored when the sent command is not recognized as the effective commands. In some embodiments, the user or users train the system to recognize their particular enunciation of the effective voice commands. In other embodiments, the system is pre-set with the range of enunciations that are effective commands.

[00130] In some embodiments, the instructions are limited to the eight commands listed above or variants thereof. In other embodiments, the instructions are limited to less than or equal to 4, 6, 8, 10, 12, 14, 16, 18, 20, or 24 commands. The commands may all be limited to instructions for selecting a target path in a lumen.

[00131] Some embodiments provide autonomous navigation with voice command. When the user starts the autonomous navigation, the user sends a voice command, “start” (S1010) and “Autonomous navigation mode” is displayed on the main display 118. The autonomous navigation system detects airways in the camera view (S1020). The centers of detected airways are displayed as diamond mark (560) in the detected airways in the camera view.

[00132] In some embodiments, in order for the user to select an airway for the steerable catheter to move in, the user sends one of voice commands from the options of “center”, “up”, “down”, “right” and “left”. When the voice command is accepted by the system, the color of “x” mark on the selected location is changed from black to red and a triangle 570 is displayed on the selected mark.

[00133] The selected location stays at the same location until a different location is accepted to the system. As the default, the “x” mark on the center is set when the autonomous navigation mode is started.

[00134] The system sets the closest airway from the selected x mark as the airway to be aimed based on the distance between the selected x mark and each diamond mark in the detected airways (S1030).

[00135] By choosing the closest airway as an operator’s intended airway from the “+” mark, which the operator moves w ith the voice commands, the operator can instruct the intended airway intuitively and accurately. The operator can always clearly confirm the distance between the “+” mark and the intended airway on the display and easily understand which airway option the autonomous system will choose on the display. Therefore, this transparency gives the operator predictable system behavior and operation confidence during autonomous operation. [00136] Also, since the position options of “+” mark include a limited number, the operator can determine the next position option easily and quickly.

[00137] Moreover, with this method, the autonomous system always has at least one intended airway until there is at least one airway candidate. This feature avoids the situation without the intended airw ay and make the system behavior robust.

[00138] The user can stop the autonomous navigation any time by sending a voice command, “stop”, and can start the manual navigation using a handheld controller 124 to control the steerable catheter and the linear translational stage. When the user takes over the control, “Manual navigation mode” is displayed on the main display 118. The user can restart the autonomous navigation when needed by sending a voice command, “start”. Alternatively, the user can stop the autonomous navigation by, for example, an interaction with the handheld controller.

[00139] While the robotic platform is bending the steerable catheter and moving the linear translational stage forward, all input signals to the robotic platform are recorded in the data storage memory⁷ HDD 188 regardless of navigation modes. When the user needs to retract the robotic platform, the user sends a voice command, “back”, then the robotic platform inversely applies the recorded input signals taken during insertion to the steerable catheter and the linear translational stage. During retraction, the system displays “back” on the display.

[00140] (Backup and alternative systems) Instead of sending voice commands, the user can send the commands using other input devices including a number pad or a general computer keyboard. The commands can be assigned as numbers on the number pad or other letters on the keyboard. The other input devices may be used along with or instead of voice commands. In some embodiments, the input device is has a limited number of keys/buttons that can be pressed by the user. For example, a numerical keypad is used where 2, 4, 6, and 8 are the four directions and 5 is center. The additional numbers on the keypad may be without function, or they may provide an angled movement. < Control >

[00141] The continuum robot and its navigation can be exemplified by a steerable catheter and particularly by a bronchoscopy procedure to diagnose tumorous tissue in a region of interest. This is described below. FIG. 11 shows the typical camera image during the autonomous navigation and FIG. 12 shows the flowchart of this exemplary’ autonomous navigation.

[00142] The autonomous navigation system receives an image from the camera and defines a position point in the image. The position point can be used as a proxy to the position of the distal tip of the continuum robot. In some embodiments, the position point is the center of the camera view. In other embodiments, the position point is offset from the center of the camera view based on, for example, a known offset of the camera within the continuum robot.

[00143] The autonomous navigation system compares the distance between the position point (e.g., the center of the camera view) and the target point with the threshold (S1050). If the distance between the position point and the target point is longer than the threshold, the robotic platform bends the steerable catheter toward the target point (S1060) until the distance between the position point and the target point is smaller than the threshold. If the distance between the position point view and the target point is smaller than the threshold, the robotic platform moves the linear translational stage forward (S1080).

[00144] In FIG. 12, the system detects the airways in the camera view (S1020). This method and system are particularly described above, and include using a depth map produced by processing one or more images obtained from the camera, fitting the lumen or lumens (e.g., airways) using an algorithm such as a circle fit, peak detection algorithm or similar, where, in instances of the camera image not able to perform the algorithm to find one or more lumen, using the deepest point. Next, the system sets the airway to be aimed (S1030). This provides a target path. This can be done by user interaction as discussed hereinabove or through an automated process.

[00145] As discussed above in reference to FIG. 6, the route of the continuum robot to reach the target is determined and (in step S605), the generated model and the decided route on the model may be stored. In this target path, a plurality of target points are used to navigate the continuum robot along the target path. The target points in the lumen (e.g., airway) define for the target path (S1040). The target point may be, for example, the center of the circle that was used to fit to the airway. In FIG. 11, the target point is shown as a “+” mark 540 and an arrow 580 connecting the “+” mark and the center of the camera view (S1040).

[00146] Then, the autonomous system must aim the distal end of the steerable catheter towards the target point and move the steerable catheter forward, towards the target point (S1050). For this step, a threshold is set. If the target point is inside of the threshold, the steerable catheter is advanced, increasing the insertion depth (Si 070) . If the target point is not inside of the threshold, then the distal end of the steerable catheter is bent towards the target point (S1060). After the steerable catheter is bent further, the controller must re-assess whether the target point is inside of the threshold (S1050). If the steerable catheter has not yet reached the region of interest, or as at the predetermined insertion depth, the steerable catheter wil 1 move forward S1080). Then these two steps of bending the continuum robot towards the target point and moving the continuum robot forward along the target path are repeated, w ith new images taken periodically to be used at each iteration to determine if the continuum robot is to move forward or to bend to more closely point to the target point. If the steerable catheter has reached the region of interest, or as at the predetermined insertion depth, the automated procedure will end (S1090).

[00147] Of note, in this process, after a particular iteration of moving forward (S1080) or bending toward the target point (S1060), the system then returns to detecting the lumen (e.g., airway) in the camera view. Thus, each iteration can be performed with a new and separate image taken by the camera. Knowledge of locations w ith i n the last image as well as knowledge obtained from prior data, such as from a pre-operative CT image, are not required. Therefore, robust registration algorithm(s) are not required to perform this automated driving function. This can be particularly advantageous in multiple situations and w ill not be effected by breathing as much as many attempts at autonomous navigation as seen in the literature.

[00148] For a bronchoscopic procedure, various parameters of the robotic platform 106 including the frame rate of the camera, the bending speed of the steerable catheter, the speed of linear translational stage 110, and the predetermined insertion depth of the linear translational stage are set. They each independently may be preset (e.g., a standard base value) or set by the user for the procedure. If set by the user, it may be based on the user’s preference, or based on specifics of the patient. The parameters may be calculated or be obtained from a look-up table. The predetermined insertion depth may be estimated as the distance from the carina to the region of interest based on, for example, one or more preprocedural CT images. These parameters may be set before the start of the automated motion portion of the procedure, such as when the steerable catheter 104 reaches the carina.

[00149] A threshold is used during autonomous function to decide whether to bend the steerable catheter to optimize the direction and/or angle of the tip or to move forward using the linear translational stage. The threshold 510 may be a constant and can be defined based on the dimensions of the camera view. The threshold relates to the distance from the center of the camera view, (e.g., the center of the a dotted circle 510 in a camera view 520, w hich is the center of the distal end of the steerable catheter) to the center of the airway 530 that has been selected as the target path (the target point 540). The threshold value is visualized in FIG. 11 as a dotted circle 510, however, it can alternatively be configured as a vector. The vector represents the direction of movement on the image plane of the image from the camera and depth map images, where the magnitude of the vector is the threshold value. The distance to be set as the threshold may be decided based on, for example, data from a lung phantom model. Alternatively, the threshold may be based on a library of threshold data. In some embodiments, the threshold set to 10%, 15%, 20%, 25%, 30%, 35%, or 40% or a value therebetween of the camera view dimension (i.e., the distance of a diagonal line across the camera view), in some embodiments, the threshold set to between 25% and 35%, or around 30%.

[00150] In some embodiments, when the user starts the autonomous navigation, an indicator of the navigation mode being used, such as displaying “Autonomous navigation mode” on the main display 118. The autonomous navigation system detects airways in the camera view (S1020). The user places a mouse pointer 550 on the airway to be aimed 530 in the camera view for the autonomous navigation system to set the airway to be aimed (S1030). The autonomous navigation system detects the target point 540 in the detected airway as described above (S1040), then the autonomous navigation system compares the distance between the center of the camera view and the target point with the threshold (Si 050). If the distance between the center of the camera view and the target point is longer than the threshold, the robotic platform bends the steerable catheter toward the target point (S1060) until the distance between the center of the camera view and the target point is smaller than the threshold. If the distance between the center of the camera view and the target point is smaller than the threshold, the robotic platform moves the linear translational stage forward (S1080).

[00151] In some embodiments, the user has the ability to stop the autonomous navigation any time by pushing a button on the handheld controller 124 and can start the man ual navigation to control the steerable catheter and the linear translational stage by the handheld controller. When the user takes over the control, an indicator of this control is provided, such as a display of “Manual navigation mode” on the main display 118. The user can restart the autonomous navigation when needed by, for example, pushing a button on the handheld controller 124. [00152] When the steerable catheter reaches a position close to the region of interest (e.g., tumorous tissue), the user has the option to switch the navigation mode to the manual navigation and, for example, deploy a biopsy tool toward the region of interest to take a sample through the working tool access port.

[00153] A Return to the Carina function may be included w ith the autonomous driving catheter and system. While the robotic platform is bending the steerable catheter and moving the linear translational stage forward, all input signals to the robotic platform may be recorded in the data storage memory HDD 188 regardless of navigation modes. When the user indicates a start of the Return to Carina function (e.g., hitting the appropriate button on the handheld controller 124), the robotic platform inversely applies the recorded input signal taken during insertion to the steerable catheter and the linear translational stage.

[00154] In some embodiments, an insertion depth may be set before driving the steerable catheter. When the linear translational stage reaches the predetermined insertion depth, the autonomous navigation system can be instructed to stop bending the steerable catheter and moving the linear translational stage forward (S1070). The robotic platform then switches the mode from the autonomous navigation to the manual navigation. This allows the user to start interacting with the region of interest (e.g., take a biopsy) or to provide additional adjustments to the location or orientation of the steerable catheter.

[00155] In some embodiments, the frame rate is set for safe movement. The steps from S1020 to S1080 in FIG. 12 can be conducted at every single frame of camera image. Thus, depending on the capability of the Central Processing Unit 182, the maximum frame rate of the camera image that can be handled for the autonomous navigation is decided. Then based on the frame rate and the acceptable risk during bronchoscopy, the speed of bending the steerable catheter and the speed of moving the linear translational stage are decided. If it is important to move the steerable catheter based on the images when the steerable catheter is moving faster than can be ‘seen’ by the images from the camera. In an example where the maximum frame rate is 10 frames per second (fps) and the acceptable amount of the airway pushed by the steerable catheter is 0.5mm, the speed of the linear translational stage may be set less than 5 [mm/sec]. Other frame rates and risk factors will suggest different speeds.

[00156] FIG. 13 shows an exemplary flowchart to set the speed of bending the steerable catheter and the speed of moving the linear translational stage based on the target point in the detected airway to be aimed.

[00157] FIG. 14 shows an exemplary' display at the parameter settings. The user can set the bending speed of the steerable catheter at two points in the camera view, Bending speed 1 and Bending speed 2. During autonomous navigation, the autonomous navigation system sets the bending speed of the steerable catheter by linearly interpolating Bending speed 1 and Bending speed 2 based on the target point in the detected airway to be aimed (S1055). In general, Bending speed 2 is slower than Bending speed 1 so that the steerable catheter does not overbend.

[00158] As shown in the exemplary display of FIG. 14, the user can set the bending speed of the linear translational stage at two points in the camera view', Moving speed 1 and Moving speed 2. During autonomous navigation, the autonomous navigation system sets the moving speed of the linear translational stage by linearly interpolating Moving speed 1 and Moving speed 2 based on the target point in the detected airway to be aimed (S1075). In general, Moving speed 1 is faster than Moving speed 2 because the closer to the center of the airway the steerable catheter is, the less risky the steerable catheter collide to the airway wall.

[00159] According to this embodiment, the steerable catheter can reach the target point faster with less risk for the steerable catheter to collide to the airw ay wall.

< Variable threshold > [00160] FIG. 15 shows an exemplary flowchart to adjust the threshold based on the location of the steerable catheter in the lung. In this workflow, the system detects the airway or airways in the camera view (S1020). The system then sets an airway to be aimed towards (S1030). Information as to which airway will be aimed may come from user input or from the controller. Next, the system detects the target point in the airway (S1040). A threshold is set (S1045) and it is determined whether the target point is inside of the threshold (S1050). In this case, as the position point is set as the center of the image, so the target point is inside of the threshold (S1050) if the target point is sufficiently close to the current position of the steerable catheter tip. If the target point is not inside the threshold, the tip of the steerable catheter in the lung is bent towards the target point (S1060). This bend may be a set amount, it may be to the approximate location of the target point, etc. After the catheter is bent, the system detects the airway in the camera view again (S1020) and the workflow repeats. If, in step S1050, the target point is inside of the threshold, it is determined whether the predetermined insertion depth is reached (S1070). If this depth is not yet reached, the steerable catheter is moved forward in the airway (S1080). Once a new location is reached (which may be a set insertion distance, a distance set by the user, etc.), the system detects the airways in the camera view (S1020) and the workflow repeats. If, in step S1070, the predetermined insertion depth is reached, the workflow is ended (S1090).

[00161] FIG. 16 shows an exemplary the display at the parameter settings, illustrating where two different thresholds are used. The user can set two thresholds to decide for the autonomous function to bend the steerable catheter or to move forw ard the linear translational stage based on the insertion depth at the parameter settings. In this example, Threshold 1 and Threshold 2 are located at the carina and at the region of interest in the planning view created from the preoperative CT image. Thus, the autonomous driving can start using Threshold 1, as shone by the large dashed circle, when the airway is large and the need for tight control and steering is not as great. Then, as the steerable catheter moves down the airways that narrow toward the peripheral area of the lung, a second threshold (Threshold 2) is needed, where Threshold 2 is smaller than Threshold 1, as shown by the smaller circle in FIG. 16. Since the airway is smaller the further into the lung the catheter moves, the robotic catheter may need to be bent more accurately towards the center of the airways before it moves forward. Thus, a smaller threshold may be required.

[00162] During autonomous navigation, the autonomous navigation system can set the threshold at each frame of the camera view , as described as S1045 in the above workflow. When two thresholds are used as discussed above for FIG. 16, the threshold can be a linearly interpolation of Threshold 1 and Threshold 2 based on the insertion depth (S1045). In other embodiments, there are two (or more) different thresholds , where ehte threshold changes from one to another when a pre-defined insertion depth or other indication of depth into the lumen is reached. In some embodiments, the thresholds are changed based on the lumen diameter at the location of the steerable catheter. According to this embodiment, the steerable catheter can move faster and spends less time to bend around the carina, leading to less time for bronchoscopy, but maintains an accurate and precise navigation further into the periphery' where a deviation from the center of the airw ay would increase risk to the patient.

< Emergency situations>

[00163] FIG. 17 shows a flowchart with an exemplary method to abort the autonomous navigation when the blood is detected in the camera view. The criterion to abort bronchoscopy may be defined as the ratio of the number of pixels indicating the blood divided by the total number of pixels in a camera image. In this embodiment, an imaging processing library', e.g. OpenCV, is used to count the number of red pixels in a RGB camera view. If the ratio of the number of pixels indicating the blood divided by the total number of pixels in a camera image exceeds the predetermined ratio, the autonomous navigation is aborted. Similar to the blood, the mucous in the airway can be detected using an imaging processing library. For detecting mucus, the number of yellow⁷ pixels in a RGB camera view can be used. According to this embodiment, the steerable catheter can be automatically stopped during bronchoscopy when an emergency situation is detected.

< Exemplary Phantom Study >

[00164] As further discussed herein, one or more methods of the present disclosure were validated on one clinically derived phantom and two ex-vivo pig lung specimens with and without simulated breathing motion, resulting in 261 advancement paths in total, and in an in vivo animal. The achieved target reachability in phantoms was 73.3%, in ex-vivo specimens without breathing motion was 77% and 78%, and in ex-vivo specimens with breathing motion was 69% and 76%. With the presented methodology(ies) and performance(s), the proposed supervised-autonomous navigation/driving approach(es) in the lung is/are proven to be clinically feasible. By potentially enhancing precision and consistency in tissue sampling, this system or systems have the potential to redefine the standard of care for lung cancer patients, leading to more accurate diagnoses and streamlined healthcare workflows.

[00165] The field of robotics has progressed and impacted numerous facets of everyday life. Notably, autonomous driving provides useful features of the present disclosure, with systems adeptly navigating intricate terrains with little or no human oversight.

[00166] Similarly, the present disclosure provides features that integrate the healthcare sector with robotic-assisted surgery (RAS) and transforms same into Minimally Invasive Surgery (MIS). Not only does RAS align well with MIS outcomes (see e.g., J. Kang, et al., Annals of surgery, vol. 257, no. 1, pp. 95-101 (2013), which is incorporated by reference herein in its entirety), but RAS also promises enhanced dexterity and precision compared to traditional MIS techniques (see e.g., D. Hu, etal., The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 14, no. 1, p. 01872 (2018), which is incorporated by reference herein in its entirety).

[00167] The potential for increased autonomy in RAS is significant and is provided for in one or more features of the present disclosure. Enhanced autonomous features of the present disclosure may bolster safety by diminishing human error and streamline surgical procedures, consequently reducing the overall time taken (3, 4) . Moreover, a higher degree of autonomy provided by the one or more features of the present disclosure may mitigate excessive interaction forces between surgical instruments and body cavities, which may minimize risks like perforation and embolization. As automation in surgical procedures becomes more prevalent, surgeons may transition to more supervisor)' roles, focusing on strategic decisions rather than hands-on execution (see e.g., A. Pore, et al., IEEE Transactions on Robotics (2023), which is incorporated by reference herein in its entirety). [00168] In addressing the aforementioned issues, at least one objective of the studies discussed in the present disclosure is to develop and clinically validate a supervised- autonomous navigation/driving approach in robotic bronchoscopy. Distinctively, one or more methodologies of the present disclosure utilize unsupervised depth estimation from the bronchoscopic image (see e.g., Y. Zou, et al., IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598 (2022), which is incorporated by reference herein in its entirety), coupled with the robotic bronchoscope (see e.g., J. Zhang, et al., Nature Communications, vol. 15, no. 1, p. 241 (Jan. 2024), which is incorporated by reference herein in its entirety), and operate devoid of any a priori knowledge of the patient’s anatomy — a significant stride forward. The inventors of the present disclosure introduce one or more advanced airway tracking method(s). These methods, rooted in the detection of airways within the estimated bronchoscopic depth map, may form the foundational perception algorithm that orchestrates the robotic bronchoscope’s movements in one or more embodiments. [00169] The propositions of the present disclosure goes beyond theory. The inventors have operationalized the method(s) into a tangible clinical tool or tools, which empowers physicians to manually delineate the robot’s desired path. This is achieved by simply placing a marker on the computer screen in the intended direction of the bronchoscopic image. Hence, while motion planning remains physician-driven, both airway detection and motion execution stand out as fully autonomous features in one or more embodiments of the present disclosure. This synthesis of manual control and autonomy is groundbreaking: to our knowledge, our tool is the pioneering clinical instrument that facilitates airway tracking for supervised-autonomous driving within the lung. Validating its effectiveness, we assessed the performance of the driving algorithm features, emphasizing target reachability and success at branching points. Our rigorous testing spanned a clinically derived phantom and two ex-vivo pig lung specimens, cumulatively presenting 168 targets. This comprehensive approach or approaches is/are part of the features of the present disclosure that contribute to features/ways to address the pressing gaps observed in previous studies.

[00170] Bronchoscopic operations were performed using a snake robot developed using the OVM6946 bronchoscopic camera (OmniVision, CA, USA). The snake robot may be a robotic bronchoscope composed of, or including at least, the follow ing parts in one or more embodiments: i) the robotic catheter, ii) the actuator unit, iii) the robotic arm, and iv) the software in one or more embodiments (see e.g., FIG. 1, FIG. 9, FIG. 12(c), etc. discussed herein) or the robotic catheter described in one or more of: U.S. Pat. 11,096,552; U.S. Pat. 11,559490; U.S. Pat. 11,622,828 ; U.S. Pat. 11,730,551; U.S. Pat. 11,926,062; US2021/0121162; US2021/0369085; US2022/0016394; US2022/0202277;

WO/2023/154825; WO/2023/154825; WO / 2023/ 164275, each of which are herein incorporated by reference in their entirety. The robotic catheter may be developed to emulate, and improve upon and outperform, a manual catheter, and, in one or more embodiments, the robotic catheter may include nine drive wires which travel through or traversethe steerable catheter, housed within an outer skin made of polyether block amide (PEBA) of 0.13 mm thickness. The catheter may include a central channel which allows for inserting the bronchoscopic camera. The outer and inner diameters (OD, ID) of the catheter may be 3 and 1.8 mm, respectively, outer and inner diameters (OD, ID) of the catheter are 3 and 1.8 mm, respectively (see e.g., J. Zhang, etal., Nature Communications, vol. 15, no. 1, p. 241 (Jan. 2024), The steering structure of the catheter may include two distal bending sections: the tip and middle sections, and one proximal bending section without an intermediate passive section/segment. Each of the sections may have its ow n degree of freedom (DOF) (see e.g., A. Banach, et al., “Medical image analysis, vol. 73, p. 102164 (2021). The catheter may be actuated through the actuator unit attached to the robotic arm and may include nine motors that control the nine catheter wires. Each motor may operate to bend one wire of the catheter by applying pushing or pulling force to the drive wire. Both the robotic catheter and actuator may be attached to a robotic arm, including a rail that allows for a linear translation of the catheter. The movement of the catheter over or along the rail may be achieved through a linear stage actuator, which pushes or pulls the actuator and the attached catheter. The catheter, actuator unit, and robotic arm may be coupled into a system controller, which allows their communication with the software. While not limited thereto, the robot’s movement may be achieved using a handheld controller (gamepad) or, like in the studies discussed herein, through autonomous driving software. The validation design of the robotic bronchoscope was performed by replicating real surgical scenarios, w here the bronchoscope entered the trachea and navigated in the airways toward a predefined target (see e.g., L. Dupourque , et al., International journal of computer assisted radiology and surgery, vo. 14, no. 11, pp. 2021-2029 (2019), which is incorporated by reference herein in its entirety).

[00171] In accordance with one or more embodiments of the present disclosure, apparatuses and systems, and methods and storage mediums for performing navigation, movement, and/or control, and/or for performing depth map-driven autonomous advancement of a multi-section continuum robot (e.c/., in one or more airways, in one or more lungs, in one or more bronchoscopy pathways, etc.), may operate to characterize biological objects, such as, but not limited to, blood, mucus, lesions, tissue, etc.

[00172] The in vivo comparison study discussed below showed that the autonomous driving took less time for bending than human operators (the median time at each bifurcation = 2.5 and 1.3 [s] for a human operator and autonomous driving, respectively). With the presented methodology(ies) and performance(s), the proposed supervised- autonomous navigation /driving approach(es) in the lung is/are proven to be clinically feasible. By potentially enhancing precision and consistency in tissue sampling, this system or systems have the potential to redefine the standard of care for lung cancer patients, leading to more accurate diagnoses and streamlined healthcare workflows.

< Perception >

[00173] The autonomous driving method feature(s) of the present disclosure relies/rely on the 2D image from the monocular bronchoscopic camera without tracking hardware or prior CT segmentation in one or more embodiments. A 200x200 pixel grayscale bronchoscopic image serves as input for a deep learning model (scGAN (see e.g., A. Banach, F. King, F. Masaki, H. Tsukada, N. Hata, “Medical image analysis, vol. 73, p. 102164 (2021), the disclosure of which is incorporated by reference herein in its entirety)) that generates a bronchoscopic depth map.

[00174] Specifically, 3cGAN’s adversarial loss accumulates losses across six levels: i<?an₆ = Lgan_lev + L_gan_lev + • • • + L_gan_lev (1)

The cycle consistency loss combines the cycle consistency losses from all three level pairs:

where A stands for the bronchoscopic image, B stands for the depth map, C stands for virtual bronchoscopic image, X represents estimation of X, and the lower index i stands for the networks level.

[00175] The merging loss of the 3cGAN combines all the networks levels:

[00176] The total loss function of the 3cGAN is:

.1/6 — Lgari + Lcyc^ + Lm (4)

[00177] The 3CGAN model underwent unsupervised training using bronchoscopic images from phantoms derived from segmented airways. Bronchoscopic operations to acquire the training data were performed using a Scope 4 bronchoscope (Ambu Inc, Columbia, MD), while virtual bronchoscopic images and ground truth depth maps were generated in Unity (Unity Technologies, San Francisco, CA). The training ex-vivo dataset contained 2458 images. The network was trained in PyTorch using an Adam optimizer on 50 epochs with a learning rate of 2 io ⁴ and a batch size of one. Training time was approximately 30 hours, and less than 0.02s for the inference of one depth map on a GTX 1080 Ti GPU.

[00178] In the inference process the depth map was generated from the 3cGAN models by inputting the 2D image from the bronchoscopic camera. The bronchoscopic image and/ or the depth map was then processed for airway detection using a combination of blob detection, thresholding, and peak detection (see e.g., FIG. 11(a) discussed below). Blob detection was performed on a depth map where 20% of the deepest area was thresholded, and the centroids of the resulting shapes w ere treated as potential points of advancement for the robot to bend and advance towards. Peak detection w as performed as a secondary detection method to detect airways that may have been missed by the blob detection. Any peaks detected inside an existing detected blob were disregarded. Direction vector control command may be performed using the directed airways to decide to employ bending and/or insertion, and/or such information may be passed or transmitted to software to control the robot and to perform autonomous advancement.

[00179] As shown in FIGS. 1-6 and 18, one or more embodiments of the present disclosure may be a robotic bronchoscope using a robotic catheter and actuator unit, a robotic arm, and/or a control software or a User Interface. Indeed, one or more robotic bronchoscopes may use any of the subject features individually or in combination.

[00180] In one or more embodiments, depth estimation may be performed from bronchoscopic images and with airway detection (see e.g., FIGS. 19(a) - 19(b). Indeed, one or more embodiments of a bronchoscope (and/ or a processor or computer in use therewith) may use a bronchoscopic image with detected airways and an estimated depth map (or depth estimation) with or using detected airways. A pixel of a set or predetermined color e.g., red or any other desired color or other indicator) 1002 represents a center of the detected airway. A cross or plus sign (+) 1003 may also be of any set or predetermined color (e.g., green or any other desired color), and the cross 1003 may represent the desired direction determined or set by a user (e.g., using a drag and drop feature, using a touch screen feature, entering a manual command, etc.) and/or by one or more processors (see e.g., any of the processors discussed herein). The line or segment 1004 (which may also be of any set or predetermined color, such as, but not limited to, blue) may be the direction vector between the center of the image/ depth map and the center of the detected blob in closer proximity to the cross or plus sign 1003.

[00181] In the inference process the depth map w as generated from the 3cGAN models by inputting the 2D image from the bronchoscopic camera. The depth map was then processed for airway detection using a combination of blob detection (see e.g., T. Kato, F. King, K. Takagi, N. Hata, IEEE/ASME Transactions on Mechatronics pp. 1-1 (2020), the disclosure of which is incorporated by reference herein in its entirety), thresholding, and peak detection (see e.g., F. Masaki, F. King, T. Kato, H. Tsukada, Y. Colson, and N. Hata, IEEE Transactions on Biomedical Engineering, vol. 68, no. 12, pp. 3534-3542 (2021), the disclosure of which is incorporated by reference herein in its entirety) (see e.g., FIGS. 19(a) - 19(b) and related discussion herein). Blob detection was performed on a depth map where 20% of the deepest area was thresholded, and the centroids of the resulting shapes were treated as potential points of advancement for the robot to bend and advance towards. Peak detection (see e.g., F. Masaki, 2021) was performed as a secondary’ detection method to detect airways that may have been missed by the blob detection. Any peaks detected inside an existing detected blob were disregarded.

[00182] The integrated control using first-person view, grants physicians the capability to guide the distal section’s motion via visual feedback from the robotic bronchoscope. For forward motion/ navigation, users may determine only the lateral and vertical movements of the third (e.g., most distal) section, along with the general advancement or retraction of the robotic bronchoscope. The user’s control of the third section may be performed using the computer mouse and drag and drop a cross or plus sign 1003 to the desired direction as shown in FIG. 20(a) and/or FIG. 20(b). A voice control may also be implemented additionally or alternatively to the mouse-operated cross or plus sign 1003. For example, an operator or user may select an airway for the robotic bronchoscope to aim using voice recognition algorithm (VoiceBot, Fortress, Ontario, Canada) via a headset (J100 Pro, Jeeco, Shenzhen, China). The options acceptable as input commands to control the robotic bronchoscope were the four cardinal directions (up, down, left, right, and center) and start/stop. For example, when the voice recognition algorithm accepted “up”, a cross 1003 was shown on top of the endoscopic camera view. Then, the system automatically selected the closest airway to the mark out of the airways detected by the trained 3cGAN model, and sent commands to the robotic catheter to bend the catheter toward the airway (see FIG. 20(b), which shows an example of the camera view in a case where voice recognition algorithm accepted “up”. The cross 1003 indicated in which direction was being selected and the line or segment 1004 showed the expected trajectory of the robotic catheter). [00183] Additionally or alternatively, any feature of the present disclosure may be used with features, including, but not limited to, training feature(s), autonomous navigation feature(s), artificial intelligence feature(s), etc., as discussed and referenced herein.

[00184] For specifying the robot’s movement direction, the target airway is identified based on its center proximity to the user-set marker visible as the cross or cross/plus sign 1003 in one or more embodiments as shown in FIGS. 19(a) - 19(b) (the cross may be any set or predetermined color, e.g., green or other chosen color). A direction vector may be computed from the center of the depth map to the center of this target detected airway. The vector may inform a 'virtual gamepad controller (or other ty pe of controller) and/or one or more processors, instigating or being responsible for the bending of the bronchoscopic tip. In one or more embodiments, the robot may advance in a straight line if this direction vector’s magnitude is less than 30% of the camera view’s width, which is called linear stage engagement (LSE). The process may repeat for each image frame received from the bronchoscopic camera without influence from previous frames. The bronchoscopic robot may maintain a set or predetermined/calculated linear speed (e.g., of 2 mm/s) and a set or predetermined/calculated bending speed e.g., of 15 deg/s).

[00185] Simultaneously, in one or more embodiments, the movements of the initial two sections (first and second sections) may be managed by the FTL motion algorithm, based on the movement history of the third section. During retraction, the reverse FTL motion algorithm may control all three sections, leveraging the combined movement history of all sections recorded during the advancement phase, allowing users to retract the robotic bronchoscope whenever necessary. By applying FTL, a most distal segment may be actively controlled with forward kinematic values, while a middle segment and another middle or proximal segment e.g., one or more following sections) of a steerable catheter or continuum robot move at a first position in the same way as the distal section moved at the first position or a second position near the first position. The FTL algorithm may be used in addition to the robotic control features of the present disclosure. For example, by applying the FTL algorithm, the middle section and the proximal section (e.g., following sections) of a continuum robot may move at a first position (or other state) in the same or similar way as the distal section moved at the first position (or other state) or a second position (or state) near the first position (or state) (e.g., during insertion of the continuum robot/ catheter, by using the navigation, movement, and/or control feature(s) of the present disclosure, etc.) . Similarly, the middle section and the distal section of the continuum robot may move at a first position or state in the same/ similar/ approximately similar way as the proximal section moved at the first position or state or a second position or state near the first position e.g., during removal of the continuum robot/catheter). Additionally or alternatively, the continuum robot/catheter may be removed by automatically and/or manually moving along the same or similar, or approximately same or similar, path that the continuum robot/catheter used to enter a target (e.g., a body of a patient, an object, a specimen (e.g., tissue), etc.) using the FTL algorithm, including, but not limited to, using FTL with the one or more control, depth map-driven autonomous advancement, or other technique(s) discussed herein. Other FTL features may be used with the one or more features of the present disclosure.

[00186] At least one embodiment of the pipeline of a workflow for one or more embodiments is shown in FIG. 20(a). For example, one or more embodiments may receive/obtain one or more bronchoscopic images (which may be input into a 3cGAN or any other Al-related architecture/structure for processing) such that a network (e.g., a neural network, a 3cGAN, a GAN, a convolutional neural network, any other Al architecture/structure, etc.) and/or one or more processors may estimate a depth map from the one or more bronchoscopic images. An airway detection algorithm or process may identify the one or more airways in the bronchoscopic image(s) and/or in the depth map (e.g., such as, but not limited to, using thresholding, blob detection, peak detection, and/ or any other process for identifying one or more airways as discussed herein and/or as may be set by a user and/or one or more processors, etc.). As aforementioned, the pixel 1002 may represent a center of a detected airway and the cross or plus sign 1003 may represent the desired direction determined by the user (e.g., moved using a drag and drop feature, using a touch screen feature, entering a manual command, etc.) and/or by one or more processors (see e.g., any of the processors discussed herein). The line or segment 1004 (which may also be of any set or predetermined color, such as, but not limited to, blue) may be the direction vector between the center of the image/depth map and the center of the detected blob closer or closest in proximity to the cross or plus sign 1003. The direction vector control command may decide between bending and insertion. The direction vector may then be sent to the robot’s control software by a virtual gamepad (or other controller or processor) which may initiate the autonomous advancement. As shown in FIG. 20(a), at least one embodiment may have a network estimate a depth map from a bronchoscopic image, and the airway detection algorithm(s) may identify the airways. The pixel 1002, the cross or plus sign 1003, and the line or segment 1004 may be employed in the same or similar fashion such that discussion of the subject features shown in FIGS. 19(a) -mi9(b) and FIG. 20(a) will not be repeated. Characteristics of models and scans for at least one study performed is shown in Table 1 below:

Table 1: Characteristics of Phantom and Ex vivo models and scans

[00187] Patient-derived phantoms and ex-vivo specimens/ animal model

[00188] Imaging and airway models: The experiments utilized a chest CT scan from a patient who underwent a robotic-assisted bronchoscopic biopsy to develop an airway phantom (see FIG. 21(b), under the IRB approval #2O2OPOO1835. FIG. 21(b) shows a robotic bronchoscope in the phantom having reached the location corresponding to the location of the lesion in the patient’s lung, using the proposed supervised-autonomous navigation. The 62-year-old male patient presented with a nodule measuring 21x21x16 [mm] in the right upper lobe (RUL). The procedure was smoothly conducted using the Ion Endoluminal System (Intuitive Surgical, Inc., Sunnyvale, CA), with successful lesion access (see FIG. 21(a) showing the view of the navigation screen with the lesion reached in the clinical phase). FIGS. 2i(ao - 21(b) illustrate a navigation screen for a clinical target location 125 in or at a lesion reached by autonomous driving and a robotic bronchoscope in a phantom having reached the location corresponding to the location of the lesion using one or more navigation features, respectively. Various procedures were performed at the lesion’s location, including bronchoalveolar lavage, transbronchial needle aspiration, brushing, and transbronchial lung biopsy. The procedure progressed without immediate complications. The inventors, via the experiment, aimed to ascertain whether the proposed autonomous driving method(s) would achieve the same clinical target (which the experiment confirmed that such method(s) would achieve the same clinical target) . Thus, one target in the phantom replicated the lesion’s location in the patient’s lung. Airway segmentation of the chest CT scan mentioned above was performed using ‘Thresholding’ and ‘Grow from Seeds’ techniques within 3D Slicer software. A physical/tangible mold replica of the walls of the segmented airways was created using 3D printing in ABS plastic. The printed mold w as later filled to produce the Patient Device Phantom using a silicone rubber compound, which w as left to cure before being removed from the mold.

[00189] The inventors, via the experiment, also validated the method features on two ex-vivo porcine lungs ith and without breathing motion simulation. A human Breathing motion w as simulated using an AMBU bag with a 2-second i nterval between the inspiration phases.

<Target and Geometrical Path Analysis> [00190] CT scans of the phantom and both ex-vivo lungs have been performed (see Table i) and airways were segmented using ‘Thresholding’ and ‘Grow from Seeds’ techniques in 3D Slicer.

[00191] The target locations were determined as the airways with a diameter constraint imposed to limit movement of the robotic bronchoscope. The phantom contained 75 targets, where ex-vivo lung #1 had 52 targets, and ex-vivo lung # 2 had 41 targets. The targets were positioned across all airways. This resulted in generating a total number of 168 advancement paths and 1163 branching points w ithout breathing simulation (phantom plus ex-vivo scenarios), and 93 advancement paths and 675 branching points with breathing motion simulation (BM) (ex-vivo) see Table 1). Each of the phantoms and specimens contained target locations in all the lobes.

[00192] Each target location was marked in the segmented model, and the Local Curv ature (LC) and Plane Rotation (PR) were generated along the path from the trachea to the target location and were computed according to the methodology described by Naito et al. (M. Naito, F. Masaki, R. Lisk, H. Tsukada, and N. Hata, International Journal of Computer Assisted Radiology and Surgery, vol. 18, no. 2, pp. 247-255 (2023), the disclosure of which is incorporated by reference herein in its entirety). LC was computed using the Menger curvature, which defines curvature as the inverse of the radius of the circle passing through three points in n-dimensional Euclidean space. To calculate the local curvature at a given point along the centerline, the Menger curvature was determined using the point itself, the fifteen preceding points, and the fifteen subsequent points, encompassing approximately 5 mm along the centerline. LC is expressed in [mm ], PR measures the angle of rotation of the airway branch on a plane, independent of its angle relative to the trachea. This metric is based on the concept that maneuvering the bronchoscope outside the current plane of motion increases the difficulty of advancement. To assess this, the given vector was compared to the current plane of motion of the bronchoscope. The plane of motion was initially determined by two vectors in the trachea, establishing a planethat intersects the trachea laterally (on the left-right plane of the human body). If the centerline surpassed a threshold of 0.75 [rad] (42 [deg]) for more than a hundred consecutive points, a new plane was defined. This approach allowed for multiple changes in the plane of motion along one centerline if the path indicated it. The PR is represented in [rad]. Both LC and PR have been proven significant in the success rate of advancement with user-controlled robotic bronchoscopes. In this study, the metrics of LC and PR have been selected as maximum values of the generated LC and PR outputs from the ‘Centerline Module’ at each branching point along the path towards the target location, and the maximum values were recorded for further analysis.

<In-vivo animal model >

[00193] An animal study was conducted as a part of the study approved by Mass General Brigham (Protocol number: 2021N000190). A 40 kg Yorkshire was sedated and tracheostomy was conducted. A ventilator (MODEL 3000, Midmark Animal Health, Versailles, OH) was connected to the swine model, and then the swine model was scanned at a CT scanner (Discovery MI, GE Healthcare, Chicago, IL). The swine model was placed on a patient bed in the supine position and the robotic catheter was inserted diagonally above the swine model. Vital signs and respiratory parameters w ere monitored periodically to assess for hemodynamic stability and monitor for respiratory distress. After the setting, the magnitude of breathing motion was confirmed using electromagnetic (EM) tracking sensors (AURORA, NDI, Ontario, Canada) embedded into the peripheral area of four different lobes of the swine model. FIG. 21(c) shows six consecutive breathing cycles measured by the EM tracking sensors as an example of the breathing motion.

[00194] Each trial of the semi-autonomous robotic advancement started with placing the robotic bronchoscope in the trachea in front of the carina. Then the operator, an engineer highly familiar with the softw are, started the program and the robot commenced the movement. From that point, the only action taken by the operator was to move (drag and drop) the green cross (see e.g., the cross 1003 in in the bronchoscopic image (see e.g., FIGS. 19(a) - 19(b) in the desired direction. The local camera coordinate frame was calibrated with the robot’s coordinate system, and the robotic software was designed to advance toward the detected airway closest to the green cross placed by the operator. One advancement per target was performed and recorded. If the driving algorithm failed, the recording was stopped at the point of failure.

[00195] The primary metric collected in this study was target reachability, defining the success in reaching the target location in each advancement. The secondary’ metric was success at each branching point determined as a binary measurement based on visual assessment of the robot entering the user-defined airway. The other metrics included target generation, target lobe, local curvature (LC) and plane rotation (PR) at each branching point, type of branching point, the total time and total path length to reach the target location (if successfully reached), and time to failure location together with airway generation of failure (if not successfully reached). Path length was determined as thelinear distance advanced by the robot from the starting point to the target or failure location.

[00196] The primary analysis performed in this study was the Chi-square test to analyze the significance of the maximum generation reached and target lobe on target reachability. Second, the influence of branching point type, LC and PR, and lobe segment on the success at branching points was investigated using the Chi-square test. Third, the Chi-square test was performed to analyze the difference in target reachability and success at branching points among the ex-vivo advancements w ith and without breathing motion simulation.

[00197] The inventors also hypothesized that the low local curvatures and plane rotations along the path increase the likelihood of success at branching points. It was also suspected that the breathing motion simulation is not going to decrease the success at branching points and, hence, total target reachability. In all tests, p-values of 0.05 or less were considered to be statistically significant. Pearson’s correlation coefficient was calculated for the linear regression analyses. All statistical analyses were performed using

Python version 3.7.

[00198] 1) Procedure: Two human operators with medical degrees (graduate of medical school and postgraduate year-3 in department of thoracic surgery) were tasked to nayigate the robotic catheter using the gamepad (Logitech Gamepad F310, Logitech, Lausanne, Switzerland) toward pseudo-tumors injected in each lobe before the study and ended the navigation when the robotic catheter reached within 20 mm from the pseudo-tumors. The human operators were allowed to move the robotic catheter forward, to retract, and to bend the robotic catheter toward any direction using the gamepad controller mapped with an endoscopic camera view. The robotic catheter was automatically bent during retraction using the reverse FTL motion algorithm.

[00199] During autonomous navigation, a navigator sending voice commands to the autonomous navigation randomly selected the airway at each bifurcation point for the robotic catheter to move in and ended the autonomous navigation when the mucus blocked the endoscopic camera view. The navigator was not allowed to change the selected airway before the robotic catheter moved into the selected airway, and not allowed to retract the robotic catheter in the middle of one attempt.

[00200] The navigation from the trachea to the point where the navigation yvas ended was defined as one attempt. The starting point of all attempts was set at 10 mm away from the carina in the trachea. To create the clinical scenario as accurate as possible during the study, the two human operators and the navigator sending voice commands yvere unaware that their input commands and force applied to each driving wire were recorded, and that the recorded data yvould be compared with each other after the study.

[00201] 2) Data collection: Time and force defined below were collected as metrics to compare the autonomous nay igation w ith the navigation by the human operators. All data points during retraction were excluded. When the robotic catheter was moved forward and bent at a bifurcation point, one data point was collected as an independent data point.

[00202] a) Time forbending command: Input commands to control the robotic catheter including moving forward, retraction and bending were recorded at too Hz. The time for bending command was collected as the summation of the time for the operator or autonomous navigation software to send input commands to bend the robotic catheter at a bifurcation point.

[00203] b) Maximum force applied to driving wire: Force applied to each driving w ire to bend the tip section of the robotic catheter was recorded at too Hz using a strain gauge (KFRB General-purpose Foil Strain Gage, Kyowa Electronic Instruments, Tokyo, Japan) attached to each driving wire. Then the absolute value of the maximum force of three driving wires at each bifurcation point was extracted to indirectly evaluate the interaction against the airway wall.

[00204] 3) Data analysis: First, box plots were generated for the time for bending command and the maximum force at each bifurcation point for human operators and autonomous navigation software. The medians w ith interquartile range (IQR) of the box plots were reported. Then the data points were divided into two locations of the lung, the central defined as the airway between the carina to the third generation and the peripheral area defined as the airway more than fourth generation. The medians with IQR at the central and the peripheral area for each operator type were reported. Mann-Whitney U test was performed to compare the difference between each operator type. P-values of 0.05 or less were considered to be statistically significant.

[00205] Second, scatter plots were generated for the both metrics with the airway generation, with regression lines and 95 % confidential intervals. The inventors analyzed the data using multiple regression models with time and force as response, generation number and operator type (human or autonomous), and their interaction as predictors. The inventors treated generation as a continuous variable, so that the main effect of operator type is the difference in intercepts between lines fit for each type, and the interaction term is the corresponding difference in slopes. The inventors tested the null hypothesis that both of these differences are simultaneously equal to zero using an F-test. The results show that the autonomous navigation keeps the catheter close to the center of the airway, which leads to a safer bronchoscopy and reduces/minimizes contact with an airway wall.

< Results >

[00206] The summary statistics are presented in Table 2 below:

Table 2: Results and Summary Statistics*

< Phantom Study>

[00207] The target reachability achieved in phantom was 73.3%. 481 branching points were tried in the phantom for autonomous robotic advancements. The overall success rate at branching points achieved was 95.8%. The branching points comprised 399 bifurcations and 82 trifurcations. The success rates at bifurcations and trifurcations were 97% and 92%, respectively. Statistical analysis using the Chi-square test revealed a significant difference (p=o.O3) between the two types of branching points in phantom (see FIGS. 22(a) - 22(c).

[00208] Furthermore, the success at branching points varied across different lobe segments, w ith rates of 99% for the left lower lobe, 93% for the left upper lobe, 97% for the right lower lobe, 85% for the right middle lobe, and 94% for the right upper lobe. The Chi- square test demonstrated a statistically significant difference (p=o.oos) in success at branching points between the lobe segments.

[00209] The average LC and PR at successful branching points were respectively 287.5 ± 125.5 [mm-¹] and 0.4 ±0.2 [rad]. The average LC and PR at failed branching points were respectively 429.5 ± 133.7 [mm-¹] and 0.9 ± 0.3 [rad]. The paired Wilcoxon signed- rank test showed no statistical significance of LC (p<o.ooi) and PR (p<o.ooi). Boxplots showing the significance of LC and PR on success at branching points are presented in FIGS. 23(a) - 23(b) together with ex-vivo data.

[00210] Using autonomous method features of the present disclosure, the inventors, via the experiment, successfully accessed the targets (as shown in FIG. 21(b). These results underscore the promising potential of our method(s) and related features of the present disclosure that may be used to redefine the standards of robotic bronchoscopy.

[00211] FIGS. 22(a) - 22(c) illustrate views of at least one embodiment of a navigation algorithm performing at various branching points in a phantom where FIG. 22(a) shows a path on which the target location (dot) was not reached e.g., the algorithm may not have traversed the last bifurcation where an airway on the right was not detected), where FIG. 22(b) shows a path on which the target location (dot) was successfully reached, and where FIG. 22(c) shows a path on which the target location was also successful reached. The highlighted squares represent estimated depth maps with detected airways at each visible branching point on paths toward target locations. The black frame (or a frame of another set/first color) represents success at a branching point and the frame of a set or predetermined color (e.g., red or other different/second color) (e.g., frame 1006 may be the frame of a red or different/second color as shown in the bottom right frame of FIG. 22(a) represents a failure at a branching point. All three targets were in RLL. Red pixel(s) (e.g., the pixel 1002) represent the center of a detected airway, green cross (e.g., the cross or plus sign 1003) represents the desired direction determined by the user (drag and drop), and the blue segment (e.g., the segment 1004) is the direction vector between the center of the image/depth map and the center of the detected blob in closer or closest proximity to the green cross (e.g., the cross or plus sign 1003).

[00212] FIGS. 23(a) illustrate graphs showing success at branching point(s) with respect to Local Curvature (LC) and Plane Rotation (PR), respectively, for all data combined in one or more embodiments. FIGS. 23(a) -23(b) show the statistically significant difference between successful performance at branching points w ith respect to LC (see FIG. 23(a) and PR (see FIG. 23(b). LC is expressed in [mm^-1] and PR in [rad].

< Ex-vivo Specimen/ animal Study>

[00213] The target reachability achieved in ex-vivo #1 was 77% and in ex-vivo #2 78% without breathing motion. The target reachability achieved in ex-vivo #1 was 69% and in ex-vivo #276% w ith breathing motion.

[00214] 774 branching points were tried in the ex-vivo#! and 583 in ex-vivo#! for autonomous robotic advancements. The overall success rate at branching points achieved was 97% in ex-vivo #1 and 97% in ex-vivo#! without BM, and 96% in ex-vivo #1 and 97% in ex-vivo#! with BM. The branching points comprised 327 bifurcations and 62 trifurcations in ex-vivo#i and 255 bifurcations and 38 trifurcations in ex-vivo#! without BM. The branching points comprised 326 bifurcations and 59 trifurcations in ex-vivo#i and 252 bifurcations and 38 trifurcations in ex-vivo#! with BM. The success rates without BM at bifurcations and trifurcations were respectively 98% and 92% in ex-vivo#i, and 97% and 95% in ex-vivo#!. The success rates with BM at bifurcations and trifurcations were respectively 96% and 93% in ex-vivo#!, and 96% and 97% in ex-vivo#!. Statistical analysis using the Chi-square test revealed a significant difference between the two types of branching points for both ex-vivo specimens (p = 0.03).

[00215] Furthermore, the success at branching points varied across different lobe segments, w ith rates (ex-vivo#!, ex-vivo#!) of (97%, 96%) for the LLL, (100%, 77%) for the LUL, (99%, 100%) for the RLL, (95%, 100%) for the RML, and (94%, 100%) for the RUL, without BM. With BM the results were as follows (ex-vivo#! with BM, ex-vivo#! with BM) of (96%, 97%) for the LLL, (100%, 50%) forthe LUL, (96%, 99%) for the RLL, (92%, 100%) for the RML, and (97%, 100%) for the RUL. The Chi-square test demonstrated a statistically significant difference (p < 0.001) in success at branching points between the lobe segments for all ex-vivo data combined.

[00216] The average LC and PR at successful branching points were respectively 211.9 ± 112.6 [mm-¹] and 0.4 ± 0.2 [rad] for ex-vivo#i, and 184.5 ± 110.4 [mm-¹] and 0.6 ± 0.2 [rad] for ex-vivo#2. The average LC and PR at failed branching points were respectively 393.7 ± 153.5 [mm-¹] and 0.6 ± 0.3 [rad] for ex-vivo# 1, and 369.5 ± 200.6 [mm-¹] and 0.7 ± 0.4 [rad] for ex-vivo#2. The paired Wilcoxon signed-rank test showed statistical significance of LC (p < 0.001) and PR (p < 0.001) for both ex-vivos on success at branching points. FIGS. 23(a) - 23(b) represent the comparison of LC and PR for successful and failed branching points, for all data (phantom, ex-vivos, ex-vivos with breathing motion) combined.

[00217] During the study, results of Local Curvature (LC) and Plane Rotation (PR) were displayed on three advancement paths towards different target locations with highlighted, color-coded values of LC and PR along the paths. Specifically, the views illustrated impact(s) of Local Curvature (LC) and Plane Rotation (PR) on one or more performances of one or more embodiments of a navigation algorithm where one view illustrated a path toward a target location in RML of ex vivo #1, which was reached successfully, where another view illustrated a path toward a target location in LLL of ex vivo #1, which was reached successfully, and where yet another view illustrated a path toward a target location in RLL of the phantom, which failed at a location marked with a square (e.p., a red square). [00218] The Chi-square test demonstrated no statistically significant difference (ex- vivo#!, ex-vivo#2) in target reachability (p = 0.37, p = 0.79) and success at branching points (p = 0.43, p = 0.8) between the ex-vivo advancements with and without breathing simulations (see e.g., FIGS. 24(a) - 24(c). These figures illustrate three advancement paths towards different target locations (see blue dots) using one or more embodiments of navigation feature(s) with and without BM. FIGS. 24(a) - 24(c) illustrate one or more impacts of breathing motion on a performance of the one or more navigation algorithm(s), where FIG. 24(a) shows a path on which the target location (e vivo #1 LLL) was reached with and without breathing motion (BM), where FIG. 24(b) shows a path on which the target location (ex vivo #1 RLL) was not reached without BM but was reached with BM (such as result illustrates that at times BM may help the algorithm(s) with detecting and entering the right airway for one or more embodiments of the present disclosure), and where FIG. 24(c) shows a path on which the target location (ex vivo #1 RML) w as reached without BM was not reached with BM (such a result illustrates that at times BM may affect performance of an algorithm in one or more situations. That said, the algorithms of the present disclosure are still highly effective under such a condition). The highlighted squares represent estimated depth maps with detected airways at each visible branching point on paths toward target locations. The black frame represents success at a branching point and the red frame represents a failure at a branching point.

< Statistical Analysis>

[00219] The hypothesis that the low local curvatures and plane rotations along the path might increase the likelihood of success at branching points was correct. Additionally, the hypothesis that breathing motion simulation will not impose a statistically significant difference in success at branching points and hence total target reachability was also correct.

< In-vivo animal study>

[00220] In total, 112 and 34 data points were collected from the human operators and autonomous navigation, respectively. Each human operator navigated the robotic catheter toward each pseudo-tumor injected into four different lobes and the autonomous navigation attempted five times, twdce toward LLL and RML, and one time toward RLL (Table 3), as follows:

Tables: Total attempts and lobes for each operator type:

[00221] 1) Time for bending command and maximum force at each bifurcation point: The median times for bending command were 2.5 [sec] (IQR = 1.0-5.6) and 1.3 [sec] (IQR = 0.7-2.3) for human operator and autonomous navigation, respectively. The Mann- Whitney U test showed statistically significant differences between human operators and autonomous navigation (FIG. 25(b). FIG. 25(a) illustrates the box plots for time for the operator or the autonomous navigation to bend the robotic catheter, and FIG. 25(b) illustrates the box plots for the maximum force for the operator or the autonomous navigation at each bifurcation point.

[00222] At the central area of the lung, the median times for bending command were 1.8 [sec] (IQR = 0.8-3.0) and 1.2 [sec] (IQR = 0.7-1.7) for human operator and autonomous navigation respectively, showing no statistically significant difference between operator type. At the peripheral area of the lung, the times for bending command were 2.9 [sec] (IQR = 1.2-7.1) and 1.4 [sec] (IQR = 0.7-2.8) for human operator and autonomous navigation respectively, showing the statistically significant differences between operator type (p = 0.030).

[00223] The medians of the maximum force at each bifurcation point were 2.8 (IQR = 1.1-3.8) [N] and 1.4 (IQR = 0.9-2.1) [N] for the human operators and autonomous navigation, respectively. The Mann-Whitney U test showed statistically significant differences between human operators and autonomous navigation (FIG. 25(b).

[00224] At the central area of the lung, the medians of the maximum force at each bifurcation point were 1.8 [N] (IQR = O.8-3.1) and 1.1 [N] (IQR = 0.9-1.4) for human operator and autonomous navigation respectively, showing no statistically significant difference between operator type. At the peripheral area of the lung, the medians of the maximum force at each bifurcation point were 3.1 [N] (IQR = 1.5-4.2) and 1-8 [N] (IQR = 1.2-2.5) for human operator and autonomous navigation respectively, showing the statistically significant differences between operator type (p = 0.005).

[00225] 2) Dependency on the airway generation of the lung: The dependency of the time and the force on the airway generation of the lung are shown in FIG. 27(a) and 27(b) with regression lines and 95% confidential intervals. For both metrics, the differences of the regression lines for each operator type become larger as the airway generation increases. FIGS. 27(a) and 27(b) show scatter plots for time to bend the robotic catheter (FIG. 27(a) and maximum force for a human operator and/or the autonomous navigation software (FIG. 27(b)), respectively. Solid lines showed the linear regression lines w ith 95% confidential intervals. While not required, jittering was applied on a horizontal axis for visualization.

[00226] There are statistically significant differences for both metrics due to operator (p = 0.006 for time, p < 0.001 for force). The null hypothesis for these tests assume not only that there is no difference in the generation slope, but also that there is no difference in the intercepts of the lines fit for the two operator types.

< Discussion >

[00227] The inventors have implemented the autonomous advancement of the bronchoscopic robot into a practical clinical tool, providing physicians with the capability to manually outline the robot’s desired path. This is achieved by simply placing a marker on the screen in the intended direction using the computer mouse (or other input device). While motion planning remains under physician control, both airway detection and motion execution are fully autonomous features. This amalgamation of manual control and autonomy is groundbreaking; according to the inventors’ knowledge, the methods of the present disclosure represent the pioneering clinical instrument facilitating airw ay tracking for supervised-autonomous driving within a target (e.y., the lung). To validate its effectiveness, the inventors assessed the performance of the driving algorithm(s), emphasizing target reachability and success at branching points. The rigorous testing encompassed a clinically derived phantom (in-vitro), two pig lung specimens (ex-vivo), and one live animal in-vivo), cumulatively presenting 168 targets. This comprehensive approach, and features discussed herein, sen e as the inventors’ response(s) to the obsened gaps in previous studies.

[00228] With the achieved performance, the presented supervised-autonomous drivi ng in the lung is proven to be clinically feasible. The inventors achieved 73.3% target reachability phantom, and 77% in ex-vivo #1 and 78% in ex-vivo #2 without breathing motion, and 69% and 76% with breathing motion. The overall success rate at branching points achieved in phantom was 95.8%, 97% in ex-vivo #1 and 97% in ex-vivo #2 without breathing motion, and 96% and 97% with breathing motion. The inventors inferred that the perpetuity of the anatomical airway structure quantified by LC and PR statistically significantly influences the success at branching points and hence target reachability. The presented method features show that, by using autonomous driving, physicians may safely navigate toward the target by controlling a cursor on the computer screen.

[00229] To evaluate the performance of the autonomous driving, the autonomous driving was compared with two human operators using a gamepad controller in a living swine model under breathing motion. Our blinded comparison study revealed that the autonomous driving took less time to bend the robotic catheter and applied less force to the anatomy than the navigation by human operator using a gamepad controller, suggesting the autonomous driving successfully identified the center of the airway in the camera view even with breathing motion and accurately moved the robotic catheter into the identified airway.

[00230] One or more embodiments of the present disclosure is in accordance with two studies that recently introduced the approach for autonomous driving in the lung (see e.g., J. Sganga, et al., RAL, pp. 1-10 (2019), which is incorporated by reference herein in its entirety, and Y. Zou, etal., IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598 (2022), which is incorporated by reference herein in its entirety). The first study reports 95% target reachability w ith the robot reaching the target in 19 out of 20 trials, but it is limited to 4 targets (J- Sganga, et al., RAL, pp. 1-10 (2019), which is incorporated by reference herein in its entirety). The only other performance metric the subject study presents is time necessary to reach the target, which is redundant not knowing the exact topological location of the target. The clinical origin of the validation lung phantom was not provided in that study. However, a robotic bronchoscope was used in the experiments of that study. The second study proposed a method for detecting the lumen center and maneuvering a manual bronchoscope by integrating it with a robotic device (Y. Zou, etal., IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598 (2022), which is incorporated by reference herein in its entirety). The subject study does not report any details on the number of targets, the location of the targets within lung anatomy, the origin of the human lung phantom, and the statistical analysis to identify the reasons for failure. The only metric used is the time to target. Both of these Sganga, et al. and Zou, et al. studies differ from the present disclosure in numerous w ays, including, but not limited to, in the design of the method(s) of the present disclosure and the comprehensiveness of clinical validation. The methods of those two studies are based on airway detection from supervised learning algorithms. In contrast, one or more methods of the present disclosure first estimate the bronchoscopic depth map using an unsupervised generative learning technique (A. Banach, F. King, F. Masaki, H. Tsukada, N. Hata, Medical image analysis, vol. 73, p. 102164 (2021), the disclosure of which is incorporated by reference herein in its entirety) and then perform standard image processing to detect the airw ays. Moreover, the clinical validation of the two studies (see e.g., J. Sganga, et al., RAL, pp. 1-10 (2019), which is incorporated by reference herein in its entirety, and Y. Zou, et al., IEEE Transactions on Medical Robotics and Bionics, vol. 4, no. 3, pp. 588-598 (2022), which is incorporated by reference herein in its entirety) is limited in vast contrast with, and w hen compared to, 261 advancements, breathing simulation and statistical analysis performed in the experiments of the present disclosure.

[00231] One or more embodiments of the presented method of the present disclosure may be dependent on the quality of bronchoscopic depth estimation by 3cGAN (see e.g., A. Banach, F. King, F. Masaki, H. Tsukada, N. Hata, “Medical image analysis, vol. 73, p. 102164 (2021), the disclosure of which is incorporated by reference herein in its entirety) or other Al-related network architecture used or that may be used (for example, while not limited hereto: in one or more embodiments, in a case where one or more processors train one or more models or Al-networks, the one or more trained models or Al-networks is or uses one or a combination of the following: a neural net model or neural network model, a deep convolutional neural network model, a recurrent neural network model with long short-term memory that can take temporal relationships across images or frames into account, a generative adversarial network (GAN) model, a consistent generative adversarial network (cGAN) model, a three cycle-consistent generative adversarial network (3CGAN) model, a model that can take temporal relationships across images or frames into account, a model that can take temporal relationships into account including tissue location(s) during pullback in a vessel and/or including tissue characterization data during pullback in a vessel, a model that can use prior knowledge about a procedure and incorporate the prior knowledge into the machine learning algorithm or a loss function, a model using feature pyramid(s) that can take different image resolutions into account, and/or a model using residual learning technique(s); a segmentation model, a segmentation model wdth post-processing, a model with pre-processing, a model with post-processing, a segmentation model with pre-processing, a deep learning or machine learning model, a semantic segmentation model or classification model, an object detection or regression model, an object detection or regression model with pre-processing or post-processing, a combination of a semantic segmentation model and an object detection or regression model, a model using repeated segmentation model technique(s), a model using feature pyramid(s), a genetic algorithm that operates to breed multiple models for improved performance, a model using repeated object detection or regression model technique(s); one or more other Al-networks or models known to those skilled in the art; etc.). One of the reasons for lack of success at branching points and hence missing the target may be that occasionally the depth estimation missed the airway when the airway was only partially visible in the bronchoscopic image. An example of such a scenario is presented in FIG. 26(a). FIGS. 26(a) - 26(d) illustrate one or more examples of depth estimation failure and artifact robustness that may be observed in one or more embodiments.

[00232] FIG. 26(a) shows a scenario where the depth map (right side of FIG. 26(a) was not estimated accurately and therefore the airway detection algorithm did not detect the airway partially visible on the right side of the bronchoscopic image (left side of FIG. 26(a). FIG. 26(b) shows a scenario where the depth map estimated the airways accurately despite presence of debris. FIG. 26(c) shows a scenario opposite to the one presented in FIG. 26(a) where the airway on the right side of the bronchoscopic image (left side of FIG. 26(c) is more visible and the airway detection algorithm detects it successfully. FIG. 26(d) shows a scenario where a visual artifact is ignored by the depth estimation algorithm and both visible airways are detected in the depth map.

[00233] Another possible scenario may be related to the fact that the control algorithm should guide the robot along the centerline. Dynamic LSE operates to solve that issue and to guide the robot towards the centerline when not at a branching point. The inventors also identified the failure at branching points as a result of lacking short-term memory, and that using short-term memory may increase success rate(s) at branching points. At branching point with high LC and PR, the algorithm may detect some of the visible airways only for a short moment, not leaving enough time for the control algorithm to react. In such scenarios, a potential solution would involve such short-term memory that ‘remembers’ the detected airways and forces the control algorithm to make the bronchoscopic camera ‘look around’ and make sure that no airways were missed. Such a ‘look around’ mode implemented between certain time or distance intervals may also prevent from missing airways that were not visible in the bronchoscopic image in one or more embodiments of the present disclosure.

[00234] In this work, the inventors developed and clinically validated the autonomous driving approach features in bronchoscopy. This is the first clinical tool providing airway tracking for autonomous driving in the lung and extensively validated on phantom and ex- vivo/ in-vivo porcine specimens. With the achieved performance, the presented method features for autonomous driving in the lung is/are proven to be clinically feasible and show the potential to revolutionize the standard of care for lung cancer patients.

<Systems, Methods, and Definitions >

[00235] The present disclosure and/or one or more components of devices, systems, and storage mediums, and/or methods, thereof also may be used in conjunction with continuum robot devices, systems, methods, and/or storage mediums and/or with endoscope devices, systems, methods, and/or storage mediums. Such continuum robot devices, systems, methods, and/or storage mediums are disclosed in at least: U.S. Pat. 11,882,365 , filed on February 6, 2022, the disclosure of which is incorporated by reference herein in its entirety. Such endoscope devices, systems, methods, and/ or storage mediums are disclosed in at least: U.S. Pat. Pub. 2022/0202502, filed on December 29, 2021, the disclosure of which is incorporated by reference herein in its entirety; and U.S. Pat. Pub. 2022/0202274, filed on December 29, 2021, the disclosure of which is incorporated by reference herein in its entirety. Any of the features of the present disclosure may be used in combination with any of the features as discussed in U.S. Prov. Pat. Pub. 2024/0112407, filed September 28, 2023, the disclosure of which is incorporated by reference herein in its entirety. Any of the features of the present disclosure may be used in combination with any of the features as discussed in U.S. Pat. Pub. No. 2023/0131269, published on April 26, 2023, the disclosure of which is incorporated by reference herein in its entirety. <AI Structures and Networks>

[00236] As aforementioned techniques of the present disclosure may be performed using artificial intelligence structure(s), such as, but not limited to, residual networks, neural networks, convolutional neural networks, GANs, cGANs, etc. In one or more embodiments, other types of Al structure(s) and/or network(s) may be used. The below discussed network/structure examples are illustrative only, and any of the features of the present disclosure may be used with any Al structure or network, including Al networks that are less complex than the network structures discussed below).

[00237] One or more processors or computers 128 (or any other processor discussed herein) may be part of a system in which the one or more processors or computers 128 (or any other processor discussed herein) communicate with other devices e.g., a database, a memory, an input device, an output device, etc.). In one or more embodiments, one or more models may have been trained previously and stored in one or more locations, such as, but not limited to, the memory, the database, etc. In one or more embodiments, it is possible that one or more models and/or data discussed herein (e.q., training data, testing data, validation data, imaging data, etc.) may be input or loaded via a device, such as the input device. In one or more embodiments, a user may employ an input device (which may be a separate computer or processor, a voice detector (e.g., a microphone), a keyboard, a touchscreen, or any other input device known to those skilled in the art). In one or more system embodiments, an input device may not be used e.g., where user interaction is eliminated by one or more artificial intelligence features discussed herein). In one or more system embodiments, the output device may receive one or more outputs discussed herein to perform coregistration, autonomous navigation, movement detection, control, and/or any other process discussed herein. In one or more system embodiments, the database and/or the memory may have outputted information {e.g., trained model(s), detected marker information, image data, test data, validation data, training data, coregistration result(s), segmentation model information, object detection/regression model information, combination model information, etc.) stored therein. That said, one or more embodiments may include several types of data stores, memory, storage media, etc. as discussed above, and such storage media, memory, data stores, etc. may be stored locally or remotely.

[00238] For regression model(s), the input may be the entire image frame or frames, and the output may be the centroid coordinates of a target, an octagon, circle (e.g., using circle fit) or other geometric shape used, one or more airways, and/or coordinates of a portion of a catheter or probe. Any of a variety of architecture of a regression model may be used. The regression model may use a combination of one or more convolution layers, one or more max-pooling layers, and one or more fully connected dense layers. The Kernel size, Width/Number of filters (output size), and Stride sizes of each layer may be varied dependent on the input image or data as well as the preferred output. Other hyperparameter search with, for example, a fixed optimizer and with a different w idth may be performed. One or more embodiments may use one or more features for a regression model as discussed in “Deep Residual Learning for Image Recognition” to Kaiming He, et al., Microsoft Research, December 10, 2015, which is incorporated by reference herein in its entirety. Other embodiments will use the features for a regression model as discussed in J. Sganga, et al., , “Autonomous Driving in the Lung using Deep Learning for Localization,” Jul. 2019 arxiv.org/abs/1907.08136vl, the disclosure of which is incorporated by reference herein in its entirety.

[00239] Since the output from a segmentation model, in one or more embodiments, is a “probability” of each pixel that may be categorized as a target or as an estimate (incorrect) or actual (correct) match, post-processing after prediction ria the trained segmentation model may be developed to better define, determine, or locate the final coordinate of catheter location and/or determine the autonomous navigation, movement detection, and/or control status of the catheter or continuum robot. One or more embodiments of a semantic segmentation model may be performed using the One- Hundred Layers Tiramisu method discussed in “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation” to Simon .Jegou, et al., Montreal Institute for Learning Algorithms, published October 31, 2017

(https://arxiv.org/pdf/1611.09326.pdf), which is incorporated by reference herein in its entirety. A segmentation model may be used. For example, by applying the One-Hundred Layers Tiramisu method(s), one or more features, such as, but not limited to, convolution, concatenation, transition up, transition down, dense block, etc., may be employed by slicing the training data set. While not limited to only or by only these embodiment examples, in one or more embodiments, a slicing size may be one or more of the following: 100 x 100, 224 x 224, 512 x 512. A batch size (of images in a batch) may be one or more of the following: 1, 2, 4, 8, 16, and, from the one or more experiments performed, a bigger batch size typically performs better e.g., with greater accuracy). The optimization of all of these hyper-parameters depends on the size of the available data set as well as the available computer/computing resources; thus, once more data is available, different hyperparameter values may be chosen. Additionally, in one or more embodiments, steps/ epoch may be 25, 50, 100, and the epochs may be greater than (>) 1000. In one or more embodiments, a convolutional autoencoder (CAE) may be used.

[00240] Further, the present disclosure and/or one or more components of devices, systems, and storage mediums, and/or methods, thereof also may be used in conjunction with continuum robotic systems and catheters, such as, but not limited to, those described in U.S. Patent Publication Nos. 2019/0105468; 2021/0369085; 2020/0375682; 2021/0121162; 2021/0121051; and 2022-0040450, each of which patents and/or patent publications are incorporated by reference herein in their entireties.

[00241] Although the disclosure herein has been described with reference to particular features and/or embodiments, it is to be understood that these features and/or embodiments are merely illustrative of the principles and applications of the present disclosure (and are not limited thereto), and the invention is not limited to the disclosed features and/or embodiments. It is therefore to be understood that numerous modifications may be made to the illustrative features and/or embodiments and that other arrangements may be devised without departing from the spirit and scope of the present disclosure. Indeed, the present disclosure encompasses and includes any combination of any of the feature(s) and/ or embodiment(s) (or component(s) thereof) discussed herein. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures, and functions.

Claims

1. An autonomous navigation robot system, comprising: a continuum robot; a camera at the distal end of the continuum robot; one or more actuators and/or motors to bend the distal end of the continuum robot and to move the continuum robot forward; a controller, the controller configured to: receive an image from the camera, define a position point in the image, determine a target point in the image based on a target path, and command the one or more actuators and/or motors, wherein, if the distance between the position point and the target point is less than a threshold value, the command is to move the continuum robot forward, and wherein, if the distance between the position point and the target point is more than the threshold value, the command is to bend the distal end of the continuum robot towards the target point.

2. The autonomous navigation robot system of claim 1, wherein the position point is the center of the image received from the camera.

3. The autonomous navigation robot system of claim 1, wherein the target point is a center of a circle that indicates a lumen as the target path.

4. The autonomous navigation robot system of claim 1, wherein the controller further comprising repeating the steps of receiving an image, defining a position point, and commanding the one or more actuators until a predetermined insertion depth is reached.

5. The autonomous navigation robot system of claim 4, wherein the controller further comprises: determining whether the continuum robot has reached a predetermined insertion depth, and stopping the movement and/or bending when the predetermined insertion depth is reached.

6. The autonomous navigation robot system of claim 1, wherein the threshold value is adjustable.

7. The autonomous navigation robot system of claim 5, wherein the threshold value is adjusted to require increasingly accurate bending before moving forward as the continuum robot progresses through a lumen.

8. The autonomous navigation robot system of claim 1, wherein a speed of bending and a speed of forw ard movement are adjustable.

9. The autonomous navigation robot system of claim 1, further comprising a user input device that, w hen activated, stops the controller from moving or bending towards the target point without further user input.

10. The autonomous navigation robot system of claim 1, wherein the controller can adjust the frame rate of images from the camera.

11. The autonomous navigation robot system of claim 1, wherein the threshold is set at betw een 20 percent and 40 percent of the diagonal length of the camera image.

12. The autonomous navigation robot system of claim 1, wherein the target path is a path in an airway, and the continuum robot is a bronchoscope.

13. An information processing apparatus to control a continuum robot comprising: at least one memory’ storing instructions; and at least one processor that executes the instructions stored in the memory to cause the information processing apparatus to perform: receiving an image, determining a target point in the image based on a target path; determining whether or not a distance from the position point to the target point in the image is more or less than a threshold value; wherein, in a case where the distance is less than the threshold value, the processor controls the continuum robot to advance, and in a case where the distance is more than the threshold value, the processor controls the continuum robot to bend so that the distance become less.

14. The information processing apparatus of claim 13, wherein the image is an image from within an airw ay, and the target point is a point in an image that is a center of the airway.

15. The information processing apparatus of claim 13, w herein the processor further performs: determining the speed of a bending of the continuum robot based on the distance.

16. The information processing system of claim 13, wherein the processor further performs: determining the speed of an advancement of the continuum robot is determined based on the determined distance.

17- The information processing system of claim 13, wherein the threshold value is determined based on a position of the continuum robot in a lumen.

18. The information processing system of claim 13, wherein the processer performs: aborting a bending or an advancement of the continuum robot in a case where a predetermined object is detected in the image.

19. The information processing system of claim 13, wherein the processer further performs: receiving an instruction from a user to sw itch mode of controlling continuum robot from an autonomous driving mode to a manual driving mode, wherein in the autonomous driving mode, a control to advance or bend the continuum robot is performed based on the distance and the threshold, and in the manual driving mode, the control of the continuum robot on the distance and the threshold is not performed.

20. The information processing system of claim 13, wherein a speed of bending and/or a speed of advancing the continuum robot is determined based on a framerate of images received from a camera.

21. The information processing system of claim 13, wherein the determination whether or not the distance is less than the threshold distance is performed frame by frame from a plurality of images.

22. The information processing system of claim 13, wherein the continuum robot is a steerable catheter with an imaging unit at the distal end of the catheter.

23. A non -transitory computer-readable storage medium storing at least one program for causing a computer to execute a method for controlling a continuum robot, the method comprising: receiving an image, determining a target point in the image based on a target path; determining whether or not a distance from the position point to the target point in the image is more or less than a threshold value; wherein, in a case where the distance is less than the threshold value, causing the continuum robot to advance, and in a case where the distance is more than the threshold value, the processor controls the continuum robot to bend so that the distance become less.

24. A method of using the autonomous navigation robot system of claim 1.

25. An autonomous navigation robot system, comprising: a steerable catheter; a camera at the distal end of the steerable catheter; one or more actuators to steer and move the steerable catheter; a user input device; and a controller, the controller configured to: in a perception step: receive camera view; identify path candidates in the camera view by processing the camera view; and determine paths among path candidates with computation, in a planning step: determine target paths among the path candidates based on concurrent user instruction from user input device and/or pre-operative instruction, in a control step: compute the commands to actuator based on the target paths and the camera view and/or the current posture of the steerable catheter and command the actuator to move the steerable catheter, wherein the actuators bend and move the steerable catheter automatically.