CN115803798A - Systems and methods for training users to operate a teleoperation system - Google Patents

Abstract

A system includes a user control system including at least one input control device for controlling movement of a virtual medical instrument through a virtual passageway. The system also includes a display for displaying a graphical user interface and a plurality of training modules. The graphical user interface includes a representation of the virtual medical instrument and a representation of the virtual pathway. The system also includes a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more computer processors. The instructions for performing operations comprise: navigating the virtual medical instrument through the virtual pathway based on a command received from the user control system; and evaluating one or more performance metrics for tracking the navigation of the virtual medical instrument through the virtual pathway.

Description

Systems and methods for training users to operate a teleoperation system

Cross References to Related Applications

This application claims priority to and benefit of U.S. Provisional Application No. 63/058,228, filed July 29, 2020, which is hereby incorporated by reference in its entirety.

technical field

The present disclosure relates to systems and methods for training a user to operate a teleoperation system, and more particularly to training a user to operate a teleoperation system through the use of a simulator system.

Background technique

Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and unwanted side effects. Such minimally invasive techniques may be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert minimally invasive medical devices, including surgical, diagnostic, therapeutic, or biopsy instruments, to reach target tissue sites. One such minimally invasive technique is the use of a flexible and/or steerable elongated device, such as a catheter, that can be inserted into an anatomical pathway and navigated to a region of interest within the patient's anatomy. Control of such elongated devices by medical personnel involves management of several degrees of freedom, including at least management of insertion and retraction of the elongated device and manipulation of the device. In addition, different operating modes are supported.

Accordingly, it would be advantageous to provide a system for training a user, such as a surgeon, to use a telesystem with features that support a flexible and/or steerable elongated device, such as a steerable catheter. Input controls for intuitive control and management, the slender device is suitable for use during minimally invasive medical techniques. It would be further advantageous to train the system to simulate movement of input controls and simulate a graphical user interface that may be used by a surgeon during a minimally invasive medical procedure.

Contents of the invention

Embodiments of the invention are best summarized by the claims of the accompanying specification.

According to some embodiments, a system is provided. The system includes a user control system including at least one input control device for controlling movement of a virtual medical instrument through a virtual pathway. The system also includes a display for displaying a graphical user interface and a plurality of training modules. The graphical user interface includes a representation of the virtual medical instrument and a representation of the virtual pathway. The system also includes a non-transitory computer readable storage medium storing a plurality of instructions executable by one or more computer processors. The instructions for performing operations include training a user to navigate a medical device through the virtual pathway. The instructions for performing operations also include determining a performance metric for tracking navigation of the virtual medical instrument through the virtual pathway.

Other embodiments include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the described methods.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide understanding of the disclosure and not to limit the scope of the disclosure. In this regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description.

Description of drawings

Figure 1A illustrates a simulator system including a user control system and a computing device, according to some embodiments.

Figure 1B shows a top view of a user control system according to some embodiments.

Figure 2A illustrates a modular graphical user interface that may be displayed on a display device, according to some embodiments.

Figure 2B illustrates a training exercise graphical user interface that may be displayed on a display device, according to some embodiments.

3A-3E illustrate various training exercises using various virtual pathways, according to some embodiments.

Figure 4 illustrates a set of instructions for performing a training exercise, according to some embodiments.

Figure 5 illustrates a method for tracking user performance of a training exercise, according to some embodiments.

6 illustrates a training exercise that may be displayed on a display device including a global view of a virtual pathway and a view from a distal tip of a virtual instrument, according to some embodiments.

7A-7G illustrate various training exercises using various virtual pathways, according to some embodiments.

8 illustrates an exercise that may be displayed on a display device, including a view from the distal tip of a virtual instrument and a contact indicator, according to some embodiments.

9A-9B illustrate a training exercise including performance metrics regarding user control of a virtual appliance, according to some embodiments.

Figure 10 illustrates a profile summary including performance metrics, according to some embodiments.

Figure 11 illustrates a graphical user interface that may be displayed on a display device in accordance with some embodiments.

Figure 12 is a simplified diagram of a computer-aided teleoperation system in accordance with some embodiments.

Embodiments of the present disclosure and advantages thereof are described in the following detailed description. It should be understood that like reference numerals are used to identify like elements in one or more of the figures for purposes of illustrating rather than limiting embodiments of the present disclosure.

Detailed ways

In the following description, specific details describe some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not limiting. Those skilled in the art may recognize other elements not specifically described but within the scope and spirit of the disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically stated otherwise or if one or more features would cause the embodiment doesn't work. In some instances, well-known methods, procedures, components and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

The emulator system can assist in accelerating user learning and improving user performance of the teleoperation system. The simulator system allows a user (eg, surgeon, clinician, practitioner, nurse, etc.) to become familiar with the controls of the teleoperated user control system. The emulator system also allows the user to become familiar with the teleoperation system's graphical user interface (GUI). Thus, during a medical procedure on a patient, the user may practice operating the telesystem via the simulator system prior to operating the telesystem. The simulator system may provide the user with a training module that teaches the user to effectively navigate challenging patient anatomy by navigating a virtual instrument, such as a virtual medical instrument (eg, a virtual endoscope) through a virtual pathway. Performance metrics can be tracked to assess user performance and further aid user training.

FIG. 1A shows system 100 including computing system 110 (which may be a computing device), computing system 120 (which may be a computing device), and user control system 130 . FIG. 1B is a top view of user control system 130 . Computing system 110 includes a display device 112 , which may include a display screen, and an optional stand 114 . Computing system 110 may include a processing system 116 including one or more processors. Computing system 110 may include power components, communication components (e.g., transmitters, receivers, transceivers) for receiving and/or transmitting data, memory/storage components for storing data, and/or for supporting computing system 110 function of other components (not shown). In some embodiments, computing system 110 is a monitor, but could be any other suitable computing system, such as a television, remote computing device (eg, laptop or mobile phone), or the like. Computing system 120 includes a display device 122 that may include a display screen. Computing system 120 may include a processing system 126 including one or more processors. Computing system 120 may include power components, communication components (e.g., transmitters, receivers, transceivers) for receiving and/or transmitting data, memory/storage components for storing data, and/or for supporting computing system 120 function of other components (not shown). In some embodiments, computing system 120 is a remote computing device (eg, laptop, mobile phone, etc.), but may be any other suitable computing system, such as a monitor, television, or the like.

Although the following discussion may be made with respect to one display device (eg, display device 122), the discussion is similarly applicable to another display device (eg, display device 112). For example, any content displayed on display device 122 may additionally or alternatively be displayed on display device 112 . In some examples, display devices 112, 122 may operate in the same manner and/or may include similar features. For example, one or both of display devices 112, 122 may include a touch screen.

Additionally or alternatively, computing system 110 may include an image capture device 118 (eg, a camera) to track a user's gaze as the user operates user control system 130 . For example, camera 118 may track the user's gaze, and processing system 116 may determine whether the user is looking at display screen 112 or display screen 122 . Additionally or alternatively, computing system 120 may include an image capture device 128 (eg, a camera) to track a user's gaze as the user operates user control system 130 . For example, camera 128 may track the user's gaze, and processing system 126 may determine whether the user is looking at display screen 112 or display screen 122 .

As shown in FIGS. 1A and 1B , user control system 130 includes housing 132 , input control device 134 , input control device 136 , status buttons 138 and spine 140 . In some embodiments, input control device 134 may be a scroll wheel and input control device 136 may be a trackball. A state button 138 may be used to control the state of the virtual appliance (eg, passive or active). In some embodiments, a spine 140 may be included to ergonomically support the user's arm/wrist while the user operates the user control system 130 . Any other ergonomic features may additionally or alternatively be included on user control system 130 . In some examples, input control device 134 has infinite travel and can rotate in either direction (eg, forward and backward). In some cases, input control device 136 has infinite travel and is rotatable about any number of axes. In some examples, the most common movement of the input control device 136 may be a combination of left and right rotation, forward rotation, and in-place rotation. In alternative embodiments, one or both of the input control devices 134, 136 may be a touch pad, joystick, touch screen, or the like.

In some examples, user control system 130 may be communicatively coupled to computing system 120 through wireless and/or wired connections. In such examples, computing system 120 may also be communicatively coupled to computing system 110 through wireless and/or wired connections. In some cases, user control system 130 may be coupled to computing system 110 via computing system 120 . In other embodiments, user control system 130 may be directly coupled to computing system 110 through wireless and/or wired connections. As will be described in further detail below, a user (eg, surgeon, clinician, nurse, etc.) may interact with one or more of computing system 110 , computing system 120 , and user control system 130 to control the virtual instrument. In some examples, the virtual instrument is a virtual medical instrument.

FIG. 2A shows a dynamic graphical user interface (GUI) 200 . GUI 200 may be displayed on display device 112, display device 122, or both. GUI 200 includes a plurality of module icons 210A-E. Each module icon 210A-210E may represent at least one module. Modules may be implemented as software executable by one or more processors of system 100. One or more modules may include one or more modules designed to familiarize users (e.g., surgeons, clinicians, nurses, etc.) with teleoperation systems. a training exercise. These exercises may provide simulations that allow a user to manipulate a virtual instrument through various virtual pathways and/or toward various virtual targets. These exercises allow the user to practice using the telesystem before using it in a medical procedure. In some embodiments, system 100 may present five training modules—an introduction module represented by module icon 210A, a basic drive 1 module represented by module icon 210B, a basic drive 2 module represented by module icon 210C, a basic drive 2 module represented by module icon 210D An airway drive 1 module is indicated and an airway drive 2 module is represented by the module icon 210E. In other embodiments, the system 100 may provide more than five or fewer than five training modules (eg, one module, two modules, three modules, four modules, six modules, seven modules, etc.). System 100 may exhibit any one or more of the modules listed above or may include any other modules not listed above. In other examples, module icons 210A-210E may represent any one or more of the above-listed modules and/or any other modules not listed. Additionally or alternatively, one or more module icons may represent a module.

Modules can be sorted based on difficulty. In some examples, the difficulty of a module may be based on the complexity of the drive path through the virtual pathway. In other examples, the difficulty of a module may be based on whether multiple control inputs are required, which may be entered via the input control devices 134, 136 while the virtual instrument traverses the virtual pathway. For example, a block that requires multiple control inputs may be more difficult than a block that requires one. Additionally or alternatively, the difficulty of a module may be based on the complexity of the control inputs. In other examples, the difficulty of a module may be based on a target time to complete the module. For example, a module with a short target completion time may be more difficult than a module with a long target completion time. Difficulty may be based on any combination of the aforementioned factors and/or any other similar factor or combination of factors. Additionally or alternatively, modules may be ordered based on one or more user learning objectives. In some examples, user learning objectives may include basic concepts (e.g., operating input control devices 134, 136, driving a virtual instrument through a relatively straight virtual path, etc.), complex concepts (e.g., driving a virtual instrument through a curved virtual path, navigating patient's virtual anatomical model, etc.), muscle memory, cognition, etc. Each module can include one or more user learning objectives.

In some cases, the Airway Drive 2 mod can be the hardest mod to complete compared to other mods. Thus, the Airway Drive 2 module can be harder than the Airway Drive 1 module, which can be harder than the Basic Drive 2 module, which can be harder than the Basic Drive 1 module, which can be harder than the Introductory module Harder. The user may be prompted to complete the modules in order of difficulty (eg, easiest to hardest), starting with the introductory module and ending with the Airway Drive 2 module. In other examples, the user may complete the modules in any order. In some examples, each module can be repeated any desired number of times. In an alternative embodiment, each module becomes available only after the user completes the previous module. For example, the Basic Drive 1 module may only be available after the user completes the introductory module. In further embodiments, a subset of modules may become available when a previous subset of modules is complete. For example, the Airway Drive 1 and 2 modules may only be available after the user completes the Basic Drive 1 and 2 modules.

As shown in FIG. 2A, each module icon 210A-210E includes a title 212A-212E indicating the general topic covered by each respective module. Each module icon 210A-210E may also include a status indicator, such as a status bar 214A-214E. As seen in FIG. 2A , the status column 214A is, for example, completely filled, which may indicate that each exercise within the introductory module has been completed. As further seen in FIG. 2A , the status bar 214B is partially filled, which may indicate that some, but not all, of the exercises within the Basic Drive 1 module have been completed. Status column 214C is empty, which may indicate that none of the exercises within the Basic Drive 2 module have started and/or completed. In some examples, one or more of the module icons 210A-210E may also include a time indicator 216A-216E. Each time indicator 216A-216E may show the estimated total time it may take the user to complete all the exercises within the module. For example, time indicator 216A may indicate that the user will take approximately 30 seconds to complete all of the exercises in the introduction module. In an alternative embodiment, each time indicator 216A-216E may show an estimated time it may take the user to complete the next available exercise in each module.

In some examples, display screen 122 may be a touch screen. In such examples, the user may select the module icon 210A, eg, by touching the module icon 210A on the display screen 122 . In other embodiments, the user may select module icon 210A using a stylus, a mouse controlling a cursor on display 122, and/or by any other suitable method (eg, voice activation, eye tracking, etc.). Any of the module icons 210A-210E may be selected using any one or more of the selection methods described above. Additionally or alternatively, display screen 112 may be a touch screen. In such examples, module icons 210A- 210E may be displayed on display screen 112 , and a user may select module icon 210A by, for example, touching module icon 210A on display screen 112 . In other embodiments, the user may select module icon 210A using a stylus, a mouse controlling a cursor on display 112, and/or by any other suitable method (eg, voice activation, eye tracking, etc.). Any of the module icons 210A-210E may be selected using any one or more of the selection methods described above.

In some embodiments, GUI 200 may also include icon 220, which may be a quick launch icon. The quick launch icon 220 may indicate the next suggested exercise set to be completed by the user. For example, if the user has completed Exercise 1 of the Basic Driving 1 module, the next exercise the user can complete is Exercise 2 of the Basic Driving 1 module. If the user exits the Basic Driver 1 module and returns to the GUI 200 (eg, "Home Screen"), the user can directly launch Exercise 2 of the Basic Driver 1 module by selecting the quick launch icon 220 . The quick launch icon 220 may provide the user with a quicker access path to select the next suggested exercise, rather than navigating to a specific module and then to a specific exercise.

GUI 200 may also include user identification information 230 . User identification information 230 may indicate which user is logged into one or both of computing systems 110 , 120 . In some embodiments, each user is associated with his or her own personal profile, which includes a unique login associated with each profile. Computing system 110 and/or computing system 120 may include any number of logins/user profiles associated with any number of users. Thus, more than one user may be logged into the computing system 110,120. In some embodiments, only one user may be logged in at a time. In other embodiments, multiple users may be logged into the same system at the same time. In some examples, a user may log in to computing system 120 using his or her profile to access modules within computing system 120 . Once a user is logged in, user identification information 230 may indicate that the user is logged in (eg, by including the user's name, username, profile ID, etc. on GUI 220). Users may log in and log out of computing system 120 at any time. If the user logs out without completing all modules/exercises, the user's progress can be saved and called when the user logs in again. This allows the user to continue completing modules/exercises without repeating modules/exercises that the user has already completed. In other examples, if the user has completed all modules/exercises, the user may log in again to repeat any one or more of the modules/exercises.

Each of the modules represented by module icons 210A-E may include multiple training exercises. For example, upon selection of module icon 210A, display screen 122 displays dynamic GUI 250, as shown in FIG. 2B. GUI 250 includes a plurality of training exercise icons 260A-E. Each exercise icon 260A-E may represent at least one training exercise. In some embodiments, exercise icons 260A-E may form a list of exercises included within the introductory module. GUI 250 may include a module identifier 270 to indicate which module the user has selected. In FIG. 2B, module identifier 270 indicates that the user has selected an introduction module, which the user can access by selecting module icon 210A. Accordingly, the GUI 250 shown in FIG. 2B illustrates the exercises included within the introductory module. In some embodiments, the introductory module may include five exercises—Exercise 1, Exercise 2, Exercise 3, Exercise 4, and Exercise 5. The number and type of exercises within each module may vary. For example, an introductory module may include more or fewer than five exercises (eg, one exercise, two exercises, three exercises, four exercises, six exercises, or any other number of exercises). In some examples, exercise icon 260A represents exercise 1 , exercise icon 260B represents exercise 2 , exercise icon 260C represents exercise 3 , exercise icon 260D represents exercise 4 , and exercise icon 260E represents exercise 5 . In other examples, exercise icons 260A-E may represent any one or more of the exercises listed above and/or any other exercises not listed. Additionally or alternatively, one or more of exercise icons 260A-E may represent more than one exercise.

Each exercise icon 260A-E may include a corresponding status indicator 262A-262E. Status indicators 262A-E may show whether a particular exercise has been completed. For example, status indicator 262A may be a check mark or any other symbol indicating that exercise is completed, and may indicate that Exercise 1 is complete. Additionally, in some examples, when the exercise is complete, a replay icon 264A may be included within the exercise icon (eg, exercise icon 260A) corresponding to the completed exercise. By selecting the replay icon 264A, the user can repeat Exercise 1 . Status indicator 262B may be a symbol representing an incomplete exercise (eg, intertwined rings, an "X", etc.), and may indicate that exercise 2 has not been completed. Because Exercise 2 has not yet been completed, exercise icon 260B may not include a replay icon. In some embodiments, a user may complete the exercises in any order, and each exercise may be repeated any number of times desired. In an alternative embodiment, each exercise becomes available only after the user completes the previous exercise. For example, Exercise 2 might only be available after the user completes Exercise 1. In further embodiments, a subset of exercises may become available when a previous subset of exercises is completed. For example, exercises 4 and 5 may only be available after the user completes exercises 1-3.

3A-3E illustrate portions of various training exercises, according to some embodiments. As shown in FIG. 3A, the display screen 112 shows a dynamic GUI 300 for insertion/retraction exercises. The insertion/retraction exercise may be the first exercise in an introductory module represented by module icon 210A. The insertion/retraction exercise may be activated when the user selects the first exercise of the introductory module. The goal of the introduction module is to familiarize the user with the user control system 130 . For example, an introduction module may teach a user how to operate the user control system 130 to control a virtual appliance. As discussed above with respect to FIG. 2B , the user can activate the introduction module by selecting the module icon 210A on the display screen 122 .

In some embodiments, the user may select Exercise 1 of the introductory module by selecting Exercise icon 260A. In some embodiments, the insertion/retraction exercise GUI 300 may be displayed on the display screen 112 when the user activates exercise 1 of the introductory module. GUI 300 may provide training for using input control device 134 . As discussed above, the input control device 134 can be scrolled forward and backward to control the insertion/retraction of the virtual instrument.

As shown in FIG. 3A , when the insertion/retraction exercise is activated, the display screen 112 displays the lumen 310 of the virtual pathway 315 defined by the surface 320 . In the embodiment shown in FIG. 3A, the lumen 310 has a rectangular cross-section, but in other embodiments, the lumen 310 may have a different cross-sectional shape, such as a circular cross-section. Target 340 is included within distal portion 330 of virtual pathway 315 . In some examples, for example, opening 335 at the end of virtual pathway 315 may enlarge when the user scrolls input control device 134 forward (representing an insertion motion of the virtual instrument). As the opening 335 gets bigger, the target 340 can then get bigger. This may give the user a sense that the virtual instrument is moving towards the target 340 as it approaches the target 340 . In some embodiments, when the virtual instrument reaches the target 340 , the display screen 112 may display an effect indicating that the virtual instrument has reached the target 340 . For example, display screen 112 may change the display of target 340, such as by causing target 340 to explode outward, target 340 to explode inward, change the opacity of target 340, change the color of target 340, and the like. Additionally or alternatively, one or more other effects may be used when the virtual instrument reaches the target 340, such as an audio signal, a textual indicator on the display screen 112, a notification to the user via the input control device 134 and/or the user control system 130. Provide haptic feedback and/or any other similar effects. In other examples, when the user scrolls the input control device 134 backward (representing a retracting motion of the virtual instrument), the opening 335 may become smaller as the virtual instrument recedes away from the target 340 . As opening 335 gets smaller, target 340 can then get smaller.

In some embodiments, the user may select Exercise 2 of the introductory module by selecting exercise icon 260B. Exercise 2 of the introductory module may be an instrument bending exercise. In some embodiments, a portion of the dynamic GUI 350 for the machine bending exercise may be displayed on the display screen 112 when the user activates the second exercise of the introductory module. GUI 350 provides training for the use of input control device 136 .

As shown in FIG. 3B , GUI 350 on display screen 112 displays virtual instrument 360 including distal portion 362 when the bending exercise is activated. In some examples, when a user scrolls input control device 136 in one direction, distal portion 362 of virtual instrument 360 bends on display screen 112 in a corresponding direction. The input control device 136 can be scrolled to actuate the virtual instrument in yaw (left and right) and pitch (up and down). For example, if the user scrolls the input control device 136 to the left (eg, in direction D1 ), the distal portion 362 of the virtual instrument 360 bends to the left. GUI 350 also includes a set of directional arrows 370 that indicate in which direction the user should scroll input control device 136 . As shown in FIG. 3B , directional arrow 370 points in the D1 direction, thereby indicating that the user should scroll the input control device 136 in the D1 direction. Progress indicator 372 shows how far the user has scrolled input control device 136 in direction D1. For example, progress indicator 372 may be shown by shading in one or more of directional arrows 370, as shown in FIG. 3A. In other examples, progress indicator 372 may be shown as a pattern, color, or any other visual indicator displayed on one or more of directional arrows 370 . In further examples, the progress indicator 372 may be a non-visual indicator, such as an audible indicator, a tactile indicator, or the like. As the user continues to scroll input control device 136 in direction D1 , progress indicator 372 may extend along directional arrow 370 , eventually reaching goal 380 . Progress indicator 372 may be a color, pattern, or any other similar indicator that may extend along, within, on, above, or below progress indicator 372 . In addition to direction D1, progress indicator 372 may also point in any other direction.

The virtual appliance 360 may be considered to have "reached" the target 380 when the user has rolled the input control device 136 a threshold distance in the direction D1. The display screen 112 may display an effect indicating that the virtual instrument 360 has “reached” the target 380 . For example, target 380 may glow/change color. Additionally or alternatively, one or more other effects may be used when virtual appliance 360 "reaches" target 380, such as an audio signal, a textual indicator on display screen 112 showing the effect (e.g., target 380 exploding outward, exploding inward, fading, disappearing, etc.), the user receiving haptic feedback through the input control device 136 and/or the user control system 130, and/or any other similar effect.

In some embodiments, after virtual instrument 360 "reaches" target 380, distal portion 362 stops bending even though the user continues to scroll input control device 136 in direction D1. In an alternative embodiment, as the user scrolls the input control device 136 in the direction D1 , the distal portion 362 of the virtual instrument 360 may continue to bend in the direction D1 beyond the target 380 .

In some embodiments, the user may select Exercise 3 of the introductory module by selecting exercise icon 260C. Exercise 3 of the introductory module may be a linear navigation exercise. In some embodiments, a portion of the dynamic GUI 400 for the linear navigation exercise may be displayed on the display screen 112 when the user activates exercise 3 of the introductory module. Linear navigation exercise GUI 400 provides training for using input control device 134 and input control device 136 at the same time.

As shown in FIG. 3C , the display screen 112 displays a linear navigation exercise GUI 400 comprising a first portion 400A and a second portion 400B. In some embodiments, first portion 400A shows a global perspective view of virtual elongated device 410 (eg, which may be a virtual catheter), virtual instrument 412 and virtual pathway 420 . As shown in FIG. 3C , virtual instrument 412 may extend from virtual catheter 410 . Dummy instrument 412 includes distal portion 414 . In some examples, second portion 400B shows a view from the distal tip of virtual instrument 412 . As the virtual instrument 412 traverses the virtual pathway 420, both the first portion 400A and the second portion 400B may be updated in real time. In some examples, first portion 400A may be displayed on display 112 alone, or second portion 400B may be displayed on display 122 alone. In other examples, the first part 400A and the second part 400B may be simultaneously displayed on the display screen 112 in a split-screen form as shown in FIG. 3C .

In a linear navigation exercise using GUI 400 , GUI 400 may provide training to teach the user to navigate virtual instrument 412 through virtual pathway 420 . In some examples, virtual pathway 420 is defined by a plurality of sequentially aligned virtual rings 420A-420C. In some embodiments, rings 420A-420C may be linearly aligned. A linear navigation exercise may be completed as the distal portion 414 of the virtual instrument 412 traverses through each of the loops 420A-420C. In some examples, system 120 and/or system 110 determines that distal portion 414 successfully traverses virtual pathway 420 when distal portion 414 passes through and/or contacts each loop 420A- 420C. In some embodiments, when the distal portion 414 passes through and/or contacts each ring 420A-420C, an effect is presented indicating that the distal portion 414 passes through and/or contacts each ring 420A-420C. For example, display 112 may show effects (e.g., each ring 420A-420C explodes outward, explodes inward, fades, disappears, etc.), an audio signal may be played, display 112 may display text indicators, rings 420A-420C Colors may change, a user may receive tactile feedback through input control device 134, input control device 136, and/or housing 132 of user control system 130, and/or any other similar indication may be presented.

As discussed above, the input control device 134 may control the insertion/retraction of the virtual instrument 412 . In some examples, rolling input control device 134 forward away from the user increases the insertion depth (insertion) of the distal end of virtual instrument 412, and rolling input control device 134 back toward the operator decreases the distal end of virtual instrument 412. Insertion depth (retraction) of the side tip. For example, when a user scrolls input control device 134 in direction D2 (FIG. IB), virtual instrument 412 may extend further outward from virtual catheter 410 in direction D3. In some examples, when the user scrolls input control device 134 in direction D4 (FIG. IB), virtual instrument 412 may retract within virtual catheter 410 in direction D5. In some embodiments, virtual pathway 420 is aligned with the longitudinal axis of virtual instrument 412 . In such embodiments, the user may only need to actuate input control device 134 to navigate virtual instrument 412 through virtual pathway 420 . In other embodiments, the virtual pathway 420 may not be aligned with the longitudinal axis of the virtual instrument 412 . In such embodiments, a user may actuate the input control devices 134 , 136 to navigate the virtual instrument 412 through the virtual pathway 420 . For example, when input control devices 134, 136 are actuated simultaneously, actuation of input control device 136 causes distal portion 414 of virtual instrument 412 to change orientation as the insertion depth of virtual instrument 412 changes. This results in a change in the orientation of the virtual instrument 412 .

In some embodiments, the user may select Exercise 4 of the introductory module by selecting Exercise icon 260D. Exercise 4 of the introductory module may be a non-linear navigation exercise. In some embodiments, a portion of the dynamic GUI 430 for the non-linear navigation exercise may be displayed on the display screen 112 when the user activates exercise 4 of the introductory module. GUI 430 provides training for using input control device 134 and input control device 136 simultaneously.

As seen in FIG. 3D , display screen 112 shows GUI 430 including first portion 430A and second portion 430B. In some embodiments, first portion 430A shows a global perspective of virtual catheter 410 , virtual instrument 412 , and virtual pathway 440 . In some examples, second portion 430B shows a view from the distal tip of virtual instrument 412 . As the virtual instrument 412 traverses the virtual pathway 440, both the first portion 430A and the second portion 430B may be updated in real time.

In a non-linear navigation exercise using GUI 430 , GUI 430 may provide training to teach the user to navigate virtual instrument 412 through virtual pathway 440 . In some examples, virtual pathway 440 is defined by a plurality of sequentially aligned virtual objects 440A-440C. As shown in Figure 3D, target 440A may include an outer ring 442A and an inner core 444A. Similarly, target 440B may include an outer ring 442B and a kernel 444B. Additionally, target 440C may include an outer ring 442C and a core 444C. Targets 440A-440C may be of any size and shape. For example, one or more of the inner cores 444A-444C may be a sphere, a cube, a pyramid, a rectangular prism, or the like. Outer rings 442A-442C may be circular, square, triangular, etc. The shape of the outer rings 442A-442C may correspond to the shape of the cores 444A-444C—for example, if the core 444A is a sphere, the outer ring 442A may be a torus. Alternatively, the shape of the outer rings 442A-442C may be different than the shape of the cores 444A-444C—for example, if the core 440A is a cube, the outer ring 442A may be a triangular ring. In an alternative example, one or more of the targets 440A- 440C may be a sphere of varying opacity, where the center of the sphere is solid and the outer edges of the sphere are translucent.

In some embodiments, targets 440A-440C may be aligned non-linearly. A non-linear navigation exercise may be completed as the distal portion 414 of the virtual instrument 412 traverses through each of the targets 440A-440C. In some examples, system 120 and/or system 110 determines virtual Distal portion 414 of instrument 412 successfully traverses virtual pathway 440 . In some cases, system 120 and/or system 110 may determine that distal portion 414 is in contact with targets 440A- 440C when contact is made within the contact threshold. The following discussion is made with respect to target 440A and applies similarly to targets 440B and 440C. In some examples, when distal portion 414 contacts core 444A of target 440A, contact may be made within a contact threshold. In other examples, contact may be made within the contact threshold when distal portion 414 contacts target 440A just inside outer ring 442A. In other examples, contact may be made within a contact threshold when distal portion 414 contacts outer ring 442A.

In some embodiments, when the distal portion 414 passes through and/or contacts each target 440A-440C, an effect may be provided indicating that the distal portion 414 passes through and/or contacts each target 440A-440C. For example, display screen 112 can show effects (e.g., each target 440A-440C explodes outward, explodes inward, fades, disappears, etc.), an audio signal can be played, display 112 can display text indicators, targets 440A-440C Colors may change, a user may receive tactile feedback through input control device 134, input control device 136, and/or housing 132 of user control system 130, and/or any other similar indication may be presented. In some examples, the effect may vary based on contact between distal portion 414 and targets 440A-440C. For example, before distal portion 414 contacts outer ring 442A, target 440A may be shown in a first display state, such as a solid color, fully opaque, or the like. When distal portion 414 first contacts outer ring 442A, target 440A may then be shown in a second display state, such as a color gradient, partially opaque, or the like. As the distal portion 414 moves closer to the core 444A, the display state of the target 440A may continue to change. For example, the color of object 440A may continue to change from the color of the first display state (eg, red) to a second color (eg, green). Additionally or alternatively, the opacity of object 440A may continue to change from that of the first display state (eg, fully opaque) to a second opacity (eg, fully translucent). When system 120 and/or system 110 determine that distal portion 414 has successfully reached target 440A—for example, when contact between distal portion 414 and target 440A is within the contact threshold discussed above—display screen 112 may show an effect (eg, target 440A explodes outward, explodes inward, fades, disappears, etc.). The above discussion applies similarly to targets 440B and 440C.

As discussed above, the input control device 136 may control the articulation of the virtual instrument 412 . In some embodiments, when the user scrolls the input control device 136 in a particular direction, the distal portion 414 of the virtual instrument 412 may bend in the corresponding direction. For example, input control device 136 may be used to simultaneously control pitch and yaw of distal portion 414 . In some examples, rotation of input control device 136 in a forward direction (eg, direction D2 ) and a rearward direction (eg, direction D4 ) may be used to control pitch of distal portion 414 . Rotation of input control device 136 in a left direction (eg, direction D6 ( FIG. 1B )) and a right direction may be used to control the yaw of distal portion 414 . For example, when a user scrolls input control device 136 in direction D6, distal portion 414 may bend in direction D7. In some examples, the user can control whether the direction of rotation is normal and/or reversed relative to the direction in which distal portion 414 moves (e.g., rotating forward to pitch down and rotating backward to pitch up versus rotating backward). to pitch down and rotate forward to pitch up). For example, when a user scrolls input control device 136 in direction D6, distal portion 414 may bend in direction D8. In some embodiments, the virtual pathway 440 is not aligned with the longitudinal axis of the virtual instrument 412 . In such embodiments, a user may actuate the input control devices 134 , 136 to navigate the virtual instrument 412 through the virtual pathway 440 .

In some embodiments, the user may select Exercise 5 of the introductory module by selecting Exercise icon 260E. Exercise 5 of the introductory module may be a pathway navigation exercise. In some embodiments, a dynamic GUI 450 for the pathway navigation exercise may be displayed on the display screen 112 when the user activates the pathway navigation exercise of the introductory module. GUI 450 provides training for using input control device 134 and input control device 136 simultaneously.

As seen in FIG. 3E , display screen 112 displays GUI 450 including first portion 450A and second portion 450B. In some embodiments, first portion 450A shows a global perspective of virtual catheter 410 , virtual instrument 412 , and virtual pathway 460 . In some examples, second portion 450B shows a view from the distal tip of virtual instrument 412 . As the virtual instrument 412 traverses the virtual pathway 460, both the first portion 450A and the second portion 450B may be updated in real time.

During the pathway navigation exercise of GUI 450 , GUI 450 may provide training to educate the user on navigating virtual instrument 412 through virtual pathway 460 . In some examples, virtual pathway 460 is defined by virtual pipe 470 . Dummy tube 470 includes a distal end 472 and defines a lumen 474 . A user may complete a pathway navigation exercise by navigating virtual instrument 412 through lumen 474 to reach distal end 472 . In some examples, system 120 and/or system 110 determines that distal portion 414 of virtual instrument 412 successfully traversed virtual pathway 460 when distal portion 414 passes through and/or contacts distal end 472 . A user may control virtual instrument 412 in a manner substantially similar to that discussed above with respect to FIG. 3C. For example, when the virtual instrument 412 reaches the distal end 472 of the virtual tube, the display screen 112 may show an effect (e.g., the distal end 472 and/or any other portion of the virtual tube 470 explodes outward, explodes inward, fading, disappearing, etc.), an audio signal may be played, the display screen 112 may display a text indicator, the virtual tube 470 may change color, and the user may receive information via the input control device 134, the input control device 136, and/or the housing 132 of the user control system 130. Haptic feedback, and/or any other similar indication may be presented.

FIG. 4 shows a set of instructions 500 for completing one or more exercises using any of the exercise GUIs 300 , 350 , 400 , 430 , 450 . For example, the set of instructions 500 may be displayed on one or both of the display screens 112, 122 after the user selects an exercise icon but before activating the exercise. In other examples, the set of instructions 500 may be displayed on one or both of the display screens 112, 122 before and/or while the exercise is activated. For example, the set of instructions 500 may be overlaid on the insertion/retraction exercise GUI 300 when the exercise GUI 300 is displayed on the display screen 112 . In other examples, the set of instructions 500 may be displayed on the display screen 112 as a picture-in-picture along with the exercise GUI 300 . In a further example, the set of instructions 500 may be displayed adjacent to the exercise GUI 300 , eg, on the display screen 112 . In some embodiments, individual instructions within instruction set 500 may be tailored to specific exercises selected by the user. As shown in FIG. 4, the set of instructions 500 may provide suggestions to the user on how to effectively control the virtual appliance. For example, set of instructions 500 may advise a user to use both hands when navigating virtual appliance 412 through a virtual pathway (eg, one or more of virtual pathways 420 , 440 , 460 ). This can aid in training the user by familiarizing the user with the process of simultaneously actuating the input control devices 134,136.

Additionally or alternatively, set of instructions 500 may provide instructions to a user on how to interact with GUI 200 . For example, the set of instructions 500 may instruct a user how to select one of the module icons 210A-210E and then how to select one of the exercise icons within the selected module. In some embodiments, instruction set 500 may provide a mix of instructions and objectives for a particular module/exercise.

Referring to FIG. 6 , in some embodiments, display screen 112 illustrates a dynamic GUI 600 for the first exercise in the Basic Drive 1 module. GUI 600 may include a first portion 600A and a second portion 600B. The Basic Drive 1 module may provide training for various virtual pathways using the user control system 130 to navigate a virtual instrument through one or more shapes. For example, a user may actuate the input control devices 134, 136 to insert, retract, and/or manipulate the virtual instrument 615 through various virtual pathways. In some embodiments, the user may activate the base driver 1 module by selecting the module icon 210B on the display screen 122 using any one or more of the selection methods discussed above. After the module icon 210B is selected, the display screen 122 may then display a graphical user interface displaying the exercises included in the Basic Drive 1 module. In some embodiments, the Basic Drive 1 module includes five exercises, but any other number of exercises may be included within the Basic Drive 1 module.

In some embodiments, the user can activate the first exercise in the Basic Drive 1 module by selecting the exercise icon corresponding to the first exercise using any one or more of the selection methods discussed above. In some embodiments, first portion 600A of GUI 600 shows a global perspective of virtual pathway 610 . In some examples, second portion 600B shows a view from the distal tip of virtual instrument 615 . Virtual instrument 615 may be substantially similar to virtual instrument 412 . As the virtual instrument 615 traverses the virtual pathway 610, both the first portion 600A and the second portion 600B may be updated in real time.

As shown in FIG. 6 , the virtual pathway 610 includes a plurality of virtual objects 620 located within the virtual pathway 610 . Virtual pathway 610 also includes a virtual final target 640 located within distal portion 612 of virtual pathway 610 . When performing an exercise using GUI 600 , a user may use input control devices 134 , 136 to navigate virtual instrument 615 through virtual pathway 610 while hitting each of targets 620 , 640 . In some examples, a user may use input control devices 134, 136 to navigate virtual instrument 615 through virtual pathway 610 and hit each of targets 620, 640 while maintaining virtual instrument 615 as close to path 630 as possible. Path 630 may be defined by target 620 . In some embodiments, path 630 may represent an optimal traverse path that virtual instrument 615 should take through virtual pathway 610 . Path 630 may be determined based on parameters such as the amount of contact between virtual instrument 615 and the walls of virtual pathway 610 or the amount of time it takes virtual instrument 615 to traverse the length of virtual pathway 610 . For example, path 630 may be determined by optimizing or minimizing such parameters. In some examples, path 630 may be substantially aligned with the longitudinal axis of virtual pathway 610 . In other examples, such as when virtual pathway 610 is a more complex shape, path 630 may not be aligned with the longitudinal axis of virtual pathway 610 . In such examples, the dummy instrument 615 may need to employ a wider approach angle than would follow the longitudinal axis of the virtual pathway 610 to reduce and/or avoid contact 610 between the dummy instrument 615 and the walls of the dummy pathway.

As further shown in FIG. 6 , display screen 112 may display instructions 650 . Although the instructions 650 are displayed at the bottom of the first portion 600A, the instructions 650 may be displayed at any suitable location on the display 112 (e.g., at the top of the display 112, on the side of the display 112, on the side of the display 112). bottom, or at any other location that may or may not be along the edge of the display screen 112). In some embodiments, the instructions 650 may change depending on the progress the user makes in the exercise using the GUI 600 . For example, instructions 650 may direct the user to move input control device 134 to begin an exercise. In some examples, after starting the exercise, the instructions 650 may change to instruct the user to control the virtual machine 615 so that the virtual machine 615 contacts each target 620 . Additionally or alternatively, instructions 650 may instruct a user to maintain virtual instrument 615 along path 630 . In some embodiments, when the user completes the exercise, instructions 650 may tell the user to return to GUI 250 to select another exercise and/or return to GUI 200 to select another module. Additionally or alternatively, any one or more of the above instructions or any additional instructions may be displayed on the display screen 122 .

In several embodiments, first portion 600A may show virtual instrument 615 advancing through virtual pathway 610 in real time. In some embodiments, an indicator may be displayed on display screen 112 to indicate the proximity of the path of virtual instrument 615 to path 630 . For example, if the path of virtual instrument 615 is substantially aligned with path 630 , virtual instrument 615 may be shown green, indicating a satisfactory proximity of virtual instrument 615 to path 630 . If the path of virtual instrument 615 deviates from path 630 , virtual instrument 615 may be shown in red, indicating an unsatisfactory proximity of virtual instrument 615 to path 630 . The proximity of the path of virtual instrument 615 to path 630 may be shown in any other suitable manner (eg, textual indicators, audible indicators, tactile feedback, etc.). In some embodiments, target 620 may no longer be displayed on display screen 112 after virtual instrument 615 contacts target 620 . Additionally or alternatively, after virtual instrument 615 contacts target 620, effects may be shown (e.g., target 620 explodes outward, explodes inward, fades, disappears, etc.), the user may receive haptic feedback, and/or may present any other similar effect.

As discussed above, second portion 600B of GUI 600 shows a view from the perspective of the distal tip of virtual instrument 615 . In some examples, second portion 600B illustrates lumen 660 of virtual pathway 610 . Target 620 may also be displayed within lumen 660 . As dummy instrument 615 is inserted further into virtual pathway 610 and approaches each target 620 , each target 620 increases in size as the distal tip of dummy instrument 615 gets closer to each target 620 . When the virtual instrument 615 contacts the target 620, an effect may be displayed on the display screen 112 (e.g., the target 620 explodes outward, explodes inward, fades, disappears, etc.), the user may receive haptic feedback, and/or any other similar effect may be presented. Contact indication effect.

In some embodiments, the display screen 112 may display a plurality of performance metrics 670 on the second portion 600B. Each performance metric of plurality of performance metrics 670 may be updated in real time as virtual instrument 615 navigates through virtual pathway 610 . Performance metrics 670 may track the user's performance as the user controls virtual appliance 615, as will be discussed in more detail below.

In several examples, virtual pathway 610 may be a virtual anatomical pathway. In some embodiments, the virtual anatomical pathway 610 may be generated by one or both of the computing systems 110 , 120 . In other embodiments, the virtual anatomical pathway 610 may represent an actual anatomical pathway in the patient's anatomy. For example, virtual anatomical pathway 610 may be generated from CT data, MRI data, fluoroscopy data, etc., which may be generated before, during, or after a medical procedure.

As discussed above, the Basic Drive 1 module may include five exercises. The Basic Drive 2 module may include three exercises in some embodiments, but may include any other number of exercises in other embodiments. Referring to FIGS. 7A-7G , dynamic GUIs 700A- 700G may be displayed on the display screen 112 for some exercises of the Basic Drive 1 module and the Basic Drive 2 module. Each exercise GUI 700A-700G may introduce the user to a virtual environment in which the operation of the user control system 130 will be practiced. Each GUI 700A-700G may be displayed in place of the first portion 600A of the GUI 600 . In some embodiments, GUIs 700A-700E may be displayed for exercises included in the base drive 1 module, and GUIs 700F and 700G may be displayed for exercises included in the base drive 2 module. Exercises may be divided between these two modules in any other suitable manner. In other embodiments, the exercises may all be included in one module. GUIs 700A-700G include various virtual pathways 710A-710G, respectively. During each exercise, the user may navigate virtual instruments 715A-715G through a corresponding one of virtual pathways 710A-710G. In some examples, one or more of virtual pathways 710A- 710G may be based on one or more anatomical pathways of the patient's anatomy. For example, one or more centerline points of virtual pathway 710A may correspond to one or more centerline points of an anatomical pathway of the patient's anatomy. Similarly, one or more centerline points of each of virtual pathways 710B- 710G may correspond to one or more centerline points of one or more anatomical pathways of the patient's anatomy.

In some examples, GUI 700A may be displayed for Exercise 1 of the Basic Driver 1 module, GUI 700B may be displayed for Exercise 2 of the Basic Driver 1 module, GUI 700C may be displayed for Exercise 3 of the Basic Driver 1 module, and GUI 700C may be displayed for Exercise 3 of the Basic Driver 1 module GUI 700D can be displayed for exercise 4 of the basic driver 1 module, GUI 700E can be displayed for exercise 5 of the basic driver 1 module, GUI 700F can be displayed for exercise 1 of the basic driver 2 module, and GUI 700G can be displayed for exercise 2 of the basic driver 2 module. In other examples, GUIs 700A-700G may be displayed for exercises included in any other modules. Additional exercises may be included in one or more of the modules discussed above or may be included in any additional modules within the computing system 110 , 120 .

Referring to FIG. 7A, an exercise GUI 700A shows a virtual pathway 710A, a plurality of virtual destinations 720A, a path 730A, and a virtual final destination 740A. Virtual target 720A may be substantially similar to virtual target 620 , and virtual final target 740A may be substantially similar to virtual final target 640 . In some embodiments, path 730A may represent an optimal path that a virtual instrument (eg, virtual instrument 615 ) may take through virtual pathway 710A. The optimal path may be determined by the processing system 116 and/or the processing system 126, determined by the user during the setup phase, or determined by the processing systems 116/126 and changed by the user during the setup phase. The processor or user may position the virtual instrument 715A at The optimal path is defined relative to the path of the optimal pose (eg, position and orientation) of the anatomical target at the end of the path. In some examples, a user may navigate virtual instrument 715A through virtual pathway 710A.

In some examples, each virtual lane 710A-710G may represent a progressively more complex virtual lane. For example, virtual pathway 710B may be more complex than virtual pathway 710A, eg, by including at least one sharper bend/curve, at least one portion with a narrower pathway width, more bends/curves, and the like. In some examples, virtual pathway 710G may be the most complex shape of virtual pathways 710A-710G. In such examples, virtual pathway 710G may be more complex than virtual pathway 710F, which may be more complex than virtual pathway 710E, which may be more complex than virtual pathway 710D, which may be more complex than virtual pathway 710C. More complex, the virtual pathway 710C may be more complex than the virtual pathway 710B, which may be more complex than the virtual pathway 710A. In other examples, any of the virtual paths 710A-710G may be of any complexity, and the complexity of the virtual paths 710A-710G may be in random order.

In some examples, virtual pathway 710A may include at least one bend 750A, which may be an S-curve, through which virtual instrument 715A must navigate to reach target 740A. Exercise GUI 700A may be used to train a user to use user control system 130 to navigate a virtual instrument through a virtual pathway, such as virtual pathway 710A, which includes one or more minor bends (eg, bends less than 45°). Accordingly, exercise GUI 700A may provide training to a user with respect to navigating a non-linear virtual pathway.

FIG. 7B shows an exercise GUI 700B that includes a virtual pathway 710B. Virtual pathway 710B may include at least one bend 750B of substantially 45° through which virtual instrument 715B must navigate to reach target 740B. Exercise GUI 700B may be used to train a user to use user control system 130 to navigate a virtual instrument through a virtual pathway, such as virtual pathway 710B, which includes at least one 45° bend. Accordingly, exercise GUI 700B may provide training to the user on navigating a non-linear virtual pathway that has a more complex shape than a virtual pathway that has only minor bends.

FIG. 7C shows an exercise GUI 700C that includes a virtual pathway 710C. Virtual pathway 710C may include at least one substantially 90° bend 750C through which virtual instrument 715C must navigate to reach target 740C. FIG. 7D shows an exercise GUI 700D that includes a virtual pathway 710D. Virtual pathway 710D may include at least one substantially 90° bend 750D through which virtual instrument 715D must navigate to reach target 740D. FIG. 7E shows an exercise GUI 700E that includes a virtual pathway 710E. Virtual pathway 710E may include at least one substantially 90° bend 750E through which virtual instrument 715E must navigate to reach target 740E. Exercise GUIs 700C-700E may each be used to train a user to use user control system 130 to navigate a virtual instrument through a virtual pathway that includes at least one 90° bend. Accordingly, exercise GUIs 700C-700E may provide training to the user on navigating a non-linear virtual pathway that has a more complex shape than a virtual pathway that has only a 45° bend. Additionally, bending may occur in any direction, which may help train users to navigate virtual pathways of different orientations.

FIG. 7F shows an exercise GUI 700F that includes a virtual pathway 710F. Virtual pathway 710F may include at least one bend 750F of substantially 180° through which virtual instrument 715F must navigate to reach target 740F. FIG. 7G shows exercise GUI 700G, which includes virtual pathway 710G. Virtual pathway 710G may include at least one bend 750G of substantially 180° through which virtual instrument 715G must navigate to reach target 740G. Exercise GUIs 700F and 700G may each be used to train a user to use user control system 130 to navigate a virtual instrument through a virtual pathway that includes at least one 180° bend. Accordingly, exercise GUIs 700F and 700G provide the user with training in navigating a non-linear virtual pathway that has a more complex shape than a virtual pathway that has only 90° bends. Additionally, bending may occur in any direction, which helps train users to navigate virtual pathways of different orientations. Additionally, exercise GUIs 700F and 700G can help train a user to navigate a virtual instrument through a virtual path that includes any linear segment of constant curvature without a virtual path.

Any one or more of virtual paths 710A-710G may include any one or more of the features discussed above and/or may include additional features not discussed above (e.g., paths that are generally straight, have different bends, and/or have different curved combined pathways, etc.).

The discussion above with respect to virtual pathway 610 is applicable to each of virtual pathways 710A-710G. For example, with respect to virtual pathway 710A, path 730A may represent the optimal path that virtual instrument 615 should take through virtual pathway 710A. Additionally, the discussion above with respect to FIG. 6 may similarly apply to any other similar features between FIG. 6 and FIGS. 7A-7G .

FIG. 8 illustrates a portion 770 of a dynamic GUI (eg, GUI 700A, 600 ) that may be displayed on display screen 112 . In some embodiments, portion 770 may be displayed on display screen 112 in place of second portion 600B of dynamic GUI 600 . As discussed above, the second portion 600B shows a view from the distal tip of the virtual instrument 615 . Similarly, section 770 shows a view from the distal tip of virtual instrument 715A. In some examples, portion 770 shows lumen 780 of virtual pathway 710A. Portion 770 also includes target 720A that may be displayed within lumen 780 . As the dummy instrument 715A is further inserted into the virtual pathway 710A and approaches each target 720A, each target 720A increases in size as the distal tip of the dummy instrument 715A gets closer and closer to each target 720A. When virtual instrument 715A contacts target 720A, an effect may be displayed on display screen 112 (e.g., target 720A explodes outward, explodes inward, fades, disappears, etc.), the user may receive haptic feedback, and/or may present any other similar Contact indication effect.

In some embodiments, display screen 112 may display multiple performance metrics 760 in portion 770 of exercise GUI 700A. Each performance measure 760A-760D of the plurality of performance measures 760 may be updated in real time as the virtual instrument 715A navigates through a virtual pathway (eg, virtual pathway 710A). Performance metrics 760 may track the user's performance as the user controls virtual appliance 615 . In some embodiments, performance metrics track a user's ability to navigate through and stay within a virtual pathway and hit virtual targets. In other embodiments, performance metrics track a user's ability or efficiency to follow an optimal path or position a virtual instrument in an optimal final position/orientation. In other embodiments, performance metrics track a user's proficiency with various input devices during navigation and actuation. In some embodiments, performance metric tracking corresponds to any combination of various types of metrics for drive within pathways/along targets, drive along optimal paths/locations, and proficiency with user input devices.

The following discussion regarding performance metrics will be made with reference to FIG. 7A. The discussion applies similarly to the virtual instruments, virtual pathways, etc. in any one or more of FIGS. 3A-3E , 6 , 7B-7G , 8 , 9A , 9B , and 11 .

In some examples, performance metrics corresponding to measuring the user's ability to navigate through and stay within the virtual pathway and hit virtual targets may be tracked and displayed or used to provide a score indicative of the user's ability to drive within the pathway. In some embodiments, plurality of performance metrics 760 may include one or more of a "goal" metric 760A, a "simultaneous drive" metric 760B, a "collision" metric 760C, and a "time to completion" metric 760D. Plurality of performance metrics 760 may also include one or more additional metrics, such as a "center drive" metric, a "miss target, reverse, then hit target" metric, a "force measure" metric, a "lean angle" metric, a "knock Hit Collision" metric, "Drag Collision" metric, "Instrument Deformation" metric, "Bending Radius" metric, etc. Any one or more of these metrics (or any other not listed metrics) may be displayed on display 112 and/or display 122 . Additionally or alternatively, any one or more of these metrics (or any other metrics not listed) may be tracked by computing system 110 and/or computing system 120, whether or not the metrics are displayed on display screen 112 and /or display screen 122 . In some examples, performance metrics 760 are not displayed on display screen 112 while the user is performing the exercise. In such examples, performance metrics 760 may be displayed as the user completes the exercise, as will be discussed in more detail below.

In some examples, as virtual implement 715A traverses virtual pathway 710A, "targets" metric 760A tracks the number of targets (eg, targets 720A) hit by virtual implement 715A out of the total number of targets within virtual pathway 710A. The number of targets hit can be updated in real time. For example, the "target" metric 760A may increase in increments of "one" when the virtual instrument 715A contacts one of the targets 720A. In some cases, the "target" metric 760A may change from "0/10" to "1/10" when the virtual instrument 715A contacts the first target 720A. In several embodiments, a "goal" metric 760A may be tracked for one or more exercises in one or more of the Basic Drive 1 module and the Basic Drive 2 module.

In some examples, the "collisions" metric 760C tracks the number of times the distal tip of the virtual instrument 715A collides with the walls of the virtual pathway 710A. For example, the "bump" metric 760C may increment its counter by one unit (eg, from 1 to 2) each time the distal tip touches a wall of the virtual pathway 710A. In some embodiments, the contact force (which may be a collision force) between the virtual instrument 715A and the walls of the virtual pathway 710A may need to reach a threshold force (e.g., a threshold collision force) to constitute a "collision" for making a "collision" Collision" metric 760C is incremented for purposes. In other embodiments, a collision of any contact force may cause the "collision" metric 760C to increment its counter. In some embodiments, the threshold force may be the force required to move the distal tip of the virtual instrument 715A two (2) millimeters beyond the wall of the virtual pathway 710A. The threshold force may be the force required to move the distal tip of the virtual instrument 715A any other distance (eg, 1 mm, 3 mm, 4 mm, etc.) beyond the walls of the virtual pathway 710A.

In some embodiments, a dummy tip (not shown) may surround the distal tip of dummy instrument 715A. The virtual tip can be a sphere, hemisphere, cube, half cube, etc. A "collision" may occur when the virtual tip contacts (eg, touches, overlaps, etc.) the walls of the virtual pathway 710A. In some examples, the virtual tip may contact the wall when the amount of overlap between the virtual tip and the wall exceeds a threshold overlap amount. The threshold overlap can be 0.25mm, 0.5mm or any other distance. In such an example, the "collision" metric may have its counter incremented when the amount of overlap exceeds a threshold amount of overlap. In some cases, this may occur before the distal tip of dummy instrument 715A contacts the wall of dummy pathway 710A. A user's goal may be to minimize the amount of collisions that occur between the virtual instrument 715A and the walls of the virtual pathway 710A. In several embodiments, the "collision" metric 760C may be tracked for one or more exercises in one or more of the base drive 1 module, base drive 2 module, airway drive 1 module, and airway drive 2 module.

In some examples, "Time To Completion" metric 760D tracks the total time elapsed from when virtual instrument 715A first begins moving until virtual instrument 715A contacts target 740A. A user's goal may be to minimize the total amount of time it takes to complete an exercise (eg, the exercise shown in GUI 700A). In several embodiments, a "time to completion" metric 760D may be tracked for one or more exercises in one or more of the Base Drive 1 module, Base Drive 2 module, Airway Drive 1 module, and Airway Drive 2 module. In an alternative embodiment, the "time to completion" metric 760D is tracked only when one or both of the input control devices 134, 136 are actuated. For example, if the user stops actuating one or both of the input control devices 134, 136 and leaves the user control system 130 in the middle of performing an exercise, the timer counting the "time to finish" may be paused. When the user returns to the user control system 130 and resumes actuating one or both of the input control devices 134, 136, the timer may start again.

In some embodiments, the "Center Drive" metric tracks the percentage of time that the distal tip of the virtual instrument 715A is in the center of the virtual pathway 710A. For example, the "Center Drive" metric compares the amount of time the distal tip of the virtual instrument 715A is in the center of the virtual pathway 710A to the total amount of time the virtual instrument 715A moves through the virtual pathway 710A. In some cases, the "Centered Drive" metric tracks the percentage of time that the distal tip of the virtual instrument 715A is in the center of the virtual pathway 710A as the virtual instrument 715A traverses one or more straight segments of the virtual pathway 710A. In some embodiments, virtual pathway 710A includes more than one straight segment. In such embodiments, the "Center Drive" metric may separately track the percentage of time that the distal tip of the virtual instrument 715A is in the center of each straight segment of the virtual pathway 710A. For example, the "Center Drive" metric may determine the percentage of the first straight segment, the percentage of the second straight segment, the percentage of the third straight segment, and so on. Additionally or alternatively, the "Center Drive" metric may track the total percentage of time that the distal tip of the virtual instrument 715A is in the center of all straight segments of the combined virtual pathway 710A. In a further alternative embodiment, the "Center Drive" metric may separately track the percentage of time that the distal tip of the virtual instrument 715A is centered in one or some (but not all) straight segments of the virtual pathway 710A. A user's goal may be to maximize the percentage of time that the distal tip of the virtual instrument 715A is centered in the virtual pathway 710A.

In some embodiments, the "miss target, reverse, then hit target" metric tracks that the virtual instrument 715A misses/passes a target (e.g., one or more targets 720A), retracts past the target, and then inserts and hits again number of goals. The number of times the virtual instrument 715A misses the target, reverses, and then hits the target can be updated in real time. For example, when the virtual instrument 715A misses the target, reverses, and then hits the target, the "misses the target, reverses, then hits the target" metric may increase in increments of "one". In some cases, the "misses target, reverses, then hits target" metric may change from "0" to "1" when virtual instrument 715A misses, reverses, and then hits target. In some examples, the "miss target, reverse, then hit target" metric may track distance traveled and time elapsed when virtual implement 715A reverses and tries to hit target again. A user's goal may be to minimize the number of missed goals.

In some embodiments, the "force measurement" metric tracks the amount of force applied by the distal tip of the virtual instrument 715A to the wall of the virtual pathway 710A when the distal tip of the virtual instrument 715A contacts the wall of the virtual pathway 710A. System 110 and/or system 120 may be based on the detected deformation of the walls of virtual pathway 710A, the approach angle of the distal tip of virtual instrument 715A relative to the walls of virtual pathway 710A, and/or the stiffness of virtual instrument 715A. The goal may be to minimize the amount of force applied to the wall, and if force is applied to the wall, minimize the length of time the force is applied to the wall. In some embodiments, deformation of the virtual pathway 710A may be determined based on the relative positions of the distal tip of the virtual instrument 715A and the walls of the virtual pathway 710A. In some embodiments, the stiffness of virtual instrument 715A may be a predetermined amount provided to system 110 and/or system 120 . Stiffness may be provided before an exercise (eg, the exercise shown in GUI 700A) is activated and/or while the exercise is activated. The goal may be to minimize the amount of deformation of virtual pathway 710A, and if virtual pathway 710A deforms, minimize the length of time virtual pathway 710A deforms.

Additionally or alternatively, the "force measurement" metric may track the amount of force applied to the gamification exercise wall by the distal tip of the virtual instrument 715A when the distal tip of the virtual instrument 715A contacts the gamification exercise wall. In some examples, gamification exercise walls represent walls of virtual pathway 710A. System 110 and/or system 120 may calculate this force to improve the accuracy of displaying the interaction between virtual instrument 715A and the walls of virtual pathway 710A (eg, on display screen 112 and/or on display screen 122 ).

In some embodiments, the "Put Up Angle" metric measures the contact angle—the angle at which the distal tip of the virtual instrument 715A contacts the wall of the virtual pathway 710A. When the distal tip of dummy instrument 715A contacts the wall of dummy pathway 710A, the wall will "brace" (eg, expand in at least a radial direction). The contact angle defines the amount of propping up. In some examples, the contact angle is gentle (eg, less than 30° with the walls of the virtual via 710A). In other examples, the contact angle is steep (eg, greater than or equal to 30° with the walls of the virtual via 710A). When the contact angle is steep, the wall may prop up more than when the contact angle is gentle. The user's goal may be to minimize the contact angle.

In some embodiments, the "knock hits" metric tracks the number of times the distal tip of the virtual instrument 715A hits the wall of the virtual pathway 710A. The knock may be a slight bounce off the wall. For example, the "tap hit" metric may increment its counter by one unit (eg, from 0 to 1 ) each time the distal tip taps the wall of the virtual pathway 710A. In some embodiments, if the contact force (which may be a collision force) between the virtual instrument 715A and the walls of the virtual pathway 710A is at or below a threshold force (e.g., the threshold collision force discussed above with respect to the "collision" metric 760C) , then the contact constitutes a "tap" for the purpose of incrementing the "tap collision" metric. If the contact force is above the threshold force, the contact constitutes a collision. A user's goal may be to minimize the number of taps that occur between the virtual instrument 715A and the walls of the virtual pathway 710A.

In some embodiments, the "drag collision" metric tracks the amount of time the virtual instrument 715A moves (forward or backward) while contacting the walls of the virtual pathway 710A. In some examples, system 110 and/or system 120 starts a timer for a "drag bump" metric when virtual instrument 715A is moving and the distal tip of virtual instrument 715A is in contact with a wall of virtual pathway 710A. Additionally or alternatively, system 110 and/or system 120 starts a timer when virtual instrument 715A is moving and any portion of virtual instrument 715A is in contact with a wall. In some cases, the "drag collision" metric may track the distance the virtual instrument 715A moves when it contacts the walls of the virtual pathway 710A. A user's goal may be to minimize the amount of time and/or distance the virtual instrument 715A moves when it contacts the walls of the virtual pathway 710A.

In some embodiments, the "Instrument Deformation" metric tracks whether the virtual instrument 715A deforms while traversing the virtual pathway 710A. For example, the "Instrument Deformation" metric may track whether the distal tip of virtual instrument 715A and/or the shaft of virtual instrument 715A experiences wedging. Wedging may occur when the distal tip and/or shaft of the dummy instrument 715A becomes stuck (eg, pinned, squeezed, etc.) against the wall of the dummy pathway 710A. The wedging portion of dummy instrument 715A may no longer be able to move in the direction of insertion through dummy pathway 710A. A display screen (eg, display screen 112 and/or display screen 122 ) may show whether virtual instrument 715A is wedged against a wall of virtual pathway 710A. For example, a user may be able to look at the display screen and see virtual instrument 715A being wedged. Additionally or alternatively, a wedge indicator may be presented when the virtual instrument 715A is wedged. The wedge indicator may be a text indicator, an audible indicator, a tactile indicator, any other indicator, or any combination thereof. Additionally or alternatively, the number of times the virtual instrument 715A is wedged may be updated in real time. For example, the "Instrument Deformation" metric may increase in increments of "one", such as from "0" to "1," when the virtual instrument 715A is wedged.

In an additional example, the "Instrument Deformation" metric tracks whether the virtual instrument 715A experiences buckling. In some cases, buckling may occur when a portion of dummy instrument 715A becomes wedged and dummy instrument 715A continues to be inserted into virtual pathway 710A. In this case, a portion of virtual instrument 715A may buckle. Additionally or alternatively, the wedged portion of virtual instrument 715A may buckle. Display 112 and/or display 122 may show whether virtual instrument 715A has buckled. For example, a user may be able to look at the display screen and see that the virtual instrument 715A has buckled. Additionally or alternatively, a flexion indicator may be presented when the virtual instrument 715A is flexed. The buckling indicator can be a textual indicator, an audible indicator, a tactile indicator, any other indicator, or any combination thereof. Additionally or alternatively, the number of buckles of the virtual instrument 715A may be updated in real time. For example, when the virtual instrument 715A buckles, the "instrument deformation" metric may increase in increments of "one", such as from "0" to "1."

In some embodiments, performance metrics track a user's ability or efficiency to follow an optimal path or position a virtual instrument in an optimal final position/orientation. The optimal path may be determined by the processing system 116 and/or the processing system 126, determined by the user during the setup phase, or determined by the processing systems 116/126 and changed by the user during the setup phase. The processor or user may position the virtual instrument 715A at The optimal path is defined relative to the path of the optimal pose (eg, position and orientation) of the anatomical target at the end of the path. In some examples, a user may navigate virtual instrument 715A through virtual pathway 710A.

Plurality of performance metrics 760 may include one or more metrics, such as an "instrument positioning" metric, a "path deviation" metric, a "drive efficiency" metric, a "parked position" metric, a "bend radius" metric, and the like. Any one or more of these metrics (or any other not listed metrics) may be displayed on display 112 and/or display 122 . Additionally or alternatively, any one or more of these metrics (or any other metrics not listed) may be tracked by computing system 110 and/or computing system 120, whether or not the metrics are displayed on display screen 112 and /or display screen 122 . In some examples, performance metrics 760 are not displayed on display screen 112 while the user is performing the exercise. In such examples, performance metrics 760 may be displayed as the user completes the exercise, as will be discussed in more detail below.

In some embodiments, the "instrument positioning" metric tracks the number of times the virtual instrument 715A is optimally positioned to prepare to turn through a curved section of the virtual pathway 710A (eg, curved section 750A). In some examples, if the virtual instrument 715A approaches the curved section at too gentle an angle, the virtual instrument 715A will not be able to smoothly traverse the curved section (eg, without retracting and/or repositioning). Instead, the virtual instrument 715A will need to be repositioned iteratively (eg, through a short sequence of insertion and retraction) as the virtual instrument 715A traverses the curved section. The number of times the virtual machine 715A is optimally positioned to prepare for the turn through the curved section can be updated in real time. For example, the "instrument positioning" metric may be increased in increments of "one" when the virtual instrument 715A is optimally positioned. In some cases, virtual pathway 710A may include two curved portions. In such cases, the "instrument position" metric may change from "0/2" to "1/2" when the virtual instrument 715A is optimally positioned. Virtual pathway 710A may include any other number of bends.

In some embodiments, the "path deviation" metric compares the traversed path of the virtual instrument 715A to the path 730A to see how closely the virtual instrument 715A followed the path 730A. In some examples, during the exercise and/or after the exercise is complete, display 112 and/or display 122 may display virtual pathway 710A including the traversed path of virtual instrument 715A and path 730A. This allows system 110 and/or system 120 to compare the traversed path of virtual instrument 715A with path 730A. In some examples, path 730A is displayed as the user performs the exercise. This allows real-time comparison of the traversed path of virtual instrument 715A with path 730A. In other examples, path 730A is displayed only after the exercise is complete. This allows the traversed path of virtual instrument 715A to be compared to path 730A after the exercise is complete. In some examples, system 110 and/or system 120 may determine that the traversed path of virtual instrument 715A deviates from path 730A when the traversed path differs from path 730A by more than a threshold distance (which may be 0.25 mm, 0.5 mm, 1 mm, etc.). A user's goal may be to maximize the time and/or length of the path traversed by virtual instrument 715A to match path 730A.

In some embodiments, the "drive efficiency" metric tracks the length of the traversed path of the virtual instrument 715A to determine how efficiently the virtual instrument 715A traverses the virtual pathway 710A to reach the target 740A. This allows system 110 and/or system 120 to compare the length of the traversed path of virtual instrument 715A to the length of path 730A. In some examples, the "drive efficiency" metric may be presented as a ratio comparing the length of the traversed path of virtual instrument 715A to the length of path 730A. For example, a ratio of "2:1" may indicate that the path traversed by virtual instrument 715A is twice as long as path 730A. Additionally or alternatively, the "drive efficiency" metric may account for the percentage of the traversed path of virtual instrument 715A that is longer than the length of path 730A.

In some cases, the "drive efficiency" metric may track the number of times virtual instrument 715A deviates from path 730A. The number of times virtual instrument 715A deviates from path 730A can be updated in real time. For example, the "Drive Efficiency" metric may increase in increments of "one", such as from "0" to "1," when the virtual instrument 715A deviates from the path 730A.

Additionally or alternatively, the "drive efficiency" metric may track the amount of time the virtual instrument 715A moves (forward or backward) while deviating from the path 730A. In some examples, when virtual instrument 715A is moving and the distal tip of virtual instrument 715A deviates from path 730A, system 110 and/or system 120 starts a timer for a "drive efficiency" metric. In other examples, system 110 and/or system 120 starts a timer when virtual instrument 715A is moving and any portion of virtual instrument 715A deviates from path 730A.

In some embodiments, the "parked position" metric tracks the number of times the virtual instrument 715A reaches a target parked position. The target parking position may represent an optimal position and/or orientation of the virtual instrument 715A to allow the virtual instrument 715A to approach a lesion or other target anatomy. In some examples, the target park location may be target 740A. In other examples, the target parking location may be represented by a clear marker located within the virtual pathway 710A. For example, the target parking location may additionally or alternatively not be visible on display screen 112 but may be known by system 110 and/or system 120 . In such cases, system 110 and/or system 120 may determine whether the parked position of the distal tip of virtual instrument 715A has reached the "invisible" target parked position.

The number of times the virtual appliance 715A reaches the target parking location can be updated in real time. For example, the "Parked Position" metric may be incremented by an increment of "one" when the virtual instrument 715A reaches a target parked position. In some cases, the "parked position" metric may change from "0/2" to "1/2" when the virtual instrument 715A reaches the target parked position. Virtual pathway 710A may include any number of optimal parking locations (eg, more or fewer than two optimal parking locations). In some embodiments, there may be more than one optimal parking position for a target anatomy. In other embodiments, there may be one optimal parking position for each target anatomy. In other embodiments, one parking location may be the optimal parking location for multiple objects.

The target parking location may be determined by processing system 116 and/or processing system 126 by determining a location that will minimize the curvature in virtual instrument 715A to ensure that the curvature is below a threshold curvature. Additionally or alternatively, the target parking location may be determined by the processing system 116 and/or the processing system 126 by determining the optimal position to place the virtual instrument 715A relative to the anatomical target. Additionally or alternatively, the target parking location may be determined by processing system 116 and/or processing system 126 by determining where to place virtual instrument 715A in an optimal pose (eg, position and orientation) relative to the anatomical target. In some examples, the target parking location may be determined by processing system 116 and/or processing system 126 by determining where to place virtual instrument 715A in an optimal shape relative to the anatomical target.

In some embodiments, the "radius of bend" metric tracks how much the distal tip of the virtual instrument 715A bends when the distal tip is articulated. The degrees may be displayed on display 112 and/or display 122 . Additionally or alternatively, the "bend radius" metric tracks whether a portion (or more than one portion) of the virtual instrument 715A is bent with too sharp a curvature to allow the device to pass through the lumen of the virtual instrument 715A. In some examples, a bend indicator may be displayed on display 112 and/or display 122 . When a portion (or portions) of the virtual instrument 715A is bent at a curvature that is too sharp to allow the device to pass through the lumen of the virtual instrument 715A, the portion of the bend indicator may turn a different color, such as yellow or red . The Bend Radius metric tracks the amount of yellow/red in the bend indicator. The number of yellow/red segments in the bend indicator can be updated in real time. For example, when a portion of the virtual instrument 715A is bent with too sharp a curvature to allow the device to pass through the lumen of the virtual instrument 715A, the "Bend Radius" metric may be increased in increments of "numbers", such as from "0" to " 1". A user's goal may be to minimize the number of yellow/red portions in the curved indicator. Additionally or alternatively, the user's goal may be to minimize the length of the yellow/red segment.

Various examples of bend indicators, and methods for monitoring other than Relevant indicators of parameters other than bending, the entire content of this patent is incorporated herein by reference. More information on bend indicators can be found in International Application No. WO 2018/195216, filed April 18, 2018 and entitled "GraphicalUser Interface for Monitoring an Image-Guided Procedure", the entire content of which patent is incorporated by reference into this article.

As discussed above, the input control device 136 controls the bending of the distal portion of the virtual instrument 715A, and the input control device 134 controls the insertion of the virtual instrument 715A. In some embodiments, multiple performance metrics track a user's proficiency with various input devices during navigation and actuation. Plurality of performance metrics 760 may include one or more additional metrics, such as "Incorrect Use of User Input Devices" metric, "Simultaneous Drives" metric 760B, "Eye Tracking" metric, "Frequency of Control Utilization" metric, "User Input Device free spins" metric, etc. Any one or more of these metrics (or any other not listed metrics) may be displayed on display 112 and/or display 122 . Additionally or alternatively, any one or more of these metrics (or any other metrics not listed) may be tracked by computing system 110 and/or computing system 120, whether or not the metrics are displayed on display screen 112 and /or display screen 122 . In some examples, performance metrics 760 are not displayed on display screen 112 while the user is performing the exercise. In such examples, performance metrics 760 may be displayed as the user completes the exercise, as will be discussed in more detail below.

In some embodiments, the "Incorrect Use of User Input Devices" metric tracks, for example, the number of times a user operates an input control device 136 incorrectly. The number of times the user incorrectly manipulates the input control device 136 to attempt to insert or retract the virtual instrument 715A can be updated in real time. For example, when a user improperly manipulates the input control device 136 in an attempt to insert or retract the virtual instrument 715A, the "Incorrect Use of User Input Device" metric may increase in increments of "one", such as from "0" to " 1". Additionally or alternatively, the "Incorrect Use of User Input Devices" metric may track the amount of time a user manipulates the input control device 136 incorrectly. This allows system 110 and/or system 120 to determine the total amount of time it took for the user to resume proper operation of input control device 136 .

In some cases, the "simultaneous actuation" metric 760B tracks the percentage of time that the input control devices 134, 136 are in motion simultaneously. Simultaneous actuation may be more efficient because simultaneous insertion and articulation of virtual instrument 715A may cause virtual instrument 715A to travel to a target (eg, target 740A) faster than if virtual instrument 715A were not inserted and articulated simultaneously. In some embodiments, the percent motion of simultaneous actuation is determined by comparing the amount of time the input control devices 134 , 136 are moving simultaneously to the amount of time only one of the input control devices 134 , 136 is moving. A user's goal may be to maximize the number of simultaneous drives and thereby increase the percentage of simultaneous drives. In several embodiments, the "simultaneous drive" metric 760B may be tracked for one or more exercises in one or more of the base drive 1 module, base drive 2 module, airway drive 1 module, and airway drive 2 module. In some examples, the "simultaneous drive" metric 760B may be tracked in one or more exercises that do not require simultaneous drive. In such examples, if a user actuates input control devices 134, 136 simultaneously, system 110 and/or system 120 may instruct the user to stop his or her "simultaneous actuation."

In some embodiments, the "free rotations of user input device" metric tracks the number of times the input control device 134 rotates at least one full revolution in less than one second. As discussed above, the input control device 134 controls the insertion of the virtual instrument 715A. The number of times the input control device 134 rotates at least one full revolution in less than one second can be updated in real time. For example, the "free rotation of user input device" metric may increase in increments of "one," such as from "0" to "1," when the input control device 134 rotates at least one full revolution in less than one second. The input control device 134 may rotate at an angular velocity greater than the threshold angular velocity when the input control device 134 rotates at least one full revolution in less than one second. In some cases, the threshold angular velocity may be 60 revolutions per minute, but may be any other suitable angular velocity. The "free rotation of user input device" metric may increase in increments of "one", such as from "0" to "1," when the input control device 134 is rotated at an angular velocity greater than the threshold angular velocity. A user's goal may be to minimize the number of times input control device 134 is rotated at an angular velocity greater than a threshold angular velocity.

In some embodiments, the "eye tracking" metric tracks the user's gaze, which allows system 110 and/or system 120 to determine which display screen (e.g., display one of screens 112, 122). System 110 and/or system 120 may also determine whether the user is looking at one or both of input control devices 134 , 136 . For example, camera 118 of system 110 and/or camera 128 of system 120 may track a user's gaze. Based on the tracked gaze, system 110 and/or system 120 may determine: (1) the percentage of time the user gazes at display screen 112 when virtual instrument 715A traverses virtual pathway 710A; and/or (3) the percentage of time the user gazes at one or both of the input control devices 134, 136 while the virtual instrument 715A traverses the virtual pathway 710A. System 110 and/or system 120 may compare these percentages to determine how often the user looks at display screen 112 as virtual instrument 715A traverses virtual pathway 710A.

In some cases, one or more indicators (eg, messages, prompts, etc.) may be presented to the user as virtual instrument 715A traverses virtual pathway 710A. Indicators may provide suggestions to the user as to where the user should direct his or her gaze. The indicators may be text indicators, audible indicators, tactile indicators, any other indicators, or any combination thereof. In examples where the indicators are text indicators, the text indicators may be displayed on one or both of the display screens 112,122. In one such example, the "eye-tracking" metric tracks whether a user looks at a text indicator. For example, camera 118 and/or camera 128 may track a user's gaze. System 110 and/or system 120 may then determine whether the user viewed the text indicator. The "eye-tracking" metric also tracks whether users follow the advice provided by text indicators.

In some embodiments, system 110 and/or system 120 may use the "eye tracking" metric to draw the user's attention to one or more suboptimal events (e.g., bleeding, perforation, occlusion, etc.) Occurs when virtual instrument 715A traverses virtual pathway 710A. For example, system 110 and/or system 120 may determine where on display screen 112 and/or display screen 122 the user's gaze is focused. System 110 and/or system 120 may then present the message to the user at the location where the user's gaze is focused. The message may instruct the user to direct his or her attention to the suboptimal event—eg, the location on display 112 and/or display 122 where the suboptimal event is displayed.

In some examples, an indicator may be presented when contact occurs between the distal tip of the virtual instrument 715A and the wall of the virtual pathway 710A. As seen in FIG. 8 , the display screen 112 may display indicators 790 along the edges of the display screen 112 . Indicator 790 may indicate a general area of contact between the distal tip of virtual instrument 715A and the wall of virtual pathway 710A. For example, based on the position of indicator 790 shown in FIG. 8 , the distal end of virtual instrument 715A touches in the general area of the lower left quadrant (e.g., -X, -Y quadrant) of virtual pathway 710A in image frame of reference I The walls of the virtual pathway 710A. In several examples, indicator 790 may overlay portion 770 . In some cases, indicator 790 may be a different color than portion 770 (eg, red, orange, yellow, etc.). Additionally or alternatively, indicator 790 may include a pattern, such as cross-hatching. In some embodiments, indicator 790 may be presented in any other suitable format (eg, text notification on display 112, audible notification, tactile feedback, etc.).

Additionally or alternatively, the indicator 790 may change due to effects, such as making the indicator 790 explode outward, causing the indicator 790 to explode inward, changing the opacity of the indicator 790, changing the color of the indicator 790, changing the color of the indicator 790, 790 fades, indicator 790 disappears, etc. Indicator 790 may be displayed by any one or more of the effects described above. In some cases, display screen 112 and/or display screen 122 may display indicator 790 to indicate the user's performance status with respect to any one or more of the performance metrics discussed above.

In some embodiments, system 110 and/or system 120 may evaluate a user's performance against any combination of the aforementioned metrics to provide an overall score for the user's performance. In some cases, one or more metrics may be weighted to emphasize the importance of certain metrics over others. In other cases, each metric may have equal weight. A total score may include one or more sub-scores. For example, the overall score may include driver sub-scores to assess how successfully the virtual instrument 715A was driven through the virtual pathway 710A. System 110 and/or system 120 may be associated with a collision between virtual instrument 715A and a wall of virtual pathway 710A, a force exerted by virtual instrument 715A on a wall of virtual pathway 710A, hitting a target (e.g., target 720A) by evaluating One or more metrics and/or any other relevant metrics or combination of metrics to determine a driver score. In some examples, the overall score may include a path navigation sub-score to assess how successfully the traversed path of the virtual instrument 715A matches the planned path (eg, path 730A). System 110 and/or system 120 may evaluate one or more metrics and/or any other relevant metrics related to the optimal traverse path, optimal parking position, optimal position, orientation, pose, and/or shape of virtual instrument 715A. or a combination of metrics to determine the path navigation subscore. Additionally or alternatively, the overall score may include input control device sub-scores to assess how successfully the user operates the input control devices 134 , 136 . System 110 and/or system 120 may determine driver sub-scores by evaluating one or more metrics related to the operation of input control devices 134, 136 and/or any other relevant metrics or combination of metrics.

Figure 5 illustrates a method 550 for controlling virtual instruments in the system 100, according to some embodiments. Method 550 is shown as a set of operations or processes 552 through 558 and is described with continued reference to at least FIG. 1A , FIG. 1B , FIGS. 3A-3E and FIGS. 6-10 . As shown in FIG. 5 , at process 552 , a virtual instrument (eg, virtual instrument 615 ) is inserted into a virtual pathway (eg, virtual pathway 610 ) in response to at least user input received from input control device 134 . During or after insertion of the virtual instrument into the virtual pathway, at process 554 the virtual instrument is directed through the virtual pathway in response to at least user input received from input control device 136 . At process 556, computing system 110 and/or computing system 120 determines at least one performance metric (e.g., "goal" metric 760A, "simultaneous drive" metric 760B, "collision" metric 760C, "time to finish") based on the manipulation of the virtual instrument. "Metric 760D, etc.). At process 558, computing system 110 and/or computing system 120 determines whether to actuate input control devices 134, 136 simultaneously. In some examples, this facilitates system 110 and/or system 120 to track "simultaneous drive" metric 760B.

FIG. 9A shows a portion 800 of a dynamic GUI (eg, GUI 700A, 600 ) that may be displayed on display screen 112 . In some embodiments, portion 800 may be displayed on display screen 112 in place of second portion 600B of dynamic GUI 600 . As discussed above, the second portion 600B shows a view from the distal tip of the virtual instrument 615 . Similarly, section 800 shows a view from the distal tip of virtual instrument 715A. Portion 800 may include number of performance metrics 810 , which may include any one or more of performance metrics 760 . Portion 800 may also include a progress bar 820 corresponding to each performance metric. In some embodiments, each progress bar 820 may indicate the progress toward completion of each performance metric. For example, a progress bar 820 corresponding to a "goal" metric 760A may indicate how much the virtual machine 715A has touched a goal (eg, goal 720A) during exercise. Progress indicator 822 of progress bar 820 may incrementally fill progress bar 820 in real time as each target is touched. Progress indicator 822 may be a color (eg, green, blue, red, etc.), pattern, or any other visual indicator for illustrating progress. In some examples, a progress bar 820 may be shown after the exercise is complete to illustrate the user's performance relative to each performance metric for the particular exercise.

FIG. 9B shows a summary report 850 that may include a statistical summary of a user's performance on a particular exercise. Report 850 may be displayed on display 112 and/or display 122 . In some embodiments, the report 850 is displayed after the user completes the exercise. In other embodiments, the report 850 may be displayed as the user performs the exercise, and the metrics 810 may be updated in real time. As shown in FIG. 9B , report 850 may also include instruction icons 860 that may provide instructions and/or tips to help the user improve his or her performance. For example, the instruction icon 860 may advise the user to actuate the input control devices 134, 136 simultaneously to improve the "simultaneous drive" score. Instruction icon 860 may provide any other suggestions/tips as needed to help improve the user's performance relative to any one or more other metrics 810 and/or any additional metrics discussed above with respect to FIG. 8 .

FIG. 10 illustrates a profile summary 900 that may be displayed on display 112 and/or display 122 in accordance with some embodiments. In some examples, profile summary 900 includes profile information 910, which can include identifying information (e.g., username, real name, password, mail, etc.). The profile summary 900 may also include module categories 920 , 940 . The module categories shown in profile summary 900 may include modules that are activated when a user logs into system 110/120. In some embodiments, presentation summaries 930A-930D, 950 may be included in module categories. The performance summaries 930A-930D, 950 may correspond to respective exercises performed by the user, and the performance summaries 930A-930D may illustrate metrics for each exercise performed by the user while the user is logged into the system.

As shown in FIG. 10, a module class 920 represents a basic driver 1 module. In some embodiments, each performance summary 930A-930D corresponds to an exercise performed by the user within the Basic Drive 1 module. For example, performance summary 930A corresponds to Exercise 1 in the Basic Drive 1 module. Performance summary 930A may include performance metrics illustrating the user's performance with respect to Exercise 1 . Performance summary 930B may correspond to Exercise 2 in the Basic Drive 1 module, performance summary 930C may correspond to Exercise 3 in the Basic Drive 1 module, and performance summary 930D may correspond to Exercise 4 in the Basic Drive 1 module. In some embodiments, performance summary 950 corresponds to exercises performed by the user within the Basic Drive 2 module. For example, performance summary 950 may correspond to Exercise 1 in the Basic Drive 2 module.

In examples where the exercise is repeated one or more times, a performance summary for each repetition of the exercise may be included within a module category corresponding to the module comprising the repeated exercise. Additionally or alternatively, when an exercise is repeated, the metrics for each exercise run can be averaged together, and the performance summary for that exercise can list the average metrics for that exercise. Additionally or alternatively, as the exercise is repeated, a measure of the user's most successful exercise run and a measure of the user's least successful exercise run may be displayed.

In some examples, one or more supervisors of the user may log into system 110 and/or system 120 to view the user's performance. For example, when a supervisor logs in, a summary chart may be displayed illustrating performance measures for one or more exercises the user has completed. The system may also display performance metrics for other users under the supervisor's supervision. In this way, the system can account for a comparison of the performance of more than one user.

FIG. 11 illustrates a graphical user interface (GUI) 1000 that may be displayed on one or both of display screens 112, 122, according to some embodiments. In some embodiments, GUI 1000 may include global airway view 1010 , reduced anatomy model 1020 , navigation view 1030 , and endoscopic view 1040 . In some examples, the global airway view 1010 includes a 3D virtual patient anatomy model 1012, which may include a plurality of virtual pathways 1014 displayed from a global perspective. Reduced anatomical model 1020 includes an elongated representation of the planned route to the target location in a simplified 2D format. Navigation view 1030 includes a magnified view of an object from 3D virtual patient anatomy model 1012 . Endoscopic view 1040 includes a view from the distal tip of virtual instrument 1016 .

GUI 1000 may be displayed when the airway drive 1 module and/or the airway drive 2 module are actuated. The goal of these modules may be to provide training to the user in navigating the medical device through various anatomical pathways while using the GUI 1000 . For example, GUI 1000 can assist a user in the guidance of a medical instrument. In some embodiments, the user can activate the airway drive 1 module by selecting the module icon 210D on the display screen 122 . Upon selection of the module icon 210D, the display screen 122 may then display a GUI displaying the exercises included in the Airway Drive 1 module. In some embodiments, the Airway Drive 1 module includes five exercises, but may include any other number of exercises. A user may activate a first exercise of the Airway Drive 1 module, which may be a first airway navigation exercise, by selecting the first exercise icon on the display screen 122 . The first exercise may be a first airway navigation exercise.

In several examples, the global airway view 1010 includes a virtual patient anatomy model 1012 , which may include a plurality of virtual pathways 1014 . In some cases, a virtual pathway in number of virtual pathways 1014 is a virtual anatomical pathway. Patient anatomical model 1012 may be generic (eg, a predetermined model stored within a computing system, such as computing system 120 , or randomly generated by computing system 110 and/or computing system 120 ). In other embodiments, the patient anatomical model 1012 may be generated from a patient database. In other embodiments, patient anatomical model 1012 may be generated from patient-specific CT data. For example, a user preparing for a particular patient procedure may load data from a CT scan acquired from the patient on whom the procedure is to be performed. In some examples, the patient anatomical model 1012 may be static during the practice of the Airway Drive 1 module.

In some embodiments, virtual instrument 1016 , which may be substantially similar to virtual instrument 615 or 715A-E, traverses patient anatomical model 1012 during different exercises in the Airway Drive 1 module. For example, patient anatomical model 1012 may include several objects 1018A-1018C. Each goal may correspond to a different exercise within the Airway Drive 1 or Airway Drive 2 modules. Thus, in some examples, when a user switches between exercises in the Airway Drive 1 module, the user may navigate virtual instrument 1016 to different targets based on which exercise is activated. For example, when the first exercise in the Airway Drive 1 module is activated, the user may navigate the virtual instrument 1016 through the virtual anatomical pathway 1014 to the target 1018A. When the second exercise in the Airway Drive 1 module is activated, the user may navigate the virtual instrument 1016 through the virtual anatomical pathway to the target 1018B. The second exercise may be a second airway navigation exercise. When the third exercise in the Airway Drive 1 module is activated, the user may navigate the virtual instrument 1016 through the virtual anatomical pathway to the target 1018C. The third exercise may be a third airway navigation exercise.

In some embodiments, when the system 100 switches from one exercise to another within the Airway Drive 1 module, the system 100 can automatically reset the distal tip of the virtual instrument 1016 to a proximal position in the patient anatomical model 1012 . For example, the distal tip of virtual instrument 1016 can be positioned at the carina. Thus, in such embodiments, each exercise begins with the virtual instrument 1016 located at the same or similar proximal location within the patient anatomical model 1012 . In other embodiments, when the system 100 switches between exercises within the Airway Drive 1 module, subsequent exercises begin with the virtual instrument 1016 at the same current position as the previous exercise ended. The system may instruct the user to retract the virtual instrument 1016 from a target the user achieved in a previous exercise (e.g., target 1018A) to the main carina or some other proximal location within the patient anatomical model 1012 (e.g., closest to the subsequent target). most recently bifurcated, such as target 1018B or target 1018C), and then navigate the virtual instrument 1016 to the target in a subsequent exercise (eg, target 1018B or target 1018C). In such embodiments, one or more intermediate targets (not shown) in the virtual pathway 1014, for example, may be presented in the GUI 1000 to help guide the user to a retraction point.

In some examples, as the virtual instrument 1016 progresses toward the target (eg, target 1018A), the reduced anatomical model view, the navigation view 1030 , and the endoscopic view 1040 may each update in real time to show the virtual instrument 1016 progressing toward the target 1018A. In several embodiments, endoscopic view 1040 shows a view from the distal tip of virtual instrument 1016 .

Endoscopic view 1040 may be substantially similar to the view shown in second portion 600B of GUI 600 (FIG. 6). In such embodiments, navigation view 1030 may represent a virtual view of endoscopic view 1040 . In some embodiments, computing system 100 and/or computing system 120 may offset navigation view 1030 from endoscopic view 1040 by a predetermined amount to simulate the difference between the navigation view and the endoscopic view in the system GUI used in an actual medical procedure. offset between. Offsets can be applied in the x-direction, y-direction, and/or diagonally. Additional information on the GUI of the system can be found in International Application No. WO 2018/195216, filed April 18, 2018 and entitled "Graphical User Interface for Monitoring an Image-Guided Procedure", the entire content of which is incorporated herein by reference .

In several embodiments, the exercises in the Airway Drive 2 module may include the same patient anatomy and the same goals as those used in the Airway Drive 1 module. As discussed above, the patient anatomical model 1012 may be static during the exercises of the Airway Drive 1 module. In some embodiments, computing system 110 and/or computing system 120 applies simulated patient motion to patient anatomical model 1012 during exercises of the Airway Drive 2 module. Simulated patient motion may be applied to simulate breathing, circulation, and/or a combination of both. Simulated patient motion may simulate how breathing and/or circulation affect (eg, deform) the patient anatomical model 1012 . To simulate patient motion, system 110 and/or system 120 may apply a sinusoidal pattern to patient anatomical model 1012 in an insertion direction (eg, an axial direction), a radial direction, and/or both insertion and radial directions. In some examples, simulated motion may be present in one or more of global airway view 1010 , reduced anatomy model 1020 , navigation view 1030 , and endoscopic view 1040 .

In some embodiments, the simulated motion may be scaled based on the position of the distal portion of the virtual instrument 1016 within the patient anatomical model 1012 . For example, if the virtual instrument 1016 is in a portion of the patient's anatomical model 1012 near the heart, the simulated motion may represent circulation rather than respiration. In other examples, as the virtual instrument 1016 moves toward a more peripheral virtual pathway of the patient anatomical model 1012, the simulated motion may represent breathing rather than circulation. In some cases, the degree of simulated motion when virtual instrument 1016 is in a distal virtual pathway may be lower than the degree of simulated motion when virtual instrument 1016 is in a more proximal virtual pathway (eg, closer to the main carina).

In some examples, the cyclic periods occur at a shorter frequency than the breathing periods. For example, four cycles may occur for each breathing cycle. Other frequencies may also be simulated, such as three cycles per breath cycle, five cycles per breath cycle, and the like. The simulated motion can be scaled to account for differences in cycle frequency. For example, simulated motion may represent cycling more often than simulated motion represents breathing.

In some embodiments, GUI 1000 may display any one or more of the performance metrics discussed above, such as a "simultaneous drive" metric, a "collision" metric, a "total time" metric, and the like. Metrics may be displayed during each exercise and/or after the user performs each exercise.

In some embodiments, the components discussed above may be used to train a user to control a telesystem in a procedure performed with the telesystem, as described in further detail below. The telesystem can be adapted for use in, for example, surgery, telesurgery, diagnostic, therapeutic or biopsy procedures. While some examples of such procedures are provided herein, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, apparatus and methods described herein can be used in animals, human cadavers, animal cadavers, parts of human or animal anatomy, non-surgical diagnostics, and in industrial systems and general robotic, general teleoperated or robotic medical systems.

As shown in FIG. 12 , the medical system 1100 generally includes a manipulator assembly 1102 for manipulating medical instruments 1104 to perform various procedures on a patient P positioned on a table T. As shown in FIG. Manipulator assembly 102 may be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with selected degrees of freedom of motion that may be motorized and/or teleoperated and may be non-motorized and/or non-teleoperated selected degrees of freedom of motion. The medical system 1100 may also include a main assembly 1106 that generally includes one or more control devices for controlling the manipulator assembly 1102 . Manipulator assembly 1102 supports medical instrument 1104 and may optionally include a plurality of actuators or motors that drive inputs on medical instrument 1104 in response to commands from control system 1112 . The actuator can optionally include a drive system that, when coupled to the medical device 1104, can advance the medical device 1104 into a natural or surgically created anatomical orifice.

Medical system 1100 also includes display system 1110 for displaying images or representations of the surgical site and medical instrument 1104 generated by the subsystems of sensor system 1108 . Display system 1110 and main assembly 1106 can be oriented such that operator O can control medical instrument 1104 and main assembly 1106 through the sense of telepresence. Additional information regarding the medical system 1100 and the medical device 1104 can be found in International Application No. WO2018/195216 filed on April 18, 2018 and entitled "Graphical User Interface for Monitoring an Image-Guided Procedure", the entire content of which patent Incorporated herein by reference.

The system 100 discussed above may be used to train a user to operate the medical instrument 1104 . For example, system 100 may provide training to a user to help the user learn how to operate main assembly 1106 to control manipulator assembly 1102 and medical instrument 1104 . Additionally or alternatively, system 100 may teach a user how to use display system 1110 to control medical instrument 1104 before and/or during a medical procedure.

The singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context dictates otherwise. And the terms "comprising", "comprising", "containing", "having" etc. specify the presence of stated features, steps, operations, elements and/or parts, but do not exclude the presence or addition of one or more other features, Steps, operations, elements, parts and/or groups. Components described as coupled may be directly coupled, electrically or mechanically, or they may be indirectly coupled via one or more intermediate components. The auxiliary verb "may" also means that the feature, step, operation, element or component is optional.

Elements described in detail with reference to one embodiment, implementation or application may optionally be included in other embodiments, implementations or applications not specifically shown or described thereof, where applicable. For example, if an element is described in detail with reference to one embodiment but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in connection with one embodiment, implementation or application can be incorporated into other embodiments, implementations or aspects unless otherwise are specifically described unless the one or more elements render the embodiment or implementation inoperable, or unless two or more elements provide conflicting functionality.

A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information in order to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programmed instructions, input information, and output information. The term "computer" and similar terms such as "processor" or "controller" or "control system" are analogous.

Although some of the examples described herein relate to surgical procedures or instruments, or medical procedures and medical instruments, the disclosed technology is applicable to non-medical procedures and non-medical instruments. For example, the devices, systems, and methods described herein may be used for non-medical purposes, including industrial uses, general robotics uses, and sensing or manipulating non-tissue artifacts. Other exemplary applications relate to cosmetic enhancement, imaging of human or animal anatomy, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include using procedures on tissue removed from the human or animal anatomy (without returning to the human or animal anatomy), and performing procedures on human or animal cadavers. In addition, these techniques can be used in surgical and non-surgical medical treatment or diagnostic procedures.

Furthermore, while some of the examples presented in this disclosure discuss teleoperated robotic systems or teleoperable systems, the disclosed technology is also applicable to computer-aided systems that are partially or fully moved directly and manually by an operator.

Additionally, one or more elements of embodiments of the present disclosure may be implemented in software for execution on a processor of a computer system, such as a control processing system. When implemented in software, the elements of the embodiments of the present disclosure are essentially the code segments to perform the necessary tasks. Programs or code segments may be stored in a processor-readable storage medium (eg, non-transitory storage medium) or device, which may have been downloaded via a computer data signal embodied in a carrier wave through a transmission medium or communication link. Processor-readable memory devices may include any medium that can store information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments may be downloaded via a computer network (such as the Internet, an intranet, etc.).

It should be noted that the processes and displays presented may not be inherently related to any particular computer or other device, and that various systems may be used with the programs in accordance with the teachings herein. The required structure for the various systems discussed above will appear as elements in the claims. Furthermore, embodiments of the present disclosure are not described with reference to any particular programming language. It should be understood that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

While certain exemplary embodiments of the present disclosure have been described and illustrated in the drawings, it is to be understood that such embodiments are by way of illustration only and are not restrictive of the broadly disclosed concepts, and that the embodiments of the present disclosure One is not limited to the exact constructions and arrangements shown and described, as various other modifications may occur to those of ordinary skill in the art.

Claims (28)

1. A system, comprising:

a user control system including at least one input control device for controlling movement of a virtual medical instrument through a virtual passageway;

a display to display a graphical user interface and a plurality of training modules, the graphical user interface including a representation of the virtual medical instrument and a representation of the virtual pathway; and

a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more computer processors for performing operations comprising:

navigating the virtual medical instrument through the virtual pathway based on a command received from the user control system; and

evaluating one or more performance metrics for tracking the navigation of the virtual medical instrument through the virtual pathway.

2. The system of claim 1, wherein the virtual path is defined by a plurality of sequentially aligned virtual targets.

3. The system of claim 1, wherein the virtual pathway is a virtual anatomic pathway.

4. The system of claim 1 or 3, wherein the performance metric tracks a number of times contact occurs between the virtual medical instrument and a wall of the virtual passageway.

5. The system of claim 4, wherein the performance metric further tracks, for each of the number of times contact occurs, a length of time the virtual medical instrument makes contact with the wall of the virtual passageway.

6. The system of claim 5, wherein the contact with the wall of the virtual passageway comprises an insertion motion of the virtual medical instrument along the virtual passageway.

7. The system of any of claims 4-6, wherein the performance metric further tracks deformation of the virtual medical instrument during the contact.

8. The system of any of claims 4-7, wherein contact is determined by the virtual medical instrument exerting an impact force on the wall of the virtual passageway that exceeds a threshold impact force.

9. The system of claim 8, wherein the collision force is based on a distance traveled by the virtual medical instrument beyond the wall of the virtual passageway.

10. The system of any of claims 4-9, wherein the graphical user interface further comprises a representation of the contact on the representation of the virtual pathway.

11. The system of any of claims 1 or 3-10, wherein the virtual passageway comprises a plurality of sequentially aligned virtual targets within a lumen of the virtual passageway.

12. The system of claim 11, wherein the plurality of sequentially aligned virtual targets are aligned along a traversal path within the virtual via, the traversal path being different from a longitudinal axis of the virtual via.

13. The system of any of claims 1-11, wherein the instructions to perform operations further comprise determining an optimal traversal path for the virtual lane.

14. The system of claim 13, wherein the optimal traversal path includes a final goal and an optimal position of the virtual medical instrument at the final goal.

15. The system of claim 14, wherein the instructions for performing an operation further comprise comparing a current position of the virtual medical instrument to the optimal position of the virtual medical instrument.

16. The system of any of claims 13-15, wherein the virtual via is defined by a plurality of sequentially aligned virtual targets aligned along the optimal traversal path of the virtual via.

17. The system of claim 11, 12, or 16, wherein the performance metric tracks a number of virtual targets of the plurality of sequentially aligned virtual targets contacted by the virtual medical instrument.

18. The system of claim 11, 12, or 16, wherein the performance metric tracks a number of virtual targets of the plurality of sequentially aligned virtual targets missed by the virtual medical instrument.

19. The system of claim 18, wherein the performance metric tracks a number of times the virtual medical instrument retracts past a missed target after insertion.

20. The system of any of claims 13-15, wherein the performance metric tracks a number of times the virtual medical instrument deviates from the optimal traversal path or a length of time the virtual medical instrument deviates from the optimal traversal path.

21. The system of any of claims 1-20, wherein the at least one input control device comprises a scroll wheel or a track ball.

22. The system of any of claims 1-21, wherein the at least one input control device includes a first input device and a second input device, and wherein the performance metric tracks an amount of time that the first input device and the second input device are actuated simultaneously.

23. The system of any of claims 1-22, wherein the performance metric tracks a number of times the at least one input control device rotates beyond a threshold angular velocity.

24. The system of any of claims 1-23, wherein the display comprises a first display device and a second display device, wherein the first display device displays the graphical user interface and the second display device displays the plurality of training modules.

25. The system of claim 24, wherein the display comprises an image capture device for tracking a user's gaze, and wherein the performance metric comprises tracking a time at which the user's gaze is on the first display device, a time at which the user's gaze is on the second display device, and a time at which the user's gaze is on the at least one input control device.

26. The system of any of claims 1-25, wherein the performance metric tracks a time taken by the virtual medical instrument to complete a traversal of the virtual pathway.

27. The system of any of claims 1-26, wherein the instructions to perform operations further comprise providing an indicator based on the evaluation of the performance metric.

28. The system of claim 27, wherein the indicator comprises an instruction or score for the user.

Images