International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

https://doi.org/10.1007/s11548-018-1751-5

ORIGINAL ARTICLE

Fast 5DOF needle tracking in iOCT

Received: 29 January 2018 / Accepted: 22 March 2018 / Published online: 30 March 2018
© CARS 2018

Abstract
Purpose Intraoperative optical coherence tomography (iOCT) is an increasingly available imaging technique for ophthalmic
microsurgery that provides high-resolution cross-sectional information of the surgical scene. We propose to build on its
desirable qualities and present a method for tracking the orientation and location of a surgical needle. Thereby, we enable the
direct analysis of instrument–tissue interaction directly in OCT space without complex multimodal calibration that would be
required with traditional instrument tracking methods.
Method The intersection of the needle with the iOCT scan is detected by a peculiar multistep ellipse fitting that takes
advantage of the directionality of the modality. The geometric modeling allows us to use the ellipse parameters and provide
them into a latency-aware estimator to infer the 5DOF pose during needle movement.
Results Experiments on phantom data and ex vivo porcine eyes indicate that the algorithm retains angular precision especially
during lateral needle movement and provides a more robust and consistent estimation than baseline methods.
Conclusion Using solely cross-sectional iOCT information, we are able to successfully and robustly estimate a 5DOF pose
of the instrument in less than 5.4 ms on a CPU.

Keywords Optical coherence tomography · iOCT · Instrument tracking · Ophthalmic tool tracking · Geometric modeling

Introduction and related work vided by iOCT combined with its ability to image tissue
structures below the surface without requiring direct contact
In ophthalmic microsurgery, the surgeon manipulates sur- could make it a natural choice for many computer-assisted
gical instruments at micron-scale precision while relying ophthalmic procedures. Information about the location of the
on visual feedback from the microscope. Although state-of- surgeon’s instrument inside the operating area is of essential
the-art devices provide high-resolution stereo view, depth importance for many assistance applications. However, prior
perception is still challenging due to the enface view. This work focused mainly on the use of microscopic RGB images.
makes it especially challenging to estimate the distance In this context, Richa et al. [3] presented tool tracking based
between the utilized surgical instrument and the anatomi- on weighted mutual information between stereo images. If
cal surface [1]. Intraoperative optical coherence tomography the 3D CAD model of the tool is known, the instrument pose
(iOCT) [2] has been introduced as an additional interven- can be recovered by projective contour modeling [4]. Sznit-
tional imaging modality, which provides high-resolution man et al. [5] classified each pixel as either background or
cross-sectional 2D images (B-Scan) and therefore depth tool part using a multiclass ensemble classifier. The precise
information in real time. The high spatial resolution pro- localization of different tool parts is subsequently obtained by
a weighted averaging on the response scores. Allan et al. [6]
B Jakob Weiss estimate the full 3D pose based on a level-set algorithm incor-
Jakob.Weiss@tum.de porating optical flow. Rieke et al. [7] propose to combine a
fast color-based tracker with a robust HoG feature-based 2D
1 Computer Aided Medical Procedures, Technische Universität pose estimator via a dual random forest. An offline learning
München, Boltzmannstr. 3, 85748 Garching, Germany
with online adaption approach further increased the general-
2 Augenklinik und Poliklinik, Klinikum rechts der Isar der ization regarding unseen backgrounds and instruments [8].
Technische Universit München, 81675 Munich, Germany
Supervised deep learning-based approaches [9–11] require
3 Carl Zeiss Meditec AG, 81379 Munich, Germany an extensive annotated data set to capture the wide range
4 Johns Hopkins University, Baltimore, MD, USA

123
788 International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

of image distortions such as blur, specular reflections and ily capable of supporting real-time applications. Throughout
limited focused field of view. experimental results on a phantom and ex vivo porcine eyes,
Despite the recent advances in instrument tracking based we demonstrate how the proposed method is able to with-
on microscopic RGB video, none of the methods can tackle a stand the iOCT specific noise and remarkably improves the
major inherent disadvantage: Even if the tracking precision robustness to instrument movement between iOCT scans.
is perfect in the microscope image, it cannot yield precise Furthermore, we exemplarily show the potential impact on
depth information through its projective imaging geometry. clinical practice by employing the 5DOF instrument tracking
Acquiring information from the high-resolution OCT at the for injection guidance.
3D location of the instrument would still not be feasible if
the accurate spatial mapping between the microscopic image
and the iOCT is unknown. Although optical microscopy and Method
iOCT can share the same optical path in a device, this align-
ment requires complex calibration routines. For the same In this section, we derive the proposed method. First, we
reason, traditional navigation solutions such as optical track- describe the setup and specify our notations. Second, we
ing or electromagnetic tracking are not applicable as they demonstrate how to robustly detect the elliptical cross sec-
usually have an accuracy in the range of 200–1400 µm. tion of the needle in OCT B-Scans and its relation to the
The intraoperative OCT, on the other hand, has an axial needle pose. Finally, we integrate this noisy and incomplete
resolution of 5–10 µm, which is close to histopathology. data into a latency-aware filter and infer the 5DOF pose of
Therefore, we propose to track the 5DOF pose of the sur- the needle.
gical instrument directly in the iOCT B-Scans and by that
completely avoid the bottleneck of calibration. OCT is a Geometric setup
fundamentally different imaging modality than conventional
microscopic imaging: It measures echo delay and intensity OCT allows real-time cross-sectional imaging of tissues
of back reflected near-infrared light waves [2]. Instead of by using a low-coherence light source and measuring the
an RGB enface view of the surgical scene, the B-Scans light reflected at tissue interfaces with different indices
provide grayscale, cross-sectional information. Due to the of refraction. Based on the interference pattern generated
underlying physics, conventional surgical instruments appear from superimposing the reflected light and the light trav-
hyperreflective and consequently any signal beneath them is eled through a reference arm, OCT reconstructs an intensity
lost (c.f. Fig. 2b). A first step toward instrument tracking in profile along the direction of the sampling laser (A-Scan).
this type of data was recently proposed by Zhou et al. [12] Standard iOCT engines have a galvanometer which allows
in terms of instrument segmentation based on a fully convo- vertical and horizontal deflection of the sampling laser. The
lution network. The method, however, requires a volumetric mirror is usually moved in a repeating pattern, and the
data set, is not applicable to real-time applications and is reconstructed signal is transferred after scanlines have been
restricted to static instruments. completed. For interventional OCT imaging, usually a fixed
In this paper, we present a novel real-time 5DOF nee- pattern of several parallel and/or orthogonal scanlines is used
dle tracking method in 2D iOCT images based on geometric to provide the surgeon with different cross-sectional views
modeling. The contribution of our work is as follows. We (B-Scans) of the working volume. From the known layout of
show how the shadows and distinct reflection caused by the this pattern, a transformation to 3D space can be computed
surgical instrument in the B-Scans are related to the inci- for each pixel. Commercially available iOCT devices typ-
dent angle of the tool. In a second step, we integrate this ically have a fixed A-Scan rate of 27–32 kHz, resulting in
information from several—not necessarily parallel—scans a B-Scan update rate of 27–32 Hz if 1000 A-Scans per B-
to infer the axis of the instrument. In order to tackle the Scan are assumed. The number and placement of B-Scans are
latency between OCT scan lines, we derive an application- determined by scanning patterns which can be flexibly inter-
specific Kalman filter that models the instrument movement changed during an intervention. To set a reference coordinate
between two acquired B-Scans. We do not track the nee- system, we define the axis of positive horizontal deflection as
dle tip explicitly, as this is impossible from single B-Scans our x axis and the vertical deflection as the y axis. As the pro-
only unless the needle tip is exactly in plane with one OCT jective field of view for small B-Scan lengths is narrow, we
scan. Nonetheless, the needle axis is already useful for many can neglect the slight projectivity of the system and assume
applications such as computing the projected tissue intersec- a euclidean coordinate system instead, thus assuming that
tion point or OCT repositioning. One of the main advantages the z axis is parallel to our A-Scan direction. Figure 1 illus-
of the method is that it is applicable to general iOCT scan- trates the geometric relationships. The plane corresponding
ning patterns such as parallel, crossing or volumetric patterns. to a B-Scan in 3D that is reconstructed from each scanline is
With a frame rate of more than 180 FPS, our method is eas- parametrized as ρ : ( #»x − #»p ) · n̄ = 0 where #»
p corresponds

123
International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796 789

Real-Time view of
OCT B-Scans

Microscope view and

d0
27G needle and n0
eye phantom p0
p1
p2
Micromanipulator p3
p4 l0

Fig. 1 Test scenario and coordinate system. Left: We use an OPMI Oberkochen. Right: Explanatory sketch describing notation and spatial
Lumera 700 surgical microscope together with a Rescan 700 iOCT sys- relationships between B-Scan direction and needle
tem and a Callisto Eye assistance system, all from Carl Zeiss Meditec,

to the top-left corner of the B-Scan image and n̄ is the plane date points over all columns is pcand . These generally
normal. As a simplification, we model the tracked needle as correspond to either the tool’s surface ( ptool ), noise
an idealized cylinder with known diameter dn and an axis ( pnoise ) or an anatomical layer ( peye ) such as the corneal
parametrized as surface in anterior segment or retinal pigment epithelium
in posterior segment images.
⎛ ⎞
sin θ cos φ 2. A Tissue Layer representing the global eye shape is fitted
 : #»
x (τ ) = #»
x + τ ⎝ sin θ sin φ ⎠ = #» ¯ τ ∈ R,
x + τ l, based on these candidate points by using the RANSAC
cos θ algorithm. For anterior surgery and posterior surgery with
non-degenerate RPE, a circular model is used. In cases
where θ and φ are azimuth and polar angles of the axis direc- where a circular model is not suitable, we use a polyno-
tion. mial of order 4 to approximate the anatomical layer more
closely.
Ellipse Detection for Orientation Estimation 3. Tool Candidate Points ptool ∗ can now be separated from
the anatomical surface peye by computing the distance
To calculate the axis of the needle in 3D, we use the of every candidate point to the tissue model d( p). To
cross section of the tool that is visible in an OCT B-Scan remove isolated candidate points that belong to pnoise or
B = (br ,c ) ∈ Rnr ×n c with nr rows and n c columns. It not-hyperreflective tissue layers, we apply a 1D morpho-
can be shown that the intersection between a cylinder and a logical opening, closing, followed by median filtering
plane in the non-degenerate case forms an ellipse, which we each with a 15 px kernel and obtain a filtered distance
parametrize through the center position in image coordinates list d ∗ ( p). The
 tool ccandidates are then
 defined as the
(C x , C y ) ∈ R2 , the length of the long (λ1 ) and short (λ2 ) axes set ptool ∗ = p ∈ pcand |d ∗ ( p) > dmin where dmin is a
and the angle α that is formed between the longer axis of the threshold to remove points close to the surface.
ellipse and the negative y-axis of the image (or equivalently, 4. (optional) For suspected pathologies (for example due
the angle between λ1 and z, c.f. Fig. 2a). Only the hyper- to preoperative imaging), we iteratively exclude from
reflective surface of the metal needle is visible on an OCT ptool ∗ all points which are closer than dmin to an already
frame while everything below is shadowed. Our algorithm excluded point, effectively removing all points which are
for ellipse detection and determination of its parameters is connected to the tissue layer.
performed through the following steps, illustrated in Fig. 2b– 5. A First Ellipse Estimate is computed by fitting an ellipse
f. to the set of tool candidate points ptool ∗ using RANSAC,
provides an inlier set ptool .
6. Ellipse Refinement is achieved by minimizing the geo-
1. Candidate Points are defined as the pixels with maxi-
metric distance between the ellipse and the points ptool .
mum intensity along each A-Scan (column) of the B-Scan
We directly assign C x = 0.5 ∗ (xr + xl ) where xl , xr are
c
image: pcand = (argmaxr (br ,c ), c). The set of all candi-

123
790 International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

(a) (b) (c) (d)

(e) (f) (g)

Fig. 2 Ellipse parameters and detection. a Schematic view showing ∗ (yellow) obtained by filtering and thresholding d( p). e
date points ptool
the relationship between ellipse parameters and the needle. b–d Steps RANSAC-fitted tool ellipse and inliers (green). f Final ellipse obtained
during ellipse detection: b input image. c Candidate points (violet + by nonlinear optimization (purple) and parameters. g Example that our
red) and fitted tissue layer (violet) with inlier margin. d Tool candi- method is still able to detect the ellipse even if it is touching the tissue

the x coordinate of the leftmost and rightmost point of The state and knowledge about our system are represented
by the state vector x = #»
T
ptool , and suppose a known needle diameter dcyl to set x T , θ, φ, ẋ T , θ̇ , φ̇ and control
λ2 = dcyl . vector u = n̄ T , #»
p T , Δt , where #»
T
x , θ and φ model the
7. Tool center line t can then be related to the ellipse param- needle axis and ẋ, θ̇ and φ̇ model its current velocity and
eters by: angular velocities, respectively. The control vector contains

cos α = z̄ I − n̄ n̄ T l¯ = cos θ the parameters defining the iOCT plane of the next measure-
ment as well as the time since the last measurement.
λ2
= cos β = n̄ · l¯
λ1 State transition
(1)
where β is the angle between the cylinder axis and the We assume that our tool is influenced by unknown accelera-
plane normal (c.f. Fig. 2a). The first relationship follows tions and angular accelerations a = ( #» x , aθ , aφ ) drawn
a T#» T

by considering the projection of l¯ into the B-Scan plane, from a zero-mean Gaussian distribution. Our preliminary
computed as I − n̄ n̄ T l. ¯ Since α is the angle between state update is thereby defined as
the A-Scan direction z̄ and this projected vector, its cosine ⎛ #»∗ ⎞ ⎛ ⎞
is equal to their dot product, which directly simplifies to xt 1 0 0 Δt 0 0
cos θ . The second equality is developed from considering ⎜ θt ⎟ ⎜0 1 0 0 Δt 0⎟
⎜ ⎟ ⎜ ⎟
the inset view of Fig. 2a: from the right triangle shown, ⎜ φt ⎟ ⎜ 0 0 1 0 0 Δt ⎟
⎜ ⎟=⎜ ⎟ · xt−1 + wt , (2)
sin γ = λλ21 follows, and sin γ = sin π2 − β = cos β = ⎜ ẋ ∗ ⎟ ⎜0 0 0 1 0 0⎟
⎜ t ⎟ ⎜ ⎟
⎝ θ̇t ⎠ ⎝0 0⎠
n̄ · l¯ from trigonometry and the dot product definition. 0 0 0 1
φ̇t 0 0 0 0 0 1
Extended Kalman Filter
where wt = ( Δt2 #» Δt 2 Δt 2 #» #»
2
x , 2 aθ , 2 aφ , Δt a x , Δtaθ , Δtaφ )
a T#» T
From the ellipse parameters in one single cross section, it is ∼ N (0, Q).
not possible to uniquely reconstruct the pose of the cylin- However, as the ellipse measurement is related to a differ-
drical needle. We use a Kalman filter [13] to fuse the noisy ent image plane at the next timestep, we move the base point
measurements from frames at different time points and infer of the line xt to that plane (c.f. Fig. 3b). Therefore, we deter-
the current pose of the needle in each frame. This section mine the final update for #» x t by intersecting the predicted
develops the state and measurement transition of the extended
Kalman Filter (EKF) that nonlinearly filters our measure-
ments.

123
International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796 791

(a) (b)

Fig. 3 Geometric modeling. a The cross section of the tool on a single is corrected by the ellipse parameters measured at time t and leads to
B-Scan allows to determine its 3D axis  up to one ambiguity. b The final estimate t . The blurred positions in image planes ρ indicate the
Kalman filter resolves this ambiguity and relates between time steps: A estimated error covariances
linear motion model of time step t − 1 gives a line prediction ˆt ; this

 T
tool center line ˆt defined by #»
x ∗t , θt and φt with the image #» λ2
ẑt = h(xt ) + vt = c tT , cos α, + vt (5)
plane ρt−1 of the next measurement which is defined by the λ1
control vector ut−1 . Solving the intersection between ˆt and
ρt results in a nonlinear state transition for #»x:
where ct are the 3D coordinates of the measured ellipse center
#» and the measurement noise vt ∼ N (0, R) is assumed as
#» p t−1 − #» x ∗t · n̄ t−1
x t = ˆt ∩ ρt−1 = #»
x ∗t + · l¯t (3) additive Gaussian noise. The above equation is motivated by
l¯t · n̄ t−1 Eq. (1), together with the fact that #» c t is on the tool axis,
#» #»
and we can thus set c t = x t . The other components of vt
With Eqs. (2) and (3), our state transition function f : xt → are not directly related to ẑt . The standard EKF innovation
f (xt−1 , ut−1 , wt ) is fully specified. Re-basing the line onto equation is St = Ht Pt|t−1 HtT +Mt RM T
 t . Again, weare able
 
the next OCT plane allows the Kalman filter to retain low to determine the Jacobian Ht = ∂∂xẑt  and Mt = ∂h ∂v  =
error covariance for the position of the line due to the result- x̂t x̂t|t−1
ing simple measurement transition, as opposed to letting the I analytically.
base point of the tool line be arbitrary and performing the Due to our choice of representation and parametrization,
line-plane intersection as part of the measurement transition. two mathematical singularities arise: Elements of Ft−1 and
The more complex update of the position implies that the Ht tend to infinity for θt → kπ, k ∈ Z as well as for t ·
new position is nonlinearly dependent on the process noise n t−1 → 0. The first implies that the needle is parallel to
variable at . Therefore, we use the formulation of the EKF the A-Scan direction, while the second represents the needle
with nonlinear noise to update the predicted error covariance parallel to the OCT plane, so for both cases we are not able
matrix as to see elliptical cross sections in the OCT frame. We avoid
numerical instabilities in our predictions by only updating the
predicted state if an ellipse has been detected and increasing
P t|t−1 = F t−1 P t−1|t−1 F 
t−1 + L t−1 Q L t−1
T
(4) the time step Δt for the next frame when no ellipse could be
detected.
where the P t−1|t−1 is the estimated error covariance matrix
of the previous time step and the Jacobian
 matrices of the

state transition function F t−1 = ∂∂ xf  and L t−1 =
 x̂ t−1|t−1 ,ut−1 Experiments and results
∂f
∂w  are derived analytically.
x̂t−1|t−1 ,ut−1 In this section, we evaluate the performance of our algorithm
regarding different aspects: We first discuss computational
Measurement transition performance and how we estimated the measurement noise
covariance matrix. Then, we analyze the algorithm in a series
From a single B-Scan, we find the parameters of the cross- of experiments on both anterior and posterior segment in
sectional ellipse as described above. We define the measure- phantom eyes and ex vivo porcine eyes in terms of movement
ment transformation stability and robustness to pathologies. Finally, we demon-

123
792 International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

strate an application example of our algorithm which consists fixed at a constant orientation, we move it only along one
of an injection guidance application. axis of the micromanipulator in order to determine the influ-
ence of translations on the pose estimation. Figure 4a shows
Parameters and initialization the effect of lateral motion on the estimated direction of the
needle. It can be seen that our method greatly reduces the
For the minimum distance of candidate points to the tissue variations in θ . Furthermore, the baseline method confuses
layer, we set dmin to 20 px, which corresponds to 50µm. lateral movement as a change of angle, resulting in a visible
The measurement noise covariance matrix R is determined correlation between φbase and the lateral movement direction,
by analyzing the covariance of the ellipse parameters across which we indicate by red/blue bar (red means vx < 0, i.e.,
several data sets where the needle is at an unknown but fixed movement to “left”). Our method does not exhibit this effect
angle with respect to the B-Scans. Based on the physical as it models not only position but also velocity. The same
interpretation of the process noise as an unknown acceler- effect can be seen for needle movement along the Z -Axis
ation induced by the surgeon (c.f. “Method” section), we (c.f. Fig. 4c), where the movement influences the azimuthal
empirically choose values for #» x = 3.0 mm/s , aθ =
a #» 2 angle θ of the baseline estimate. Variation in θ is higher com-
aφ = 60 deg/s and derive Q based on the definition of
2 pared to φ due to the higher uncertainty of ellipse detection
wt . These parameters are used across all experiments. In in the axial than in the lateral direction. As we cannot pro-
all our experiments, we initialize the Kalman filter with duce precise rotation with the mechanical micromanipulator,
x0 = ( #»
x 0 , θ0 , φ0 , 0, 0, 0) with parameters of the line fitted we instead record an image sequence during which the OCT
through the centers of the first two detected ellipses. scan pattern is rotated around the z axis and then manipulate
the metadata to ignore the known rotation, yielding an image
Computational performance sequence that is equivalent to rotating the tool around the z
axis. The analysis of the fitted orientation in Fig. 4b shows
The prediction and estimation steps for the EKF reduce to that our algorithm is able to reliably reconstruct this rotation
matrix operations on matrices not larger than 10 × 10 ele- while being less susceptible to noise in the ellipse estimation
ments, which can be implemented very efficiently. Therefore, of each frame. A slightly delayed angular adaptation of our
the most computationally intensive part is the processing of method is noticeable due to the smoothing property of the
each B-Scan to detect the ellipse and find its parameters. Kalman filter. However, we argue that a rotation as strong as
Our CPU-based native implementation (C++) with circle in these data set rarely occurs in ophthalmic surgery, where
fitting and without pathology handling is able to process needle movement is generally very slow and controlled.
1024 × 1024 px B-Scans in under 5.4 ms (186 FPS), on
a Notebook with an Intel Core i7-6820HQ CPU @ 2.70 GHz Ex vivo experiment
and 16GiB RAM. We are therefore are easily able to process
the OCT framerates of current iOCT engines, which range To evaluate the transfer toward real scenarios, we performed a
around 27–32 FPS at this resolution. similar experiment on enucleated porcine eyes. We acquired a
series of 15 anterior data sets from 5 different eyes, each with
Movement stability evaluation the same setup with a fixed needle angle and lateral move-
ment. Figure 5a shows that our algorithm can still robustly
We evaluate the movement stability of the proposed method estimate the translation and greatly reduce the variance in the
on both phantom and ex vivo porcine eyes. Since a compari- estimated angle.
son to optical tracking methods or other traditional methods
is not feasible, we employ a mechanical micromanipulator Irregular tissue and ellipse detection
for generating ground truth in terms of known, precise 3DOF
movements along an axis with fixed direction for the surgical To investigate the robustness of our method in more chal-
tool. To evaluate the quality of our estimation, we compare lenging cases, we have tested the following modifications
our method to line fitting through the ellipse centers of two on one of the ex vivo data sets with lateral needle move-
subsequent images. ment (c.f. Fig. 5b–d): To simulate a needle being too close to
the tissue to be found by our ellipse detection, we force the
Phantom experiments ellipse detection to fail in two of five B-Scans. It can be seen
that our method is able to retain stable tracking. To simulate
A first set of experiments was performed with a 27 G needle pathologies, we shift the candidate points pcand to resem-
in an otherwise empty field of view with an OCT scanning ble an irregularly shaped retina (Fig. 5c). The experiment
pattern of five parallel B-Scans to assess the stability of our shows that our method using a circular tissue model alone
algorithm to different kinds of movements. With the needle (θ∅,path , φ∅,path ) is problematic in the presence of patholo-

123
International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796 793

(c)
[°] [°] [°] [°]
ours ours base base ours ours base base 130
(a) (b)
100 160 120

110
90
140
100
80

angle [°]
120 90

angle [°]
70 80
100
60 70

80 60
50
50
60
40
40

30 40
0 5 10 15 20 0 5 10 15 ours base ours base
t [s] t [s]

Fig. 4 Phantom evaluation. Parameters with subscript base are from Needle rotation around the z-axis simulated by rotation of the OCT scan-
line fitting while ours indicate the proposed method. a Needle move- ning pattern (green). The known rotation angle φ is recovered robustly
ment lateral to the B-Scan direction. Due to the fixed needle orientation, while our method shows better stability regarding the expected con-
θ and φ are expected to be constant. The baseline method exhibits higher stant angle θ. c Box plot of estimated angles during axial movement.
variation which, in the case of φbase , is correlated with the lateral move- Our method shows much reduced variation and therefore better results
ment direction, while our method retains a more stable orientation. b regarding the reconstructed orientation

(a) (b)
0.1
ø ø ø,path ø,path ø,skip ø,skip ~,enh ~,enh
ours
0.09 ours
(d)
100
base

0.08 base

80
0.07

0.06 60
Var [rad 2 ]

0.05
40
(c)
0.04

20
0.03

0.02 0

0.01
-20

0
0 5 10 15 10 11 12 13 14 15 16 17 18 19 20
Data Set Number t [s]

Fig. 5 Ex vivo evaluation. Reconstruction of the needle orientation dur- tion (Step 4) can distinguish pathology candidates (blue) from ellipse
ing lateral movement. Movement stability: a analysis of the variance of points (green). d The circle fit (θ∅ , φ∅ ) performs worse for the same
the estimated orientation during lateral movement with fixed orienta- movement if pathologies are present (θ∅,path , φ∅,path ). A polynomial
tion. Our method shows reduced variance for both angles in all data sets. tissue model with pathology handling can reconstruct the needle ori-
Robustness to irregular tissue or ellipse detection failure: b Robustness entation (θ∼,enh , φ∼,enh ). Needle axis stability is also maintained when
to failing ellipse detection is verified by simulating failed detection in ellipse can only be detected in three of five B-Scans due to the needle
B-Scans marked as red. c Polynomial fit and additional pathology detec- touching the tissue in the other scans (θ∅,skip , φ∅,skip )

123
794 International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

(b)
ours
[°]
ours
[°]
base
[°]
base
[°] D [10-1 mm]

(a)
100

80

60

40

20

0
0 1 2 3 4 5 6 7
t [s]

Fig. 6 Freehand movement in ex vivo experiment. a Analysis of provide a stable tracking. b Microscope view with OCT scanning loca-
freehand needle movement in posterior segment while scanning a cross- tion overlaid in blue. Yellow circles indicate the centers of the detected
pattern of two perpendicular B-Scans. D (green line) is the distance ellipse from the two B-Scans. Orange line is the estimated line from the
between estimated tool axis and intersecting line of the two B-Scans. baseline method. Green line is our estimated, highlighting the benefit
Linear fitting fails to compute a reliable pose while our method can still of using the ellipse shape for more stable tracking

gies. Thus, if they are expected from preoperative data (or this injection point to give the surgeon a good impression
the tradeoff can be generally accepted), enhanced pathol- of the current needle depth. Manual repositioning is, how-
ogy handling (φ∼,enh , φ∼,enh ) can successfully recover stable ever, not feasible. We use the proposed algorithm to track the
tracking, at the cost of higher per-frame processing time to injection needle and show the projected injection point over-
7.1ms. layed on the microscope image. To estimate the intersection
point, we first reconstruct the target surface by using the tis-
Pattern comparison and freehand movement sue surface points peye of the ellipse detection stage of each
B-Scan, which correspond to pixels on the RPE for posterior
We performed an experiment with freehand movement of images (c.f. Fig. 2b). We reproject the points peye from sev-
the needle inside the OCT region while scanning with a pat- eral B-Scans to 3D space and fit a sphere using RANSAC.
tern consisting of only two perpendicular B-Scans. Figure 6a The intersection points of the tracked tool axis with the esti-
shows the orientation of the sequence once again compared to mated sphere are projected to the camera coordinate system
the baseline method. This shows the baseline method being using the 2D calibration provided by the manufacturer, which
unable to provide a meaningful estimate when subsequent is valid in the current microscope focus plane. We draw a
points are too close together, which is the case when the circle corresponding to the needle thickness to indicate the
needle moves closer to the intersection of the two B-Scans injection point. Thus, we provide distance perception with-
(Fig. 6b). It can be seen that our method is susceptible to out explicitly tracking the needle tip, as the surgeon can infer
bad initialization by the linearly fitted line through the first the distance of the needle to the surface by the distance of the
frames; however, it is able to converge to a stable tracking projected intersection point and the instrument tip visible in
after a few seconds and retain this pose even when the needle the camera image (Fig. 7).
is close to the center. This demonstrates that our estimator can
still infer the needle orientation from the ellipse shape when
the ellipse centers alone do not provide enough information. Conclusion

Injection guidance application We presented a novel algorithm for tracking a surgical needle
in 3D space solely using the high-resolution, cross-sectional
As an example application, we have designed an assistance OCT view. The method makes no assumptions on the layout
application that provides injection guidance during subretinal of the scanning pattern and is therefore easily integratable
injection by showing the surgeon the projected intersection into existing systems with dynamically changing scanning
point of the tracked needle with the target layer. During an patterns. We avoid expensive computations by geometric
actual injection, the OCT would be optimally placed through modeling, and consequently, our method is able to update at

123
International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796 795

Fig. 7 Screenshot of the injection guidance application. Left: Aug- and blue bars for illustrative purposes. Right: schematic view of the 3D
mented view of the surgical scene, showing the camera view with the relationships between B-Scans (blue), current needle estimate (green),
overlaid OCT scanning locations as well as the projected intersection and intersection point with the target surface (red). These relationships
point with the RPE layer. Current and last B-Scan are marked with white cannot easily be inferred from a simple 2D microscope image

more than 180 FPS, which easily matches the high framerates References
provided by today’s OCT engines. Experimental evaluation
on both phantom and ex vivo porcine eyes shows that the 1. Roodaki H, di San Filippo CA, Zapp D, Navab N, Eslami A (2016)
A surgical guidance system for big-bubble deep anterior lamellar
method is able to compensate needle movement between sub-
keratoplasty. In: MICCAI. Springer, pp 378–385
sequent B-Scans, thus showing increased robustness toward 2. Ehlers J (2016) Intraoperative optical coherence tomography: past,
B-Scan latency and generating a more stable estimate com- present, and future. Eye 30(2):193–201
pared to line fitting. We demonstrate the usefulness of our 3. Richa R, Balicki M, Meisner E, Sznitman R, Taylor R, Hager G
(2011) Visual tracking of surgical tools for proximity detection in
tracking algorithm by providing a simple augmented real-
retinal surgery. In: IPCAI, pp 55–66
ity scenario for subretinal injection. As our ellipse detection 4. Baek YM, Tanaka S, Kanako H, Sugita N, Morita A, Sora S,
fails for B-Scans where the needle is inside the tissue, we Mochizuki R, Mitsuishi M (2012) Full state visual forceps tracking
currently rely on the needle to be visible above the tissue under a microscope using projective contour models. In: Proceed-
ings of IEEE ICRA, pp 2919–2925
in at least one B-Scan to maintain our tracking. A possible 5. Sznitman R, Becker C, Fua P (2014) Fast part-based classification
future extension is to extend the ellipse detection to sup- for instrument detection in minimally invasive surgery. In: Golland
port cases where the needle is already penetrating tissue to P, Hata N, Barillot C, Hornegger J, Howe R (eds) MICCAI 2014.
improve tracking accuracy during this critical phase. Fur- LNCS, vol 8673. Springer, Heidelberg, pp 692–699
6. Allan M, Chang PL, Ourselin S, Hawkes DJ, Sridhar A, Kelly
thermore, the needle guidance application can be extended J, Stoyanov D (2015) Image based surgical instrument pose esti-
by a more robust target surface reconstruction to provide a mation with multi-class labelling and optical flow. In: MICCAI.
more precise intersection point visualization and perform a Springer, pp 331–338
validation study on the benefits of such a system. 7. Rieke N, Tan DJ, di San Filippo CA, Tombari F, Alsheakhali M,
Belagiannis V, Eslami A, Navab N (2016) Real-time localization of
articulated surgical instruments in retinal microsurgery. Med Image
Compliance with ethical standards Anal 34:82–100
8. Rieke N, Tan DJ, Tombari F, Page Vizcaíno J, di San Filippo
CA, Eslami A, Navab N (2016) Real-time online adaption for
robust instrument tracking and pose estimation. In: International
Conflict of interest The authors declare that they have no conflict of conference on medical image computing and computer-assisted
interest. intervention. Springer, pp 422–430
9. Garcia Peraza Herrera L, Li W, Gruijthuijsen C, Devreker A, Atti-
Ethical statement All procedures performed in studies involving ani- lakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T,
mals were in accordance with the ethical standards of the institution Ourselin S (2016) Real-time segmentation of non-rigid surgical
or practice at which the studies were conducted. This article does not tools based on deep learning and tracking. In: CARE work-
contain patient data. shop at international conference on medical image computing and
computer-assisted intervention. Springer
10. Laina I, Rieke N, Rupprecht C, Vizcaíno JP, Eslami A, Tombari
F, Navab N (2017) Concurrent segmentation and localization
for tracking of surgical instruments. In: International conference

123
796 International Journal of Computer Assisted Radiology and Surgery (2018) 13:787–796

on medical image computing and computer-assisted intervention 12. Zhou M, Roodaki H, Eslami A, Chen G, Huang K, Maier M,
2017. Springer, pp 664–672. The first two authors contributed Lohmann C, Knoll A, Nasseri M (2017) Needle segmentation in
equally to this paper volumetric optical coherence tomography images for ophthalmic
11. Kurmann T, Neila PM, Du X, Fua P, Stoyanov D, Wolf S, Sznitman microsurgery. Appl Sci 7(8):748
R (2017) Simultaneous recognition and pose estimation of instru- 13. Jazwinski AH (2007) Stochastic processes and filtering theory.
ments in minimally invasive surgery. In: International conference Courier Corporation, New York
on medical image computing and computer-assisted intervention.
Springer, pp 505–513

123