ORIGINAL ARTICLE

Computer Vision Tool and Technician as First Reader

of Lung Cancer Screening CT Scans
Alexander J. Ritchie, MD,a,b Calvin Sanghera,a Colin Jacobs, PhD,c Wei Zhang, MD,a,d
John Mayo, MD,e Heidi Schmidt, MD,f,g Michel Gingras, MD,h Sergio Pasian, MD,h
Lori Stewart, MD,i Scott Tsai, MD,i Daria Manos, MD,j Jean M. Seely, MD,k
Paul Burrowes, MD,l Rick Bhatia, MD,m Sukhinder Atkar-Khattra, BS,a
Bram van Ginneken, PhD,c Martin Tammemagi, PhD,n Ming Sound Tsao, MD,o,g
Stephen Lam, MD,a,* for the Pan-Canadian Early Detection of Lung Cancer
Study Group
a

Department of Integrative Oncology, British Columbia Cancer Research Center, Vancouver, Canada
Royal Brisbane Hospital, Brisbane, Australia
c
Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center,
Nijmegen, The Netherlands
d
Peking University First Hospital, Beijing, China
e
Department of Radiology, Vancouver Coastal Health, Vancouver, Canada
f
Department of Radiology, University Health Network-Princess Margaret Cancer Centre, Toronto, Ontario, Canada
g
Toronto General Hospital, Toronto, Ontario, Canada
h
Department of Radiology, Institut Universitaire de Cardiologie et de Pneumologie de Qubec, Quebec, Canada
i
Department of Radiology, Juravinski Hospital and Cancer Center, Hamilton, Ontario, Canada
j
Department of Radiology, Dalhousie University, Halifax, Nova Scotia, Canada
k
Department of Radiology, The Ottawa Hospital and University of Ottawa, Ottawa, Canada
l
Department of Radiology, University of Calgary, Calgary, Alberta, Canada
m
Department of Radiology, Memorial University, Newfoundland, Canada
n
Department of Health Sciences, Brock University, St. Catharines, Ontario, Canada
o
Department of Pathology, University Health Network-Princess Margaret Cancer Centre, Toronto, Ontario, Canada
b

Received 10 December 2015; revised 15 January 2016; accepted 27 January 2016

ABSTRACT
Objectives: To implement a cost-effective low-dose
computed tomography (LDCT) lung cancer screening
program at the population level, accurate and efcient
interpretation of a large volume of LDCT scans is needed.
The objective of this study was to evaluate a workow
strategy to identify abnormal LDCT scans in which a technician assisted by computer vision (CV) software acts as a
rst reader with the aim to improve speed, consistency, and
quality of scan interpretation.
Methods: Without knowledge of the diagnosis, a technician
reviewed 828 randomly batched scans (136 with lung cancers, 556 with benign nodules, and 136 without nodules)
from the baseline Pan-Canadian Early Detection of Lung
Cancer Study that had been annotated by the CV software
CIRRUS Lung Screening (Diagnostic Image Analysis Group,
Nijmegen, The Netherlands). The scans were classied as
either normal (no nodules
1 mm or benign nodules) or
abnormal (nodules or other abnormality). The results were
compared with the diagnostic interpretation by PanCanadian Early Detection of Lung Cancer Study radiologists.

Results: The overall sensitivity and specicity of the technician in identifying an abnormal scan were 97.8% (95%
condence interval: 96.498.8) and 98.0% (95% condence
interval: 89.599.7), respectively. Of the 112 prevalent
nodules that were found to be malignant in follow-up,
92.9% were correctly identied by the technician plus CV
compared with 84.8% by the study radiologists. The
average time taken by the technician to review a scan after
CV processing was 208 120 seconds.
Conclusions: Prescreening CV software and a technician as
rst reader is a promising strategy for improving the

*Corresponding author.
Disclosure: The authors declare no conict of interest.
Address for correspondence: Stephen Lam, MD, British Columbia
Cancer Agency, University of British Columbia, 675 West 10th Ave.,
Vancouver, BC, V5Z 1L3, Canada. E-mail: slam@bccancer.bc.ca
2016 International Association for the Study of Lung Cancer.
Published by Elsevier Inc. All rights reserved.
ISSN: 1556-0864
http://dx.doi.org/10.1016/j.jtho.2016.01.021

Journal of Thoracic Oncology

Vol.

No.

-: ---

2 Ritchie et al

consistency and quality of screening interpretation of LDCT

scans.
 2016 International Association for the Study of Lung
Cancer. Published by Elsevier Inc. All rights reserved.
Keywords: Lung cancer; Screening; Early detection; Computed
tomography

Introduction
On the basis of the U.S. Preventive Services Task Force
recommendations1,2 and the decision by the Centers of
Medicare and Medicaid Services to cover low-dose
computed tomography (LDCT) screening of Americans
aged 55 to 77 years who have smoked for at least 30
pack-years,3 lung cancer screening is being implemented
in the U.S. health care system and in many other countries. In addition to annual screening CT scans, both the
National Lung Screening Trial (NLST) and the DutchBelgian Lung Cancer Screening Trials found that additional scans were required to evaluate suspicious or
indeterminate abnormalities in approximately 20% of
individuals after the baseline round of screening and in
4% to 8% of individuals after annual repeat screening
LDCT.48 An estimated 8.6 million Americans are potentially eligible for LDCT screening.9 To enable implementation of the LDCT portion of a cost-effective lung
cancer screening program, a number of practical barriers
must be addressed, including the following: (1) accurate
interpretation of the large volume of LDCT scans, (2)
provision of a quality assurance program for scan interpretation, and (3) consistent application of standardized
evidence-based follow-up management of screendetected lung nodules.
Computer-aided nodule detection or computer vision
(CV) has the potential to improve both the efciency and
accuracy of LDCT scan interpretation. Automated interpretation and characterization of nodules discovered by
LDCT using CV followed by review by a trained technician
can potentially select a subset of scans that require
radiologist review. CV has the advantage of being able
to process large volumes of scans around the clock.
Because more than 96% of screening CT scans do not
harbor a lung cancer, we hypothesize that a trained
technician assisted by CV software could triage and optimize selection of screening CT scans for subsequent review by radiologists. This strategy is similar to that use in
manual interpretation of Papanicolaou tests in cervical
cancer screening programs, according to which only
abnormal Papanicolaou smears are interpreted by cytopathologists. Normal smears are read and signed off
by trained technologists. The objective of our study was to
explore an alternative strategy for reading screening LDCT
scans by determining whether a nonradiologist technician

Journal of Thoracic Oncology

Vol.

No.

assisted by CV software can accurately and efciently

identify and triage abnormal LDCT scans for subsequent
review by a radiologist to reduce the workload and subsequently improve the speed, consistency, and quality of
scan interpretation. A secondary objective of our study
was to determine the proportion of scans that would be
triaged as abnormal using different cutoff criteria
(NLST,4,5 Lung CT Screening Reporting and Data System
[Lung-RADS],10 and Pan-Canadian Early Detection of Lung
Cancer Study [PanCan] risk score11,12) and the sensitivity
of each method in identifying scans with lung cancer.

Materials and Methods

LDCT Scan Selection
From 2537 candidates in the previously reported
PanCan Study,11 828 baseline LDCT scans were selected
to sample three distinct groups: pathologically proven
malignant lung nodules, follow-upproven benign lung
nodules, and no nodule detected prospectively. In the
entire PanCan study there were a total of 141 pathologically proven (by biopsy or resection) malignant lung
nodules, 136 of which were included. Images of four
participants were not available, and one CT scan could
not be processed by the CV software because of problem
with the Digital Imaging and Communications in Medicine data (Fig. 1). The remaining 692 cases were
randomly selected from two other groups: 556 CT scans
from 1821 participants with follow-upproven benign
nodules with a range of nodule sizes and 136 CT scans
from 575 participants without lung nodules 1 mm or
larger as prospectively determined by the PanCan study
radiologists. The absence of malignancy in the benign
CV Soware Processing (n=828)
136 with proven cancer

556 with benign nodules

136 with no nodules

Technician reviews scans that have been analyzed and marked by CV Soware
1. Review CV markings, accept or reject
2. Add any addional nodules
3. Scan classied as Abnormal or Normal

Abnormal Scan
Nodules
Other concerning feature
(e.g. lymphadenopathy, ILD)

Normal Scan:
No nodules
No concerning features

Categorisaon of LDCT Scans compared to PanCan Radiologist Determinaon.

Discordant scan categorizaon reviewed in a joint reading with experienced
chest radiologist (J.RM.).

Figure 1. Study design. CV, computer vision; ILD, interstitial

lung disease; LDCT, low-dose computed tomography; PanCan,
Pan-Canadian Early Detection of Lung Cancer Study.

---

2016

nodule and no nodule groups was conrmed by biopsy

or through a minimum follow-up of at least 2 years
(mean 4.9 1.2 years).

LDCT Scan Technique

LDCT scans were obtained at the nine sites by using
multidetector row CT scanners with at least four detector rows, greater than 1.25-mm detector collimation, 120
kVp, 40 to 50 mAs, and less than a 1-second tube rotation time. Contiguous high and intermediatespatial
frequency algorithm images were reconstructed in the
transaxial plane using a thickness of 1.25 mm or less.
Coronal and sagittal reformats and maximum intensity
sliding slab images were generated at the discretion of
the reviewing radiologist. In the prospective reading, a
designated trained subspecialty chest radiologist at each
site reviewed the CT scans.10

CIRRUS CV Lung Screening Software

A research workstation specically designed to read
screening LDCT scans (CIRRUS Lung Screening, Diagnostic
Image Analysis Group, Department of Radiology and
Nuclear Medicine, Radboud University Medical Center,
Nijmegen, the Netherlands) was used. CIRRUS Lung
Screening integrates three computer-aided detection
(CAD) software packages that identify solid and subsolid
lung nodules. The performance of the CV software has
been previously published.1315 The program also generated a structured report highlighting nodule features and
incorporating the calculated nodule malignancy risk score
according to McWilliams et al. (Supplementary Fig. 1).11

Technician Training
The rst part of this study involved a training process
for a nonradiology technician (C.S.) with no prior experience in reading chest CT scans. Initially, the technician
was trained to operate the lung screening workstation.
Next, the technician was trained to identify potential
lung abnormalities in low-dose screening thoracic CT
scans using a training set of 40 cases that were separate
from the study set. The training set contained representative archetypal abnormalities of calcied granulomas and solid, part-solid (semisolid), and nonsolid
nodules. The training set was processed automatically by
the software. These processed scans with annotation
marks were rst reviewed by the technician independently and then reread under the supervision of one of
the study pulmonologists (A. R.) during interactive
feedback sessions. All nodules annotated by the technician were reviewed and any nodules missed or overcalled were addressed. All marked nodules were dened
by type (solid, semisolid, nonsolid, perissural, or calcied), segmental lobar location, and morphology (well

CV Reading of Screening CT

dened, lobulated, or spiculated). The presence or

absence of emphysema (
5%) was recorded in a binary
fashion without reference to severity. The technician was
not trained to recognize mediastinal or hilar adenopathy
or incidental ndings in the breasts or upper abdomen.

Screening LDCT Scan Analysis

The study workow is illustrated in Figure 1.
The anonymized scans were randomly allocated into
nine batches for processing by the CV software. The scans
were batched for the purposes of workow management
and to determine any learning effects. Group allocation
and nal diagnoses were concealed from the technician.
The technician reviewed the processed cases and the
annotated markings of abnormalities by the CV software.
CAD marks on true nodules were accepted and falsepositive CAD marks were deleted. The technician then
reviewed the scans and manually marked any additional
nodules that were not identied by the CV software. The
software performed an automatic segmentation for both
solid and subsolid nodules that could be optimized
manually. After reviewing the scan, the technician categorized the study as abnormal (requiring radiologist review) or normal (potentially not requiring radiologist
review). An abnormal study was dened as one containing 1-mm or larger nodules or other abnormalities (such
as brosis). Normal studies did not have any nodules or
other abnormalities identied by the software or by the
technician, or they contained only completely calcied
nodules. Scans with lung nodules also have automated
display of the lung nodule malignancy risk score using the
method previously reported (Supplemental Figure 1).11
Discordant ndings between the readings by the technician and the PanCan radiologist were later reviewed with
an experienced chest radiologist (J.R.M.) with 14 years
experience in lung cancer screening to resolve differences
after all the scans had been read and the results recorded.

Outcomes
The primary outcome of this study was to determine
the sensitivity and specicity of a technician assisted by
CV software in categorizing screening scans with and
without lung cancer as normal or abnormal compared
with the sensitivity and specicity of the PanCan Study
radiologists. Secondary outcomes were to determine the
proportion of scans that would be classied as abnormal
when using different guidelines and the sensitivity of
each method for identifying scans with lung cancer. Scans
with (1) one of more nodules at least 1 mm in diameter,
(2) one of more nodules at least 4 mm in diameter (the
NLST criteria4,5), (3) solid or semisolid nodules at least
6 mm in diameter or nonsolid (ground-glass) nodules at
least 20 mm in diameter (the Lung-RADS criteria10), or

4 Ritchie et al

(4) lung nodules with a malignancy risk score of 2% or

higher (the PanCan model11,12) were compared. The time
taken by the technician to review each scan that was
recorded automatically by the CIRRUS Lung Screening
software was also examined to determine the potential
workload implications of using a technician augmented
by CV software as the rst reader.

Data Analysis
Descriptive statistics were prepared using contingency table analysis for categorical data and Fishers
exact test. The 95% condence intervals (CIs) for proportions were estimated using the binomial exact
method. The sensitivity and specicity of the technician
augmented by CV software in triaging abnormal scans
for reading by a radiologist were calculated. Stata statistical software (version 14.1 MP, StataCorp LP, College
Station, Texas) was used to calculate statistics.

Results
The clinical characteristics of the 828 participants
whose CT scans were used in this study are shown in
Table 1. Of the 136 lung cancer cases, 112 were prevalent cancers with an identiable nodule on the baseline
scan. Twenty-four were incident cancers, in which the
cancer developed in an area that did not contain a lung
nodule on the baseline LDCT scan. Fifty-seven of the 136
cancers were diagnosed within 12 months whereas the
remaining 79 cancers were diagnosed on repeat imaging
studies at least 12 months after the baseline study. Two
of the prevalent cancers were endobronchial tumors
found by autouorescence bronchoscopy as part of a
parallel autouorescence bronchoscopy study. One of
the endobronchial tumorsa 5-mm carcinoid tumor
was retrospectively identied on the LDCT scan. The
other was a squamous cell carcinoma in situ in the left
upper lobe bronchus that was not visible on the LDCT
scan. The average time to diagnosis from baseline screen
was 16.8 16.2 months (range 0.461.9 months) for the
prevalent cancers and 35.7 14.7months (range 10.3
54.7 months) for the incident cancers.

Technician Performance
Of the 136 scans that were classied as normal
(containing no lung nodules) by the PanCan radiologists,
85 were found to have noncalcied nodules by the
technician and CV software (Supplemental Table 1). The
average size of these nodules was 4.7 4.8 mm. Twentyseven of these nodules (31%) were at least 4 mm. Of the
556 scans that were classied by PanCan radiologists as
containing lung nodules and turned out to be benign, 15
(2.7%) were misclassied as normal by the technician.
With regard to the 136 scans from participants with lung

Journal of Thoracic Oncology

Vol.

No.

cancer, the technician triaged all 136 scans as abnormal,

including the 24 cases of incident lung cancer, because of
the presence of other nodules on the scan. The overall
sensitivity and specicity of the technician plus CV in
identifying an abnormal scan using the 1 mm or larger
nodule size cutoff were 97.8% (95% CI: 96.498.8) and
98.0% (95% CI: 89.599.7), respectively. In addition to
determining the performance of the technician plus CV
software in identifying abnormal scans, we also examined the proportion of malignant lung nodules identied
by the technician plus CV software versus by the radiologists. With regard to the 112 prevalent lung cancer
cases, the technician identied 104 of 112 malignant
nodules (92.9%) compared with 95 of 112 (84.8%)
identied by the PanCan radiologists. The characteristics
of the lung nodules not annotated by the technician plus
CV software and/or by the PanCan radiologists are
shown in Table 2. Examples of cancer nodules missed
are shown in Figure 2.

Implication of Different Triage Criteria

The proportions of CT scans that would be triaged as
abnormal and the sensitivities in identifying CT scans
from participants in whom lung cancer was diagnosed
within 12 months of the rst CT scan if the technician
were instructed to use the NLST4,5 or Lung-RADS10 criterion for a positive screen or the PanCan nodule
malignancy risk threshold of 2% or higher11,12 are shown
in Table 3. Compared with if the cutoff criterion of 1 mm
or larger were used, the proportion of all CT scans that
would be classied as abnormal by the technician
decreased from 92.1% to 75.4% with the NLST criteria, to
64.4% with the Lung-RADS criteria, and to 59.7% with
the PanCan malignancy risk threshold of 2% or higher.
The corresponding sensitivities of identifying scans from
subjects in whom lung cancer was diagnosed within 12
months were 98.2%, 98.2%, 94.6%, and 98.2%, respectively. The characteristics of cancers missed by each criterion are identied by footnotes in the third column
of Table 3. The two ground-glass opacities that would
not be classied as abnormal on the basis of current LungRADS criteria but were found to be invasive adenocarcinoma are shown in Figure 3. One of these two adenocarcinomas contained a micropapillary component
an adenocarcinoma subtype with poor prognostic
outcomeaccounting for approximately 10% of it.17

Technician Reading Efciency

Screening by the technician was quite time efcient.
The average time required for the technician to review a
LDCT scan for nodules and generate a structured report
was 208 seconds (range 471018). Supplemental Figure 2
demonstrates the rapid learning curve of the technician.

---

2016

CV Reading of Screening CT

Table 1. Characteristics of the Participants in the Study

Baseline LDCT Scan Group Allocation Determined by the PanCan Radiologists
Characteristic

Cancera

Benign Lung Nodule

No Nodule or
Calcied Nodule

No. Subjects
Average age, y
Female sex, (%)
Emphysema present, (%)
Active smoker, (%)
Average pack-years SD
Nodule count per scan
Median
Range
Maximum nodule size, mm (mean SD)b
Nodule typec
Nonsolid (GGN)
Part solid (SSN)
Solid
Follow-up, y
Mean SD
Histopathologic ndingse
AIS
Invasive adenocarcinoma
Squamous cell carcinoma
NSCLC NOS
Small cell
Other
Cancer stage at diagnosisf
AIS
IAf
IB
IIA
IIB
IIIA
IIIB
IV
Small cell

136
64 6
71 (52%)
101 (74%)
86 (63%)
55.6 23.4

556
63 6
238 (43%)
348 (61%)
344 (62%)
54.5 23.1

136
62 6
55 (40%)
83 (61%)
86 (63%)
55.0 24.5

4.8
117
14.1 9
Total
158
59
406

4.8
123
9.6 6

Not applicable

% malignant (n)d
18% (29)
42% (25)
16.5% (66)

4.8 1.3

425
134
2165
4.99 1.0
Not applicable

5.05 0.7
Not applicable

Not applicable

Not applicable

4
99
19
4
7
3
2
73
15
5
4
14
7
11
6

Includes 24 participants with incident cancer. They had a CT scan for follow-up of other nodules that did not turn out to be malignant.
Largest nodule if more than one nodule in the same CT scan.
All nodules in each CT scan were included.
d
Of the 111 prevalent lung cancer cases, two were CT occult (one endobronchial tumor and one carcinoma in situ found by autouorescence bronchoscopy). In
the remaining 109 lung cancer cases, there were a total of 120 lung cancers.
e
World Health Organization classication.
f
International Association for the Study of Lung Cancer Staging Guidelines, 7th edition.16
LDCT, low-dose computed tomography; PanCan, Pan-Canadian Early Detection of Lung Cancer Study; SD, standad deviation; GGN, ground-glass nodule; SSN,
semi-solid nodule; NSCLC, nonsmall cell lung cancer; NOS, not otherwise specied; AIS, adenocarcinoma in situ.
b
c

Discussion
Using a strategy that is similar to that used in cervical
cancer screening programs (in which normal Papanicolaou tests are read and signed off by trained technologists and only abnormal tests are interpreted by
cytopathologists), our study demonstrates that a
technician screener assisted by CV software can efciently and accurately review and triage abnormal
LDCT scans for radiologist review with a sensitivity of
97.8% (95% CI: 96.498.8) and a specicity of 98.0%
(95% CI: 89.599.7). Because of the presence of other

abnormalities, 100% of the scans containing malignant

lung nodules were correctly triaged as abnormal and
requiring review by a radiologist. When the PanCan
nodule malignancy risk cutoff of 2% or more was used,
40% of the screening CT scans were categorized as lowrisk scans that could have a different level of clinical action by radiologists, either a quick review or no review
with quality assurance check. The average time required
for a technician to prescreen a scan was only 208 seconds
(range 471018), making it possible for abnormal CT
scans to be rapidly triaged to a radiologist for reading.

6 Ritchie et al

Journal of Thoracic Oncology

Vol.

No.

Table 2. Malignant Nodules in Baseline Scan Not Annotated by Technician Plus CV Software and/or PanCan Study
Radiologists
Scan No.

Nodule Type

Maximum Transverse
Diameter, mm

Nodules not annotated by both technician CV software and radiologists

1
GGO
9
2
Solid
7
3
Solid
2
5
4
Solida
5
5
Endobronchial Lesion found
on bronchoscopyb
Nodules not annotated by technician CV alone
6
Solid
18
7
Solid
25
8
Solid
6
Nodules identied on retrospective review of baseline CT by radiologists
9
Semisolid
10
10
Solid
7
11
Solid
20
12
Solid
8
13
Solid
4.2
14
Solid
5
15
Solid
9.2
16
Solid
8
17
Solid
7
18
GGO
10
19
Solid
5
20
Solid
5.8

Time to
Diagnosis, mo

Pathologic Diagnosis

Stage

51.1
25.3
13.5
17
4

Nonmucinous AIS
Adenocarcinoma
Squamous
Squamous
Carcinoid

IA
IIIA
IA
IIIA
IA

1.2
0.4
22.4

Adenocarcinoma
Adenocarcinoma
Large cell neuroendocrine

IA
IIB
IA

28
13.7
15.2
50.4
23.9
20.4
6.4
16.5
14
62
23.4
16

Nonmucinous MIA
Adenocarcinoma
Typical carcinoid
Adenocarcinoma, papillary
Adenocarcinoma, acinar
Adenocarcinoma
Squamous
Small cell
Adenocarcinoma, acinar
Nonmucinous MIA
Squamous
Squamous

IA
IA
IA
IA
IA
IA
IV
Limited
IA
IA
IA
IA

A 5-mm nodule between blood vessels with a borderline enlarged 11-mm 4R lymph node that was not initially reported. The participant dropped out of study
and presented clinically 17 months later with stage IIIA squamous cell carcinoma.
b
Participants were enrolled in a parallel autouorescence bronchoscopy screening study.
CV, computer vision; GGO, ground-glass opacity; AIS, adenocarcinoma in situ; CT, computed tomogrpahy, MIA, minimally invasive adenocarcinoma.

Scans with actionable ndings can potentially be

communicated to the patient and referring physician for
referral to a specialist to fast-track further investigation.
A rapid learning curve was seen for the technician, as
evidenced by the improvement in average scan-reading
time after the rst batch. This is likely due to increasing
familiarity with the software and the screening process.
Although all the scans containing malignant nodules were
triaged as abnormal by the technician because of the
presence of nodules elsewhere, two stage IA malignant
nodules and one mass lesion were missed in addition to
one small endobronchial carcinoid tumor and four 2- to
9-mm-diameter lung nodules that turned out to be malignant on follow-up and were also missed by the radiologists. Without the advantage of the CV technology, 17
of 112 of the nodules that turned out to be malignant
(15.2%) were not annotated by the radiologists in the
baseline scan reading. Although all except one of the
nodules (case 15 [Table 2]) that were missed were
identied in retrospective review of the baseline screen
and were stage IA tumors when diagnosed, a 9.2-mm
nodule that was missed turned out to be a stage IV
squamous cell carcinoma 6.4 months later. Signicant

CT scan reader variability was also found in the NLST,

with false-negative rates ranging from 3.5% to 8.1% and
false-positive rates ranging from 3.8% to 69.0%.18 The
strategy of having a technician aided by CV technology as
rst reader will likely improve the consistency and
quality of scan interpretation. Traditionally, CV studies
have focused on detecting and characterizing abnormal
lung nodules, but as a stand-alone tool, this approach has
not been widely adopted clinically because of insufcient
accuracy of nodule detection and the presence of falsepositive annotations.19 False-positive annotations can
increase radiologist workload. However, if a radiologist
can read an annotated scan with false-positive CV annotations removed and false-negative ndings added by a
technician, the workload will not increase and this may
overcome the barrier to adopting CV software technology. The nodule malignancy scores and nodules sizes
automatically generated by the CV software allow preset action thresholds to further reduce the number of
false-positive annotations as well to assist radiologists in
making clinical management recommendations. Although
the advantage of having a technician remove the falsepositive CV annotations and add false-negative ndings

---

2016

CV Reading of Screening CT

Figure 2. Examples of malignant lung nodules missed by the technician plus computer vision (CV) and/or Pan-Canadian Early
Detection of Lung Cancer Study radiologists. (A and B) Malignant nodules missed by the technician plus CV software as well as
by the radiologists. (A) A 5-mm endobronchial carcinoid tumor of the left lower lobe (box). (B) A 7-mm solid nodule (arrow)
diagnosed as stage IIIA adenocarcinoma 25.3 months later. (C and D) Malignant nodules missed by the technician plus CV
software. (C) An 18-mm solid nodule adjacent to left hilar vessels (arrow). Stage IA adenocarcinoma diagnosed 1.2 months
later. (D) A 25-mm stage IIB adenocarcinoma (box). (E and F) Malignant nodules missed by the radiologist. (E) A 9.2-mm solid
nodule of the right lower lobe (arrow). Stage IV squamous cell carcinoma diagnosed 6.4 months later. (F) An 8-mm solid
nodule of the right lower lobe (box). Limited-stage small cell lung cancer diagnosed 16.5 months later.

could be a temporary advantage that may disappear

when software becomes better and more reliable, prescreening by a technician aided by CV technology
serves as a quality assurance tool to minimize falsenegative manual readings. In this study, three nodules
(9.2-, 5-, and 8-mm nodules) missed by the radiologist
were later diagnosed as stage IV squamous cell carcinoma,
stage IIIA squamous cell carcinoma, and limited-stage
small cell lung cancer 6.4, 17, and 16.5 months later,
respectively (Table 2 and Fig. 2).
Regardless of the method used to stratify risk, it is
inevitable that some participants with very early lung
cancer on the baseline scan and rapidly growing
incident cancers will be categorized as low risk. We
assessed three such methodsthe NLST, Lung-RADS,
and PanCan criteria. In this retrospective analysis, we
found that the PanCan criterion of a nodule malignancy
risk of 2% or more performs the best in terms of
reducing the number of scans classied as abnormal
while maintaining the highest sensitivity in identifying
scans with lung cancer (Table 3). It should be noted
that the NLST, Lung-RADS, and PanCan criteria are not
entirely validated. Prospective evaluation of these
lung nodule management guidelines is needed to dene
the optimal action threshold. A limitation of the
study is that incidental ndings such as those in the
adrenals, liver, breasts, or mediastinum, as well as
nodules attached to hilar vessels, are not automatically
annotated by the CV software. However, recognition of
these blind spots would alert the radiologists to focus
on these areas during reporting. With rapid advances in

CV technology, identication of incidental abnormalities such as coronary artery calcication can be automated.20,21 The technician in this study received
minimal training in reading chest CT scans and yet was
able to achieve a high degree of accuracy with the CV

Table 3. Proportion of CT Scans Triaged as Abnormal by

Using Different Cutoff Criteria

Classication
Nodule
1 mm
Nodule
4 mm
(NLST)4,5
Solid/semisolid
nodule
6 mm
or GGO
20 mm
(Lung-RADS)15
Risk
2%
(PanCan)9,17
a

Proportion of
828 Scans
Classied as
Abnormal

Lung Cancers Diagnosed

at 12 mo (N 57) and
Sensitivity of Selecting
Scans with Lung Cancer
Using Each Set of
Classication Criteria

763 (92.1%)
624 (75.4%)

56 (98.2%)a
56 (98.2%)a

533 (64.4%)

54 (94.6%)a,b,c

494 (59.7%)

56 (98.2%)a

A 4-mm pT1aN0 endobronchial carcinoid tumor diagnosed by bronchoscopy

but missed by the technician plus CV software and also by the study
radiologist.
b
A 12-mm GGO pT1aN0 moderately differentiated adenocarcinoma of mixed
subtype (75% acinar, 25% lepidic) diagnosed 3.5 months after baseline LDCT.
c
A 19-mm GGO pT1aN0 adenocarcinoma (90% lepidic, 10% micropapillary)
diagnosed 2.7 months after baseline LDCT.
CT, computed tomography; NLST, National Lung Screening Trial; GGO, groundglass opacity; Lung-RADS, Lung CT Screening Reporting and Data System;
PanCan, Pan-Canadian Early Detection of Lung Cancer Study; LDCT, low-dose
computed tomography.

8 Ritchie et al

Journal of Thoracic Oncology

Vol.

No.

Figure 3. Invasive adenocarcinoma presented as ground-glass nodules less than 20 mm in diameter. (A) A 12-mm ground-glass
opacity. A pT1aN0 moderately differentiated adenocarcinoma of mixed subtype (75% acinar and 25% lepidic) diagnosed 3.5
months later (box). (B) A 19-mm ground-glass opacity (arrow). A pTIaN0 adenocarcinoma with a 90% lepidic and 10%
micropapillary growth pattern diagnosed 2.7 months later.

software. A more structured training program will

likely improve the accuracy further and minimize
missing signicant ndings.

Conclusion
A technician assisted by CV software can accurately
categorize abnormal scans for review by a radiologist.
Prescreening by a technician and CV software is a
promising strategy for reducing workload and improving
the speed, consistency, and quality of scan interpretation
of screening chest CT scans. To our knowledge, this is the
rst time that a nonradiologist screener has been tested
for this purpose. This strategy could enable large volumes of scans to be processed around the clock with
improved speed and consistency in quality, saving
valuable time of the limited number of expert chest radiologists to concentrate on abnormal scans. This strategy may facilitate health care system uptake of
population-based lung cancer screening programs and
overcome some of the barriers to cost-effective
screening programs to reduce lung cancer mortality.

Acknowledgments
Funding for this project was provided by the Terry Fox
Research Institute, the Canadian Partnership Against
Cancer, and the British Columbia Cancer Foundation.

Supplementary Data
Note: To access the supplementary material accompanying this article, visit the online version of the Journal of
Thoracic Oncology at www.jto.org and at http://dx.doi.
org/10.1016/j.jtho.2016.01.021.

References
1. Moyer VA, U. S. Preventive Services Task Force. Screening for lung cancer: U.S. Preventive Services Task
Force recommendation statement. Ann Intern Med.
2014;160:330338.

2. Humphrey L, Deffeback M, Pappas M, et al. Screening for

Lung Cancer: Systematic Review to Update the U.S.
Preventive Services Task Force Recommendation Statement. Evidence Synthesis No. 105. Rockville, MD: Agency
for Healthcare Research and Quality; 2013, AHRQ Publication No. 13-05196-EF-1.
3. Final National Coverage Determination on Screening for
Lung Cancer with Low Dose Computed Tomography
(LDCT)(CAG-00439N). http://www.cms.gov/.../details/
nca-proposed-decision-memo.aspx?NCAId274. Accessed February 5, 2015.
4. National Lung Screening Trial Research Team, Aberle DR,
Adams AM, et al. Reduced lung-cancer mortality with
low-dose computed tomographic screening. New Engl J
Med. 2011;365:395409.
5. National Lung Screening Trial Research Team, Church TR,
Black WC, et al. Results of initial low-dose computed
tomographic screening for lung cancer. New Engl J Med.
2013;368:19801991.
6. van Klaveren RJ, Oudkerk M, Prokop M, et al. Management of lung nodules detected by volume CT scanning.
New Engl J Med. 2009;361(23):22212229.
7. Aberle DR, DeMello S, Berg CD, et al. Results of the two
incidence screenings in the National Lung Screening
Trial. New Engl J Med. 2013;369:920931.
8. Horeweg N, van der Aalst CM, Vliegenthart R, et al.
Volumetric computed tomography screening for lung
cancer: three rounds of the NELSON trial. Eur Respir J.
2013;42:16591667.
9. Ma J, Ward EM, Smith R, Jemal A. Annual number of lung
cancer deaths potentially avertable by screening in the
United States. Cancer. 2013;119:13811385.
10. American College of Radiology. Lung CT Screening and
Reporting System (Lung-RADs). www.acr.org/QualitySafety/Resources/LungRADS. Accessed August 11, 2014.
11. McWilliams A, Tammemagi MC, Mayo JR, et al. Probability of cancer in pulmonary nodules detected on rst
screening CT. New Engl J Med. 2013;369:910919.
12. Tammemagi MC, Lam S. Screening for lung cancer using low dose computed tomography. BMJ. 2014;348:
g2253.
13. Jacobs C, Sanchez CI, Saur SC, et al. Computer-aided
detection of ground glass nodules in thoracic CT images

---

14.

15.

16.

17.

2016

using shape, intensity and context features. Med Image

Comput Comput Assist Interv. 2011;14:207214.
Jacobs C, van Rikxoort EM, Twellmann T, et al. Automatic detection of subsolid pulmonary nodules in
thoracic computed tomography images. Med Image Anal.
2014;18:374384.
Murphy K, van Ginneken B, Schilham AM, et al.
A large-scale evaluation of automatic pulmonary nodule
detection in chest CT using local image features and
k-nearest-neighbour classication. Med Image Anal.
2009;13:757770.
Chansky K, Sculier JP, Crowley JJ, et al. The International Association for the Study of Lung Cancer Staging
Project: prognostic factors and pathologic TNM stage in
surgically managed nonsmall cell lung cancer. J Thorac
Oncol. 2009;4:792801.
Yeh YC, Kadota K, Nitadori JI, et al. International Association for the Study of Lung Cancer/American Thoracic

CV Reading of Screening CT

18.

19.

20.

21.

Society/European Respiratory Society classication predicts occult lymph node metastasis in clinically mediastinal node-negative lung adenocarcinoma. Eur J
Cardiothorac Surg. 2016;49:e9e15.
Pinsky PF, Gierada DS, Nath PH, et al. National Lung
Screening Trial: variability in nodule detection rates in
chest CT studies. Radiology. 2013;268:865873.
Godoy MC, Cooperberg PL, Maizlin ZV, et al. Detection
sensitivity of a commercial lung nodule CAD system in a
series of pathologically proven lung cancers. J Thoracic
Imaging. 2008;23:16.
Isgum I, Prokop M, Niemeijer M, et al. Automatic coronary calcium scoring in low-dose chest computed tomography. IEEE Trans Med Imaging. 2012;31:23222334.
Jacobs PC, Gondrie MJ, van der Graaf Y, et al. Coronary
artery calcium can predict all-cause mortality and cardiovascular events on low-dose CT screening for lung
cancer. AJR Am J Roentgenol. 2012;198:505511.