Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 62

EXAMINING THE COMPELLINGNESS OF ELECTRONIC LOW

VISIBILITY TAXI CHARTS
Michelle Yeh,1 Joseph Jaworski,2 Suzanne Thomas,2 Andrew Kendra,3 Danielle Hiltunen4
1
Federal Aviation Administration
2
Cherokee CRC, LLC
3
United States Department of Transportation, Volpe National Transportation Systems Center
4
Stinger Ghaffarian Technologies, Inc.

The purpose of this study was to examine whether the presentation of own-aircraft (ownship) position was
compelling when presented on electronic low visibility taxi charts. Although airport charts showing ownship
position have been in use for some time, ownship position was not available on low visibility taxi charts
because these charts were not geo-referenced. Twenty Airline Transport Pilots (ATP) (10 flightcrews)
participated in a simulator study in which they performed six taxi scenarios in three different levels of
visibility (1200 Runway Visual Range (RVR), 600 RVR, 300 RVR) using an electronic chart application on
an iPad. Ownship position was shown on the chart for half the scenarios. In one scenario, we simulated a
position error. We collected objective data (taxi speed, taxi time, fixation and dwell time), and pilot opinions
on the usability of the electronic chart application. The results showed that no incursions/excursions were
committed. All flightcrews noticed the error in ownship position, when it occurred; in fact, they also noticed
other errors in ownship position that were not planned as part of the experiment design. Captains looked more
often at the electronic chart when ownship position was presented than when it was not, regardless of
visibility conditions. Additionally, Captains’ percentage of fixations were almost equal between the
electronic chart and out-the-window. Such behavior may reflect the perceived utility of showing ownship on
the electronic chart and may be an indication of the compelling nature of that information source.

INTRODUCTION extent that the imagery “captures” the pilots’ attention at the
cost of information beyond the symbology (the out-the-
Pilots are presented with a variety of information sources window view; Yeh & Wickens, 2001).
that must be integrated, from the out-the-window view to The issue of compellingness has also been explored
installed avionics to portable electronic devices (PEDs). The relative to use of electronic flight bag (EFB) applications and
attention allocated to each of these information sources may PEDs. Much of the evidence for the compellingness of these
be driven by several factors, including the effort to access displays comes from the general aviation domain (Part 23
information, the effort to extract the information, and the operations) where use of PEDs has led to increased head-down
perceived usefulness of the source. The value associated with time. Reports from the Aviation Safety Reporting System
each of these factors will vary depending on the task being (ASRS) provide examples where general aviation pilots
performed (Wickens, Goh, Helleburg, Horrey, & Talleur, committed errors (e.g., airspace violation) because they paid
2003). Underlying all these factors is the compellingness of more attention to incorrect or out-of-date global positioning
the information presentation. system (GPS) information on their portable device (e.g., smart
Not subject to U.S. copyright restrictions. DOI 10.1177/1541931218621014

The term “compelling” is defined in this paper as a phone, tablet, laptop) rather than their installed avionics
display property that attracts interest or attention potentially at displays. That is, information that was provided for strategic
the cost of attention to other information sources, such as other purposes was used tactically. Additionally, pilots noted they
displays or out-the-window. Compellingness can be observed were learning to use the device during the flight or that they
as attentional tunneling. Several factors contribute to making a became lost in the wealth of information available on the PED
display appear compelling, such as salience, realism, and (Joslin, 2013).
resolution, to name a few (Wickens & Alexander, 2009). A Given the perspective provided in the general aviation
display that is compelling can have both positive and negative domain, we were interested in understanding how flightcrews
effects on human performance. Compellingness can be use EFB applications in air transport operations (Part 121) for
beneficial, when attention is drawn to information when that position awareness only. The purpose of this study was to
information is time-critical (e.g., alerts) or when information is examine whether the presentation of own-aircraft (ownship)
presented in such a way that it reduces the “cost” of accessing position contributes to compellingness in the context of low
or integrating that information. However, compellingness can visibility taxi scenarios. Airport markings (e.g., geographic
be problematic if the information display is distracting (for position markers (GPMs), runway guard lights, and stop bars)
example, a display that attracts attention at the cost to other as well as special low visibility taxi charts help flightcrews
information sources) or if that information is used maneuver in low visibility conditions. The presentation of
inappropriately. For example, the presentation of information ownship position on a low visibility taxi chart could facilitate
superimposed on a head-up display may be more compelling position awareness but bring the pilot’s eyes away from the
than information presented in a head-down location to the out-the-window view for a longer than optimal period. Even in
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 63

low visibility conditions, it is the out-the-window view that glasses, a video camera was placed in the simulator to capture
contains important information to determine aircraft position pilot head movement and voice data. These data were
and avoid obstacles. analyzed separately from the eye tracking data.
Flightcrews conducted 6 taxi scenarios (3 taxi in, 3 taxi
out) in 3 levels of visibility (1200 Runway Visual Range Experiment Design
(RVR), 600 RVR, and 300 RVR) in nighttime lighting
conditions using specially-designed prototype geo-referenced The study was a 2 (Ownship: on, off) x 3 (RVR: 1200,
electronic low visibility taxi charts. For half the scenarios, 600, 300) x 2 (Group: 1, 2) mixed design. Ownship and RVR
ownship position was presented on the electronic chart. Low were manipulated within subjects, so each flightcrew saw all
visibility taxi charts are different from traditional airport three visibility levels and two levels of ownship presentation
diagrams in that they identify specific routes and cues to help (on/off).
pilots maintain position awareness when the visibility of Group (1 or 2) was a between-subjects variable that
traditional out-the-window cues (e.g., taxi signs, edge line described the manipulation of ownship presentation within the
lights) is limited. Although airport charts showing ownship are scenarios. Six scenarios were developed for this study,
in widespread operational use, electronic low visibility taxi consisting of one taxi-in and one taxi-out scenario for each
charts did not show ownship position at the time we conducted RVR level. The scenarios were specific to the RVR level
this study because the charts were not geo-referenced and thus because only certain taxi routes are available as visibility
could not meet the accuracy required by the Federal Aviation decreases. Each flightcrew had ownship on for either the taxi
Administration (see Advisory Circular (AC) 120-76D, in or the taxi out scenario, but not both. The Group variable is
Authorization for Use of Electronic Flight Bags). further described in Table 1 below.
This study was originally intended to replicate an
experiment conducted by a charting provider, in which the Table 1. Experiment Design
sample size and data analysis were limited due to time
constraints. This replication was an attempt to combine the Group 1 1200 RVR 600 RVR 300 RVR
data samples from the two studies to allow for significance
Ownship on IN OUT IN
testing. However, we made some modifications to the
experiment design to correct some issues identified during the Ownship off OUT IN OUT
conduct of the previous study. For example, we re-designed
some taxi routes so that the direction in which own-aircraft Group 2 1200 RVR 600 RVR 300 RVR
was taxiing was consistent with that shown on the low
Ownship on OUT IN OUT
visibility taxi charts, and we shortened the length of some
scenarios so that the study could be conducted within the time Ownship off IN OUT IN
allotted. These modifications prevented any combination of
the data. Scenarios

METHOD Each flightcrew completed a taxi-in and taxi-out at

Denver International Airport (DEN) for each RVR condition;
Participants a total of six scenarios. All taxi-out scenarios began on the
apron. All taxi-in scenarios started on the Instrument Landing
Twenty Airline Transport Pilots (ATP) participated in System (ILS) final approach three miles out from the
the study. The mean age for the participants was 45, with a touchdown position of the runway in use, with the aircraft
range from 31 to 53 years old. Pilots were paired to form 10 fully configured for landing (i.e., gear-down, flaps–full, final
two-person flightcrews. Each participant flew the same checklist checked, and aircraft stabilized for landing). When
position in the study as their position in flight operations, with ownship position was shown, the ownship icon was displayed
one exception in which one Captain participated as a First only when the aircraft speed was below 80 knots. Ownship
Officer. position was not shown on approach charts during flight.
All trials were conducted in a simulated night environment.
Simulators No traffic aircraft were introduced in the scenarios. However,
the scenarios were still operationally realistic because air
The study was conducted using two Boeing 737 traffic control would limit the number of aircraft on active
commercial aircraft simulators, without motion. Two crews taxiways and runways and increase the distance aircraft
were run simultaneously. Electronic taxi charts were presented maintain from one another in low visibility conditions.
on two iPads with 9.7” screens located in the areas of the left Consequently, traffic aircraft would not be visible to own-
and right side flight deck window with the GPS signal aircraft.
simulated through a portable Bluetooth device. The iPads were The duration of each trial was approximately 10-15
fixed in place. minutes. Each set of six scenarios was completed during an
Eye tracking glasses were available for only one of the approximate two-hour simulator time block. The order in
simulators. The glasses recorded pilot’s gaze behavior in real- which the visibility conditions (and hence the scenarios) were
time for six crews. For the simulator without eye tracking presented was counterbalanced for each group.
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 64

To gather data on whether ownship position was Taxi Speed and Taxi Time
compelling, we presented flightcrews with one off-nominal
scenario in which we simulated a GPS failure and ownship We analyzed the data for taxi speed and taxi time using a
position froze on the electronic airport chart. This presentation 2 (ownship) x 3 (RVR) ANCOVA. Simulator data were
was a deviation from FAA policy, which requires ownship collected at a sampling rate of 10 Hz. Direct comparisons by
position to be removed when the GPS signal is lost. This GPS RVR were not possible because the taxi route differed
failure was presented to flightcrews only once, and on the final depending on the RVR level. Consequently, we introduced
trial in which ownship position was depicted. two variables (taxi route length and complexity) as covariates
to “normalize” the differences in the taxi route. Although nine
Procedure flightcrews were included in the analysis for each trial, we
excluded flightcrews for trials in which a simulated GPS error
The study took approximately two-and-a-half hours to occurred, or problems with the simulator data collection or
complete. Flightcrews began with a 15-minute pre-briefing, failures to follow experimental protocol were observed. The
followed by approximately two hours in the simulator, and a number of flightcrews used for the analysis is shown in Table
15-minute de-brief to pilots. During the pre-brief, flightcrews 2 below.
completed the informed consent form, received an overview of
what to expect in the simulator, and completed a background Table 2. Number of Flightcrews for Taxi Speed and Taxi
questionnaire. Once in the simulator, a researcher calibrated Time Analysis
the eye tracking glasses for each pilot if the simulator was
equipped with the eye tracking system. RVR Ownship On Ownship Off
Flightcrews first completed a practice scenario in 1200
RVR daytime conditions to familiarize themselves with the 300 n=4 n=7
airport layout and simulator. Following the practice trial,
flightcrews completed the six taxi scenarios in a nighttime 600 n=5 n=5
environment. Pilots selected the charts for use during each
scenario, and were allowed to switch between low visibility 1200 n=5 n=6
taxi charts, and airport diagrams as needed. The experimenter
pre-loaded the chart for the first trial and the pilots loaded all Neither ownship position nor RVR significantly affected taxi
subsequent charts. Assigned breaks were not included; time or taxi speed.
however, crews were instructed to inform the experimenter We also examined the number of errors committed by
should they want a break. counting the number of incorrect or missed turns. Each
One researcher acted as an air traffic controller for each scenario contained between 4-8 turns. There were a total of
simulator, providing all taxi instructions and responding to any 101 potential opportunities for error in the scenarios when
calls the crews made to Air Traffic Control (ATC). The ownship was available and 159 when ownship was not. Only
researcher was seated inside the simulator with the crew. In three taxi errors were committed overall – all occurring when
the off-nominal scenario, this researcher also initiated the GPS ownship was not shown (1.9%). Two of the errors occurred in
failure using a smartphone application to toggle the Bluetooth 1200 RVR conditions, possibly because the taxi route for that
connection to the simulator off. This simulated a GPS scenario spanned two charts and required more flightcrew
“freeze.” Once the flightcrew noticed the disparity in the interaction than the other scenarios. The third error occurred in
ownship presentation, the researcher turned the Bluetooth 300 RVR conditions. All three errors were identified by the
connection back on, which simulated the GPS signal being flightcrews and reported to ATC.
reacquired.
After each scenario, each flightcrew member completed Eye Tracking
an electronic post-scenario questionnaire to gather subjective
data about the use of the electronic chart and general usability Eye movement was collected at 60 Hz. Eye tracking data
issues. Following the sixth trial, each flightcrew member also were collected for only six flightcrews (totaling 72 videos).
completed a post-test questionnaire to gather overall From this data set, we excluded 12 videos related to the GPS
impressions regarding the presentation of ownship. error, four videos due to violations of the experimental
protocol, and one video due to equipment malfunctions. In
RESULTS total, we examined 55 eye tracking videos to examine where
pilots looked, how often (number of fixations) and for how
Our data set consisted of objective data collected from long (dwell time). We defined four areas of interest: out-the-
the simulator (taxi speed and taxi time) and eye trackers window (OTW), instruments/controls, own EFB, and other
(number of fixations, dwell time on areas of interest) and (fixations that did not fall within one of the previous three
subjective data from the post-scenario and post-test areas of interest). Figure 1 shows the areas of interest defined
questionnaires. Data for nine flightcrews were included in the for the Captain (left seat).
data analysis; one flightcrew was excluded because the
Captain was not current.
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 65

OTW

Figure 1. Areas of interest as defined for the Captain.

The eye tracking data were analyzed separately for
Captains and First Officers because of the different tasks they
performed. In our study, Captains maneuvered the aircraft
whereas First Officers were responsible for visually
monitoring the progress of the aircraft and ATC
communications. Thus, we expected what they looked at and
for how long to differ. The percentage of fixations and overall Figure 2. Percent of fixations for each area of interest for
dwell time were analyzed using multiple one-way ANCOVAs. Captains.
In first examining the data for the Captains, the results showed
that Captains had significantly more fixations on the EFB
when ownship was presented than when it was not, F(1, 25) =
9.61, p = .005. Captains also spent more time looking at the
EFB overall when ownship was presented than when it was
not, F(1, 25) = 6.42, p = .019. We found no difference in the
mean dwell time for each fixation to the electronic chart as a
function of ownship position (F(1, 25) = 0.87, p = 0.36).
Figure 2 shows the percent fixations in each area of interest.
Figure 3 shows the percent dwell time as a function of
ownship (on vs. off). As Figure 2 shows, Captains looked at
their EFB almost six times more when ownship was on than
when it was off. Figure 3 shows that the amount of time
Captains looked at the EFB also increased by almost six times
when ownship was present than when it was not, and that this
time spent looking at the EFB was “borrowed” from time
spent looking out-the-window. Finally, the figures show that
the number of times (i.e., percent of fixations) and the amount
of time Captains spent looking at their instruments did not
change significantly as a function of ownship presence (or
absence).
This data suggests that the presentation of ownship
position was compelling but does not speak to why. We Figure 3. Percent dwell time for each area of interest for
hypothesize that ownship position increased the perceived Captains.
usefulness of the electronic chart/EFB to the Captain. This is In examining the data for First Officers, there was no
supported by subjective data from flightcrews who indicated significant difference in the percentage of fixations or dwell
that electronic charts with ownship position increased the time time as a function of ownship presentation. Overall, 33% of
available for crew duties. Because Captains were responsible First Officers’ fixations were on the electronic chart, 48% out-
for taxiing the aircraft (i.e., eyes-out), the “cost” of the-window, and 8% were on their instruments. They spent
information access—looking at the electronic chart and 32% of their dwell time on the EFB, 48% out-the-window,
correlating the chart to the out-the-window view to verify and 8% on their instruments. In considering the data in the
position—was significantly lower when ownship was present context of the “cost” for information access, we find that there
than when it was not. was no significant difference. First Officers, who were
responsible for visually monitoring the progress of the aircraft,
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 66

were continuously referencing the aircraft’s position on the showing ownship on the electronic chart was compelling. In
airport surface, regardless of ownship presentation. this case, we believe that compellingness was due to the
There was no significant difference in either the number perceived utility of ownship position, so compellingness in
of fixations or dwell time for Captains or First Officers by this sense should not be interpreted as negative; rather, the
RVR. presentation of ownship reduced the information access cost
for determining position. This interpretation is consistent with
GPS Error the finding that there were no negative consequences to any
other performance measure associated with ownship depiction.
We conducted an analysis of the scenario where ownship This study was only a first step to defining factors
position was “frozen” on the display. From the set of nine contributing to display compellingness. The data in this study
flightcrews, two flightcrews were excluded from this analysis: are limited; we only collected data from 9 flightcrews, so
one due to a GPS/Bluetooth failure, and another because the additional data are needed to develop a broader picture.
GPS error occurred when the flightcrew was switching charts, Specifically, a better understanding is needed as to whether
and this action caused a different location to be shown on the there are implications of the increased fixations and dwell time
chart. All seven flightcrews included in the analysis noticed on the electronic chart when ownship is presented. Thus,
the GPS error in approximately 1.5 minutes (range = 30 further research is recommended to understand the impact of
seconds–3 minutes). No incursions were committed. compellingness (a display property) in terms of human
Our intent in simulating the GPS error was to create one performance (attention allocation) in light of the various tasks
condition to examine how pilots would respond if ownship being performed. Additionally, a metric to quantify the nature
position was incorrect, particularly after consistently seeing of compellingness, the contribution of the various factors that
“perfect” positioning. Here, our simulator study was effective create the sense of compellingness, and the positive and
at simulating real-world conditions because several negative impacts would be of value when designing and
flightcrews identified ownship position errors in the “normal” evaluating new avionics.
scenarios due to a faulty Bluetooth signal. These errors were
not planned as part of the experiment design. ACKNOWLEDGMENTS

Subjective Data The research was conducted under the Flight Deck
Program Directive/Level of Effort Agreement between the
We analyzed the data from the post-scenario FAA Human Factors Division (ANG-C1) NextGen Program
questionnaire using a paired t-test to examine ownship effects. and the Aerospace Human Factors Research Division (AAM-
Data for all nine flightcrews were included in this analysis. 500) of the FAA Civil Aerospace Medical Institute (CAMI).
The results showed that pilots felt that they referred to the The authors would like to thank our program manager, Katrina
electronic chart significantly more often when it showed Avers, as well as Cathy Swider (FAA); Janine King, Hunter
ownship than when it did not (t(17) = 3.60, p < 0.05). Klevgard, Daniel Jack, Esther Devanney, and Shijing Liu of
Additionally, pilots felt the electronic chart significantly Cherokee CRC, LLC; and Stephanie Chase of the US DOT
helped them in following their taxi route more when it showed Volpe Center for their contributions. Special thanks goes to
ownship position than when it did not (t(17) = 4.3, p < 0.05), Will Ware at Southwest Airlines and Jeff Williams, Samantha
and that using the chart with ownship position significantly Schwartz, Katie Reid, and Jason Clark at Jeppesen for their
increased the time available for other crew duties than when assistance in planning, developing, and executing this project.
ownship position was not shown (t(17) = 3.5, p < 0.05). Finally, we would like to thank all the pilots who participated.
The views expressed herein are those of the authors and
CONCLUSIONS do not necessarily reflect the views of the Federal Aviation
Administration or the United States Department of
This study provides a glimpse into pilot behavior when Transportation.
using ownship position on electronic taxi charts in low
visibility conditions. While there were three taxi errors REFERENCES
committed across all the scenarios, no unsafe acts
(incursions/excursions) were observed for any of the Federal Aviation Administration. (2017). Advisory Circular (AC) 120-
76D, Guidelines for the Certification, Airworthiness, and Operational Use of
flightcrews, regardless of whether ownship was presented or Electronic Flight Bags. FAA: Washington, D.C.
not. We found no difference in taxi time or taxi speed due to Joslin, R. E. (2013). Human Factors Hazards of IPADS in General
the presentation of ownship or the visibility level. Eye Aviation Cockpits. In Proceedings of the Human Factors and Ergonomics
tracking data showed that the presentation of ownship position Society Annual Meeting (Vol. 57, No. 1, pp. 56-60). SAGE Publications.
Wickens, C. D. and Alexander, A. (2009). Attentional tunneling and
influenced only the Captains’ looking behavior, such that task management in synthetic vision displays. International Journal of
Captains looked at the electronic chart more when ownship Aviation Psychology, 19(2), 182–199.
was present than when it was absent. In the study, the Captain Wickens, C. D., Goh, J., Helleburg, J., Horrey, W. J., and Talleur, D.
was responsible for taxiing/maneuvering the aircraft, so we A. (2003). Attentional models of multi-task pilot performance using advanced
display technology. Human Factors, 45, 360–380.
anticipated that the Captain would be looking out the window Yeh, M. and Wickens, C. D. (2001). Display Signaling in Augmented
(eyes-out) most of the time. The fact that the Captain looked Reality: Effects of Cue Reliability and Image Realism on Attention Allocation
more at the EFB when ownship was available suggest that and Trust Calibration. Human Factors, 43(3), 355–365.