Page 1 of 12

STC 2009

iPhone Application Testing

By
Dipali Vaidya
Principal, phoneTesting
dipali.vaidya@phonetesting.mobi
 Table of Contents Page 2 of 12
Introduction..........................................................................................................................................3
Unique features of mobile platform ....................................................................................................3
Compact Screen Size........................................................................................................................3
Limited Memory...............................................................................................................................3
One Screen at a time.........................................................................................................................3
One Application at a time.................................................................................................................4
Minimal User Help...........................................................................................................................4
Types of iPhone Applications...............................................................................................................4
Productivity Application...................................................................................................................4
Utility Application............................................................................................................................4
Immersive application......................................................................................................................4
iPhone Application Testing Strategies and Types.................................................................................5
Testing on iPhone Simulator -..........................................................................................................5
Testing on device (iPhone / iPod Touch)..........................................................................................5
iPhone Testing Methodology................................................................................................................5
Application Acquisition - ................................................................................................................5
Study of Application functionality - ................................................................................................6
Page 12 of 12....................................................................................................................................6
Design Test Cases - ..........................................................................................................................6
Test and Report Defects....................................................................................................................6
Testing types recommended.................................................................................................................6
Installation Testing...........................................................................................................................6
Usability Testing...............................................................................................................................6
Functional Testing - .......................................................................................................................10
Domain Testing...............................................................................................................................10
Conclusion .........................................................................................................................................11
References..........................................................................................................................................12
Author Biography...............................................................................................................................12
 Page 3 of 12
Abstract

The "fourth" screen has certainly become an integral part of people's lives. Apple's iPhone is
one of the iconic leaders in the smart-phone category due to its extreme focus on usability, an
equally cool design quotient and the numerous applications available on its App Store. The
number of iPhone applications on App Store is expected to cross the 100,000 mark by end of
2009. Only recently Apple reported completion of 2 billion downloads from App Store. This is
by no means a meagre feat and the future holds a great promise for iPhone development
companies. This consequently presents a unique opportunity to independent testers /
enterprises to validate these applications in order to improve their user experience and degree
of robustness.
This paper presents a methodology for testing iPhone applications and provides an
introduction to the concept of Usability Index which bears a direct co-relation to the sales
performance numbers for an Application on the App Store.

Introduction

iPhone is the brand name of Apple's mobile handset. It was first unveiled in the summer of
2007 amongst tremendous anticipation and frenzy. Apple Inc is known to introduce cool
products and their design value has always been respected the world over. iPod has
revolutionized the media consumption pattern of masses and has made it accessible through
multiple variants like the Shuffle, Touch, Nano etc. “Podcasts”, a term which was originally
coined for the iPod is used as a generic term. The iPod Touch is iPhone without the phone
functionality. They share a common iPhone OS and all applications which run on iPhone and
do not utilize the “calling” feature, are compatible with iPod Touch.
App Store is the sales portal for iPhone/iPod Touch applications. To date there are a
staggering 85,000 apps available for sale - spanning across categories and price points. Two
billion downloads of these apps have happened over a short period and this brings forth the
important and serious customer base for the iPhone application market. The customer base is
evolving and upgrading continuously and so are their expectations. The developer community
needs to meet the challenge of increasing user expectations and need to offer a product that is
value for money, easy to use, aesthetically pleasing and emotionally satisfying.
This paper describes the considerations for testing iPhone / iPod Touch applications.

Unique features of mobile platform

It is essential to keep in mind that an iPhone OS–based device is not a desktop computer, and
an iPhone application is not the same as a desktop application. iPhone applications differ
considerably from their desktop counterparts due to the following reasons -

Compact Screen Size

The iPhone screen is of a high resolution and of size 480 X 320 pixels. This compact size
heavily limits the number of GUI elements that can be placed on the screen. This needs to be
kept in mind during the application design so that the interface appears attractive and
overcrowding is avoided.

Limited Memory
Memory is a critical resource on iPhone and applications have to judiciously reduce the
memory footprint and memory leaks. Low memory conditions could abruptly terminate
applications which will reduce their reliability value.

One Screen at a time

iPhone applications can have as many screens as necessary but users can access them
sequentially, never simultaneously. This is huge paradigm shift from the normal desktop
application.

One Application at a time

Page 4 of 12
Only one iPhone application can run at a time, and third-party applications never run in the
background. This means that when users switch to another application, answer the phone, or
check their email, the application they were using quits. It’s important to make sure that users
do not experience any negative effects because of this reality.

Minimal User Help

Mobile users need to access applications fast and they do not have much time to read help
documentation or go though a steep learning curve. Hence the applications on iPhone need to
be extremely intuitive by nature and usable.

Types of iPhone Applications

Based on the visual and behavioural characteristics, data model and user experience, iPhone
applications can be classified into three distinct types. The designer needs to decide the style
of her application by understanding the user need and type. The tester's job is to experience
the feel of the application and judge as to whether the style is appropriate for the job at hand.

Productivity Application
A productivity application enables tasks that are based on the organization and manipulation of
detailed information. People use productivity applications to accomplish important tasks.
Productivity applications often organize user data hierarchically. In this way, people can find
information by making progressively more specific choices until they arrive at the desired level
of detail. iPhone OS provides table elements that make this process extremely efficient on
iPhone OS devices. Mail / Photo Album are good examples of Productivity applications.

Utility Application
A utility application performs a simple task that requires a minimum of user input. People open
a utility application to see a quick summary of information or to perform a simple task on a
limited number of objects. A utility application tends to organize information into a flattened list
of items; users do not usually need to drill down through a hierarchy of information. The user
interaction model for a utility application is very simple: Users open the application to scan a
summary of information and, optionally, change the configuration or source of that information.
The standard “Weather” application on iPhone is a perfect example of a Utility application.

Immersive application
An immersive application offers a full-screen, visually rich environment that’s focused on the
content and the user’s experience with that content. People often use immersive applications
to have fun, whether playing a game, viewing media-rich content, or performing a simple task.
An immersive application tends to hide much of the device’s user interface, replacing it with a
custom user interface that strengthens the user’s sense of entering the world of the
application. Instead, immersive applications present information in the context of the game-
play, story, or experience. Also for this reason, immersive applications often present custom
navigational methods that complement the environment, rather than the standard, data-driven
methods used in utility or productivity applications. Gaming applications are typically of the
Immersive type
iPhone Application Testing Strategies and Types

Testing on iPhone Simulator -

iPhone applications are developed using the language Objective-C in Xcode – the Integrated
Development Environment provided by Apple. The IDE comprises among other things, an
iPhone Simulator tool which demonstrates the running code as if it were executed on the
device. Many developers perform testing
Page 5 of 12
themselves using this tool. However this has many limitations.
1. The user experience with the simulator and the device is completely different. The
device has a small screen based keypad while if the simulator is used, the full-sized
keyboard is at ones command.
2. Certain actions cannot be performed on simulator like invoking the Accelerometer,
getting the effect of touch of more than 2 fingers etc
3. The Location service which is available through the device's carrier is not accessible
on the simulator
4. Memory limitations on the iPhone cannot be simulated on the Simulator realistically.
This means that applications which work perfectly fine on Simulator could crash on the
device if memory management is not done efficiently.

Testing on device (iPhone / iPod Touch)

After unit testing and testing on the simulator, the developer would have the application beta
tested by users/ beta testers for testing on device. This highlight issues related to usage
patterns, memory limitations, network availability, battery consumption etc.

iPhone Testing Methodology

Application Acquisition -
The following flowchart describes the process by which an iPhone application reaches a beta
tester and the steps to install it on to the device

Beta Tester Tester connects

sends UDID to device to
Application PC/Mac and
Developer does a “Sync”
operation in
iTunes
Developer
creates Mobile
provisioning file

Tester copies the Application

Developer sends application and installed
application provisioning file on device
software & in iTunes
Provisioning file Library on a PC
to Tester or Mac
UDID is the unique device ID for iPhone/iPod Touch assigned by Apple and can be found
using the iTunes software.

Study of Application functionality -

Beta Tester gets familiar with application working by playing around or reading the Help
documentation if available. It would be good if the beta tester is familiar with the domain of the
application so that her findings would be more realistic and similar to the actual user.

Page 6 of 12

Design Test Cases -

Test cases are designed which check whether the application functions in a proper manner
and user goals are met. The techniques used for doing this would be the same as those used
for desktop/web applications.

Test and Report Defects

The application is tested using the testing types described in the following section and defects
are reported to the developer. In addition to the usual fields of a defect management system
like priority, severity etc., every bug found during beta testing of iPhone applications could be
classified under one or more of the following heads. This would provide a meaningful feedback
to developer and help improve quality of t he application within the shortest possible
turnaround time.
1. Installation issue
2. Initialization issue
3. Navigational issue
4. Functional error
5. Confusing labels / button names
6. Incorrect handling of abnormal / boundary / blank inputs and inputs given using Copy-
Paste functionality
7. Unnecessary / Erroneous warnings / Error messages
8. Incorrect data / content
9. Inconsistent behaviour due to rapid user input (especially in Games Applications)
10. Application crash
11. Error in connectivity with external applications like Social networking sites, Game
score keepers etc

Testing types recommended

Installation Testing
This involves successful transfer of the application on the device. Installation may not happen
due to the following reasons -
1. Device not compatible with the application. For e.g. if the application is supposed to
work on iPhone and you try to install it on an iPod Touch.
2. Application not compatible with the iPhone OS version
3. Mobile provision file does not include UDID of the device on which it is meant to be
installed.
4. Mobile provision file created incorrectly

Usability Testing
Post Installation, Usability testing is the most important type of testing to be done on iPhone
applications. Apple has defined very elaborate Human Interface guidelines for mobile device
applications and it is very much recommended that designers incorporate those. We have
discussed the unique features of the mobile platform earlier. The iPhone screen being compact
and user being mobile and “always in hurry”, the application designers must place the
“usability” criterion above all.
Following are the parameters contributing to a better usability of any iPhone application.

Simplicity and Ease of Use

Simplicity and ease of use are fundamental principles for all types of software, but in iPhone
applications these are absolutely critical. iPhone OS users are probably doing other things
while they are simultaneously using the application. If the users can’t quickly figure out how to
use the application, they’re likely to move on
Page 7 of 12
to a competitor’s application and not come back.

Focus on Primary Task

An iPhone application that establishes and maintains focus on its primary functionality is
satisfying and enjoyable to use. A tester needs to check whether every component that is a
part of the application is absolutely required and does not create any diversion from the
objective that has been set to achieve. The beta tester should pretend to be a user of the
application and report any confusions faced while working with it.

Effective Communication
Communication and feedback are as important in iPhone applications as they are in desktop
computer applications. Users need to know whether their requests are being processed and
when their actions might result in data loss or other problems. It’s also important to avoid
communication overload, for example, alerting the user to conditions that aren’t really serious
or asking for confirmation too often. Any communication that takes user closer to the goal is
required and that which takes him away from it should be avoided.

Appropriate User Gesture Support

Users perform specific movements, called gestures, to get particular results. For example,
users tap a button to select it and flick or drag to scroll a long list. iPhone users understand
these gestures because the built-in applications use them in a consistent manner. To benefit
from users’ familiarity, therefore, and to avoid confusing them, designers should use these
gestures appropriately in their applications. To ensure that the application is easy to use, only
familiar gestures like tap or drag should be used. In most applications, it’s equally important to
avoid defining new gestures, especially if these gestures perform actions users already
associate with the standard gestures. The primary exception to this recommendation is an
immersive application, in which custom gestures can be appropriate.

Usability Index
We can derive a “Usability Index” for the application based on the above parameters using the
following framework -
The scores are given as 3, 2, 1 meaning “Yes”, “Sometimes”, “No” respectively.
 Page 8 of 12

Usability Parameter Sub-Parameter Score (1/2/3)

Obvious Usage (no Help required)

Frequently used information at top of the screen

Simplicity and Ease of Use Minimal text input

Essential information clearly displayed

Fingertip size target area for tappable elements

Average Score for Simplicity and Ease of Use

Quick access to primary functionality

Focus on Primary Task
Right mode of user input

Average Score for Focus on Primary Task

Timely feedback for all user actions

Processing status displayed

Effective Communication Unwanted feedback avoided

Warning before potentially destructive action by

user

Appropriate language (avoiding technical jargon)

Average Score for Effective Communication

Usage of familiar gestures

User Gesture Support
Avoidance of non-standard and complex gestures

Average Score for User Gesture Support

Ease of Use Index (U Index) = Average Score for Simplicity and Ease of Use
Focus Index (F Index) = Average Score for Focus on Primary Task
Communication Index (C Index) = Average Score for Effective Communication
Gesture Index (G Index) = Average Score for User Gesture Support
Usability Index of the Application can now be computed as below
Usability Index = Average (U Index, F Index, C Index, G Index)
iPhone applications across various categories were rated using this framework and the results
are as below -

Page 9 of 12
Application Name Category Usability Index* App Store Rating
iSignals Finance 2.475 Not in AppStore
lyricalTweet Music 2.625 3
MainEvent Entertainment 2.225 1.5
MayanFlyer Games 2.9 Not in App Store
MiniTaxi Games 1.9 2.5
TricTrac Games 2.75 No rating yet
RailAdventure Games 2.6 Not in AppStore
iLectricFB Games 1.875 Not in AppStore
Hockey Games 2.9 Not in AppStore
DepthCharge Games 2.45 Not in AppStore
YouOweMe Finance 2.525 No Rating Yet
RedSquare Games 2.85 3
hLog Health & Fitness 2.625 4
MoodTunes Music 2.45 Not in AppStore
iBurn Entertainment 2.075 2.5
HistoricEarth Reference 1.775 Not in AppStore
MagneticBanana Books 2.9 No Rating yet

*Computation of Usability index is based on the impression of the application during its beta
testing. There could be changes in the application from the time it was beta tested to the time it
went up on the App Store.

The graphical representation of the above table is as follows –

 Usability Index v/s App Store Rating

4.5

3.5

3
Rating value

2.5
Usability Index
AppStore Rating
2

1.5

0.5

0
er
t

na
t

re

e
e

n
s

FB

es
i

rth
c
ee

ax
en
al

ke

r
rg

eM

ur
o
a
y

tu

ua

un

na
Tr

hL

Ea
n

ric
iT
Fl
w

ha

iB
oc
en
ig

nE

Sq
ic
lT

in
an

Ba
dT
ct

ric
C
H
iS

O
Tr

dv
M
ica

ai

ed
ay

th

oo

tic
u

to
iL
M

lA

Yo
ep
lyr

is
R
M

ne
M
ai

H
D

ag
R

M
Application Name

Page 10 of 12
It can be determined that the Usability index bears a direct correlation to the rating these
applications got at the App Store after they were up for sale (in majority of cases). Thus
Usability Index is a powerful tool which a beta tester can leverage to provide meaningful
feedback to the developer and which would directly impact the application sales figures.

Functional Testing -
This involves
1. Executing each functionality of the application using the test cases designed.
2. Verifying every screen to check for consistencies in style and presentation
3. Feeding various input types (normal, abnormal, boundary)
4. Executing real life user scenarios (multiple functionalities in serial manner)
5. Checking Navigational aspects
6. Checking functionalities specific to iPhone technology – Accelerometer response,
Push notifications, location specific functionalities, localization specific features (if
application uses non-English languages) etc.
7. Behaviour of application when a phone call is received (sudden quitting action)

Domain Testing
This is probably the most ignored aspect of iPhone application testing and yet very important.
Every application is only as meaningful as the content it carries. A highly usable application
coupled with robust functionality will not sell if the content is uninteresting or inaccurate. Hence
it would help if the beta tester has at least a preliminary knowledge about the concerned
domain and reveal problems there.
Conclusion
In this paper we have analysed the unique design challenges of iPhone applications and their
recommended types. These application types are flexible and an enterprise might chose to
use a combination as per the business need. Testing professionals for iPhone applications can
use these strategies and testing types to report more meaningful defects. Implementation of
the Usability Index framework can be leveraged to improve the application quality and
subsequently its sales on the App Store. This framework can be further refined to
accommodate the features, application categories catering to various business needs.
 Page 12 of 12

References
1. Apple's Website http://www.apple.com
2. iPhone Human Interface Guidelines from Apple
3. iPhone Development Guide from Apple
4. Stackoverflow.com discussion forums
5. Application names mentioned in the paper were beta tested through the site
www.ibetatest.com

Author Biography
Dipali Vaidya, the Principal of phoneTesting is a veteran in the IT field of over 17 years of
experience across various domains. Her specialization is Software Quality Assurance and
Testing. She has worked collaboratively with Global Teams to deliver high quality products for
her clients. She has worked on applications catering to various domains - Finance,
Manufacturing, Medical, Social Networking etc
Dipali holds a BE (Electrical) degree from the University of Mumbai and PGCPM from IIM
Kozhikode.