Reliability Mobile
Reliability Mobile
net/publication/259232374
CITATIONS READS
18 3,198
3 authors:
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Ali Bou Nassif on 09 January 2014.
Abstract—: Smartphones have become the most used successfully applied to desktop (classic/standard) applications
electronic devices. They carried out most of the functionalities of to assure high reliability, as was the case with Hardware
desktops, allowing various useful applications that suit the users’ Reliability Growth Models applicability to both
needs. Therefore, instead of the operator, the user has become desktops/laptops and mobile phone hardware [1]. We suspect
the number one controller of the device and its applications and SRGM is similarly applicable to both desktops/laptops and
thus its reliability becomes an emergent need. We aim to Smartphones. To the best of our knowledge, there is no
investigate and evaluate the efficacy of Software Reliability published work that addressed the application of the usual
Growth Models (SRGMs) when applied to Smartphone SRGMs to Smartphone applications. In the present paper, we
application failure data and check whether they achieve the same
attempt to answer the research question “Is it possible to
success as in the desktop/laptop area. We selected three of the
successfully apply SRGMs to Smartphone applications as it is
most used SRGMs and applied them to three different
Smartphone applications. None of the selected models were able the case for desktop applications? How accurate are they and
to account for the data satisfactorily. Their failure is traced back what are the challenges?”
to the specific features of mobile applications compared to The rest of the paper is organized as follows: in Section II
desktop applications. Thus, a suitable model for Smartphone we stress the need for the reliability of Smartphone
applications is still needed to improve their reliability. applications. In Section III we provide a short roadmap of the
existing models that we will use. We devote section IV and
Keywords—Smartphone applications; software reliability; Section V to describe our dataset and experiments. Finally, in
NHPP; Musa-Basic; Musa-Okumoto model; SRGM model. Section VI, we evaluate the application of the existing models
to Smartphone application failure data. We conclude, in
I. INTRODUCTION Section VII, and outline our future work.
Smartphones are so important nowadays and they are
overselling PCs. The high usage and trust in these devices and II. RELIABILITY FOR SMARTPHONE APPLICATIONS
their applications make their reliability a critically important Smartphones have become personal devices that are used
goal to achieve. Thus, manufacturers are competing to release almost anywhere, at any time and for everything; checking e-
the most reliable devices and they successfully achieved high mails, gas prices, browsing the Internet, banking and even for
reliability in terms of hardware by applying traditional and health services. These high expectations and trust placed on the
enhanced Hardware Reliability Growth Models (HRGM) [1]. mobile environment make it useful more than just a phone and
These HRGMs have been useful for classic mobile phones. exceeds the functionalities of only sending SMS or making
Today’s cellular phones, called Smartphones, are almost voice calls. A few years ago, Smartphones use was only limited
like pocket PCs, meaning that their functionalities far exceed to business. Lately, thanks to network and mobile technology
those of the classic mobile phones. This kind of change improvements and progress, those smart devices started to gain
requires Smartphone application developers to pay more tractions and they got a remarkable acceptance in the users
attention and spend more effort on the reliability and security market. Since then, it has been a significant increase in the
of their applications. The rapid increase in developing smart Smartphone technology. Thus, countless Smartphone
phones relies on Software Product Lines (SPL). In SPL, applications have been designed and developed. In step with
software is not re-developed from scratch where similar this, the market has been rapidly growing, and market analysis
products can be developed using common set of core assets confirmed that it will continue growing to the point that it will
[2][3]. exceed the desktop/laptop growth and oversell many other
Software Reliability deals with “the probability that electronic devices in particular laptops [7].
software will not cause the failure of a system for a specified As a result, to assure the continuing growth in this
time under a specified condition. It differs from hardware competitive market, there are various types of customers’
reliability in that it reflects the design perfection, rather than needs to be satisfied because, based on previous studies, those
manufacturing perfection” [4]. Reliability is an important needs differ from one region to another and from one user to
attribute of software quality in addition to other attributes such another, thus, here is where the reliability of Smartphone
as performance, usability and fault prediction [5]. Software applications will play an important role to keep the trust in
testing techniques can impact reliability measurement [6]. one’s device.
Software Reliability Growth Models (SRGMs) have been
III. EXISTING RELIABILITY MODELS where N(t) is the cumulative number of failure by time t and
μ(t) is the mean value of N(t) or the expected cumulative
In this section we will shortly present three different number of failure :
SRGMs that will be used in our experiments; the Non-
Homogenous Poisson Process (NHPP) model, the Musa-Basic μ(t) = E(N(t)) (2)
model and the Musa-Okumoto model.
The instantaneous failure intensity is defined as:
As a matter of fact, hundreds of models exist but those
models were chosen because according to Software Reliability ࢊஜሺ࢚ሻ
ࣅሺ࢚ሻ ൌ (3)
modeling survey [11][12], they are the most useful and ࢊ࢚
successful models in the computer applications domain. Hence, The NHPP model implemented in the RGA7 tool is the
one of our research goals is to check if those models will NHPP-Crow model [14] where the probability density function
achieve the same success when applied to Smartphone for the failure time is given by:
applications failure data as it was the case for the desktop and ࢼ
Hardware Growth Reliability Models [1]. ࢌሺ࢚ሻ ൌ ࣅሺ࢚ሻࢋିஜሺ࢚ሻ ൌ ࣅࢼ࢚ࢼି ࢋିࣅ࢚ (4)
The purpose behind developing models is the measurement,
where > 0 ; > 0 are the two parameters of the model.
estimation and prediction of Software Reliability which has
Finally, to implement this model, either the fault counts or
become an important major target for companies because it
the time between failures are required [11]. A detailed study of
was shown reliability has a significant effect in each stage of
NHPP model can be found in [11][13].
the DLC of an application; from the design to the maintenance
[2]. B. Musa’s basic execution tim Model
An SRGM usually results in a set of mathematical The Musa-Basic model, also termed the exponential model,
equations that fit accurately the collected failure data [12]. Any is given by the following mean value [11]:
model relies on simplifying assumptions. However, some of
ࡱ
these assumptions may not be useful in real situations. Table 1 μ (t) = ࢼࡱ ቀ െ ࢋିࢼ ࢚ ቁ (5)
from [2] presents an assessment of the most used assumptions
and their conformity to real observations. where ࢼࡱ : is the expected number of failures and ࢼࡱ is the
Due to the limited space of this paper we are not going to hazard rate or in other words “the amount that each fault
present all the assumptions of the different models used later contributes to the overall failure rate”.
but only the basic ones that are shared by all the models. The
156
This model is used especially for execution time data but it • Name of the crashed application
can also be applied to calendar time data by applying a
conversion from calendar to execution time. The required data • Type
to build this model are either the time of failure or time • Hardware type (device an iPhone, iPad or iPod
between failures. Touch) this information is needed to check
Based on the Software Reliability modeling survey from whether the crash is of an application of the same
the Handbook of SRE [11], this model is considered as one of device or same application from a different device.
the most widely used models.
There are several similar models that have been developed. • Date/Time of the crash (which is the most
Moreover, Musa mentioned that “the basic execution model important information in the crash log for our
generally appears to be superior in capability and applicability research work)
to other published models” [15].
• The version of the OS.
C. Musa-Okumoto logarithmic Poisson Model The crash logs of Apple devices are transferred to a hidden
The Musa-Okumoto model, also termed the logarithmic folder located/created in the PC that is used for the
model, is one of the most extensively applied models [11][12]. synchronization of the device. It contains the crash logs of all
Besides that, Musa himself confirmed that this model is more the applications installed on the device as well as reports about
accurate comparing to the exponential model [15]. the battery, memory and other features; however we are
As for the previous models, the mean value is extracted interested in the crash files. Thus, we ignored the other files.
from the model’s proper assumptions [11] and given by:
μ (t) = ࢼࡸ ࡸ൫ ࢼࡸ ࢚൯ (6)
157
It is evident to note that June, July and August are the
months with the highest crash rate. Since this application is
developed for the purpose of locating bicycle stations, it is used
during the summer period more than in winter, which explains
the high crash rate during the hot season. This reflects the fact
that the type of an application and its usage play an important
role in its reliability. From this graph we extracted the failure
data during six months (177 data points).
We used two Software Reliability tools for this application
to check the results. The first tool is the RGA 7 from ReliaSoft
and the second one is SMERFs (Statistical Modeling and
Estimation of Reliability Functions for Software). We
configured our tools as follows: “1” for the severity level of all
failures and the unit selected is hour. As the time scales of the
Fig. 2. Output of the JAVA program
three applications are very different, we choose to normalize
V. EXPERIMENTS our data between ܼ [0,1] using the following equations:
The reliability demonstration of Smartphone applications is ܻ = value of the raw target variable Y for the training
carried out through the traditional testing, failure data case ܼ = standardized value corresponding to Y
collection, and the application of the most used SRGMs for
standard applications to observe and check the adequacy of Range = upper bound of Y – lower bound of Y (7)
these models, in the mobile area for Smartphone applications.
For this purpose we used two applications for iOS and one ࢅ ି࢝ࢋ࢘࢈࢛ࢊࢌࢅ
for Windows mobile phone to test the models with different ࢆ ൌ ࢘ࢇࢍࢋ
(8)
platforms. We couldn’t collect enough data from Android As the RGA tool doesn’t accept the zero value as a time to
phones but we are still collecting to have enough data to test event, we entered 0.001 instead of 0 as the first value to be able
the models on Android applications. The first iPhone to have results. For the severity, 1 was selected because the
application is one of the most popular applications in applications used are not going to cause harmful consequences
communication, Skype, which has been tested and used for a if they fail. But it is not the case with other applications. When
year (from 01 Nov. 2011 to 11 Nov. 2012). Hence, the data working with applications such as online banking and health,
have been collected during a year with some missing values the severity of the failure has to be taken into consideration.
due to the non-use of the application occasionally. Therefore,
we were able to collect 46 data points. The second application VI. EVALUATION
is Vtok (an application for Google talk). This application has Smartphone applications reliability is a challenge. Thus, it
been continuously used during two months (from 19 Sep. 2012 is a necessity for reliability methods being applied elsewhere to
to 25 Nov. 2012). Hence, we were able to collect failures every be evaluated and to assess their validity in the mobile area. One
day (80 data points). Each of the above mentioned SRGM of the main goals of this work is to check if the most accurate
models was applied to Skype and Vtok failure data which and used SRGMs for desktop applications have the same
represent two different situations: Skype used during a year but accuracy when applied to Smartphone applications.
with some missing values, and Vtok application used for two Fig. 4, 5 and 6, present respectively the cumulative number
months every day with the possibility of collecting more than of failures per time, the failure intensity per time and the Mean
one failure per day. This is an instance of testing the efficiency Time Between Failure (MTBF) per time, for the Skype
and goodness of the models in different situations with application when applying the NHPP model. The RGA tool
different types of data. On the other hand, the Windows phone indicates an evident failure.
application, for finding bicycle stations, was continuously used
and tested for six months (from March 2012 to August 2012).
The crash count of the application is illustrated by Fig. 3.
Fig. 3. Windows phone crash count Fig. 4. Cumulative number of failures per time (Skype)
158
Fig. 5. Failure intensity per time (Skype)
Fig. 9. MTBF per time (Vtok)
.
Fig. 7. Cumulative number of failures per time (Vtok)
Fig. 10. NHPP model applied to Skype Time Between Failure (TBF) data
Fig. 11, 12 and 13 show the results of the same data from
Skype and Vtok application respectively when applying the
Musa-Basic and Musa-Okumoto models. Each time the models
fail to fit the data. The models failed completely to fit the Vtok
failure (Fig. 13). Fig. 14, 15 and 16 represent the results of the
application of the NHPP model to the Windows phone failure
data. Once again the RGA tool indicates the failure of the
model
159
One explanation of the failuress of the models is the Goodness-
of-fit tests that failed for eachh model application. Two tests
were used; the Cramer-Von Mises (CVM) and the Chi-Square
test. For both tests, the statisticc (test value) has to be less than
the critical value in order to haave a successful fit otherwise it
is a failure as it is the case in our experiments (this is
confirmed in Fig. 7, by perforrming the CVM test where the
model failed because the statiistic (0.358) is greater than the
critical value (0.173) same in Fig.14
F when performing the Chi-
Square test; the statistic (12822.005) is higher than the critical
value (180.094) thus the modell didn’t fit the data) [14].
Fig. 11. Skype failure data and failure of the Muusa-basic model
Fig. 12. Skype failure data and failure of the Musaa-Okumoto model
Fig. 13. Vtok failure data and failure of the three selected models Fig. 16. MTBF per time (Windows phone data)
160
Another reason is that the DLC of a mobile application is Based on different surveys and studies, reliability was
short and the programmer aims to develop the application as identified as the most important quality attribute of the
quickly to satisfy the time to market constraint which leads to application software. Thus, the reliability of Smartphone
skip phases from the DLC such as the design phase which is applications needs to be assured since everyone is using their
the most important phase in the DLC of the application. Thus, own Smartphones for daily life activities and tasks more than
it would be difficult to identify the causes of errors, during the PCs. Our study confirms that a reliability growth model
execution time, and find a convenient solution to fix them. adapted to Smartphone applications is needed since the
Besides that, the failure or unreliability of the application may traditional reliability models turned out to be inefficient.
be caused by the technology used during the development
process. Also, the skills of the developer and the tester play a VII. CONCLUSIONS AND FUTURE WORK
huge role in the reliability of the application. Moreover, the Our work is a step toward the application and evaluation of
device itself and its hardware characteristics such as the size of traditional Software Reliability models in the mobile area. We
the screen, the performance, the keyboard, etc. can have a selected three of the most used models that are known for their
direct effect on the reliability of the application. For example to efficiency in the desktop area: the NHPP, Musa-Basic, Musa-
adjust the map size to a certain zoom level, a zoom in/out Okumoto models. We used two iPhone applications, Skype and
function is needed. However, to assure a perfect usage of this Vtok that were used and tested differently to evaluate the
function the performance of the device has to be taken into models under different conditions, and one Windows phone
consideration [9]. application that we didn’t mention the name because of the
Other reasons that may explain this dynamic aspect of the company’s confidential policies. It turned out that none of the
Smartphone applications are summarized in Table 1 that gives selected SRGMs was able to account for the failure data
an idea about the different causes, external and internal, of the satisfactorily.
unreliability of the application. Due to the space limitation of Our study also highlighted the causes of the failure of the
the paper, a complete list of the causes and their description models and the need for a meticulous Software Reliability
and examples can be found in [20]. Growth Model for Smartphone applications; this is because the
existing Software Reliability approaches are developed for
TABLE I. CAUSES OF APPLICATIONS CRASH traditional desktop software applications that are static and
stable during their execution which it is not the case for
Cause of Failure Description
Failures arise when not taking into consideration
Smartphone applications which have unknown operational
Code the limited resources of the device such as power profile, highly dynamic configuration and changing execution
and memory conditions. On a continuous background, the smartphone
Interfaces
WAP Gateway fails when converting WTP failures come in relatively short bursts from time to time which
request to HTTP request explain the abrupt changes in the observed cumulative failure
Various models of devices: developers should number curves. This particular feature cannot be
Hardware take into consideration the specific platform and
performance of each device
accommodated by the used SRGMs. Thus, in order to evaluate
Failure to open the help, demonstration or the reliability of Smartphone applications, new models,
Non-executable files principles and tools are needed to incorporate the underlying
samples files of an application
Thanks to the SOA, many application interfaces uncertainties of such applications [21].
are located on a server. Thus, mobile applications Our investigation of Smartphone application reliability
Interaction have to connect to the server to accomplish data through the use of well-known available growth models, suited
transfer and carry out tasks. Failure in the server
may cause the crash of the application primarily to desktop applications, is twofold: (i) highlight the
The application has to be developed in a manner versatile nature of mobile applications, their dynamic
Data input that the data input has to be optimized to ensure configuration, unknown operational profile and varying
maximum efficiency for the user. execution conditions in contrast to the static and stable desktop
Smartphone application architecture uses third- ones, (ii) stress the need for the design of new reliability
party software applications (for example as
Third-party software Facebook and Adobe Photoshop Express to be
models suited for mobile applications which take into account
failures able to modify and upload pictures). A the inherent versatility of such applications.
crash/problem in the third-party application may As is well known, reliability is one of the most important
cause the failure of the other application. features of an application and great efforts have been devoted
The sudden loss of connection or failure in to tailor and predict it through the study of recorded failure
Wireless Network configuration may cause the failure of the data. A non-reliable application means dissatisfied customers,
application.
Failure to connect to the database due to an error
loss of market share and significant costs to the supplier. For
Mobile Database critical applications, such as banking or health monitoring,
occurred in the database server.
Some Smartphone applications may not be non-reliability can lead to great damage. Therefore, it is of
OS version
compatible with upgraded OS version (for great necessity to insure early detection and resolution of
example Gas Prices Canada application is no reliability issues in desktop applications as well as, now
longer available for iOS 6)
increasingly, in mobile applications.
Upgrading from a version to another may fix
problems but cause others as it was the case with Our future work will focus on analyzing more in depth
Software upgrades Skype 4.2.2601 and Skype 4.2.2604, the updated these selected SRGMs and try to modify the closest one to the
version crashes more often than the previous data and adapt in to Smartphone applications. Moreover, we
version when making calls will check if we need to have for each type of applications a
161
specific model or one model is applicable to all the categories Software Engineering and its Applications, Jeju Island, Korea, pp. 168-
of Smartphone applications taking into consideration the 175, 2009.
severity of the failure. [10] A. Wasserman, “Software Engineering Issues for Mobile Application
Development,” Worshop on the Future of Software Engineering
Another future purpose is to evaluate the possibility of Reasearh (FoSER), Santa Fe, NM, USA, ACM Press, 2010.
applying more than one model on the same failure data such as [11] M. R. Lyu, “Handbook of Software Reliability Engineering”, Chapter 3,
the Windows phone crash count by dividing the data into two McGraw-Hill, 1996.
or more categories and applying the convenient model to each [12] Y. K. Malaiya and J. Denton, “What do the Software Reliability Growth
category to predict the reliability of the application. Further Model Parameters Represent?”, 8th IEEE International Symposium on
investigations of Android failure data are also underway. Software Reliability Engineering, Albuquerque, NM, USA, pp. 124-135,
1997.
REFERENCES [13] R. Lai and M. Garg, “A Detailed Study of NHPP Software Reliability
[1] U.D. Perera, “Reliability of Mobile Phones”, Annual Reliability and Models”, Journal of Software, Vol. 7, No. 6, pp. 1296-1306, 2012.
Maintainability Symposium, Washington, DC, USA, pp. 33-38, 1995. [14] ReliaSoft, “Reliability Growth & Repairable System Data Analysis
[2] F. Ahmed and L.F. Capretz, “Managing the Business of Software Reference”, http://rga.reliasoft.com/, 2010.
Product Line: An Empirical Investigation of Key Business Factors”, [15] J.D. Musa, A.Iannino and K. Okumoto, “Software Reliability –
Information and Software Technology, Vol. 49, pp. 194-208, 2007. Measurement, Prediction, Applications”, McGraw-Hill, 1987.
[3] F. Ahmed, L.F. Capretz and J. Samarabandu, “Fuzzy Inference System [16] W. Xia, L.F. Capretz and D. Ho, “A Neuro-Fuzzy Model for Function
for Software Product Family Process Evaluation”, Information Sciences, Point Calibration”, WSEAS Transactions on Information Science and
Vol. 178, pp. 2780-2793, 2008. Applications, Vol. 5, pp. 22-30, 2008.
[4] J. Pam, “Software Reliability”, http://www.ece.cmu.edu/~koopman [17] A.B. Nassif, D. Ho and L. F. Capretz. "Towards an Early Software
/des_s99/sw_reliability/, 1999. Estimation Using Log-Linear Regression and a Multilayer Perceptron
Model", Journal of Systems and Software, Vol. 86, pp. 144-160, 2013.
[5] J. Xu, D.Ho and L.F Capretz, “An Empirical Validation of Object-
Oriented Design Metrics for Fault Prediction,” Journal of Computer [18] A.B. Nassif, L. F. Capretz and D. Ho, "Software Estimation in the Early
Science, Vol. 4, No. 7, pp. 571-577, ANSInet, 2008. Stages of the Software Life Cycle", International Conference on
Emerging Trends in Computer Science, Communication and
[6] H.F. El Yamany, M.A.M. Capretz and L.F. Capretz, “A Multi-Agent Information Technology, Nanded, India, pp.5-13, 2010.
Framework for Testing Distributed Systems”, 30th IEEE International
Computer Software and Applications Conference (COMPSAC), [19] A.B. Nassif, L. F. Capretz and D. Ho, "Estimating Software Effort
Chicago, MI, USA, Vol. II, pp. 151-156, 2006. Based on Use Case Point Model Using Sugeno Fuzzy Inference
System”, 23rd IEEE International Conference on Tools with Artificial
[7] Mharmer, “Modern Mobile Apps”, Intelligence (ICTAI), Boca Raton, FL, USA, pp. 393-398, 2011.
http://blog.modernmobileapps.com/category/chips.aspx, 2011.
[20] A.K. Jha, “A Risk Catalog for Mobile Applications”, Chapter 5, Florida
[8] L.F. Capretz and P.A. Lee, "Reusability and Life Cycle Issues within an Institute of Technology, Melbourne, FL, USA, 2007.
Object-Oriented Methodology", Proceedings of the Eight International
Conference on Technology of Object-Oriented Languages and Systems, [21] S. Malek, R. Roshandel, D. Kilgore and I. Elhag, “Improving the
Prentice-Hall, Santa Barbara, CA, USA, pp. 139-150, 1992. Reliability of Mobile Software Systems through Continuous Analysis
and Proactive Reconfiguration”, 31st IEEE International Conference on
[9] S. Jang and E. Lee, “Reliable Mobile Application Modeling Based on Software Engineering, Vancouver, BC, Canada, Vol. ICSE-Companion,
Open API”, Proceedings of International Conference on Advanced pp. 275-278, 2009.
162