A SYSTEMATIC APPROACH TO PERFORMANCE EVALUATION

Most performance problems are unique. The metrics, workload, and evaluation techniques
used for one problem generally cannot be used for the next problem. Nevertheless, there are
steps common to all performance evaluation projects that help you avoid the common
mistakes in performance evaluation. These steps are as follows.
1. State Goals and Define the System: The first step in any performance evaluation project is
to state the goals of the study and define what constitutes the system by delineating system
boundaries. Given the same set of hardware and software, the definition of the system may
vary depending upon the goals of the study. Given two CPUs, for example, the goal may be
to estimate their impact on the response time of interactive users. In this case, the system
would consist of the timesharing system, and the conclusions of the study may depend
significantly on components external to the CPU. On the other hand, if the two CPUs are
basically similar except for their Arithmetic-Logic Units (ALUs) and the goal is to decide
which ALU should be chosen, the CPUs may be considered the system’s and only the
components inside the CPU may be considered part of the system.
The choice of system boundaries affects the performance metrics as well as workloads used
to compare the systems. Therefore, understanding the system boundaries is important.
Although the key consideration in setting the system boundaries is the objective of the study,
other considerations, such as administrative control of the sponsors of the study, may also
need to be taken into account. If the sponsors do not have a control over some components,
they may want to keep those components outside the system boundaries.
2. List Services and Outcomes: Each system provides a set of services. For example, a
computer network allows its users to send packets to specified destinations. A database
system responds to queries. A processor performs a number of different instructions. The
next step in analyzing a system is to list these services. When a user requests any of these
services, there are a number of possible outcomes. Some of these outcomes are desirable and
some are not. For example, a database system may answer a query correctly, incorrectly (due
to inconsistent updates), or not at all (due to deadlocks or some similar problems). A list of
services and possible outcomes is useful later in selecting the right metrics and workloads.
3. Select Metrics: The next step is to select criteria to compare the performance. These
criteria are called metrics. In general, the metrics are related to the speed, accuracy, and
availability of services. The performance of a network, for example, is measured by the speed
(throughput and delay), accuracy (error rate), and availability of the packets sent. The
performance of a processor is measured by the speed of (time taken to execute) various
instructions. The selection of the correct metrics will be discussed later.
4. List Parameters: The next step in performance projects is to make a list of all the
parameters that affect performance. The list can be divided into system parameters and
workload parameters. System parameters include both hardware and software parameters,
which generally do not vary among various installations of the system. Workload parameters
are characteristics of users’ requests, which vary from one installation to the next.
The list of parameters may not be complete. That is, after the first pass of the analysis, you
may discover that there are additional parameters that affect the performance. You can then
add these parameters to the list, but at all times keep the list as comprehensive as possible.
This allows the analyst and decision makers to discuss the impact of various parameters and
determine what data needs to be collected before or during the analysis.
5. Select Factors to Study: The list of parameters can be divided into two parts: those that will
be varied during the evaluation and those that will not. The parameters to be varied are called
factors and their values are called levels. In general, the list of factors, and their possible
levels, is larger than what the available resources will allow. Otherwise, the list will keep
growing until it becomes obvious that there are not enough resources to study the problem. It
is better to start with a short list of factors and a small number of levels for each factor and to
extend the list in the next phase of the project if the resources permit. For example, you may
decide to have only two factors: quantum size and the number of users. For each of these two
factors you may choose only two levels: small and large. The working set size and the type of
workload may be fixed.
The parameters that are expected to have a high impact on the performance should be
preferably selected as factors. Like metrics, a common mistake in selecting the factors is that
the parameters that are easy to vary and measure are used as factors while other more
influential parameters are ignored simply because of the difficulty involved. In selecting
factors, it is important to consider the economic, political, and technological constraints that
exist as well as including the limitations imposed by the decision makers’ control and the
time available for the decision. This increases the chances of finding a solution that is
acceptable and implementable.

6. Select Evaluation Technique: The three broad techniques for performance evaluation are
analytical modeling, simulation, and measuring a real system. The selection of the right
technique depends upon the time and resources available to solve the problem and the desired
level of accuracy.
7. Select Workload: The workload consists of a list of service requests to the system. For
example, the workload for comparing several database systems may consist of a set of
queries. Depending upon the evaluation technique chosen, the workload may be expressed in
different forms. For analytical modeling, the workload is usually expressed as a probability of
various requests. For simulation, one could use a trace of requests measured on a real system.
For measurement, the workload may consist of user scripts to be executed on the systems. In
all cases, it is essential that the workload be representative of the system usage in real life. To
produce representative workloads, one needs to measure and characterize the workload on
existing systems.
8. Design Experiments: Once you have a list of factors and their levels, you need to decide on
a sequence of experiments that offer maximum information with minimal effort. In practice,
it is useful to conduct an experiment in two phases. In the first phase, the number of factors
may be large but the number of levels is small. The goal is to determine the relative effect of
various factors. In most cases, this can be done with fractional factorial experimental
designs. In the second phase, the number of factors is reduced and the number of levels of
those factors that have significant impact is increased.
9. Analyze and Interpret Data: It is important to recognize that the outcomes of
measurements and simulations are random quantities in that the outcome would be different
each time the experiment is repeated. In comparing two alternatives, it is necessary to take
into account the variability of the results. Simply comparing the means can lead to inaccurate
conclusions.
Interpreting the results of an analysis is a key part of the analyst’s art. It must be understood
that the analysis only produces results and not conclusions. The results provide the basis on
which the analysts or decision makers can draw conclusions. When a number of analysts are
given the same set of results, the conclusion drawn by each analyst may be different, as seen
in earlier lectures.
10. Present Results: The final step of all performance projects is to communicate the results
to other members of the decision-making team. It is important that the results be presented in
a manner that is easily understood. This usually requires presenting the results in graphic
form and without statistical jargon. The graphs should be appropriately scaled. The issue of
correct graph plotting is discussed further in subsequent classes. Often at this point in the
project the knowledge gained by the study may require the analyst to go back and reconsider
some of the decisions made in the previous steps. For example, the analyst may want to
redefine the system boundaries or include other factors and performance metrics that were
not considered before. The complete project, therefore, consists of several cycles through the
steps rather than a single sequential pass. The steps for a performance evaluation study are
illustrated in below.

Case Study Consider the problem of comparing remote pipes with remote procedure calls. In
a procedure call, the calling program is blocked, control is passed to the called procedure
along with a few parameters, and when the procedure is complete, the results as well as the
control return to the calling program. A remote procedure call is an extension of this
concept to a distributed computer system. A program on one computer system calls a
procedure object on another system. The calling program waits until the procedure is
complete and the result is returned. Remote pipes are also procedure like objects, but when
called, the caller is not blocked. The execution of the pipe occurs concurrently with the
continued execution of the caller. The results, if any, are later returned asynchronously.

The following project plan was written before starting the study.
1. System Definition: The goal of the case study is to compare the performance of
applications using remote pipes to those of similar applications using remote procedure calls.
The key component under study is the so-called channel. A channel can be either a procedure
or a pipe. The system consists of two computers connected via a network. The requests are
sent via the channel from the client computer to the server computer. Only the subsets of the
client and server computers that offer channel services is considered to be part of the system.
The study will be conducted so that the effect of components outside the system is
minimized.
2. Services: The services offered by the system are the two types of channel calls—remote
procedure call and remote pipe. The resources used by the channel calls depend upon the
number of parameters passed and the action required on those parameters. In this case study,
data transfer is chosen as the application and the calls will be classified simply as small or
large depending upon the amount of data to be transferred to the remote machine. In other
words, the system offers only two services: small data transfer or large data transfer.
3. Metrics: Due to resource limitations, the errors and failures will not be studied. Thus, the
study will be limited to correct operation only. For each service, the rate at which the service
can be performed, the time taken for the service, and the resources consumed will be
compared. The resources are the local computer (client), the remote computer (server), and
the network link. This leads to the following performance metrics:
(a) Elapsed time per call
(b) Maximum call rate per unit of time, or equivalently, the time required to complete a block
of n successive calls
(c) Local CPU time per call
(d) Remote CPU time per call (e) Number of bytes sent on the link per call
4. Parameters: The system parameters that affect the performance of a given application and
data size are the following:
(a) Speed of the local CPU (b) Speed of the remote CPU (c) Speed of the network
(d) Operating system overhead for interfacing with the channels (e) Operating system
overhead for interfacing with the networks (f) Reliability of the network affecting the number
of retransmissions required The workload parameters that affect the performance are the
following:
(a) Time between successive calls (b) Number and sizes of the call parameters (c) Number
and sizes of the results (d) Type of channel
(e) Other loads on the local and remote CPUs (f) Other loads on the network
5. Factors: The key factors chosen for this study are the following:
(a) Type of channel. Two types—remote pipes and remote procedure calls—will be
compared. (b) Speed of the network. Two locations of the remote hosts will be used—short
distance (in the campus) and long distance (across the country).
(c) Sizes of the call parameters to be transferred. Two levels will be used—small and large.
(d) Number n of consecutive calls. Eleven different values of n—1, 2, 4, 8, 16, 32, ..., 512,
1024—will be used.
The factors have been selected based on resource availability and the interest of the sponsors.
All other parameters will be fixed. Thus, the results will be valid only for the type of CPUs
and operating systems used. The retransmissions due to network errors will be ignored (not
included in the measurements). Experiments will be conducted when there is very little other
load on the hosts and the network.
6. Evaluation Technique: Since prototypes of both types of channels have already been
implemented, measurements will be used for evaluation. Analytical modeling will be used to
justify the consistency of measured values for different parameters.
7. Workload: The workload will consist of a synthetic program generating the specified types
of channel requests. This program will also monitor the resources consumed and log the
measured results. Null channel requests with no actual work but with monitoring and logging
activated will be used to determine the resources consumed in monitoring and logging.
8. Experimental Design: A full factorial experimental design with 23 × 11 = 88 experiments
will be used for the initial study.
9. Data Analysis: Analysis of Variance will be used to quantify the effects of the first three
factors and regression will be used to quantify the effects ofthe number n of successive calls.
10. Data Presentation: The final results will be plotted as a function of the block size n. This
case study was completed successfully and reported by Glasser (1987).