Performance Analysis of VxWorks and RTLinux

Benjamin Ip
COMS W4995-2, Languages of Embedded Systems
Department of Computer Science, Columbia University, NY

1. Abstract execution. Besides this unique feature, RTLinux

is freely available to the public1. As more
This paper compares and evaluates the development tools are geared towards RTLinux,
suitability of two real-time operating systems, it will become a dominant player in the
the commercially available VxWorks and the embedded market.
publicly available RTLinux. Holding the
hardware constant and using different
measurement methodologies, we measured the 3. Performance Metrics
overheads incurred during operating systems
Our project goal is to study the performance
context switching, interrupt processing, object
analysis of these two operating systems by
synchronization, and message passing. We also
measuring the following key metrics.
examine their effectiveness in terms of how they
handle priority inversion problem.
3.1. Context Switch
Our finding illustrates that both VxWorks and Both RTOSes are designed to support
RTLinux provide good raw performance. multitasking. This feature is important for real-
However, VxWorks is more predictable and time applications that are frequently
deterministic, thereby making it more suitable as implemented with multiple asynchronous tasks
an operating system platform for developing and of execution. During task scheduling, a context
running soft and hard real-time applications. switch is needed to suspend one task and
immediately resume the other. Therefore, it is
fundamental to analyze the average context
2. Introduction switch latency in order to measure operating
2.1. Overview of VxWorks systems responsiveness.
VxWorks is by far the most widely adopted
commercial RTOS in the embedded industry. It 3.2. Priority Inversion
is developed by WindRiver with the intention to Priority inversion occurs when a high-priority
design an operating system with fast, efficient, task is blocked, waiting for a low-priority task to
and deterministic context switching. Its Wind release a resource shared by the high priority
micro-kernel can support preemptive and round task. Priority inversion is a serious problem in
robin scheduling policies, and unlimited number real-time system since it often leads to deadlock.
of tasks with a maximum of 256 priority levels. Both RTOSes incorporate their own priority
VxWorks is also well known for its rich tool inheritance protocol and one of the project goals
chain and run time library that significantly is to examine the effectiveness of these
reduce the amount of time for application protocols.
development. Despite the extensive features
from VxWorks, it bares a high premium for 3.3. Interrupt Latency
royalty fee.
Interrupt Latency is defined as the sum of
2.2. Overview of RTLinux interrupt blocking time during which the kernel
Unlike Linux, RTLinux provides hard real-time is pending to respond to an interrupt, saving the
capability. It has a hybrid kernel architecture tasks context, determining the interrupt source,
with a small real-time kernel coexists with the and invoking the interrupt handler. For a
Linux kernel running as the lowest priority task. particular interrupt, the latency also includes the
This combination allows RTLinux to provide execution time of other nested interrupt handlers.
highly optimized, time-shared services in parallel Since most embedded systems are interrupt-
with the real-time, predictable, and low-latency
1
RTLinux is distributed by Finite State Machine
driven, low interrupt latency will drastically and its board support package. These
increase system throughput. significantly reduce development time spent on
configuring the hardware.
3.4. Synchronization
A full suite of synchronization methods is 4. Related Work
provided by VxWorks and RTLinux to allow
exclusive access of shared resources. Acquiring Performance Analysis of operating systems has
and releasing semaphores to protect shared long been an interesting subject among research
objects do incur penalty. Such penalty is often groups. Levine [1] and his peers have presented
associated with adding and removing the their benchmark results on context switch time
requested tasks into and out of the object lock and priority inversion protocol latency of a real-
queues. As such, measuring synchronization time CORBA2 architecture. Levine’s [1] method
overhead is another way to determine the of detecting and observing priority inversion is
viability of a real-time operating system. complicated. A straightforward way to create a
priority inversion scenario is explained in
3.5. Inter-Process Communications Obenland’s [4] article and will be described in
next section.
Modern real-time applications are constructed as
a set of independent, cooperative tasks. Along Sohal [3] took both the analytical and empirical
with high-speed semaphores, VxWorks and approaches to measure different phases of
RTLinux also provide message queue as higher- interrupt latency of a real-time operating system.
level synchronization mechanism to allow Due to time constraint, we chose only one of
cooperating tasks to communicate with each Sohal’s interrupt tests that typically reveals the
other. Because of the implementation performance of interrupt handling. However, we
complexity, using this service imposes the did not apply Sun’s [4] approach to measure
greatest amount of latency and thus is a key interrupt latency because Linux interrupt
metric to operating system study. mechanism is implemented vastly different from
RTLinux (with no distinction of top and bottom
3.6. Measurement Process half of interrupt service routine).
Measuring the above metric requires certain
degree of resolution, accuracy, and granularity. We also learnt from Obenland’s [2] experience
Throughout our project, both hardware and that prior to executing any IPC test, the message
software logic analyzers were used to capture queue should have no message pending and the
and record measurement samples. We receiving task must be blocked waiting for the
emphasized on the use of the hardware logic message.
analyzer because it gives the finest resolution,
least obtrusion to real-time code, and more In Stewards [4] paper, he devoted much of his
important it is platform independent. In most time to review and explain various measurement
test cases the software analyzer was used for techniques that produce results of different
verification. We also developed small firmware granularity. We are convinced by Stewards [4]
code to setup the memory maps and interrupt that hardware logic analyzer is preferable to
vector tables, tune the system clock, and disable other tools and techniques for measurements
the hardware cache. All test functions and throughout the project.
system calls written to initialize tasks,
semaphores and message queues are POSIX 5. Test Methods and Experimental
compliant. Finally, each test was measured with
a sample size of 25 to ensure that the data Results
collected are statistically sound. We modified some of the test methods
referenced in the previous section to achieve fair
3.7. Test Environment and accurate results. This section describes the
Our tests are conducted under VxWorks version measurement outcomes along with the methods
5.4 and RTLinux version 3.0. We executed all that we used to test different metrics.
of our tests on evaluation boards manufactured
by FSM and WindRiver. Each evaluation board
comes with a single MPC8260 microprocessor 2
Common Open Brokerage Architecture
5.1. Context Switch
We configured both RTOSes to use round-robin
scheduling policy to determine context switch
time. Figure 1 shows that with round-robin
policy, we simply need to create two tasks and
let the scheduler to execute them alternately,
without the need of prioritizing them.

Task Task Task Figure 2: Measuring Priority Inversion Using

1 2 1 Software Analyzer
Context Context We took several time measurements between
Switch Switch tCyclicTask requesting the resource and
Time Time
tSlowTask releasing it, and the results are given
Figure 1: Context Switch Test Setup in the table 2.

Both tasks under test have the same function;

each contains an infinite empty loop to avoid VxWorks RTLinux
additional computation. Table 1 shows the Priority Inversion (µS)
123 (1.67) 108 (0.41)
average (and standard deviation) context switch Mean (Std)
time measured in microseconds.
Table 2: Priority Inversion Measurments

VxWorks RTLinux An important characteristic of RTOSes is

Context Switch (µS) predictability. Although RTLinux takes less
11 (0.04) 13.4 (0.6) time to resolve a priority inversion problem, both
Mean (Std)
figures appear to be in an acceptable range. This
Table 1: Context Switch Time Measurements indicates that both RTOSes have implemented an
effective priority inheritance protocol to ensure
The context switch time measured on VxWorks that critical deadlines are met.
is consistently low, with a standard deviation of
0.04. On the contrary, the RTLinux context 5.3. Interrupt Latency
switch time is 18% higher than and it is not as
consistent (with std of 0.6) as VxWorks. Thus, In this experiment, we configured the MPC8260
context switch time for VxWorks is more hardware timer with a period of 50 MHz to
deterministic. The lower score achieved by generate a timer interrupt every 20 µs. An
RTLinux seems to imply that running both real- interrupt service routine that updates a system
time and non real-time tasks in parallel may not tick count is hooked to the interrupt vector table.
be the most feasible solution to embedded We use the hardware logic analyzer to measure
products. the time between the assertion of the timer
interrupt and the execution of the ISR (Figure 3).
5.2. Priority Inversion
We created the priority inversion scenario by Timer Task Timer
running three tasks at low, medium, and high Interrupt Context ISR
priorities, with the low and high priority tasks Assertion Saved Invocation
competing for the same resource. Below is an
occurrence of priority inversion (the yellow Latency
arrow) that captured from a software analyzer.
The tCyclicTask, tWorkTask, and tSlowTask Figure 3: Interrupt Latency Test Setup
correspond to tasks with high, medium, and low
priorities.
Noticed that all other system interrupts are
disabled so that our measurements are not
affected by nested-interrupts. The average and
standard deviation of both systems interrupt
latencies are recorded in Table 3.
 VxWorks RTLinux 5.5. Inter-Process Communication
Interrupt Latency (µS)
98 (0.55) 132 (1.2) This test is to measure the communication delay
Mean (Std)
necessary for a task to send a message to another
Table 3: Interrupt Latency Measurements task via a message queue as shown in Figure 2.

It is not surprised that VxWorks has much lower

interrupt latency (35%) than RTLinux.
Data
Traditional Linux is notorious for having high Send Rcv
interrupt latency. It appears that even though Task Message Queue Task
RTLinux had been added with real-time
capability, it still exhibits some non real-time
behaviour. Delay

Figure 5: Message Queue Test Setup

5.4. Synchronization
In our test, we only focused on measuring the We began this test by creating and activating (or
time to acquire a binary semaphore in both open) a message queue. Next, we spawned a
systems. To measure the semaphore overhead, receiving task from which the message receive
we first created and initialized the semaphore function is invoked. The receive system call
itself to make it unavailable. blocks the receiving task and put it in the wait
state (since the message queue is empty). While
Semaphore the receiving task was waiting for the message,
we spawned a sending task to send a message via
the same message queue. The time between the
sending task to call the message send function
and the receiving task to receive message
Semaphore Semaphore notification is given in Table 6.
Acquire Release
Task Task
VxWorks RTLinux
Delay Msg Queue Delay (µS)
118 (0.9) 113 (1.8)
Mean (Std)
Figure 4: Binary Semaphore Test Setup
Table 5: Message Queue Measurements
We then spawned two tasks to release and
acquire the semaphore respectively, in the exact
order. Finally, we measured the time (Figure 4) In terms of message send/receive latency,
RTLinux achieves a better score than VxWorks
during which the task made the system call to
by a small margin. As mentioned earlier, these
acquire the semaphore. This task should not be
figures can vary greatly depending on the IPC
blocked waiting since the first task should
implementation (IPC can be implemented using
release the semaphore prior to execution of the
second task. Table 4 shows the average shared memory).
overhead for VxWorks and RTLinux to
successfully acquire a semaphore. 6. Conclusions and Future Work
In this project, we measured several real-time
VxWorks RTLinux
operating system key metrics to evaluate the
Semaphore Take (µS) performance of VxWorks and RTLinux. The
13 (0.29) 15 (0.08)
Mean (Std) results presented in this paper roughly matches
Table 4: Binary Semaphores Take with the characteristics of the two operating
Measurements systems. Our overall analysis shows that both
operating systems are suitable for real-time
These figures show that the RTLinux takes application. In particular, VxWorks is more
slightly longer to obtain a binary semaphore than deterministic and predictable,
VxWorks.
Due to time constraint and limited resources, we
can focused only on studying the heart of the [9] P. Wilshire, “Installing RTLinux”,
operating system – the kernel level performance http://rtlinux.org/documents/installation_ju
that unveil the true system behaviour. Modern ne_2000.html, 2000
real-time operating systems often packaged with
powerful run-time libraries, scalable networking [10] WindRiver Systems Inc, Tornado User’s
components and flexible file system. A broad Guide, Alameda,CA: WindRiver Systems,
range of tests that cover these aspects will Inc, 1999
provide us a comprehensive result in terms of
performance versus cost. Thereby, it is difficult [11] WindRiver Systems Inc, VxWorks
to conclude which operating system is superior Programmer’s Guide, Alameda,CA:
to the other without an exhaustive comparison. WindRiver Systems, Inc, 1999

7. Bibliography
[1] D. Levine, S. Flores-Gaitan, C. D. Dill, and
D. C. Schmidt, “Measuring OS Support for
Real-Time CORBA ORBs”, in 4th IEEE
International Workshop on Object-oriented
Real-Time Dependable Systems 00’, Santa
Babara, California, Jan. 27-29.

[2] K. Obenland, “Real-Time Performance of

Standards Based Commercial Operating
Systems”

[3] V. Sohal, “How To Really Measure Real-

Time”, Embedded System Conference,
Spring 2001

[4] Jun Sun, ”Interrupt Latency”, Monta Vista

Software,http://www.mvista.com/realtim
e/latency/

[5] D. Stewart, “Measuring Execution Time

and Real-Time Performance”, Embedded
System Conference, Spring 2001

[6] Real Time magazine, “Evaluation Report

Definition”, http://www.realtime-info.be,
March 1999

[7] R. Appleton, “Understanding a Context

Switch Benchmark”, Linux Journal
http://www2.linuxjournal.com/ljissues/is
sue57/2941.html, Jan. 1997

[9] V. Yodaiken, “An Introduction to Real-

Time Linux”,
http://www.rtlinux.org/documents/RTLinu
x.ppt

[8] Victor Yodaiken, “The RTLunix Approach

to Hard Real-Time”,
http://rtlinux.org/documents/papers/whitep
aper.html, Oct. 1997