The Embedded Design Life Cycle

Iteration and Implementation

(Before Hardware and Software Teams Stop Communicating)

The iteration and implementation part of the process represents a somewhat blurred area between
implementation and hardware/software partitioning in which the hardware and software paths diverge.
This phase represents the early design work before the hardware and software teams

build “the wall” between them. The design is still very fluid in this phase. Even though major blocks might
be partitioned between the hardware components and the software components, plenty of leeway
remains to move these boundaries as more of the design constraints are understood and modeled. In
Figure 1.2 earlier in this chapter, Mann represents the iteration phase as part of the selection process. The
hardware designers might be using simulation tools, such as architectural simulators, to model the
performance of the processor and memory systems. The software

designers are probably running code benchmarks on self-contained, single-board computers that use the
target micro processor. These single-board computers are often referred to as evaluation boards because
they evaluate the performance of the microprocessor by running test code on it. The evaluation board
also provides a convenient software design and debug environment until the real system hardware
becomes available.

You’ll learn more about this stage in later chapters. Just to whet your appetite, however, consider this:
The technology exists today to enable the hardware and software teams to work closely together and keep
the partitioning process actively engaged longer and longer into the implementation phase. The teams
have a greater opportunity to get it right the first time, minimizing the risk that something might crop up
late in the design phase and cause a major schedule delay as the teams scramble to fix it.

Detailed Hardware and Software Design

This book isn’t intended to teach you how to write software or design hardware. However, some aspects
of embedded software and hardware design are unique to the discipline and should be discussed in detail.
For example, after one of my lectures, a student asked, “Yes, but how does the code actually get into the
microprocessor?” Although well-versed in C, C++, and Java, he had never faced having to initialize an
environment so that the C code could run in the first place. Therefore, I have devoted separate chapters
to the development environment and special software techniques.

I’ve given considerable thought how deeply I should describe some of the hardware design issues. This is
a difficult decision to make because there is so much material that could be covered. Also, most electrical
engineering students have taken courses in digital design and microprocessors, so they’ve had ample
opportunity to be exposed to the actual hardware issues of embedded systems design. Some issues are
worth mentioning, and I’ll cover these as necessary.

Hardware/Software Integration
The hardware/software integration phase of the development cycle must have special tools and
methods to manage the complexity. The process of integrating embedded software and hardware is an
exercise in debugging and discovery. Discovery is an especially apt term because the software team now
finds out whether it really understood the hardware specification document provided by the hardware
team.

Big Endian/Little Endian Problem

One of my favorite integration discoveries is the “little endian/big endian” syndrome. The hardware
designer assumes big endian organization, and the software designer assumes little endian byte order.
What makes this a classic example of an interface and integration error is that both the software and
hardware could be correct in isolation but fail when integrated because the “endianness” of the interface
is misunderstood. Suppose, for example that a serial port is designed for an ASIC with a 16-bit I/O

bus. The port is memory mapped at address 0x400000. Eight bits of the word are the data portion of the
port, and the other eight bits are the status portion of the port. Even though the hardware designer might
specify what bits are status and what bits are data, the software designer could easily assign the wrong
port address if writes to the port are done as byte accesses .

If byte addressing is used and the big endian model is assumed, then the algorithm should check the status
at address 0x400001. Data should be read from and written to address 0x400000. If the little endian
memory model is assumed, then the reverse is true. If 16-bit addressing is used, i.e., the port is declared
as

unsigned short int * io_port ;

then the endianness ambiguity problem goes away. This means that the software might become more
complex because the developer will need to do bit manipulation in order to read and write data, thus
making the algorithm more complex.

The Holy Grail of embedded system design is to combine the first hardware prototype, the application
software, the driver code, and the operating system software together with a pinch of optimism and to
have the design work perfectly out of the chute. No green wires on the PC board, no “dead bugs,” no
redesigning the ASICs or Field Programmable Gate Arrays (FPGA), and no rewriting the software. Not likely,
but I did say it was the Holy Grail.

Note Here “dead bugs” are extra ICs glued to the board with their I/O pins facing up. Green wires are then
soldered to their “legs” to patch them into the rest of the circuitry.

You might wonder why this scenario is so unlikely. For one thing, the real-time nature of embedded
systems leads to highly complex, nondeterministic behavior that can only be analyzed as it occurs.
Attempting to accurately model or simulate the behavior can take much longer than the usable lifetime of
the product being developed. This doesn’t necessarily negate what I said in the previous section; in

fact, it is shades of gray. As the modeling tools improve, so will the designer’s ability to find bugs sooner
in the process. Hopefully, the severity of the bugs that remain in the system can be easily corrected after
they are uncovered. In Embedded Systems Programming [1], Michael Barr discusses a software
architecture that anticipates the need for code patches and makes it easy to insert them without major
restructuring of the entire code image.

Debugging an Embedded System

In most ways, debugging an embedded system is similar to debugging a hostbased application. If the
target system contains an available communications channel to the host computer, the debugger can
exist as two pieces: a debug kernel in the target system and a host application that communicates with it
and manages the source database and symbol tables. (You’ll learn more about this later on as well.)
Remember, you can’t always debug embedded systems using only the methods of the host computer,
namely a good debugger and printf() statements.

Many embedded systems are impossible to debug unless they are operating at full speed. Running an
embedded program under a debugger can slow the program down by one or more orders of magnitude.
In most cases, scaling all the real-time dependencies back so that the debugger becomes effective is
much more work than just using the correct tools to debug at full speed.

Manufacturers of embedded microprocessors also realize the difficulty of controlling these variables, so
they’ve provided on-chip hooks to assist in the debugging of embedded systems containing their
processors. Most designers won’t even consider using a microprocessor in an embedded application
unless the silicon manufacturer can demonstrate a complete tool chain for designing and debugging its
silicon.

In general, there are three requirements for debugging an embedded or real-time system:

• Run control — The ability to start, stop, peak, and poke the processor and memory.
• Memory substitution — Replacing ROM-based memory with RAM for rapid and easy code
download, debug, and repair cycles.
• Real-time analysis — Following code flow in real time with real-time trace analysis.

For many embedded systems, it is necessary also to integrate a commercial or inhouse real-time
operating system (RTOS) into the hardware and application software. This integration presents its own
set of problems (more variables); the underlying behavior of the operating system is often hidden from
the designers because it is obtained as object code from the vendor, which means these bugs are now
masked by the RTOS and that another special tool must be used.

This tool is usually available from the RTOS vendor (for a price) and is indispensable for debugging the
system with the RTOS present. The added complexity doesn’t change the three requirements previously
listed; it just makes them more complex. Add the phrase “and be RTOS aware” to each of the three

listed requirements, and they would be equally valid for a system containing a RTOS. The general
methods of debugging that you’ve learned to use on your PC or workstation are pretty much the same as
in embedded systems. The exceptions are what make it interesting. It is an exercise in futility to try to
debug a software module when the source of the problem lies in the underlying hardware or the
operating system. Similarly, it is nearly impossible to find a bug that can only be observed when the
system is running at full speed when the only trace capability available is to single-step the processor.
However, with these tools at your disposal, your approach to debugging will be remarkably similar to
debugging an application designed to run on your PC or workstation.

Product Testing and Release

Product testing takes on special significance when the performance of the embedded system has life or
death consequences attached. You can shrug off an occasional lock-up of your PC, but you can ill-afford a
software failure if the PC controls a nuclear power generating station’s emergency system. Therefore, the

testing and reliability requirements for an embedded system are much more stringent than the vast
majority of desktop applications. Consider the embedded systems currently supporting your desktop PC:
IDE disk drive, CD-ROM, scanner, printer, and other devices are all embedded systems in their own right.
How many times have they failed to function so that you had to cycle power to them?

From the Trenches For the longest time, my PC had a nagging problem of crashing in the middle of my
word processor or graphics application. This problem persisted through Windows 95, 95 Sr-1, 98, and 98
SE. After blaming Microsoft for shoddy software, I later discovered that I had a hardware problem in my
video card. After replacing the drivers and the card, the crashes went away, and my computer is behaving
well. I guess hardware/software integration problems exist on the desktop as well.

However, testing is more than making sure the software doesn’t crash at a critical moment, although it is
by no means an insignificant consideration. Because embedded systems usually have extremely tight
design margins to meet cost goals, testing must determine whether the system is performing close to its
optimal capabilities. This is especially true if the code is written in a high-level language and the design
team consists of many developers. Many desktop applications have small memory leaks. Presumably, if
the application ran long enough, the PC would run out of heap space, and the computer would crash.
However, on a desktop machine with 64MB of RAM and virtual swap space, this is unlikely to be a problem.
On the other side, in an embedded system, running continuously for weeks at a time, even a small memory
leak is potentially disastrous.

Who Does the Testing?

In many companies, the job of testing the embedded product goes to a separate team of engineers and
technicians because asking a designer to test his own code or product usually results in erratic test results.
It also might lead to a “circle the wagons” mentality on the part of the design team, who view the testers
as a roadblock to product release, rather than equal partners trying to prevent a defective product from
reaching the customer.

Compliance Testing
Compliance testing is often overlooked. Modern embedded systems are awash in radio frequency (RF)
energy. If you’ve traveled on a plane in the last five years, you’re familiar with the requirement that all
electronic devices be turned off when the plane descends below 10,000 feet. I’m not qualified to discuss
the finer points of RF suppression and regulatory compliance requirements; however, I have spent many
hours at open field test sites with various compliance engineering (CE) engineers trying just to get one
peak down below the threshold to pass the class B test and ship the product.

I can remember one disaster when the total cost of the RF suppression hardware that had to be added
came to about one-third of the cost of all the other hardware combined. Although it can be argued that
this is the realm of the hardware designer and not a hardware/software design issue, most digital
hardware designers have little or no training in the arcane art of RF suppression. Usually, the hotshot
digital wizard has to seek out the last remaining analog designer to get inclued in on how to knock down
the fourth harmonic at 240MHz. Anyway, CE testing is just as crucial to a product’s release as any other
aspect of the test program.

CE testing had a negative impact on my hardware/software integration activities in one case. I thought
we had done a great job of staying on top of the CE test requirements and had built up an early
prototype especially for CE testing. The day of the tests, I proudly presented it to the CE engineer on
schedule. He then asked for the test software that was supposed to exercise the hardware while the RF
emissions were being monitored. Whoops, I completely forgot to write drivers to exercise the hardware.
After some scrambling, we pieced together some of the turn-on code and convinced the CE engineer
(after all, he had to sign all the forms) that the code was representative of the actual operational code.

Referring to Figure 1.4, notice the exponential rise in the cost to fix a defect the later you are in the
design cycle. In many instances, the Test Engineering Group is the last line of defense between a smooth
product release and a major financial disaster.

The percentage of project time spent in each phase of the embedded design life cycle. The curve shows
the cost associated with fixing a defect at each stage of the process. Like debugging, many of the
elements of reliability and performance testing map directly on the best practices for host-based
software development. Much has been written about the correct way to develop software, so I won’t
cover that again here. What is relevant to this subject is the best practices for testing software that has

mission-critical or tight performance constraints associated with it. Just as with the particular problems
associated with debugging a real-time system, testing the same system can be equally challenging.

Maintaining and Upgrading Existing Products

The embedded system tool community has made almost no effort to develop tools specifically targeted to
products already in service. At first blush, you might not see this as a problem. Most commercially
developed products are well documented, right?

The majority of embedded system designers (around 60 percent) maintain and upgrade existing products,
rather than design new products. Most of these engineers were not members of the original design team
for a particular product, so they must rely on only their experience, their skills, the existing documentation,
and the old product to understand the original design well enough to maintain and improve it. From the
silicon vendor’s point of view, this is an important gap in the tool chain because the vendor wants to keep
that customer buying its silicon, instead of giving the customer the chance to do a “clean sheet of paper”
redesign. Clean sheets of paper tend to have someone else’s chip on them.

From the Trenches One can hardly overstate the challenges facing some upgrade teams. I once visited a
telecomm manufacturer that builds small office phone systems to speak to the product-support team. The
team described the situation as: “They wheel the thing in on two carts. The box is on one cart, and the
source listings are on the other. Then they tell us to make it better.” This usually translates to improving
the overall performance of the embedded system without incurring the expense of a major hardware
redesign.

Another example features an engineer at a company that makes laser and ink-jet printers. His job is to
study the assembly language output of their C and C++ source code and fine-tune it to improve
performance by improving the code quality. Again, no hardware redesigns are allowed.
Both of these examples testify to the skill of these engineers who are able to reverse-engineer and improve
upon the work of the original design teams. This phase of a product’s life cycle requires tools that are
especially tailored to reverse engineering and rapidly facilitating “what if …” scenarios. For example, it’s
tempting to try a quick fix by speeding up the processor clock by 25 percent; however, this could cause a
major ripple effect through the entire design, from memory chip access time margins to increased RF
emissions. If such a possibility could be as easily explored as making measurements on a few critical code
modules, however, you would have an extremely powerful tool on your hands.

Sometimes, the solutions to improved performance are embarrassingly simple. For example, a data
communications manufacturer was about to completely redesign a product when the critical product
review uncovered that the processor was spending most of its time in a debug module that was
erroneously left in the final build of the object code. It was easy to find because the support teams had
access to sophisticated tools that enabled them to observe the code as it executed in real time. Without
the tools, the task might have been too time-consuming to be worthwhile. Even with these test cases,
every marketing flyer for every tool touts the tool’s capability to speed “time to market.” I’ve yet to hear
any tool vendor advertise its tool as speeding “time to reverse-engineer,” although one company claimed
that its logic analyzer sped up the “time to insight.” Embedded systems projects aren’t just “software on
small machines.” Unlike application development, where the hardware is a fait accompli, embedded
projects are usually optimization exercises that strive to create both hardware and software that
complement each other. This difference is the driving force that defines the three most characteristic
elements of the embedded design cycle: selection, partitioning, and system integration. This difference
also colors testing and debugging, which must be adapted to work with unproven, proprietary hardware.

While these characteristic differences aren’t all there is to embedded system design, they are what most
clearly differentiate it from application development, and thus, they are the main focus of this book. The
next chapter discusses the processor selection decision. Later chapters address the other issues.