Why is Linux so pervasive? And why does something as simple as a TV need to run something as complex as Linux just to display streaming video on a screen?

The simple answer is Moore’s law. Gordon Moore, cofounder of Intel, observed in 1965 that the density of components on a chip doubles approximately every two years. That applies to the devices that we design and use in our everyday lives just as much as it does to desktops, laptops, and servers. At the heart of most embedded devices is a highly integrated chip that contains one or more processor cores and interfaces with main memory, mass storage, and peripherals of many types. This is referred to as a System on Chip (SoC). SoCs are increasing in complexity in accordance with Moore’s law. A typical SoC has a technical reference manual that stretches to thousands of pages.

Your TV isn’t simply displaying a video stream like the analog sets of old. The stream is digital, possibly encrypted, and needs processing to produce an image. Your TV is (or soon will be) connected to the internet. It can receive content from smartphones, tablets, laptops, desktops, and home media servers. It can be used to play games, stream video, and display live feeds from security cameras. You need a full OS to manage this degree of complexity.

Here are some points that drive the adoption of Linux:

• Linux has the necessary functionality. It has a good scheduler, a good network stack, support for USB, Wi-Fi, Bluetooth, many kinds of storage media, multimedia devices, and so on. It ticks all the boxes.
• Linux has been ported to a wide range of processor architectures, including some that are very commonly found in SoC designs – Arm, RISC-V, x86, PowerPC, and MIPS.
• Linux is open source, so you have the freedom to get the source code and modify it to meet your needs. You or someone working on your behalf can create a board support package for your device. You can add protocols, features, and technologies that may be missing from the mainline source code. You can remove features that you don’t need to reduce memory and storage requirements. Linux is flexible.
• Linux has an active community (in the case of the Linux kernel, very active). There is a new release of the kernel every 8 to 10 weeks, and each release contains code from more than 1,000 developers. An active community means that Linux is up to date and supports current hardware, protocols, and standards. The Linux Foundation is a non-profit organization with backing from big tech.

The foundation acts as a steward for several major open source projects besides Linux, including Kubernetes and PyTorch. It also hosts yearly events around the world like the Open Source Summit and Linux Plumbers Conference.

• Open source licenses guarantee that you have access to the source code. There is no vendor lock-in.

For these reasons, Linux is an ideal choice for complex devices. But there are a few caveats I should mention here. Complexity makes it harder to understand. Coupled with the fast-moving development process and the decentralized structures of open source, you need to put some effort into learning how to use it and to keep on re-learning as it changes. I hope that this book helps in the process.

Is Linux suitable for your project? Linux works well where the problem being solved justifies the complexity. It is especially good where connectivity, robustness, and complex user interfaces are required. However, it cannot solve every problem, so here are some things to consider before you jump in:

• Is your hardware up to the job? Compared to a traditional Real-Time Operating System (RTOS) such as VxWorks or QNX, Linux requires a lot more resources. It needs at least a 32-bit processor and lots more memory. I will go into more detail in the Selecting hardware for embedded Linux section.
• Do you have the right skill set? The early parts of a project, the board bring-up, require detailed knowledge of Linux and how it relates to your hardware. Likewise, when debugging and tuning your application you will need to be able to interpret the results. If you don’t have the skills in-house you may want to outsource some of the work. Of course, reading this book helps!
• Is your system real time? Linux can handle many real-time activities as long as you pay attention to certain details, which I cover in depth in Chapter 21.
• Will your code require regulatory approval (medical, automotive, aerospace, and so on)? The burden of regulatory verification and validation might make another OS a better choice. Even if you do choose Linux for use in these environments, it may make sense to purchase a commercially available distribution from a company that has supplied Linux for existing products like the one you are building. These commercial Linux vendors include Siemens, Timesys, and Wind River.

Consider these points carefully. Probably the best indicator of success is to look around for similar products that run Linux and see how they did it, and follow best practices.

Where does open source software come from? Who writes it? In particular, how does it relate to the key components of embedded development – the toolchain, bootloader, kernel, and basic utilities found in the root filesystem?

• Open source community: This, after all, is the engine that generates the software you are going to be using. The community is a loose alliance of developers many of whom are funded in some way by a non-profit organization, an academic institution, or a commercial company. They work together to further the aims of the various projects. There are many of them – some small, some large. Some that we will make use of are Linux itself, U-Boot, BusyBox, Buildroot, The Yocto Project, and the many projects under the GNU umbrella.
• CPU architects: These are the organizations that design the CPUs we use. The important ones here are Arm/Linaro (Arm Cortex-A), Intel (x86 and x86-64), SiFive (RISC-V), and IBM (PowerPC). They implement or at the very least influence support for the basic CPU architecture.
• SoC vendors: These include Broadcom, Intel, Microchip, NXP, Qualcomm, TI, and many others. They take the kernel and toolchain from the CPU architects and modify them to support their chips. They also create reference boards: designs that are used by the next level down to create development boards and working products.
• Board vendors and OEMs: These people take the reference designs from SoC vendors and build them into specific products like set-top boxes or cameras. They also create more general-purpose development boards such as those from Advantech and Kontron. An important category is the cheap Single-Board Computers (SBCs) such as BeagleBoard and Raspberry Pi, which have created their own ecosystems of software and hardware add-ons.
• Commercial Linux vendors: Companies such as Siemens, Timesys, and Wind River offer commercial Linux distributions that have undergone strict regulatory verification and validation across multiple industries (medical, automotive, aerospace, and so on).

These form a chain, with your project usually at the end, which means that you do not have a free choice of components. You cannot simply take the latest kernel from kernel.org, except in rare cases, because it does not have support for the chip or board that you are using.

This is an ongoing problem with embedded development. Ideally, the developers at each link in the chain would push their changes upstream but they don’t. Developers are under constant time pressure and getting patches accepted into the Linux kernel takes major effort. It is not uncommon to find a kernel that has many thousands of patches that are not merged. In addition, SoC vendors tend to actively develop open source components only for their latest chips, meaning that support for any chip more than a couple of years old will be frozen and not receive any updates.

The consequence is that most embedded designs are based on old versions of software. They do not receive security fixes, performance enhancements, or features that are in newer versions. Problems such as Heartbleed (a bug in the OpenSSL library) and Shellshock (a bug in the Bash shell) go unfixed.

What can you do about it? First, ask questions of your vendors (NXP, TI, and Xilinx to name just a few): what is their update policy, how often do they revise kernel versions, what is the current kernel version, what was the one before that, and what is their policy for merging changes upstream? Some vendors are making great strides in this direction. You should prefer their chips.

Secondly, you can take steps to make yourself more self-sufficient. The chapters in Part 1 explain the dependencies in more detail and show you where you can help yourself. Don’t just take the package offered to you by the SoC or board vendor and use it blindly without considering the alternatives.

This book is divided into five sections that reflect the phases of a project. The phases are not necessarily sequential. Usually, they overlap, and you will need to jump back to revisit things that were done previously. However, they are representative of a developer’s preoccupations as the project progresses:

• Elements of Embedded Linux (Chapters 1 to 5) will help you set up the development environment and create a working platform for the later phases. It is often referred to as the board bring-up phase.
• Building Embedded Linux Images (Chapters 6 to 8) shows you how to automate the process of building an embedded Linux image by leveraging a build system like Buildroot or The Yocto Project. Automating complex build tasks accelerates the project life cycle so that teams can deliver higher-quality products in less time.
• System Architecture and Design Choices (Chapters 9 to 14) will inform some of the design decisions you will have to make concerning the storage of programs and data, how to divide work between kernel device drivers and applications, and how to initialize the system.
• Developing Applications (Chapters 15 to 18) shows you how to package and deploy Python applications, make effective use of the Linux process and thread model, and manage memory in a resource-constrained device. What do packaging and deploying Python applications have to do with embedded Linux? The answer is “not much”, but bear in mind that the word “development” also happens to be in the title of this book. And Chapters 15 and 16 have everything to do with modern-day software development.
• Debugging and Optimizing Performance (Chapters 19 to 21) describes how to trace, profile, and debug your code in both the application and the kernel. The last chapter explains how to design for real-time behavior when required.

Now, let’s focus on the four basic elements of embedded Linux that comprise the first section of the book.

Every project begins by obtaining, customizing, and deploying these four elements: the toolchain, the bootloader, the kernel, and the root filesystem. This is the topic of the first section of this book.

• Toolchain: This is the cross compiler and other tools needed to create code for your target device. A cross compiler generates machine code for a target CPU architecture while running on a different host CPU architecture.
• Bootloader: This is a bare metal program that initializes the board and the Linux kernel. The term “bare metal” means the program runs directly on the CPU, not on top of an OS.
• Kernel: This is the heart of the system, managing system resources and interfacing with the hardware.
• Root filesystem: This contains the libraries and programs that are run once the kernel has completed its initialization.

There is also a fifth element not mentioned here. That is the collection of programs specific to your embedded application that make the device do whatever it is supposed to do, be it weighing groceries, displaying movies, controlling a robot, or flying a drone.

Typically, you will be offered some or all of these elements as a package when you buy your SoC or board. But for the reasons mentioned earlier, they may not be the best choices for you. In the first eight chapters, I will give you the background to make the right selection and introduce two tools that automate the whole process for you: Buildroot and The Yocto Project.

The components of embedded Linux are open source so now is a good time to consider what that means, why open source licenses work the way they do, and how this affects the often proprietary embedded device you will be creating from it.

Licenses

When talking about open source the word free is often used. People new to the subject often take it to mean nothing to pay and open source software licenses do indeed guarantee that you can use the software to develop and deploy systems for no charge. However, the more important meaning here is freedom since you are free to obtain the source code, modify it in any way you see fit, and redeploy it in other systems. Open source licenses give you this right, but some also require you to share these changes with the public.

Compare that with freeware licenses, which allow you to copy the binaries for no cost but do not give you the source code. Other licenses allow you to use the software for free under certain circumstances, for example, for personal use, but not commercial. These are not open source.

I will provide the following comments in the interest of helping you understand the implications of working with open source licenses, but I would like to point out that I am an engineer and not a lawyer. What follows is my understanding of the licenses and how they are interpreted.

Open source licenses fall broadly into two categories:

• Copyleft licenses such as the GNU General Public License (GPL)
• Permissive licenses such as the BSD and MIT licenses

The permissive licenses say, in essence, that you may modify the source code and use it in systems of your own choosing as long as you do not modify the terms of the license in any way. In other words, apart from that one restriction, you can do with it what you want, including building it into possibly proprietary systems.

The GPL licenses are similar but have clauses that compel you to pass the rights to obtain and modify the software on to your end users. In other words, you share your source code. One option is to make it completely public by putting it onto a public server. Another is to offer it only to your end users by means of a written offer to provide the code when requested.

The GPL goes further to say that you cannot incorporate GPL code into proprietary programs. Any attempt to do so would make the GPL apply to the whole. In other words, you cannot combine GPL and proprietary code in the same program. Aside from the Linux kernel, the GNU Compiler Collection and GNU Debugger, as well as many other freely available tools associated with the GNU project, fall under the umbrella of the GPL.

So, what about libraries? If they are licensed with the GPL, any program linked with them becomes GPL also. However, most libraries are licensed under the GNU Lesser General Public License (LGPL). If this is the case, you are allowed to link with them from a proprietary program.

The GPL v3 and LGPL v3 have their problems though. There are security issues. If the owner of a device has access to the system code, then so might an unwelcome intruder. Often the defense is to have kernel images signed by an authority such as the vendor so that unauthorized updates are not possible. Is that an infringement of my right to modify my device? Opinions differ.

If you are designing or selecting hardware for an embedded Linux project, what do you look out for?

• First, a CPU architecture that is supported by the kernel – unless you plan to add a new architecture yourself of course! Looking at the source code for Linux 5.15 there are 23 architectures each represented by a subdirectory in the arch/ directory. They are all 32-or 64-bit architectures, most with an MMU, but some without. The ones most often found in embedded devices are Arm, RISC-V, PowerPC, MIPS, and x86 each in 32-and 64-bit variants all of which have Memory Management Units (MMUs).
• Most of this book is written with this class of processor in mind. There is another group that doesn’t have an MMU and that runs a subset of Linux known as a microcontroller Linux or uClinux. These processor architectures include ARC (Argonaut RISC Core), Blackfin, MicroBlaze, and Nios. I will mention uClinux from time to time, but I will not go into detail because it is a rather specialized type.
• Second, you will need a reasonable amount of RAM. 16 MB is a good minimum, although it is quite possible to run Linux using half of that. It is even possible to run Linux with 4 MB if you are prepared to go to the trouble of optimizing every part of the system. It may even be possible to get lower, but there comes a point at which it is no longer Linux.
• Third, there is non-volatile storage, usually flash memory. 8 MB is enough for a simple device such as a webcam or basic router. As with RAM, you can create a workable Linux system with less storage if you really want to, but the lower you go the harder it becomes. Linux has extensive support for flash storage devices, including raw NOR and NAND flash chips, and managed flash in the form of SD cards, eMMC chips, USB flash memory, and so on.
• Fourth, a serial port is very useful, preferably a UART-based serial port. It does not have to be fitted on production boards but makes board bring-up, debugging, and development much easier.
• Fifth, you need some means of loading software when starting from scratch. Many microcontroller boards are fitted with a Joint Test Action Group (JTAG) interface for this purpose. Modern SoCs can also load boot code directly from the removable media, especially SD and microSD cards, or serial interfaces such as QSPI or USB.

In addition to these basics, there are interfaces to the specific bits of hardware your device needs to get its job done. Mainline Linux comes with open source drivers for many thousands of different devices, and there are drivers available (of variable quality) from the SoC manufacturer and from the OEMs of third-party chips that may be included in the design.

Remember my comments on the commitment and ability of some manufacturers. As a developer of embedded systems, you will find that you spend quite a lot of time evaluating and adapting third-party code, if you have it, or liaising with the manufacturer if you don’t. Finally, you will have to write the device support for the interfaces that are unique to the device or find someone to do it for you.

The examples in this book are intended to be generic. To make them relevant and easy to follow I have had to choose specific hardware. I have chosen three exemplary devices: the Raspberry Pi 4, BeaglePlay, and QEMU. The first is by far the most popular Arm-based SBC on the market. The second is a widely available SBC that can also be used in serious embedded hardware. The third is a machine emulator that can be used to create a range of systems that are typical of embedded hardware.

It was tempting to use QEMU exclusively, but like all emulations, it is not quite the same as the real thing. Using the Raspberry Pi 4 and BeaglePlay, you have the satisfaction of interacting with real hardware and seeing real LEDs flash. The BeaglePlay, like the BeagleBone Black before it, is open source hardware, unlike the Raspberry Pi 4. This means that the board design materials are freely available for anyone to build the BeaglePlay or a derivative into their products.

In any case, I encourage you to try out as many of the examples as you can, using either of these three platforms or any embedded hardware you may have on hand.

The Raspberry Pi 4

From June 2019 until October 2023, the Raspberry Pi 4 Model B was the flagship SBC produced by the Raspberry Pi Foundation. The Raspberry Pi 4’s technical specs include the following:

• A Broadcom BCM2711 1.5 GHz quad-core Cortex-A72 (Arm v8) 64-bit SoC
• 2, 4, or 8 GB DDR4 RAM
• 2.4 GHz and 5 GHz 802.11ac wireless, Bluetooth 5.0, BLE
• A serial port for debugging and development
• A microSD slot, which can be used as a boot device
• A USB-C connector to power the board
• Two full-size USB 3.0 and two full-size USB 2.0 host ports
• A Gigabit Ethernet port
• Two micro HDMI ports for video and audio output

In addition, there is a 40-pin expansion header for which there are a great variety of daughter boards known as Hardware Attached on Top (HATs) that allow you to adapt the board to do many different things. However, you will not need any HATs for the examples in this book.

In addition to the board itself you will require the following:

• A microSD card and a means of writing to it from your development PC or laptop
• A USB-to-TTL serial cable with a 3.3 V logic level
• A 5 V USB-C power supply capable of delivering 3 A
• An Ethernet cable and a router to plug it into as some of the examples require network connectivity

The BeaglePlay

The BeaglePlay is an open source hardware design for an SBC produced by the BeagleBoard.org Foundation. The main points of the specification are:

• A TI AM6254 1.4 GHz Arm quad-core Cortex-A53 (Arm v8) 64-bit Sitara SoC
• 2 GB DDR4 RAM
• 16 GB eMMC on-board flash
• 2.4 GHz and 5 GHz MIMO Wi-Fi, BLE, Zigbee
• A serial port for debugging and development
• A microSD slot, which can be used as a boot device
• A USB-C connector to power the board
• A full-size USB 2.0 host port
• A Gigabit Ethernet port
• A full-size HDMI port for video and audio output

Instead of a large expansion header, the BeaglePlay has mikroBUS, Grove, and Qwiic interfaces for connecting add-on boards.

In addition to the board itself, you will require the following:

• A microSD card and a means of writing to it from your development PC or laptop
• A USB-to-TTL serial cable with a 3.3 V logic level
• A 5 V USB-C power supply capable of delivering 3 A
• An Ethernet cable and a router to plug it into as some of the examples require network connectivity

In addition to the above, Chapter 12 also requires the following:

• A MikroE-5764 GNSS 7 Click add-on board
• An external active GNSS antenna with an SMA connector
• A MikroE-5546 Environment Click add-on board
• A MikroE-5545 OLED C Click add-on board

QEMU

QEMU is a machine emulator. It comes in different flavors, each of which can emulate a processor architecture and various boards built using that architecture. For example, we have the following:

• qemu-system-arm: 32-bit Arm
• qemu-system-aarch64: 64-bit Arm
• qemu-system-mips: MIPS
• qemu-system-ppc: Power PC
• qemu-system-x86: x86 and x86-64

For each architecture, QEMU emulates a range of hardware that you can see by using the -machine help option. Each architecture emulates most of the hardware that would normally be found on that board. There are options to link hardware to local resources, such as using a local file for the emulated disk drive. Here is a concrete example:

$ qemu-system-arm -machine vexpress-a9 -m 256M -drive file=rootfs.ext4,sd -net nic -net use -kernel zImage -dtb vexpress-v2p-ca9.dtb -append "console=ttyAMA0,115200 root=/dev/mmcblk0" -serial stdio -net nic,model=lan9118 -net tap,ifname=tap0

The options used in the preceding command line are as follows:

• -machine vexpress -a9: Creates an emulation of an Arm Versatile Express development board with a Cortex-A9 processor.
• -m 256M: Populates it with 256 MB of RAM.
• -drive file=rootfs.ext4,sd: Connects the SD interface to the local rootfs.ext4 file, which contains a filesystem image.
• -kernel zImage: Loads the Linux kernel from the local file named zImage.
• -dtb vexpress-v2p-ca9.dtb: Loads the device tree from the local vexpress-v2p-ca9.dtb file.
• -append "…": Appends the string in quotes as the kernel command line.
• -serial stdio: Connects the serial port to the terminal that launched QEMU so that you can log on to the emulated machine via the serial console.
• -net nic,model=lan9118: Creates a network interface.
• -net tap,ifname=tap0: Connects the network interface to the virtual network interface tap0.

To configure the host side of the network you need the tunctl command from the User Mode Linux (UML) project. On Debian and Ubuntu, the package is named uml-utilities:

$ sudo tunctl -u $(whoami) -t tap0

This creates a network interface named tap0 that is connected to the network controller in the emulated QEMU machine. You configure tap0 the same way as any other network interface.

All these options are described in the following chapters. I will be using Versatile Express for most of my examples, but it should be easy to use a different machine or architecture.

I have only used open source software for both the development tools and the target OS/applications. I assume you will be using Linux on your development system.

I tested all the host commands using Ubuntu 24.04 LTS, so I recommend using that version throughout the book to prevent any unexpected problems.

Besides Ubuntu, The Yocto Project only supports a select few Linux distributions: Fedora, Debian, openSUSE, AlmaLinux, and Rocky. If you absolutely cannot use Ubuntu, then make sure to choose one of those supported distros for The Yocto Project exercises.

Embedded hardware continues to get more complex following the trajectory set by Moore’s law. Linux has the power and flexibility to make use of hardware in an efficient way. Together, we will learn how to harness that power so we can build robust products that delight our users. This book will take you through the five phases of an embedded project’s life cycle, beginning with the four elements of embedded Linux.

The sheer variety of embedded platforms and the fast pace of development lead to isolated pools of software. In many cases, you will become dependent on this software, especially the Linux kernel that is provided by your SoC or board vendor, and to a lesser extent, the toolchain.

Some SoC manufacturers are getting better at pushing their changes upstream and the maintenance of these changes is getting easier. Despite these improvements, selecting the right hardware for your embedded Linux project is still an exercise fraught with peril. Open source license compliance is another topic you need to be aware of when building products atop the embedded Linux ecosystem.

In this chapter, you were introduced to the hardware and some of the software you will use throughout this book (namely QEMU). Later on, we will examine some powerful tools that can help you create and maintain the software for your device. We cover Buildroot and dig deep into The Yocto Project. Before we tackle these build tools, we will deconstruct the four elements of embedded Linux, which you can apply to all embedded Linux projects regardless of how they are built.

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers: https://packt.link/embeddedsystems