Contents

Summary 3

Introduction 4

Kernel Archeology 5

Impact of Development Process Best Practices 7

Adoption of Maintainer Hierarchy 8

Version Control Systems 9

Removing Unused Code 12

Highly Diversified Corporate Contributors 13

Tagging and adoption of the Developer Certificate of Origin 15

Release Model with Predictable Release Cycle Cadence 17

Improving Automated Testing the Kernel 18

Stable Release Process 19

Longterm Release Kernels 20

Conclusion 21

References 23
 Impact of Development Process
Best Practices
In 2015, The Linux Foundation’s Core Infrastructure The kernel maintainers meet in person every year
Initiative (CII) program introduced a best practices (except this one) at the Maintainer Summit to discuss
badge for open source projects to publicly determine how to improve their ways of working together. Topics
their software development and security practices10. for the summit are determined by developers raising
The Linux kernel was one of the first projects to get a them on the ksummit-discuss mailing list13. After
badge. As time has evolved, additional practices have a healthy discussion, those that aren’t resolved are
been identified, and these were incorporated into selected to be discussed in person. These meetings are
higher badge levels for projects to strive for11. In June vital to the continuous improvement of the processes
of this year, the Linux kernel joined the small handful that the community follows. Over time there has been
of projects with a gold badge, the top badge level12. a steady growth in the number of contributors and
This is a visible recognition of what’s been there for a commits to the kernel each year. While it is not easy
long time now; the Linux kernel community continues to quantify the impact of these process improvement
to lead on establishing best practices in the areas of meetings, they are likely a positive factor.
software engineering and secure development at scale.

Contributors to the Linux Kernel per Year Commits to the Linux Kernel per Year

2020 Linux Kernel History Report The Linux Foundation 7

 Adoption of Maintainer Hierarchy
One of the best practices that have helped the kernel mailing lists and USENET groups. Before the linux-
scale over time is the adoption of a maintainer hierarchy. kernel mailing list was hosted on vger.kernel.org,
If we look back through the kernel release archives, the it was at vger.rutgers.edu, which was only one of a
first instance of the MAINTAINERS file was committed number of the lists where the discussions occurred,
as part of v1.3.68 in January 199614. The file was only with significant discussions still on USENET. One archive
107 lines long, and there were three maintainers listed, source is http://lkml.iu.edu/hypermail/linux/
Alan Cox, Jon Naylor, and Linus Torvalds. That version kernel/, containing linux-kernel mailing list (“LKML”)
of the MAINTAINERS file ends with: archives going back to 1995, but even it contains some
key gaps. If anyone reading this has access to early
REST: Linux discussion threads, please reach out to the lore.
P: Linus Torvalds kernel.org administrators15 so we can add more early
S: Buried alive in email development discussions to the archives16.

In trying to understand the discussions that led to In contrast, today’s v5.8 MAINTAINERS file is now 19,033
the formation of this file, early Linux development lines long and has 150117 maintainers listed18. The v5.8
mailing lists were consulted. Unfortunately, only partial MAINTAINERS file ends with:
records of the discussions are publicly available before
1997, as Linux development took place across multiple THE REST
M: Linus Torvalds <torvalds@linux-foundation.org>
L: linux-kernel@vger.kernel.org
S: Buried alive in reporters
Q: http://patchwork.kernel.org/project/LKML/list/
T: git git://git.kernel.org/pub/scm/linux/
kernel/git/torvalds/linux.git
F: *
F: */

As the number of maintainers has increased, the

number of email messages on LMKL has increased as
well, but it is no longer mentioned as a problem by
Linus, at least.

Emails on LKML per Year

2020 Linux Kernel History Report The Linux Foundation 8

 The Linux Kernel community members continue to look
for ways to improve the diversity of contributors, and
it has been a topic at past Kernel Summits24. Kernel
developers volunteer their own time to mentor new
developers participating in programs like Outreachy25,
and the Linux Foundation’s Kernel Mentorship(LKMP)
program26. LKMP is part of the mentorship program
on CommunityBridge27 which has been helping
developers new to open source to experiment, learn,
and contribute to open source communities. It has
ultimately helped them connect with experts in open
source communities as well as employers. The program
is designed to build a healthy ecosystem around the
open source software we all care about and rely on by
funding projects, securing code, and connecting with
talented developers from diverse backgrounds.

We at the Linux Foundation have also been reaching

Estimated Female Contributors to the Linux Kernel per Year out to new women contributors asking for suggestions
for improving the materials and guidance available to
new contributors. A few takeaways from the feedback:

• Online resources are inadequate or outdated and

require updates.

• More FAQs on subsystems patch submission

guidelines will be helpful.

• More FAQs or blogs on best practices for

contributing to Linux Kernel will be helpful.

• Communication about patch acceptance status can

be improved.

• More beginner-friendly courses and videos such as

Shuah Khan’s LFD10328 and Greg Kroah-Hartman’s
“Write and Submit your first Linux kernel Patch”29
are helpful for students with limited resources.

% of Contributors to Linux Kernel that are Estimated to be Female

2020 Linux Kernel History Report The Linux Foundation 11

 Release Model with Predictable Release Cycle Cadence
The Linux Kernel Release Model has evolved, and The length of the release cycle cadence has undergone
releases are considered to be classified into one of four much discussion and experimentation by the community.
categories: Prepatch (or “-rc”) kernels, Mainline, Stable, Still, since 2011 the community seems to have found
and Long Term Stable38. Details and links to the active a release model that works and supports the quality
Kernel releases can be found on https://www.kernel.org/. they seek. In fact, the release cadence has become so
predictable, that a “Pointy-Haired-Boss tool” 39, was
created so developers could quickly answer their corporate
managers’ question of “when will it be released?”

Each release cycle now starts with a two week “merge

window” when new functionality can be included with the
appropriate review into the git repository for the upcoming
release. Once it is tagged rc1, then the cycles of integration
testing, debugging, and stabilizing commence. A weekly
rc candidate is then tagged until the target quality and
stability is reached. The release is made, and the cycle
commences again with the next merge window.

More details on the kernel release model and how it

occurred can be found in Greg Kroah-Hartmann’s blog on
the history of how the community came up with this model40.

Left: source: https://www.kernel.org/ on 2020-08-20

Linux Releases per Year

2020 Linux Kernel History Report The Linux Foundation 17

 Longterm Release Kernels
The introduction of Longterm release kernels, based Sasha Levin on creating a tool using machine-learning
on stable releases, has further increased Linux’s techniques to identify patches that look like useful
popularity with product makers and increased fixes59, have been significantly improving in recent
adoption in new market segments. Greg Kroah- years, and are now finding more bug fixes that can be
Hartmann wrote an excellent blog on which kernel he’d applied to the longterm release kernels.
choose for a task in 201857, and it provides a useful
overview for developers looking to select a kernel for In 2019, 18,668 commits were made to the various
their project today. Details of the supported long term LTS kernels; that’s over 2.15 commits per hour. More
stable releases and period until the end of the support changes were made to the LTS releases in 2019 than
(Projected EOL) can be found at kernel.org. to the 2.6.12 kernel, 15 years ago.

Longterm release kernels start off as a stable kernel There is market pressure for even longer support
with a commitment to maintaining for an extended windows to support some of the applications
period. Distributions like SUSE, Ubuntu, Red Hat, etc. where Linux is being used. For example, the Civil
helped to pioneer this concept, and it was formalized in Infrastructure Platform (CIP) project60 members have a
201058 with the formalization of “longterm” kernels and common interest in having an infrastructure that needs
creation of the stable@kernel.org mailing list. to be maintained for extended periods. The developers
participating in this project have decided to support
Bugs found in the current linux stable kernel are fixed 4.4 and 4.19 as Super Long-term Stable (SLTS) kernel
upstream and then applied to the stable kernel. When release61, 62. It will be interesting to see what tools and
a fix is determined to be applicable to one of the processes these developers create to help make this
longterm release kernels, it is backported and applied. vision possible in the years to come.

Improvements in tooling to detect when a fix can

apply to a longterm stable kernel, like the work by

Version Maintainer Released Projected EOL

5.4 Greg Kroah-Hartman & Sasha Levin 2019-11-24 December 2025
4.19 Greg Kroah-Hartman & Sasha Levin 2018-10-22 December 2024
4.14 Greg Kroah-Hartman & Sasha Levin 2017-11-12 January 2024
4.9 Greg Kroah-Hartman & Sasha Levin 2016-12-11 January 2023
4.4 Greg Kroah-Hartman & Sasha Levin 2016-01-10 February 2022

Source: https://www.kernel.org/category/releases.html Tools Detect More Fixes for Longterm Linux Kernels

2020 Linux Kernel History Report The Linux Foundation 20

 References
1 19
https://lore.kernel.org/lkml/CAHk-=wj+mDPbj8hXspXRAksh+1TmPj https://www.linuxfoundation.org/blog/2015/04/10-years-of-git-
ubc9RNEbu8EVpYyypX=w@mail.gmail.com/ an-interview-with-git-creator-linus-torvalds/
2
https://lwn.net/Articles/827735/ 20
https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/
3
https://www.linuxfoundation.org/publications/2008/04/linux- 21
https://repo.or.cz/davej-history.git
kernel-development-report-2008/
22
https://git.kernel.org/pub/scm/linux/kernel/git/history/history.
4
https://cregit.linuxsources.org/ git//?h=1.0&id=13f97bf0ff18e367f94a5ebcf6e89998ecedb80f
5
https://github.com/cregit/cregit 23
https://github.com/tue-mdse/genderComputer
6
https://github.com/dmgerman/linux-pre-history/ 24
https://lwn.net/Articles/662911/
commit/9417d4148f0ddc5ee2cc1114ce97c71c5e4cb4b7
25
https://www.outreachy.org/
7
https://archive.org/details/git-history-of-linux
26
https://wiki.linuxfoundation.org/lkmp
8
https://repo.or.cz/davej-history.git/commit/
bb441db1a90a1801ef4e6546417a8d907c55d92f 27
https://communitybridge.org/
9
Architecture counts taken from boot test coverage counts from 28
https://training.linuxfoundation.org/training/a-beginners-guide-
https://github.com/groeck/linux-build-test/tree/master/rootfs to-linux-kernel-development-lfd103/
10
https://www.coreinfrastructure.org/announcements/linux- 29
https://www.youtube.com/watch?v=LLBrBBImJt4
foundations-core-infrastructure-initiative-seeks-community-
30
input-on-new-security-focused-badge-program/ http://www.kroah.com/log/blog/2018/02/05/linux-kernel-release-
model/
11
https://www.linuxfoundation.org/blog/2020/06/why-cii-best-
31
practices-gold-badges-are-important/ https://lwn.net/Articles/756031/
12 32
https://www.linuxfoundation.org/blog/2020/06/linux-kernel- https://github.com/AlDanial/cloc
earns-cii-best-practices-gold-badge/
33
German, D.M., Adams, B. & Stewart, K. cregit: Token-level blame
13
https://lore.kernel.org/ksummit-discuss/ information in git version control repositories. Empir Software Eng
24, 2725–2763 (2019). https://doi.org/10.1007/s10664-019-09704-x
14
https://repo.or.cz/davej-history.git/
34
commit/8bf26ec84e6362618e1abe641ac7f026c2674372 git://git.lwn.net/gitdm.git
15 35
https://www.kernel.org/lore.html https://lwn.net/Articles/222773/
16 36
https://korg.docs.kernel.org/lore.html https://developercertificate.org/
17 37
grep “M:” MAINTAINERS | sort | uniq | wc -l https://www.wired.com/2004/05/linux-whose-kernel-is-it/
18 38
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ https://www.kernel.org/category/releases.html
tree/MAINTAINERS?h=v5.8.2

2020 Linux Kernel History Report The Linux Foundation 23

 39 55
http://phb-crystal-ball.org/ https://lkft.linaro.org/tests/
40 56
http://www.kroah.com/log/blog/2018/02/05/linux-kernel-release- https://kernelci.org/
model/
57
http://kroah.com/log/blog/2018/08/24/what-stable-kernel-
41
https://sparse.docs.kernel.org/ should-i-use/
42
http://smatch.sourceforge.net/ 58
https://lwn.net/Articles/418580/
43
https://bottest.wiki.kernel.org/coccicheck 59
https://lwn.net/Articles/789225/
44
https://bottest.wiki.kernel.org/ 60
https://www.cip-project.org/
45
https://01.org/lkp 61
https://www.cip-project.org/blog/2020/08/17/cip-kernel-team-
46
helping-cip-sustain-industrial-grade-systems
mailto:hulkci@huawei.com
62
47
https://lwn.net/Articles/749530/
https://github.com/kernelslacker/trinity
63
48
https://www.linuxfoundation.org/blog/2020/06/linux-kernel-
https://lwn.net/Articles/536173/ earns-cii-best-practices-gold-badge/
49
https://lwn.net/Articles/804119/ 64
https://www.techdesignforums.com/practice/technique/medical-
50
linux-security/
https://lore.kernel.org/stable/
65
51
https://www.automotivelinux.org/about/
https://lore.kernel.org/lkml/20200714184115.844176932@
linuxfoundation.org/ 66
https://www.zdnet.com/article/from-earth-to-orbit-with-linux-
52
and-spacex/
https://kerneltests.org
67
53
https://elisa.tech/
https://kerneltests.org/one_line_per_build
54
https://lkft.linaro.org/

2020 Linux Kernel History Report The Linux Foundation 24