About Jorge!

● Ops background from an .edu

● 20+ years as an OSS Community Manager
○ Canonical
○ VMware/Heptio
● OSS Projects
○ Ubuntu
○ Kubernetes
○ Kubeflow
○ Cloud Custodian
● Currently at the CNCF working for
○ Our 184(!) projects
○ And thousands of OSS contributors
About Colin

● Emacs startup screen contributed to me

being here
● Debian, GNOME, Red Hat, OpenShift, Fedora,
ostree, bootc
● Somehow ~20+ years later working on the
operating system in the FOSS space is still
fun!
One unified model for Linux
The desktop/client as an example use case
Universal Blue: Year One

● KubeCon | CloudNativeCon NA 2022

● Passion Project
○ Jorge, Marco, Adam, and Wayne
○ I already had a “fix it script”
● Initial prototype finished in a weekend
○ Podman in a for loop!
● “It looks like a community manager wrote this.”
Universal Blue: Year Two

● The Nvidia moment

○ Thanks Josh Stone (Red Hat) and Alex Diaz (Independent)
● Splitting into base layers, more contributors
● Focused on growth and covering lots of use cases
● SREs/sysadmins rewrite things
● Hyper focus on automation, “punch above our weight”
● Finding our target audience
Universal Blue: Going GA (General Availability)

● Redirecting users away from base images and custom tooling

● Removing low-use images
● Focusing on infrastructure and contributor sustainability
● Automation and explicit removal of tech debt
● Focus on end user experiences instead of “a slightly better Fedora”
Users don’t want images they
want experiences
They will insist that they want images
Users don’t want images they want experiences
Thanks opensauced.pizza
Universal Blue: The Basics

FROM scratch AS config

- Service units, udev rules, shared justfiles

FROM fedora AS main

- Ingest multiple GUI desktops + QoL

FROM main AS bluefin

- Strongly opinionated image for general use cases

FROM bluefin AS bluefin-dx

- “The perfect UNIX workstation”

Universal Blue: The GitHub Actions

Thanks @bsherman, @KyleGospo, @p5, @m2, and others …

Universal Blue: Applying application container patterns to OS images
Universal Blue: The ISO Generator Action

Thanks Noel Miller (Red Hat) and Jason Nagin (Red Hat)
Speaking of experiences …
Building off of base images let’s us focus
Pre-existing relationship with Fedora

We polish the minutia

The reliability of a Chromebook with the

power of Fedora

A viable alternative to MacOS for

systems engineers
Podman on
the Metal
Prototype faster: Include drivers on your custom image
Prototype faster: ollama integration
Prototype Faster: Ptyxis Terminal and Podman

Thanks Luca di Maio (Independent) and Christian Hergert (Red Hat)

Prototype Faster: The “Developer Experience” Pattern

● Built in k8s, QEMU/KVM

● VSCode with devcontainers
● Tailscale
● Homebrew
● Docker
● Virt-manager
● Great monospace fonts
● Quick-fire setup for jetbrains,
pytorch, etc.
Prototype Faster: Swapping out kernels
Future directions: Alignment with dnf

RUN if [[ KERNEL_FLAVOR = “fsync” ]];

then dnf -y install kernel-fsync && dnf
clean all; fi
Stay Safe: Live mitigation of upstream regressions
Strongest Patterns

● One common language between dev and ops

● Sharing base images leads to efficiency
● Consistent image to the end user
● Hardware Enablement
○ ASUS, Framework, Lenovo, and Valve hardware
○ We ship newer kernels to the Steam Deck than Valve
● Gating regressions at the image level, not needing
to do OS/distro work
Purposeful Simplicity

● Admins can go so far with bash and a little Python

● Explicit non-scope mission statements
● Encourage more innovation outside of Universal Blue
● “Not a distro” forces people to go fix deeper technical issues in
Fedora itself
● “We are not Bitnami”
Silly UNIX tricks

● Shared community aliases via `just`

● Dynamic MOTD
● ujust update
● ujust toggle-nvk
● ujust setup-cockpit
● ujust changelogs
● ujust device-info
● Non-trivial amount of effort in terminal
artwork. 😃
Challenges: The Linux Desktop’s Docker Moment (powered by Podman)

● Runtime --> build time customization

● Cloud Native tooling v. distribution tooling
● Container-focused development continues
to challenge new users
○ Podman Desktop
○ Devcontainers
● Sigstore and SBOMs
Things we’d love to see in container native Fedora

● Container diffs
○ The whole industry needs this
● More consumption of cloud native tooling
● Better automated testing
● Gating kernels via CoreOS cadence
● Command Line experience in general
● Management of /etc
Benefits for experts AND new explorers
Safety switches lead to positive
interactions between users and OSS
Contributors in our issue trackers.

They deserve that.

Intermission
 There’s lots of Linux out there

● Gitlab CI, GitHub Action

● Servers: virthost-bos01…08.examplecorp.com
● Cloud
● Desktops
● Edge/IoT
Image-based/Immutable vs General purpose

● (Title is a false dichotomy but we’ll get to that later)

● Brief history:
○ …
○ RHEV Image based hypervisor ISO (rhel6, ~2011)
○ …
○ Docker invented
○ …
○ (CoreOS) Container Linux
○ RHEL Atomic Host (rhel7)
○ RHEL CoreOS (rhel8+9)
○ Lots more around: Flatcar, Ubuntu Core, etc.
 Why has OSTree stayed around?

● Fancy wrapper around link(), plus bootloaders: But it works

● Mutable /etc 3 way merge is not perfect, but better than alternatives
● Filesystem based adds a ton of flexibility
● Network-efficient static deltas
Lots of comp/co-opetition in the ecosystem

● Ubuntu Core ● Google COOS

● SUSE ALP ● NixOS (and Guix)
● Balena ● Amazon Bottlerocket
● Fedora CoreOS/RHCOS
● Systemd: DDIs
● Also Fedora Atomic Desktops + ublue
● Kairos
● Talos
● Flatcar
● Docker LinuxKit
● Yocto generating images
● Lots more
 virthost-bos01…08 needs good tools

● https://grahamc.com/blog/erase-your-darlings/ “There are lots of cases in

which immutable infrastructure doesn’t work, and the dirty secret is those
servers need good tools the most.”
● Atomic Host and kernel-debug; throwaway or not
 Reality is: there’s a spectrum

● Ephemeral ● Long lived

● Reprovisionable (cattle) My workstation ● Pet

Split OS vs data, keep

hostname + static IPs
 CoreOS/Atomic and rpm-ostree

● ostree is a teenager: AuthorDate: Sun Oct 9 17:03:08 2011 -0400 (predates

docker, etc.); goal was “image based” by default w/transactional updates
● Yes, you can “rpm-ostree override replace kernel-debug.rpm” on
virthost01, but you can also confidently reset back to the “golden image”;
having your cake and eating it too
● High level config (CRDs, Kube driven) vs Ignition vs machine-specific state
● …vs deeper OS customization and agents
 rpm-ostree learned containers!

● Let’s talk about that deep customization

● FROM quay.io/fedora/fedora-coreos:stable or FROM
quay.io/fedora/fedora-silverblue:40
● OCP CoreOS Layering
● Enabled Universal Blue
● But… but…
 Changing engines is hard

● Gasoline vs electric ⇔ package vs ostree/container

● The name rpm-ostree is very literal and no longer makes sense
● Creation of http://github.com/containers/bootc that is all-in on containers
but allows seamless switch from prior ostree-oriented system
● Thinking hard about a future where we have dnf + bootc, less rpm-ostree
 Architecture diagram
Current architecture
More realistic architecture diagram
Current architecture
Demo: local builds w/bootc
 Next: Containers and Fedora

● Goal: Containers are an equal center of gravity for Fedora derivatives

● But: rpm packages continue to exist; also continue to adopt good ideas
from other communities/projects
● Also Kubernetes vs standalone continues to exist.
 Next: Deeper binding bootc+podman

● Deltas!
○ Req: https://github.com/containers/bootc/pull/215
● bootc+podman “lifecycle bound” images
● “System extensions” continuation (systemd-sysext binding/frontend)
● Host-toolbox idea
 More realistic architecture diagram
How it could look
Current architecture
 Building an ecosystem

● CoreOS is super opinionated: Thou shalt use Ignition

● https://github.com/coreos/layering-examples/tree/main/ansible-firewalld
● What if we built an Ansible Galaxy like thing for this?
● What if it was just normal to boot Fedora Linux and derivatives as a
container?
 Next: Time to do a fedora+centos-bootc?

● bootc has been a side project until recently, we are working to start to build
out a fedora-bootc in Fedora+derivatives, look for that soon!
Sustainability in Open Source