I wanted to share the CI/CD pipeline I created in my homelab for building my own personal derivative of Fedora Kinoite with Nvidia drivers and a few package swaps. While I do this on my homelab, it can be done entirely on a single system.

The reason for making this was in part because I simply thought it was cool, but also in part because I did not and do not like the ways the Universal Blue project(s) deviate from upstream Fedora. As an example, while I understand why things like Bazaar exist and are used, I don't like that it's implemented *in lieu of* Discover, regardless of how much better than Discover it may be.

As I have no intentions of uploading anything to Github and operate out of my own private Forgejo server in my homelab, I'll be using Pastebin to share things with you today. If there's a better alternative, I'm all for hearing about them.

The Pipeline

On my homelab I have my desktop and a server running Forgejo. Using a Forgejo Runner, my server automatically runs build jobs nightly to produce new containers provided the following is true:

1. Upstream updated their image since the last build.

2. No errors occurred at build-time.

To make sure the runner can push the resultant OCI image to Forgejo itself you have to create an auth token for the runner to use to push the OCI image.

sudo mkdir -p /root/.config/containers/
sudo podman login --authfile /root/.config/containers/auth.json your.forgejo.local

With that set up, Podman can push to your container registry without you having to authenticate every time. Here's what my build.yml ends up looking like.

The Containerfile

My objective when I started this project for myself was to remove the need to layer packages to get what I wanted out of my atomic installation. More specifically, I wanted to:

1. Add the Nvidia drivers and support services to the image.

2. Swap out toolbox for distrobox which I find to be a better and more flexible utility for the kinds of work I do.

3. Swap out firefox entirely for zen, necessitating a one-shot systemd unit (which we'll get to)

4. Add a bunch of other utilities Fedora doesn't ship by default, such as utilities for my Yubikey, Nextcloud, fonts and codecs, gaming stuff, and the like.

To achieve this I had to break the whole build down into three distinct steps:

- Extract the kernel version of the latest upstream image to build the Nvidia drivers against

- Build the Nvidia drivers against the above kernel version specifically, pulling from Koji directly in case the latest kernel version packages weren't available on Fusion yet

- Assemble the entire thingJohoBlue

The Containerfile could stand to be a lot more optimized than it is, but for the sake of simplicity I separated out steps logically so I could follow along myself during development. In short, the first stage is designed to gather kernel version information which we'll need to properly and reliably build modules against. To make sure we have access to the latest kernel packages at all times, we pull directly from Koji instead of Fusion repos which can sometimes lag behind a little bit (relative to Koji, obviously). We then build the kernel modules against the specific kernel version being shipped in the latest upstream image, and inject them in the third stage, adding in any other software we want at that point.

With this staged approach we can catch any mismatches between what kernel version the Nvidia drivers need to build against, which will simply cause the build to fail and no updated image to be pushed to our registry until the needed package versions align again.

Building Against Fedora Atomic Beta

Taking this a step further, I branched my private repo to add support for Fedora Kinoite 44. I had to add a stage to the build process to support this - I was running into weird TLS errors while trying to build the kernel modules for the Nvidia drivers that does not occur with Fedora 43 images, so I had to work around that in it's own stage. Again, this could be much better optimized, but for the sake of being able to logically read through it easily I separated out several RUN commands for myself.

To make sure both the latest and beta containers built nightly, I modified the build.yml file in the main branch to pull both branches and build against them individually.

The systemd Flatpak One-Shot Service

To support the installation of Flatpaks on an initial installation I added a one-shot systemd unit to automate the installation and then create a stub file to detect if the unit had ever run at any point on the system (which should be false on a new installation).

---

If you're dabbling in Fedora Atomic and want to try building your own images, it's a pretty fun experience seeing it all come together. My laptop and desktop both are always in identical system states and I don't have to worry about updates breaking anything as breakage is mitigated against during build-time. I have my own variants of Fedora IoT as well incorporating and enabling Cockpit without layering and adding FIDO2 support for Yubikey LUKS unlocks. The sky is the limit.

My next project will be learning to assemble the base images completely locally from scratch. To do this I'll need to mirror the upstream packages called for in the build recipies. Once I've successfully done that, my plan is to build those RPMs from source with compiler optimizations, adding a little Gentoo to my Fedora blood, all in the name of some fun and exploration.