u/IcyConversation7945

Hey fellow homelabers!

As mentioned in the title, I finally got OpenStack deployed on my three-node cluster ! 🎉🎉🎉

OpenStack is not as widely known in the homelab world as Proxmox, so here is a quick overview: OpenStack is an open-source cloud platform that lets you build something close to an AWS-style private cloud on your own hardware. Instead of managing individual VMs directly like you usually would in Proxmox, OpenStack gives you a full cloud control plane: compute scheduling with Nova, networking with Neutron, images with Glance, block storage with Cinder, identity management with Keystone, and so on.

In practice, you upload images, define flavors, create networks, assign floating IPs, and let OpenStack decide where the instances should run across the cluster. It is much more complex than a traditional homelab hypervisor, but it also exposes you to the same kind of concepts used in real cloud environments: tenant isolation, virtual networking, distributed storage, APIs, quotas, availability zones, and automated provisioning.

I deployed it using Kolla Ansible, which makes the whole process much more approachable by running the OpenStack services as Docker containers. It still requires a proper understanding of networking, storage, and service configuration, but compared to deploying all the components manually, it removes a lot of the pain and makes the setup much easier to reproduce.

It is definitely a bit overkill for a home setup, but that is also part of the point. I wanted to get a deeper understanding of how real cloud infrastructures are built and operated. Another interesting aspect is that OpenStack can integrate with Kubernetes, allowing the Kubernetes scheduler to provision VMs when horizontal scaling is needed.

When used through Terraform you basically get all of the customization options you would get from a cloud provider: multi tenant networking, floating ips, instance and storage customization and more.

For reference, here is the original post where I shared the full homelab build.

So far, everything has been working really well. The main issue I had was related to CPU C-states: I had to disable them for now because the nodes sometimes had trouble waking up properly from deep idle states.

For networking, the nodes are connected using Mellanox ConnectX-4 Lx SFP28 cards, each cooled with a custom 3D-printed shroud. They are connected through a MikroTik SFP+ switch.

There is still some work left to do. I need to buy a proper router, and I also have a second RJ45 switch on the way. The goal is to create a separate link over rj45 for management and other OpenStack-related traffic that is not bandwidth critical, so that it does not clutter the main high-speed SFP+ link.

If you have any networking/gear/openstack related advice let me know, I'll be happy to learn more 🫡🫡

Thanks to all of you for your advices on my previous posts. As always don't hesitate if u have any question !

Take care 🙂

Hey there,

Just finished assembling these cute 3d printed shrouds for the​ Mellanox qfp28 cards that came for my homelab. I had my doubts but it works great with these 40mm noctua fans. Cards now reach 40 celsius max.

If you have any question let me know I'll be happy to share !

Printable link: https://www.printables.com/model/1715543-mellanox-connectx-4-lx-shroud

That's all 🐧❤️

Hi everyone,

After one year of scavenging, doubts, dreams and eating noodles it's finally done !

I just published the first article about my slightly unreasonable custom watercooled homelab build.

It is a three-node cluster built around Minisforum BD-series motherboards, but almost nothing about the build ended up being standard. The boards do not use regular Intel or AMD cooler mounting, so I had to use custom handmade waterblocks. The cases also needed quite a bit of CAD work and 3D printing: custom rear I/O panels, quick-disconnect pass-throughs, C14 power inlets, internal PSU holders, airflow brackets, and various mounts for the pump, controller, radiator, and fans.

The loop is built around three custom made ​CPU waterblocks, a large 200 mm Alphacool radiator, an EK D5 pump/reservoir/manifold, quick-disconnect fittings, and an Aquacomputer QUADRO running the pump and fan curves independently from the nodes. The whole thing is designed so the cooling system keeps running even if one node goes down.

It is not the prettiest thing from the front yet, but the empty middle section will eventually be filled by the 10G SFP+ switch and router, which should hide most of the cables. I also plan to add a small touchscreen in front of the top radiator. The good news: it works. Idle CPU temps sit around 26°C, and with all three nodes running full benchmarks, the hottest temperatures I have seen are around 75°C.

It's built for openstack so I stuffed it with all the Ram I could get, around 384G of crucial DDR5 right before the AI crunch. With 96 cores in total it should be able to handle most loads.

I’ll post follow-ups with more details about performance, BIOS setup, thermals, networking, and the final 10G SFP+ setup once the remaining gear arrives.

Full article/ tutorial and parts ​here: https://medium.com/@armeldemarsac/how-i-built-my-own-private-cloud-1-aa7fc6e9b87b

I’d love to hear suggestions or advice. I know that nothing about this build is reasonable 🤣This is my first PC build, so there is definitely still a lot to learn. Thanks for all the help that I got around here and the inspiration !