Not quite there yet, but it's shaping up! I'll make a diagram to update this once it's stable 🙂
#Openstack deployment for my 100% as-code homelab
woven forge
Everything is containerized (cause its awesome!) Ceph + Openstack (cephadm + kolla-ansible) on 3 DIY 1u chassis, totalling 60 cores, 192GB RAM, 3TB SATA SSD, 12TB NVME SSD.
This should hopefully host pretty much all the openstack services in the not so distant future. For now it's only os-core, to make sure it stabilize, then:
barbican (with hashicorp vault backend)
senlin (clusters)
magnum (k8s clusters)
trove (databases)
octavia (load-balancers)
manilla (filesystems)
mazakari (VM live migration)
swift (object storage)
update: it's alive !
remote pond
@woven forge Always love seeing more OpenStack in the wild!
Especially with Ceph!
Don't actually run Swift, use your Ceph RGW and have swift service endpoint point to your RGW nodes
You'll also need Octavia before you do Magnum
woven forge
Hey ! glad you like it ! To address some of your points, yes, I plan on setting up at least trove, senlin and octavia, masakari, barbican, manilla (the ones that have pretty much 0 dependencies) before doing anything else. As for swift, I have some drives to do native swift-on-disks, but I have been looking to do it the way you advised, I just don't have enough storage on ceph for now... I might wait until I add some drives to ceph
remote pond
Hey ! glad you like it ! To address some of your points, yes, I plan on setting ...
you can technically do it from using the same disks as it will just create a new set of pools, but yeah I get that. I actually have a completely different setup for my RGW backends, I use spinning disks with WAL/DB located on NVMe disks
woven forge
how many nodes do you run at home ?
remote pond
Eight Ceph nodes, 2 LB nodes, 3 controller nodes, 5 compute nodes
woven forge
oh nice !
woven forge
yh I saw the pictures, I didnt see mentions of the archi for os/ceph
GPUs with cyborg ? or native nova ?
woven forge
ouch
remote pond
Cyborg is in the Dev lab
remote pond
I use OpenStack-Ansible for deployment and no Cyborg role yet 😐
woven forge
oh, that's why...
remote pond
although still having same issues with Cyborg that I do with Nova with Ampere-based cards
woven forge
I opted for kolla-ansible so that the hosts could be wiped fast
with cephadm for ceph
wanted to keep it as code and use ceph-ansible, but they're pushing people to cephadm so I didn't want to end up with a dead project...
remote pond
yeah my Ceph deployment is cephadm
although there is a Cephadm-Ansible in the works
but I'm coding my own version
woven forge
but I'm coding my own version
oh nice
I've done that for my core cluster (the one that hosts the openstack dependencies like git, CICD runners, etc...)
with hashistack
aka vault nomad consul
remote pond
Very nice!
woven forge
remote pond
what are you using for Neutron networking?
woven forge
linuxbridge
it's experimental but it works fine in my experience at work
and I had some perofrmance issues with openvswitch so I didn't bother
remote pond
lol, well technically it's "deprecating", the community is moving to OVN
woven forge
in case anything is of interest to you
remote pond
which is what I use
woven forge
lol, well technically it's "deprecating", the community is moving to OVN
I didnt know that, it's tsgged as experimental tho
but yh I noticed everyone is kind of pushing towards ovn now
remote pond
yeah that was done recently because they don't want it necessarily "deprecated" because a lot of users still use linuxbridges
they jsut are no longer going to be developing for lbr anymore
remote pond
are you doing provider only networking or are you doing self-service networking?
woven forge
basically the plan is to have a /21 (I think) that's the "public" range which is routed to my physical network, with quotas on it
then whatever you want in the projects
remote pond
with "provider" networking, you, as the admin, provide customers a network (usually a VLAN or flat interface that's segregated at a hardware level). With self-service customers can create their own tunneled network.
which allows customer networks to have overlapping IPs
then you use a neutron router to "get out" to the outer world
woven forge
which allows customer networks to have overlapping IPs
yh that's the plan
woven forge
then you use a neutron router to "get out" to the outer world
except people can use routers or "public" IPs for their LBs and stuff
remote pond
so with my mine I have a "public" network connected to my ISPs with my pool of static IPs and then customer create their own private network and attach their routers to the "Public" network
woven forge
so with my mine I have a "public" network connected to my ISPs with my pool of s...
yup that's pretty much what I plan on doing, except since the public range is still private, they just have quotas
50 or so public IPs
they do whatever with it
routers, LBs, etc..
I'd like to try a lab with ipv6 so that people can have internet routable IPs, but I first needs to have my main ipv4 cluster working lol
been having stupid issues all day with hypervisors not registering
remote pond
yeah I've still avoided ipv6
woven forge
I also need to figure out how to route the traffic from my "public" VLAN to the projects
cause it implies adding route tables to the core router which is not something people will be able to do
remote pond
that's why I was looking at the Arista switches I got because they have both an L2 and L3 driver
woven forge
at first it shouldn't matter since I will be owning all the projects, but since i plan on offering hosting to friends and stuff who are also labbing, I need to find some way to do it
worldly ether
Not quite there yet, but it's shaping up! I'll make a diagram to update this onc...
I would ** LOVE** to see your diagram, and if you could point me to a really good OpenStack "How To" or set of videos, I'd appreciate it! I'd like to get a lab setup on this and perhaps move it to production in my home DC.
woven forge
I would ** LOVE** to see your diagram, and if you could point me to a really goo...
Hey ! sorry for the late reply ! I don't know of any very good deep dive videos on openstack. I know The guy from Learn Linux TV made a bunch of videos last year, lawrence systems as well, but it wasnt very in-depth. There is also a fun video series from this guy named sean shuping about openstack dpeloyment at home. There is also the guy Daniel Persson (https://www.youtube.com/@DanielPersson) who did a lot of openstack install from scratch, this might actually be the most in-depth I've seen on youtube. You shouldn't for your own mental health deploy it like he does, but it teaches you a great deal about how it works.
once you're familiar with the topics, I would recommend looking at deployment tools, kolla-ansible (the one I use), openstack-ansible, or juju (this one is kind of weird and I don't like it, but some people do)
once you're familiar with the way it works, tbh, the documentation is very very extensive, and you should find anything you could possibly want in here