Home Lab 7 - RancherOS on Proxmox (fail)

I started with Kubernetes... the storage was very frustrating and maintaining it was a pain in the ass. I want my home lab to just keep itself updated while I forget about it for months on end, but I still need to be able to use my services all the time so I don't want shit going down and having to spend time on it... Kubernetes had those issues. To migrate away from that, I just started running docker-compose stacks. I didn't want them all on one host though so I could control turning certain ones off and so the storage was nice a segregated for backups where I care more about some than others. Well, now I have 5 different VM's that are just running docker containers that I have to maintain - what a pain! I just want one VM that I can throw excess resources at, and run all my containerized applications. Then I can harden that bad boy and maintain the single host quite easily. I'll still have to separately maintain the two Proxmox OS's, Security Onion and pfSense meaning I'll be maintaining four OS's. But it'll be one step closer! So I've decided I'm going to try RancherOS. I did think about going back to Kubernetes again, but I'm only running workloads on one machine so the extra virtualization to get a HA cluster is just a Toy... it's not providing any real benefit because it sits on the same metal.

RancherOS

Installation

There are a ton of methods to install this, and they actually have a specific iso for Proxmox! So I logged into my worker node and downloaded the iso:

$ cd /var/lib/vz/template/iso/
$ wget https://releases.rancher.com/os/latest/proxmoxve/rancheros.iso
--2023-02-11 19:01:23--  https://releases.rancher.com/os/latest/proxmoxve/rancheros.iso
Resolving releases.rancher.com (releases.rancher.com)... 54.192.51.27, 54.192.51.44, 54.192.51.73, ...
Connecting to releases.rancher.com (releases.rancher.com)|54.192.51.27|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162529280 (155M) [application/x-iso9660-image]
Saving to: ‘rancheros.iso’
rancheros.iso                          100%[==========================================================================>] 155.00M  73.5MB/s    in 2.1s
2023-02-11 19:01:25 (73.5 MB/s) - ‘rancheros.iso’ saved [162529280/162529280]

It should be present in Node -> Storage -> Local -> ISO Images:
Begin creating a new VM on your desired node, call it whatever you want:
Select the iso we downloaded earlier:
For hardware settings, just make sure you have at least 1280Mb RAM. I'm going to allocate a lot more out of the gate but it's simple to increase/decrease this later. Defaults for everything will be fine.



Your network configuration will also obviously change based on your network topology. I'm going to have this machine sit in my LAN VLAN so I assign it VLAN tag 10!
Check start after created and let it boot:
Watch the VM boot and get into a console as quick as you can, you'll see the BIOS screen:
If you miss it don't worry - the automatic boot option is what we want! I was surprised to see that it drops us to a shell after it boots - there's no installation process?
Checking a few things out:
Looks good! Now let's actually install it. First we'll create a cloud-init script to load our ssh key on there.

#cloud-config
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEbbftPXN+ERjMxhDQH3AdSDF/AM5hAiYphsiBf1Nml1QPkTgpfjT8XQFKERzqNECBbZHfLKSgWQK4zA7q1ikgOALhHvYAjjV8ua5TicfrQNcP3dlhvmmVXaUlzxYD8SC1B7hmaLdvOLWgF3RF007isJUUOTGdc+rPLGBwXi8uf0xocsaPJu1P6MPgHVvto+GU3uDQv9KuCvAGsNGopiY6dFCfvk6DHlYW6sNtDJh5B4e5AqIyS1TKndn16ySqbvG2RMeBMJ93Y7AREcH8mWVJMNSkiG40o0hgzQMrjlJ7xCPehQgMBWUz817ycNBTrCPyGWcATxxeKj+FIlr8m9s1YjfhP9h2cAhJnFKbdAGs72FvTYzNdrfFDVf2HP60i1bSmVDi/kgUWyHzmDwxz8KeRQ45Q8l1eZGRUaIrfoqKKSl2Qom6x0atlKUN/87RcYq4dCRjQAqX8b+W7dj2R0w0gKeGOMO9ti7UWnk8pVer2uU9YT0jRzFZlvPYtHfco+c= matth@DESKTOP-HP4PFU8

We need to write this to /var/lib/rancher/conf/cloud-config.yml but you can't copy into the VNC window in Proxmox... so we'll have to run some commands from the Proxmox host to copy this into the guest system. First, note your VMID. You can find this via CLI by running qm list or through the WebGUI. Mine is 111. Then execute this, replacing with your ssh public key and VMID:

qm guest exec 111 -- /bin/bash -c 'echo "#cloud-config\nssh_authorized_keys\n  - ssh-rsa YOURKEY" > /var/lib/rancher/conf/cloud-config.yml'

The result should be:

{
   "exitcode" : 0,
   "exited" : 1
}

Confirm in VNC that the file is written as expected:
Okay it wasn't, the newlines didn't work. Just open it with vi and fix it:

Now we can install with this command:

cd /var/lib/rancher/conf
sudo ros install -c cloud-config.yml -d /dev/sda


The installation didn't take long at all:
Remove the installation media and let it reboot:
You should now be able to ssh in, as long as you've configured your firewall to allow it and you're using the key!

ssh rancher@10.20.0.10
[rancher@rancher ~]$ id
uid=1100(rancher) gid=1100(rancher) groups=1100(rancher),1101(docker),1103(sudo)
[rancher@rancher ~]$

Next Steps

This completely depends on what you plan on doing, here's some of the next steps I preformed:

  • Added a static DHCP lease in pfSense
  • Added a DNS entry for the host
  • Applied firewall rules to the host
  • Installed Wazuh agent ........... It was at this point I realized... RancherOS is unmaintained and incompatable due to it's base architecture with many of the security services I had planned to install here. How sad... They explain why they've dropped support here: https://rancher.com/docs/os/v1.x/en/support/ I do have to agree, everyone I talk to is dropping Docker lately.

Conclusion

Back to the drawing board I suppose! In the next blog perhaps I'll come up with something that better suits my needs. This really is the joy of a homelab, I don't have anyone waiting on something to be delivered, I just get to try things out and explore offerings as I feel like it, slowly improving my skills and environment as the years go by. It seems like I should stop looking for something purpose built and just go set up what I need myself on Alpine or Ubuntu like I was already doing! Oh well! As an improvement, I could build a new base image on Alpine, harden it, and install Portainer. I'm finding it a little slow managing all the different applications spead out over different VM's so I'd like Gitops to be present, which portainer could do for me. Then I could really just have one big host with portainer and all my services running. The last piece I want to achieve is serverless services. I.E. I only want my accounting service to be running when I'm actually using it, I don't mind waiting 2-3 minutes for it to boot when I first open it up in my browser. I know how to achieve this with Kubernetes, is there a way to do this with portainer on a containerd daemon? I'll explore this in the next blog.