Home Lab 8 - Minimalist Container host with autoscale and gitops
In the last blog I tried to get going with RancherOS as a secure base for running all my applications on a proxmox worker node. Well, when it came down to actually securing the system I found it was going to take a long time to install NIDS/SIEM agents, and then learned it was unsupported by the Rancher team. In this blog I'll actually come up with a solution. I'll cover Architecture and setup of the Alpine container host in this blog. If you follow the series the end result will be a minamalist container host with AlpineOS with efficient hardware usage thanks to autoscaling via Traefik/Sablier and automatic secure updates thanks to Gitops via Gitea/Woodpecker/Portainer.
- Secure base OS that I could install NIDS/HIDS/SIEM agents onto
- Easy to run and manage on Proxmox
- Single Container Runtime
- Efficient use of hardware (Scale services to 0 when not in use)
Where the magic comes in is within the container environment. I've swapped out nginx for Traefik which enables the autoscaling feature and added portainer to gain Gitops.
When I say
GitOps I'm referring to the ability to update infrastructure through interacting with nothing but
git. To achieve this On Premise without Kubernetes, I'll be using Portainer, Woodpecker and Gitea. Here's how it works:
What I'm looking for here is for services to be scaled to 0 when they're not in use for some duration. Then, when a request hits Traefik for that service hostname the container is started up. This is how it works:
What is Alpine Linux anyway? Alpine Linux is a free and open-source Linux distribution based on the musl C library and BusyBox utilities. It is designed to be lightweight and security-focused, with a small disk footprint and minimal attack surface. Alpine Linux is widely used as a base image for containers, due to its small size, speed and security features. It is also used as a minimalist distribution for servers and other systems that require a highly secure and minimal operating environment. The distribution uses its own package manager, apk-tools, which is designed to be simple and efficient, with a focus on security. Alpine Linux also uses a read-only root file system, which helps to prevent tampering and improve overall system security.
What is Portainer? I actually used this within it's first few releases and had a terrible experience, although it wasn't entirely portainer's fault, it was mostly due to the instability of Docker Swarm. Portainer is a open-source management solution for Docker and Swarm-based container environments. It provides a simple and intuitive web-based interface for managing and monitoring Docker containers, images, networks, and volumes, as well as managing and deploying Swarm stacks. With Portainer, users can easily manage containers, images, networks, and volumes, perform basic operations such as start, stop, and restart containers, view logs, and monitor resource usage. It also provides a visual representation of the containers and their relationships, making it easier to understand and manage complex container environments. Portainer is lightweight and can be run as a standalone container, making it easy to deploy and manage. It also provides role-based access control and a RESTful API, making it suitable for use in large, multi-user environments. Overall, Portainer offers a simple and user-friendly solution for managing and monitoring Docker and Swarm environments, and is a popular choice for users looking for a graphical interface for managing containers. Personally, a GUI is nice but I don't mind sticking in a shell - what I'm going to gain out of this is hopefully a Gitops integration where I can make all my infrastructure changes in a Git repo and just use the GUI to watch the changes rollout or do debug.
Traefik is a popular open-source reverse proxy and load balancer that is designed to handle incoming network traffic and distribute it to multiple services or containers in a dynamic and efficient way. It is often used in cloud-native and containerized environments, such as Kubernetes clusters, where it can automatically discover and configure routes for new services as they are deployed or removed. Traefik is known for its easy-to-use configuration syntax, powerful routing capabilities, and support for a wide variety of backends, including HTTP, TCP, and UDP services. It also supports a range of features like SSL termination, rate limiting, circuit breaking, and request retries, making it a versatile tool for managing network traffic. Traefik is written in Go and is available under the permissive MIT license, making it an attractive option for organizations and developers who want a high-performance and reliable proxy that they can customize and extend to meet their specific needs.
Sablier is a middleware plugin for Traefik that enables the use of dynamic timeouts for HTTP requests. It is designed to help manage and optimize the use of resources by setting appropriate timeouts for requests, based on the expected response time of the upstream service. In traditional load balancing, timeouts are usually set statically for all requests, which can lead to either slow response times or increased resource usage. Sablier helps address this issue by dynamically setting timeouts for each request, based on its expected response time, which can lead to better resource utilization and faster response times. Sablier works by using the Time-Based One-Time Password (TOTP) algorithm to generate a unique timeout value for each request. The generated value is used as the timeout duration for the request, and it is calculated based on the expected response time of the upstream service. Sablier is easy to configure with Traefik, and it can be used in conjunction with other middleware plugins and features to create a powerful and flexible load balancing solution. It is available as open-source software under the permissive MIT license.
Create Proxmox VM
Upload the Alpine Virtual ISO to Proxmox.
Create a VM from the
Note: Before creating this, I generated a MAC and used it to create a DHCP lease and DNS record on pfSense so it would be accessable via hostname rightaway.
Proxmox VNC Console
Login as root, no password. Execute
setup-alpine which will walk through initial setup.
In my configuration I added a vlan tagged interface, so I can just setup
eth0 and get going from my DHCP lease. You may have a different network configuration.
Set a temporary root password (change this after you get a shell you can paste into). The mirro doesn't overly matter, try to pick one closest to you. Setup a user for yourself, leave ssh key unpopulated for now as we'll add this later and set defaults for ssh config.
sys for disk type, the other options run from RAM.
Reboot when the installation finishes, you can remove the
iso from the mount path in proxmox if you want now.
Now we can log in via SSH!
# Give yourself root priv for setup over ssh (Clean up later) su root adduser -g "matt" matt adduser matt wheel apk add doas echo "permit persist :wheel" >> /etc/doas.d/doas.conf exit # Disable root account login doas vi /etc/passwd # Change: # root:x:0:0:root:/root:/bin/ash # to # root:x:0:0:root:/root:/sbin/nologin # Test doas su root This account is not available # Add our SSH key cd ~ && mkdir .ssh chmod 700 .ssh cd .ssh && touch authorized_keys chmod 600 authorized_keys cat << EOF > authorized_keys YOUR_SSH_KEYS EOF # Harden OpenSSH configuration doas vi /etc/ssh/sshd_config # Add: # PermitEmptyPasswords no # PermitRootLogin no # Protocol 2 # ClientAliveInterval 300 # ClientAliveCountMax 2 # PasswordAuthentication no # PubkeyAuthentication yes # AllowUsers YOUR_USERNAME # X11Forwarding no # Restart sshd doas service sshd restart # Logout to test ssh config exit # Check that we cannot login as root ssh email@example.com firstname.lastname@example.org: Permission denied (publickey,keyboard-interactive). # Check that we cannot login as user without a ssh key (no password auth) ssh -o PasswordAuthentication=yes -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no email@example.com firstname.lastname@example.org: Permission denied (publickey,keyboard-interactive). # Log in with our key ssh -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes email@example.com
community repo is enabled in
Install and configure:
# Install Docker & Compose doas apk update doas apk add docker docker-cli-compose # Enable at boot and right away doas rc-update add docker boot doas service docker start
In the next blog we install Portainer, Traefik and Sablier to configure a stack deployment environment (Portainer) and configure our routing (Traefik) and autoscale mechanism (Sablier).