Home Lab 8 - Minimalist Container host with autoscale and gitops

In the last blog I tried to get going with RancherOS as a secure base for running all my applications on a proxmox worker node. Well, when it came down to actually securing the system I found it was going to take a long time to install NIDS/SIEM agents, and then learned it was unsupported by the Rancher team. In this blog I'll actually come up with a solution. I'll cover Architecture and setup of the Alpine container host in this blog. If you follow the series the end result will be a minamalist container host with AlpineOS with efficient hardware usage thanks to autoscaling via Traefik/Sablier and automatic secure updates thanks to Gitops via Gitea/Woodpecker/Portainer.

Architecture

Requirements

  • Secure base OS that I could install NIDS/HIDS/SIEM agents onto
  • Easy to run and manage on Proxmox
  • Single Container Runtime
  • Efficient use of hardware (Scale services to 0 when not in use)
  • Gitops

OS Stack

Service Stack

Where the magic comes in is within the container environment. I've swapped out nginx for Traefik which enables the autoscaling feature and added portainer to gain Gitops.

GitOps

When I say GitOps I'm referring to the ability to update infrastructure through interacting with nothing but git. To achieve this On Premises without Kubernetes, I'll be using Portainer, Woodpecker and Gitea. Here's how it works:

Autoscaling

What I'm looking for here is for services to be scaled to 0 when they're not in use for some duration. Then, when a request hits Traefik for that service hostname the container is started up. This is how it works:

Background

AlpineOS

What is Alpine Linux anyway? Alpine Linux is a free and open-source Linux distribution based on the musl C library and BusyBox utilities. It is designed to be lightweight and security-focused, with a small disk footprint and minimal attack surface. Alpine Linux is widely used as a base image for containers, due to its small size, speed and security features. It is also used as a minimalist distribution for servers and other systems that require a highly secure and minimal operating environment. The distribution uses its own package manager, apk-tools, which is designed to be simple and efficient, with a focus on security. Alpine Linux also uses a read-only root file system, which helps to prevent tampering and improve overall system security.

Portainer

What is Portainer? I actually used this within it's first few releases and had a terrible experience, although it wasn't entirely portainer's fault, it was mostly due to the instability of Docker Swarm. Portainer is a open-source management solution for Docker and Swarm-based container environments. It provides a simple and intuitive web-based interface for managing and monitoring Docker containers, images, networks, and volumes, as well as managing and deploying Swarm stacks. With Portainer, users can easily manage containers, images, networks, and volumes, perform basic operations such as start, stop, and restart containers, view logs, and monitor resource usage. It also provides a visual representation of the containers and their relationships, making it easier to understand and manage complex container environments. Portainer is lightweight and can be run as a standalone container, making it easy to deploy and manage. It also provides role-based access control and a RESTful API, making it suitable for use in large, multi-user environments. Overall, Portainer offers a simple and user-friendly solution for managing and monitoring Docker and Swarm environments, and is a popular choice for users looking for a graphical interface for managing containers. Personally, a GUI is nice but I don't mind sticking in a shell - what I'm going to gain out of this is hopefully a Gitops integration where I can make all my infrastructure changes in a Git repo and just use the GUI to watch the changes rollout or do debug.

Traefik

Traefik is a popular open-source reverse proxy and load balancer that is designed to handle incoming network traffic and distribute it to multiple services or containers in a dynamic and efficient way. It is often used in cloud-native and containerized environments, such as Kubernetes clusters, where it can automatically discover and configure routes for new services as they are deployed or removed. Traefik is known for its easy-to-use configuration syntax, powerful routing capabilities, and support for a wide variety of backends, including HTTP, TCP, and UDP services. It also supports a range of features like SSL termination, rate limiting, circuit breaking, and request retries, making it a versatile tool for managing network traffic. Traefik is written in Go and is available under the permissive MIT license, making it an attractive option for organizations and developers who want a high-performance and reliable proxy that they can customize and extend to meet their specific needs.

Sablier

Sablier is a middleware plugin for Traefik that enables the use of dynamic timeouts for HTTP requests. It is designed to help manage and optimize the use of resources by setting appropriate timeouts for requests, based on the expected response time of the upstream service. In traditional load balancing, timeouts are usually set statically for all requests, which can lead to either slow response times or increased resource usage. Sablier helps address this issue by dynamically setting timeouts for each request, based on its expected response time, which can lead to better resource utilization and faster response times. Sablier works by using the Time-Based One-Time Password (TOTP) algorithm to generate a unique timeout value for each request. The generated value is used as the timeout duration for the request, and it is calculated based on the expected response time of the upstream service. Sablier is easy to configure with Traefik, and it can be used in conjunction with other middleware plugins and features to create a powerful and flexible load balancing solution. It is available as open-source software under the permissive MIT license.

Alpine Installation

Create Proxmox VM

Upload the Alpine Virtual ISO to Proxmox.
Create a VM from the .iso
Note: Before creating this, I generated a MAC and used it to create a DHCP lease and DNS record on pfSense so it would be accessable via hostname rightaway.

Proxmox VNC Console

Login as root, no password. Execute setup-alpine which will walk through initial setup.
In my configuration I added a vlan tagged interface, so I can just setup eth0 and get going from my DHCP lease. You may have a different network configuration.
Set a temporary root password (change this after you get a shell you can paste into). The mirro doesn't overly matter, try to pick one closest to you. Setup a user for yourself, leave ssh key unpopulated for now as we'll add this later and set defaults for ssh config.
Choose sys for disk type, the other options run from RAM.
Reboot when the installation finishes, you can remove the iso from the mount path in proxmox if you want now.

Initial Configuration

Now we can log in via SSH! 

# Give yourself root priv for setup over ssh (Clean up later)
su root
adduser -g "matt" matt
adduser matt wheel
apk add doas
echo "permit persist :wheel" >> /etc/doas.d/doas.conf
exit
# Disable root account login
doas vi /etc/passwd
# Change:  
#   root:x:0:0:root:/root:/bin/ash
#   to
#   root:x:0:0:root:/root:/sbin/nologin
# Test
doas su root
  This account is not available
# Add our SSH key
cd ~ && mkdir .ssh
chmod 700 .ssh
cd .ssh && touch authorized_keys
chmod 600 authorized_keys
cat << EOF > authorized_keys
YOUR_SSH_KEYS
EOF
# Harden OpenSSH configuration
doas vi /etc/ssh/sshd_config
# Add:
#   PermitEmptyPasswords no
#   PermitRootLogin no
#   Protocol 2
#   ClientAliveInterval 300
#   ClientAliveCountMax 2
#   PasswordAuthentication no
#   PubkeyAuthentication yes
#   AllowUsers YOUR_USERNAME
#   X11Forwarding no
# Restart sshd
doas service sshd restart
# Logout to test ssh config
exit
# Check that we cannot login as root
ssh root@container0.salmon.sec
root@container0.salmon.sec: Permission denied (publickey,keyboard-interactive).
# Check that we cannot login as user without a ssh key (no password auth)
ssh -o PasswordAuthentication=yes -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no matt@container0.salmon.sec
matt@container0.salmon.sec: Permission denied (publickey,keyboard-interactive).
# Log in with our key
ssh -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes matt@container0.salmon.sec

Docker installation

Ensure community repo is enabled in /etc/apk/repositories
Install and configure:

# Install Docker & Compose
doas apk update 
doas apk add docker docker-cli-compose
# Enable at boot and right away
doas rc-update add docker boot
doas service docker start

Next

In the next blog we install Portainer, Traefik and Sablier to configure a stack deployment environment (Portainer) and configure our routing (Traefik) and autoscale mechanism (Sablier).