Home Lab 10 - Minimalist Container host with Autoscale and Gitops - GitOps Mechanism

You should have an Alpine Linux host running Docker. If you don't then head back to the first blog to get that setup. Ontop of this you should have Portainer, Traefik and Sablier installed. If you don't then head back to the last blog to get that setup. Today we'll polish off our environment, at least in terms of functionality, with addition of the Gitops mechanism.

Application stacks

How you configure this is really entirely up to you, I'll show you how I've set it up and you can copy that if you wish. I do suggest reading the documentation to verify my settings.

Postgresql

We'll be needing a postgres database for Gitea. Gitea can use a built-in DB if you wish but this is a more robust/scalable setup. Here is the stack file:

version: '3.1'
# Database sits only in data_tier
# which should only be accessible from app_tier
networks:
  data_tier:
     external: true
     
services:
  postgres:
    image: postgres:15-alpine
    container_name: postgres
    restart: always
    environment:
      # Create secret values `postgres_user` and `postgres_password`
      # to have these variables swapped out
      POSTGRES_USER: $POSTGRES_USER
      POSTGRES_PASSWORD: $POSTGRES_PASSWORD
    networks:
      - data_tier
    volumes:
      - postgres:/var/lib/postgresql/data
    # Don't allow Traefik to route to here
    # We want to rely only on built-in internal
    # networking of Docker
    labels:
      - traefik.enable=false
volumes:
  postgres:
    external: false

Go ahead and create this stack in Portainer.

Configuration

You can lock this down as much as you wish, but these instructions will get you running.

# Login to the postgresql container
doas docker exec -it postgres /bin/sh
# `psql` into the database
$ psql -u $POSTGRES_USER
# Create a user for Gitea
> create user gitea with encrypted password 'DEADBEEF';
# Allow access to public schema
> grant all on schema public to gitea;
# Create database and give owner to gitea
> create database gitea with owner gitea;

Gitea

Here is the stack yaml file:

version: "3.1"
# Gitea must access postgres so it needs both app tier and data tier
networks:
  app_tier:
     external: true
  data_tier:
     external: true
    
services:
  gitea:
    image: gitea/gitea:1.18.3
    container_name: gitea
    restart: always
    networks:
      - app_tier
      - data_tier
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=postgres:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      # Create secret 'gitea__database__pass' with the password
      # you set in your user creation
      - GITEA__database__PASS=$GITEA__database__PASS
      - GITEA__webhook__ALLOWED_HOST_LIST=*.salmon.sec
      - GITEA__webhook__SKIP_TLS_VERIFY=true
    ports:
      - 3000 # HTTP
      - 2222 # SSH
    volumes:
      - data:/data
volumes:
  data:
    external: false

You can launch this stack now, but won't be able to access the installation until we reconfigure traefik.

Woodpecker

Now let's add our CI software! Here is the stack:

# docker-compose.yml
version: '3.1'
networks:
  app_tier:
    external: true
  data_tier:
    external: true
services:
  server:
    image: woodpeckerci/woodpecker-server:next-alpine
    ports:
      - 8000
      - 9000
    container_name: woodpecker-server
    volumes:
      - data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=https://woodpecker.salmon.sec
      - WOODPECKER_ADMIN=admin
      - WOODPECKER_REPO_OWNERS=admin
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=https://git.salmon.sec
      - WOODPECKER_GITEA_CLIENT=${WOODPECKER_GITEA_CLIENT}
      - WOODPECKER_GITEA_SECRET=${WOODPECKER_GITEA_SECRET}
      - WOODPECKER_GITEA_SKIP_VERIFY=true
      - WOODPECKER_DATABASE_DATASOURCE=${WOODPECKER_DATABASE_DATASOURCE}
      - WOODPECKER_DATABASE_DRIVER=postgres
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
    networks:
      - app_tier
      - data_tier
  agent:
    image: woodpeckerci/woodpecker-agent:next-alpine
    command: agent
    restart: always
    depends_on:
      - server
    container_name: woodpecker-agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}
      - WOODPECKER_MAX_PROCS=4
      - WOODPECKER_LOG_LEVEL=trace
    networks:
      - app_tier
    labels:
      - traefik.enable=false
volumes:
  data:
    external: false

I suggest you take a look at woodpecker installation docs for details on what to change here for your environment.

Reconfigure Traefik

The last thing we need is to reconfigure Traefik to route to the new services we've created. In your dynamic.yml for Traefik add the following:

...
http:
  # New middleware for Gitea/Woodpecker
  middlewares:
    sablier-gitea:
      plugin:
        sablier:
          dynamic:
            displayName: The Git Bash (っ^‿^)っ
            refreshFrequency: 10s
            showDetails: "true"
            theme: ghost
          names: gitea
          sablierUrl: http://sablier:10000
          sessionDuration: 1h
    sablier-gitea-ssh:
      plugin:
        sablier:
          blocking:
            default-timeout: 2m
          names: gitea
          sablierUrl: http://sablier:10000
          sessionDuration: 1h
    sablier-woodpecker:
      plugin:
        sablier:
          dynamic:
            displayName: The CICD Bird
            refreshFrequency: 10s
            showDetails: "true"
            theme: ghost
          names: woodpecker-server, woodpecker-agent
          sablierUrl: http://sablier:10000
          sessionDuration: 1h
    # Proto/host forwarding for Woodpecker
    woodpecker-headers:
      headers:
        customRequestHeaders: 
          X-Forwarded-Proto: https
          X-Forwarded-For: woodpecker.salmon.sec
  # New Routers
  routers:
    gitea_http:
      service: gitea_http_svc
      rule: "Host(`git.salmon.sec`)"
      tls: false
      entrypoints:
        - websecure
      middlewares:
      - "sablier-gitea"
    gitea_ssh:
      service: gitea_ssh_svc
      rule: "Host(`git.salmon.sec`)"
      entrypoints:
        - ssh
      middlewares:
      - "sablier-gitea-ssh"
    woodpecker_http:
      service: woodpecker_http_svc
      rule: "Host(`woodpecker.salmon.sec`)"
      tls: false
      entrypoints:
        - websecure
      middlewares:
      - "woodpecker-headers"
      - "sablier-woodpecker"   
  # New Services
  services:
    gitea_http_svc:
      loadbalancer:
        servers:
        - url: http://gitea:3000
    gitea_ssh_svc:
      loadbalancer:
        servers:
        - url: http://gitea:2222
    woodpecker_http_svc:
      loadbalancer:
        servers:
        - url: http://woodpecker-server:8000

Write that file, traefik will hot-reload and you'll now be able to reach your services!

GitOps

With these services installed, head into Gitea and create yourself an account and push the local repo you've been using for all these stack files up there. We'll be using this repo to execute CI jobs to update the stacks. For our example, we'll use CI to update itself! We'll use a webhook on the Woodpecker stack to update it from Woodpecker. As your environment grows you would want to add additional jobs here for each stack you want to keep up to date on each change in Git. In your repo, create a Woodpecker CI file .woodpecker.yml:

clone:
  git:
    image: woodpeckerci/plugin-git
    settings:
      skip_verify: true # Allow us to quickly clone from our internal git repo
pipeline:
  woodpecker:
    image: curlimages/curl:latest
    commands:
      - curl --insecure --retry 5 --retry-delay 30  --retry-max-time 360 -v -X POST $WOODPECKER_STACK_WEBHOOK
    secrets: [ woodpecker_stack_webhook ]
    when:
      path: woodpecker/docker-compose.yml

Before this will work, we need to use the web UI's to do some configuration

Webhook URL

Load up portainer, go into the Woodpecker stack we created and enable 'automation' then capture the Webhook URL.
Load the woodpecker UI, enable the stack repo you created and create a secret for the woodpecker_stack_webhook.
Now when you push a change to the woodpecker/docker-compose.yml file it will automatically update from Git!

Conclusion

Now we've created a wonderful lightweight container environment that efficiently uses our hardware and is easily updated through us just interacting with Git. This gives us an awesome base system to really tightly lock down, we'll get into that in future blogs of this series!