Home Lab 5

With my GitOps enabled Kubernetes up and running, it's time to setup a CICD mechanism and get my blog running there to test the setup. With everything already configured to pull from a repository of manifests, all I need to do is define YAML! Kubernetes == YAML! We're going to:

  • Deploy a Docker Image Registry
  • Deploy Drone for a CICD Runner
  • Deploy this blog First, we define the entry point for Flux Sync, /clusters/testing. Here I'll define two 'sub-modules' that'll have their own directories.

applications.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: applications
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: birtast
  path: ./applications
  prune: true
  validation: client

infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: birtast
  path: ./infrastructure
  prune: true
  validation: client

Now, at each of the newly defined directories, I add a kustomization.yaml file to declare the resources I'll place in each:

./applications/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - blog

./infrastructure/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - registry
  - drone

Registry

Deploying the Registry contains a few pieces

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert.yaml
  - deployment.yaml
  - ingress.yaml
  - namespace.yaml
  - storage.yaml

cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: registry-cert
  namespace: istio-system
spec:
  # Secret names are always required.
  secretName: self-signed-registry-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  commonName: registry.salmon.sec
  dnsNames:
  - registry.salmon.sec
  issuerRef:
    name: selfsigned-issuer

ingress.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: registry-gateway
  namespace: registry
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: self-signed-registry-tls # must be the same as secret
    hosts:
    - registry.salmon.sec
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: registry-https
  namespace: registry
spec:
  hosts:
  - registry.salmon.sec
  gateways:
  - registry-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 5000
        host: registry-http
---
apiVersion: v1
kind: Service
metadata:
  name: registry-http
  namespace: registry
spec:
  selector:
    app: registry
  ports:
  - port: 5000
    targetPort: 5000

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: registry
  labels:
    istio-injection : enabled

storage.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: registry
  name: docker-registry-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: rook-ceph-block

Drone

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert.yaml
  - deployment.yaml
  - ingress.yaml
  - namespace.yaml
  - runner.yaml

cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: drone-cert
  namespace: istio-system
spec:
  # Secret names are always required.
  secretName: self-signed-drone-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  commonName: drone.salmon.sec
  dnsNames:
  - drone.salmon.sec
  issuerRef:
    name: selfsigned-issuer

deployment.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: drone-data-pvc
  namespace: drone
spec:
  storageClassName: rook-ceph-block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone
  namespace: drone
  labels:
    app: drone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone
  template:
    metadata:
      labels:
        app: drone
    spec:
      containers:
      - name: drone
        image: drone/drone:1
        env:
        - name: DRONE_SERVER_HOST
          value: drone.salmon.sec
        - name: DRONE_SERVER_PROTO
          value: https
        - name: DRONE_STASH_SKIP_VERIFY
          value: "true"
        - name: DRONE_GITEA_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: client-id
              name: drone-gitea-oauth
        - name: DRONE_GITEA_CLIENT_SECRET
          valueFrom:
              secretKeyRef:
                key: client-secret
                name: drone-gitea-oauth
        - name: DRONE_GITEA_SERVER
          value: "https://git.salmon.sec"
        - name: DRONE_RPC_SECRET
          valueFrom: 
                secretKeyRef:
                  key: "secret"
                  name: drone-runner-secret
        - name: DRONE_RUNNER_VOLUMES
          value: "/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt"
        volumeMounts:
        - name: tls-cert
          mountPath: "/etc/ssl/certs"          
          readOnly: true
        - name: drone-data
          mountPath: "/data"          
          readOnly: false
      volumes:
      - name: tls-cert
        secret:
          secretName: ca-cert
          items:
          - key: cert
            path: ca-certificates.crt
      - name: drone-data
        persistentVolumeClaim:
          claimName: drone-data-pvc

ingress.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: drone-gateway
  namespace: drone
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: self-signed-drone-tls # must be the same as secret
    hosts:
    - drone.salmon.sec
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: drone-https
  namespace: drone
spec:
  hosts:
  - drone.salmon.sec
  gateways:
  - drone-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: drone
---
apiVersion: v1
kind: Service
metadata:
  name: drone
  namespace: drone
spec:
  selector:
    app: drone
  ports:
  - port: 80
    targetPort: 80

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: drone
  labels:
    istio-injection : enabled

runner.yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: drone
  namespace: drone
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - create
  - delete
  - list
  - watch
  - update
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: drone
  namespace: drone
subjects:
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: drone
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-runner
  namespace: drone
  labels:
    app.kubernetes.io/name: drone-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: drone-runner
  template:
    metadata:
      labels:
        app.kubernetes.io/name: drone-runner
    spec:
      containers:
      - name: drone-runner
        image: drone/drone-runner-kube:latest
        env:
        - name: DRONE_RPC_HOST
          value: drone
        - name: DRONE_RPC_PROTO
          value: http
        - name: DRONE_NAMESPACE_DEFAULT
          value: drone
        - name: DRONE_DEBUG
          value: "true"
        - name: DRONE_RPC_SECRET
          valueFrom:
            secretKeyRef:
              name: drone-runner-secret
              key: secret

1

Blog

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert.yaml
  - deployment.yaml
  - flux.yaml
  - ingress.yaml
  - namespace.yaml

cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: blog-cert
  namespace: istio-system
spec:
  # Secret names are always required.
  secretName: self-signed-blog-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  commonName: blog.salmon.sec
  dnsNames:
  - blog.salmon.sec
  issuerRef:
    name: selfsigned-issuer

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
  namespace: blog
  labels:
    app: blog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
        - name: blog
          image: registry.salmon.sec/blog:32-55003d9a4915beab58fba9cebb570a499e56395f # {"$imagepolicy": "flux-system:blog"}

flux.yaml

apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImageRepository
metadata:
  name: blog
  namespace: flux-system
spec:
  image: registry.salmon.sec/blog
  interval: 1m0s
  certSecretRef: 
    name: registry-cert
---
apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImagePolicy
metadata:
  name: blog
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: blog
  filterTags:
    pattern: '^(?P<id>\d+)-\S+$'
    extract: "$id" 
  policy:
    numerical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImageUpdateAutomation
metadata:
  name: blog
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: birtast
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: master
  update:
    path: ./applications/blog
    strategy: Setters

ingress.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: blog-gateway
  namespace: blog
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: self-signed-blog-tls # must be the same as secret
    hosts:
    - blog.salmon.sec
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: blog-https
  namespace: blog
spec:
  hosts:
  - blog.salmon.sec
  gateways:
  - blog-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 8080
        host: blog-http
---
apiVersion: v1
kind: Service
metadata:
  name: blog-http
  namespace: blog
spec:
  selector:
    app: blog
  ports:
  - port: 8080
    targetPort: 8080

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: blog
  labels:
    istio-injection : enabled

2 Finally, on the blog I have to define the Drone pipeline!

---
kind: pipeline
name: salmonsec_blog
namespace: drone
type: kubernetes
# Disable default clone step
clone:
  disable: true
# Disable Istio Injection
metadata:
  annotations:
    sidecar.istio.io/inject: false
steps:
- name: clone
  image: alpine/git
  commands:
  - cat /etc/resolv.conf
  - apk update && apk add git
  - git clone http://gitea-http.gitea.svc.cluster.local:3000/matt/salmonsec.git .
  - git checkout $DRONE_COMMIT
- name: build
  image: banzaicloud/drone-kaniko
  settings:
    registry: registry.salmon.sec
    repo: blog
    skip_tls_verify: true
    tags: ${DRONE_BUILD_NUMBER}-${DRONE_COMMIT_SHA}
  when:
    branch:
      exclude:
      - master
- name: release
  image: alpine/git
  commands:
  - git remote rename origin upstream
  - git remote add origin https://xxx:$TOKEN@github.com/xxx/salmonsec
  - git push origin master
  environment:
    TOKEN:
      from_secret: github_personal_access_token
  when:
    branch:
    - master

As you can see, I'm not using this cluster to actually host my public blog - just internally for testing and for fun. I don't plan on opening any holes into my home network! Now, when I push any changes, I can see the whole system working together to build the repo, push it to the image registry and update the running image version of the blog on kubernetes: 3 Very satisfying indeed!