alt text

The Dream

I've spent a huge portion of my life on a computer, like many others in my generation. We take it for granted that we have all this information at our fingertips.There's a feeling that it's always been there, and that everything works just because it does... But I think this needs to change for one simple reason, we're all incredibly insecure. New age threats are fought on the battleground described by the OSI model, and western countries seem like they're losing. Even on a smaller scale, the little guys are losing every day. The rate of cyber attacks increases every year, and becomes more profitable too. The general population just doesn't have the tools to combat this, and they never will unless it's applied at some much higher level than they control. I don't know how yet, but I'm going to change this one day. I've followed countless rabbit holes, finding new depths of knowledge that I never knew existed. In a sense I feel I have an unsatiable curiosity. This blog is going to detail my venture into the world of cyber security. Likely, nobody will ever read this. But, If I succeed in my dream in any way maybe someone will want to follow in my footsteps. Imagine that!

The platform

I guess it's fitting to start by documenting how I got this blog up and running. First, I had in my head the components I'd need to sort out:

A domain name
Cheap infrastructure suitable for the ~0 traffic this blog will get
Some primitive (free) CICD
A base repository that I can mould into my blog website
The end goal is to have a container running somewhere that I can update by pushing to git, that renders markdown documents as blog pages without much further interaction required from my part. I also want to write all the code and manage the infrastructure myself, just because that's more fun.

Domain Name

I've never been much for naming things, whenever I name internal projects I just convert the functional goal of the project into a bunch of languages on google translate until one is short and easy to spell and roll with that. For this though, I decided to try to use some sort of online generator. I found Wordoid which inspired a few things which points out domain names that are available. I almost named it mattnoseworthy but honestly my name is pretty common and I don't love the sound of it. I wanted something playful, something a little dumb. So, SalmonSec it is. I just wanted the .com domains so I went to godaddy and confirmed that they're free. They were. Before checking out, I did some quick searching for Coupon codes around the web. As usual I found a ton, I saw one that seemed too good to be true: ".com domain for $1 for the first year" code=GDD1dom. I slapped it in on Godaddy and it worked, sweet! With such easy success I felt I could go keep looking but it's already so cheap, who cares. I made sure I had everything extra turned off, I don't need all that crap. I know from past experiences that Godaddy's nameserver records are dreadfully slow to update and the UX of their Record table is pretty meh. So, the first thing I did after buying the domain was point it to Digital Ocean's nameservers. Since the last time I had used GoDaddy they changed their UI, clearly more geared towards inexperienced users - took me about 5 minutes to find where to change name servers (They don't let you do it from DNS records yourself :eyeroll:) Once I finally found it, I just set the nameserves to:

ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com
Sweet, confirming that I can now leave GoDaddy until I have some sort of billing problem, a quick whois:

whois salmonsec.com
Domain Name: salmonsec.com
Registry Domain ID: 2545487141_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.godaddy.com
Registrar URL: http://www.godaddy.com
Updated Date: 2020-07-12T17:01:41Z
Creation Date: 2020-07-12T17:01:40Z
Registrar Registration Expiration Date: 2021-07-12T17:01:40Z
Registrar: GoDaddy.com, LLC
Registrar IANA ID: 146
Registrar Abuse Contact Email: abuse@godaddy.com
Registrar Abuse Contact Phone: +1.4806242505
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited http://www.icann.org/epp#clientUpdateProhibited
Domain Status: clientRenewProhibited http://www.icann.org/epp#clientRenewProhibited
Domain Status: clientDeleteProhibited http://www.icann.org/epp#clientDeleteProhibited
Registrant Organization:
Registrant State/Province: Newfoundland
Registrant Country: CA
Registrant Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=salmonsec.com
Admin Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=salmonsec.com
Tech Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=salmonsec.com
Name Server: NS1.DIGITALOCEAN.COM
Name Server: NS2.DIGITALOCEAN.COM
Name Server: NS3.DIGITALOCEAN.COM
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2020-07-12T17:00:00Z <<<

Looks good to me, bye GoDaddy!

Cheap Infrastructure

I've used most of the common Cloud Service Providers before at work and for personal projects, I like Digital Ocean and it's cheap so that's what I'll use for this. Honestly I really want to run all this on K8s and use Terraform, but I know Kubernetes is overkill for the zero traffic I'll get and I don't really want to spend much on this project, so I'm just going to setup a simple Docker host. I'd usualy use ansible or something to manage a VM, mostly to ensure I'm properly documenting what I'm doing so my work can be audited if something goes wrong with the benefit of it beinig reproducable. But, this is just a personal project and I'm the only one going to be ever looking at this, I'll just use the UI and ssh in. So I create a new key pair:

ssh-keygen

And use that to create a $5, absolute base tier VPC. First, I want to start typing my new domain name so the first thing I do is point the root A record of my domain to the server.

ssh root@salmonsec.com -i ~/.ssh/salmonsec
root@ubuntu-s-1vcpu-1gb-tor1-01:~#

Sweet, let's spend a few minutes hardening the box. Nothing fancy, will come back to this later just want the bare essentials for now.

#Update
apt update && apt upgrade
# Set root password
passwd
# Create myself a user
useradd matt
usermod -aG sudo matt
mkdir /home/matt
chown -R matt:matt /home/matt
passwd matt
# Set default login shell to bash
usermod -s /bin/bash matt
# Configure ssh access to the new user
cp /root/.shh /home/matt/.ssh
chmod 700 /home/matt/.ssh
chmod 644 /home/matt/.ssh/* 
# -- go test access via matt --
exit
ssh matt@salmonsec.com -i ~/.ssh/salmonsec
# Lock down ssh: no root login, only via key pair, only from single IP, no empty passwords
vim /etc/ssh/sshd_config
# Disable root account
sudo passwd -l root
# Drop fail2ban
sudo apt-get install fail2ban
echo "[sshd]
> enabled = true
> port = 22
> filter = sshd
> logpath = /var/log/auth.log
> maxretry = 3" > /etc/fail2ban/jail.local
# Secure shared memory
echo "tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0" >> /etc/fstab
# Setup an extreemly basic firewall with IPtables for now
iptables -F
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
ufw enable
# Setup auto-updates
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
# Reboot
reboot now

Good enough for now. We'll come back later and secure the kernel more, lock down tmp, add some monitoring services and things like that. I know I'm going to just run docker on this host, so let's take a second to get that going too:

# Install docker
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add –
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu  $(lsb_release -cs)  stable"
sudo apt-get update
sudo apt-get install docker-ce
# Start on boot
sudo systemctl start docker
sudo systemctl enable docker
# Add docker group
sudo groupadd docker
sudo usermod -aG docker matt
# Test
docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Good to go.

CICD

At my job we're using a full-fledged CICD stack with kubernetes. As mentioned earlier, I don't want to pay for clusters for this. So I'm going to go out and look for something I can run on this one VPC. My end goal:

I'll have a local git repository on the VPC
I'll run a private container registry
When I push to git, I want something to build an image and push it to the registry
When a new image is pushed, I want whatever container orchestrator I'm using to update the running image Since I'm not expecting this site to actually grow at all, I'll likely just prop this all up on a single host so a lot of complexities are removed from this bringup - I think I can just use docker-compose and then docker webhooks to update the image when something is done building.

Watchtower

So, I'm going to start with figuring out what something is. Searching around for something that'll build images for me (I don't care about testing this thing), Watchtower seems like a decent choice. Let's try it out. Reading the documentation, seems pretty straight forward. You run it as a container with the Docker socket mounted into it and it monitors images of container, checks for updates on the registry and updates the running image if available, cool! I just want to throw everything together and have it work, so I'm just going to move on and come back here once I'm ready for things to update.

Docker registry

Well, I had planned to setup a private registry on this VPC. When researching how to do so, I realied that to use letsencrypt I'd need to have multiple domains pointing to this box, and thus I'd need to setup domain-based routing to multiple services mapped from 443 on the box. This is supposed to be quick and dirty... So, I had a quick look, and DO currently has a free private container registry. Love to hear it - I'll just use that. Ah, SIKE it requires a $5/month storage plan. Whatever I want the instant gratification, lets do it. Good job DO marketing team, you got me. Setup is quick, auth seem easy, moving on!

Application

I love golang and it's incredibly simple to get a simple webserver going, so here goes! Just starting top-down here from execution path, I begin by defining a main function with what is now just undefined function:

package main
import (
	"html/template"
	"net/http"
	"github.com/gorilla/mux"
	log "github.com/sirupsen/logrus"
)
// Server - simple struct to pass around data for the server
type Server struct {
	// Logger configured with various production-grade tags and params
	Log *log.Entry
	// Templates with go template language
	Templates map[string]*template.Template
	// Markdown Blog Posts
	Blogs map[string]string
}
func main() {
	log.Info("Booting Server...")
	// Create Server struct
	s := Server{}
	// Load templates
	s.Templates = LoadTemplates("./templates")
	s.Blogs = LoadPosts("./content")
	// Define router
	router := mux.NewRouter()
	// Register handlers
	RegisterHandlerFunctions(router, &s)
	// Start Server
	s.Log.Info("Coming online, port 8080")
	http.ListenAndServe("0.0.0.0:8080", nil)
}

Before going onwards, I grab the most basic bootstrap skeleton I can find and pull it apart into files that I'll use more later. For now there'll be two main pages:

Home
Blogs Home & Blogs will both have the same header/footer/navbar/sidenav, but the body will just either contain a pagenated list of blog cards, or content of a blog. Easy enough! For template loading we're just dragging in a bunch of html files that contain golang templating markup, and feeding each template a set of elements they can use to build up the full page in a modular way.

// LoadTemplates - Load our templates
func LoadTemplates(templateDir string) map[string]*template.Template {
	globals := []string{templateDir + "/footer.html", templateDir + "/header.html", templateDir + "/topnav.html", templateDir + "/rightnav.html"}
	return map[string]*template.Template{
		"home": template.Must(template.ParseFiles(append([]string{templateDir + "/home.html"}, globals...)...)),
		"blog": template.Must(template.ParseFiles(append([]string{templateDir + "/blog.html"}, globals...)...)),
	}
}

To load my blog posts, I really want to write them in markdown because it's just a comfortable workflow for me. So, here's what I ended up using:

// LoadPosts - Load markdown posts
func LoadPosts(contentDir string) map[string]string {
	// List of paths to load markdown from
	paths := []string{"blog_creation.md"}
	// Declare return data
	ret := make(map[string]string, 0)
	// Setup markdown parser
	extensions := parser.CommonExtensions | parser.AutoHeadingIDs
	parser := parser.NewWithExtensions(extensions)
	// Iterate over given paths
	for _, path := range paths {
		// Read from the file
		content, err := ioutil.ReadFile(contentDir + "/" + path)
		if err != nil {
			log.Fatal(err)
		}
		// Convert it to markdown
		html := markdown.ToHTML(content, parser, nil)
		// Strip .md from path
		name := strings.Replace(path, ".md", "", -1)
		ret[name] = string(html)
	}
	return ret
}

I'll likely come back to this in the future and add a more flexible way of loading these off disk and organizing them into memory, but for now this will get the job done! It just takes a list of strings that will point to my markdown documents, iterates through converting each one to a html string and dropping those into memory so I can serve them up later. Not the best code but I'm trying to be fast here. Now we just need to setup routing:

// RegisterHandlerFunctions - Handler functions
func RegisterHandlerFunctions(router *mux.Router, s *Server) {
	/* Define endpoint handlers */
	log.Debug("Configuring HTTP Routing functions")
	// Configure security headers at the routing top-level
	router = ConfigureSecurityHeaders(router, s)
	router.Use(handlers.CompressHandler)
	// Add non-authed routing functions to default router
	router.HandleFunc("/", HandleIndex(s))
	router.HandleFunc("/blogs/{name}", HandleBlogPage(s))
	router.HandleFunc("/health-check", HealthCheckHandler)
	// Configure CSRF (This has to be manualy validated on POST requests)
	csrf := nosurf.New(router)
	//csrf.ExemptRegexp("/*")
	csrf.SetFailureHandler(http.HandlerFunc(HandleCsrfFail))
	// Add router to root
	http.Handle("/", csrf)
}

Notice the router.HandleFunc("/blogs/{name}", HandleBlogPage(s)) which is going to pass forward the name of the blog to our handler! We also serve up static files and configure some security stuff that I'm going to gloss over for now and cover more in depth in the future as I pentest this site and box. All that's left is a few handler functions:

// HandleIndex - Handle home page
func HandleIndex(s *Server) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		switch r.Method {
		case "GET":
			// Start logging fields
			f := logrus.Fields{"Handler": "HandleIndex"}
			// Load template data
			data := struct {}{}
			// Execute template
			if err := s.Templates["home"].Execute(w, data); err != nil {
				s.Log.WithFields(f).Errorf("Error rendering projects template: %v", err)
			}
		default:
			s.Log.Debug("? @ HandleIndex")
			w.WriteHeader(http.StatusUnauthorized)
			return
		}
	}
}
// HandleBlogPage - Generic blog page rendering
func HandleBlogPage(s *Server) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		switch r.Method {
		case "GET":
			// Capture request vars
			// Careful to not inject this somehow onto the page and introduce XSS
			vars := mux.Vars(r)
			// Start request logging
			f := logrus.Fields{"Handler": "HandleBlogPage", "Name": vars["name"]}
			// Resolve blog path from name
			content := ""
			for path, html := range s.Blogs {
				if vars["url"] == path {
					content = html
				}
			}
			// If nothing is found, return 404
			if content == "" {
				w.WriteHeader(http.StatusNotFound)
				return
			}
			// Inject blog  HTML into blog template data
			data := struct {
				Content template.HTML
			}{
				Content: template.HTML(content),
			}
			// Respond and execute template
			if err := s.Templates["blog"].Execute(w, data); err != nil {
				s.Log.WithFields(f).Errorf("Error rendering projects template: %v", err)
			}
		default:
			s.Log.Debug("? @ HandleIndex")
			w.WriteHeader(http.StatusUnauthorized)
			return
		}
	}
}

Simple stuff, right?! Now let's get a simple dockerfile together and get this thing running!

# ---- Make Build environment -----
FROM golang:1.14.3-alpine3.11 as build
# Set workdir within gopath
WORKDIR /go/src/git.salmonsec.com/salmonsec
RUN apk update \
    && apk upgrade \
    && apk add --no-cache \
    git ca-certificates \
    && update-ca-certificates 2>/dev/null || true
# Add source code
COPY . .
# Install dependancies
RUN  go get ./...
# Run build script
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '"-X main.version=${TAG}" -extldflags "-static"'  -o main .
# ---- Bundle Production Container  ----
FROM alpine:3.11
# Expose port to webserver
EXPOSE 8080
# Copy built binary & static files
COPY --from=build /go/src/git.salmonsec.com/salmonsec/main .
COPY ./static ./static
COPY ./content ./content
COPY ./templates ./templates
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
# Cache timezone db
RUN apk add --no-cache tzdata
# Run as non-root user
RUN adduser -D notroot
USER notroot
# Expect mounted folder @ /env with .env file and cacert.pem
ENTRYPOINT ./main

Deploying

Not quite my intended goal, but I'm happy enough just building locally and pushing an image up, good enough for now!
Build and push the image to our DO repository:

brew install doctl
doctl auth init
docker build . -t salmonsec
docker tag salmonsec:latest registry.digitalocean.com/salmonsec/salmonsec:latest
docker push  registry.digitalocean.com/salmonsec/salmonsec:latest

For now, let's just get the thing running on the server and worry about tls and proper nginx setup later

ssh matt@salmonsec.com -i ~/.ssh/salmonsec
doctl auth init
docker run -d -p 80:8080 --name blog registry.digitalocean.com/salmonsec/salmonsec

Updating

The docs say that this thing just works like black magic, so let's try:

$ docker run -d \
  --name watchtower \
  -v /home/matt/.docker/config.json:/config.json \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower blog --debug
Unable to find image 'containrrr/watchtower:latest' locally
latest: Pulling from containrrr/watchtower
e42677bd61ba: Pull complete
f6f75156942d: Pull complete
7d65721eea54: Pull complete
Digest: sha256:b26dfbdda14acac2b5cc862691a0e8248f510a1671532b55dabb2a5231126800
Status: Downloaded newer image for containrrr/watchtower:latest
2bfa20dac2a5af95e56145d9502cc660acb28e6e904a1975eb78b1f71588aa03
$ docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS               NAMES
2bfa20dac2a5        containrrr/watchtower   "/watchtower"       3 seconds ago       Up 2 seconds        8080/tcp            watchtower

Make a change, rebuild, push it up again... waiting...

docker logs -f watchtower
time="2020-07-13T02:05:00Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2020-07-13T02:05:01Z" level=debug msg="Retrieving running containers"
time="2020-07-13T02:05:01Z" level=debug msg="There are no additional watchtower containers"
time="2020-07-13T02:05:01Z" level=info msg="Starting Watchtower and scheduling first run: 2020-07-13 02:10:01 +0000 UTC m=+300.184533820"
time="2020-07-13T02:10:01Z" level=debug msg="Checking containers for updated images"
time="2020-07-13T02:10:01Z" level=debug msg="Retrieving running containers"
time="2020-07-13T02:10:01Z" level=debug msg="Pulling registry.digitalocean.com/salmonsec/salmonsec:latest for /blog"
time="2020-07-13T02:10:01Z" level=debug msg="Loaded auth credentials for user ddcc7b4c2327b02966c04beeccca08c643d50ddc6602fab4b6c1cfad6f14f975, on registry registry.digitalocean.com/salmonsec/salmonsec:latest, from file /config.json"
time="2020-07-13T02:10:01Z" level=debug msg="Got image name: registry.digitalocean.com/salmonsec/salmonsec:latest"
time="2020-07-13T02:10:05Z" level=info msg="Found new registry.digitalocean.com/salmonsec/salmonsec:latest image (sha256:ac785c6f39a86b5bf8ffdf9d7c7b7039c488c0a480d6f578cb7e161c3f358251)"
time="2020-07-13T02:10:05Z" level=info msg="Stopping /blog (0e4a1315cf0c7df1fbebbeff4ae892bd19445db2274a801268f472bf5acd8e31) with SIGTERM"
time="2020-07-13T02:10:06Z" level=debug msg="Removing container 0e4a1315cf0c7df1fbebbeff4ae892bd19445db2274a801268f472bf5acd8e31"
time="2020-07-13T02:10:06Z" level=info msg="Creating /blog"
time="2020-07-13T02:10:06Z" level=debug msg="Starting container /blog (5be72b4ccd58095b0d98a7d7ec95936bec18536f0121e82400f568ccf578c1a2)"
time="2020-07-13T02:10:07Z" level=debug msg="Scheduled next run: 2020-07-13 02:15:01 +0000 UTC"

She worked!

TLS

Now, these days there's a hundred different ways to get a webserver properly handling TLS traffic. I'm not looking for anything very robust here, I don't care about downtime nobody is going to look at this site! So, I'm just going to install certbot on the server and mount certificates into the container!

apt install certbot
certbot certonly  --standalone
cp /etc/letsencrypt/archive/salmonsec.com ~/certs
docker run  -d -p 443:8080 --name blog -v /home/matt/certs:/certs registry.digitalocean.com/salmonsec/salmonsec

There we go, now we've got it all running and I can update it by pushing a new image to the DO Container registery, yay!