BOSS kuberneter cluster

To make hosting for free as easy as pie, BOSS has its own Kubernetes cluster. This is hosted on servers donated to us by the Department of Computer Science and is hosted within their server room in 1W. This allows us to host databases and any docker container for free, which all is configured within our terraform repo.

Deployment

If you are wanting to use this cluster for hosting projects, please get in contact with committee. However, this should be part of the process with adding a project.

When deploying, a module and related configuration will be added to our terraform repo, which then is deployed by a sysadmin to the staging cluster. Once we have validated that it works on staging, it will finally be deployed to production.

Machine Structure

The more technical detail about the cluster is stored within BOSS's internal wiki in GitLab and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.

Due to hw2210 bricking an SSD in 6 months on a 5 node cluster due to disk intensive operations with longhorn (a storage manager that shares drives across nodes), it was decided that BOSS would only host a 1 node cluster with K3S's basic storage manager to extend the life of their finite disks as much as possible.

From experience, it was also decided to would host the clusters in VMs with immutable distros to allow more flexibility when moving VMs between machines, and stability (if it continuously crashes, you can get access to the console without plugging into host).

ZFS was chosen due to extremely good data protection.

Therefore the final structure is:

• Production host machine running FreeBSD with ZFS
  • Production VM with CoreOS running K3S
• Staging host machine running FreeBSD with ZFS
  • Staging VM with CoreOS running K3S

The VMs themselves are then backed up to a backup server allowing for easy whole cluster reversions and recovery without any painful redeployment.

CoreOS

CoreOS was chosen as it is an immutable distro based off of Fedora with SELinux support and is extremely lightweight. When first booting up, it uses an ignition file to configure itself (including network access), the ignition files can be found in the ignition repo. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.

Immutable means that the root directory cannot be altered, only /var can be altered during the running of the distro. /etc can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in /var/home/<username> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now rpm-ostree install ....

Each alteration to the installed packages or /etc creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow. If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:

sudo ostree admin pin 0
rpm-ostree status # See pinned versions
sudo ostree reset 2 # Reset back to the pinned version

FreeBSD

FreeBSD was specifically chosen as it offers two benefits:

• The best support for ZFS on root (Linux doesn't like ZFS's license)
• Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).

ZFS was also chosen specifically because it has world-class data retention and backup tooling for RAID based systems, making it perfect for servers running with multiple large drives. We chose RAIDZ-1, which means one drive can fail and we can still recover all our data.

FreeBSD is like Linux in many ways as it is Unix-like and subsequently many linux tools exist on FreeBSD and can be installed. However there are minor cultural differences, such as sudo and systemd are disliked. Instead doas or mdo is recommended. Additionally VMs/containers are called jails and have many more useful features.

K3S

K3S is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the hardened configuration on a IPv4 only cluster (Bath sadly does not provide IPv6 support) as well as running with SELinux enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).

Helper services

Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the Kubernetes page. To help with this, the cluster runs a range of "helper" applications which help with deployment and add cool features to your application:

• Traefik - this is the http proxy software, managing routing traffic to the right application. This comes with the ability to create middlewares to filter traffic based off a range of things, and combined with authelia, allows adding a login page for applications which don't have one.
• Cert manager - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.
• Authelia - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through Bath's group manager.
• Docker mailserver - This is configured with SPS, DKIM and DMARC to allow us to send emails under bathcs.com for free and makes sure the emails are likely to actually reach the destination without going into spam.
• Cloud native's Postgres Operator - This is just a tool to easily deploy postgres databases within the cluster and manage them.
• Prometheus operator - This allows easy monitoring of the whole cluster and emails us if anything is going wrong
• Grafana - cool graphs right?
• LLDAP - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.
• K3S upgrades - keeps the cluster up to date
• k8up - Automatically backs up our critical Persistent Volumes to a Scaleway S3 bucket.
• Security Profile Operator - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)