View source for BOSS/Hosting/Cluster

To make hosting for free as easy as pie, [[Bath Open Source Society|BOSS]] has its own [[Kubernetes]] cluster. This is hosted on servers donated to us by the Department of Computer Science and is hosted within their server room in 1W. This allows us to host databases and any docker container for free, which all is configured within [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo].

== Deployment ==
If you are wanting to use this cluster for hosting projects, please get in contact with committee. However, this should be part of the process with adding a project.

When deploying, a module and related configuration will be added to [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo], which then is deployed by a sysadmin to the staging cluster. Once we have validated that it works on staging, it will finally be deployed to production.

== Machine Structure ==
{{Note|text=The more technical detail about the cluster is stored within [https://gitlab.bath.ac.uk/cs/boss/int-wiki BOSS's internal wiki in GitLab] and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.}}
Due to me (hw2210) bricking an SSD in 6 months on my own 5 node cluster due to disk intensive operations with [https://longhorn.io/ longhorn] (a storage manager that shares drives across nodes), it was decided that we would only host a 1 node cluster with K3S's basic storage manager to extend the life of our finite disks as much as possible.

From experience, it was also decided that we would host the clusters in VMs with immutable distro's to allow more flexibility with moving VMs between machines and stability (if it continuously crashes, we can get access to the console without plugging into host).

Therefore our final structure was:

* Production host machine running FreeBSD with ZFS (for data protection as ZFS is king)
** Production VM with CoreOS running K3S
* Staging host machine running FreeBSD with ZFS
** Staging VM with CoreOS running K3S

The VM's themselves are then backed up to a backup server allowing for easy whole cluster revertions and recovery without any painful redeployment.

=== CoreOS ===
CoreOS was chosen as it is an immutable distro based off of Fedora with [[SELinux]] support and is extremely lightweight. When first botting up, it uses an ignition file to configure itself (including network access), our ignition files can be found in [https://gitlab.bath.ac.uk/cs/int/ignition our ignition repo]. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.

By immutable, we mean that the root directory cannot be altered, only <code>/var</code> can be altered during the running of the distro. <code>/etc</code> can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in <code>/var/home/<username></code> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now <code>rpm-ostree install ...</code>.

Each alteration to the installed packages or <code>/etc</code> creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow.  If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:<syntaxhighlight lang="shell">
sudo ostree admin pin 0
rpm-ostree status # See pinned versions
sudo ostree reset 2 # Reset back to the pinned version
</syntaxhighlight>

=== FreeBSD ===
[[wikipedia:FreeBSD|FreeBSD]] was specifically chosen as it offers two benefits:

* The best support for [[wikipedia:ZFS|ZFS]] on root (Linux doesn't like ZFSes license)
* Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).

[[wikipedia:ZFS|ZFS]] was also chosen specifically because it has world-class data retention and backup tooling for RAID based systems, making it perfect for servers running with multiple large drives. We chose RAIDZ-1, which means one drive can fail and we can still recover all our data.

FreeBSD is like Linux in many ways as it is Unix-like and subsequently many linux tools exist on FreeBSD and can be installed. However there are minor cultural differences, such as <code>sudo</code> and <code>systemd</code> are disliked. Instead <code>doas</code> or <code>mdo</code> is recommended. Additionally VMs/containers are called jails and have many more useful features.

== K3S ==
[https://k3s.io K3S] is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the [https://docs.k3s.io/security/hardening-guide hardened configuration] on a IPv4 only cluster (Bath sadly does not provide an IPv6 support) as well as running with [[SELinux]] enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).

== Helper services ==
Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the [[Kubernetes|Kubernetes page]]. To help with this, we also deploy a range of "helper" applications which help with deployment and add cool features to your own application:

* [https://doc.traefik.io/traefik/ Traefik] - this is our http proxy software, managing routing traffic to the right application. This comes with the ability to [https://doc.traefik.io/traefik/expose/kubernetes/advanced/#create-middlewares create middlewares] to filter traffic based off a range of things, and combined with authelia, allows us to add a login page for applications which don't have one.
* [https://cert-manager.io/ Cert manager] - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.
* [https://www.authelia.com/ Authelia] - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through [https://www.bath.ac.uk/services/group-manager/ Bath's group manager].
* [https://docker-mailserver.github.io/docker-mailserver/latest/ Docker mailserver] - This is configured with SPS, DKIM and DMARC to allow us to send emails under <code>bathcs.com</code> for free and makes sure the emails are likely to actually reach the destination without going into spam.
* [https://cloudnative-pg.io/ Cloud native's Postgres Operator] - This is just a tool to easily deploy postgres databases within the cluster and manage them.
* [https://prometheus-operator.dev/ Prometheus operator] - This allows easy monitoring of the whole cluster and emails us if anything is going wrong
* [https://grafana.com/ Grafana] - cool graphs right?
* [https://github.com/lldap/lldap LLDAP] - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.
* [https://docs.k3s.io/upgrades/automated K3S upgrades] - keeps our cluster up to date
* [https://k8up.io/ k8up] - Automatically backs up our critical PVs to a Scaleway S3 bucket.
* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)