BOSS kuberneter cluster - Revision history

Pm2022: Pm2022 moved page BOSS/Hosting/Cluster to BOSS kuberneter cluster

2026-08-26T08:06:28Z

Pm2022 moved page BOSS/Hosting/Cluster to BOSS kuberneter cluster

← Older revision	Revision as of 08:06, 26 August 2026
(No difference)

Pm2022: reduce first person

2026-08-26T08:06:06Z

reduce first person

Pm2022 at 07:54, 26 August 2026

2026-08-26T07:54:09Z

← Older revision		Revision as of 07:54, 26 August 2026
Line 59:		Line 59:
	* [https://k8up.io/ k8up] - Automatically backs up our critical PVs to a Scaleway S3 bucket.		* [https://k8up.io/ k8up] - Automatically backs up our critical PVs to a Scaleway S3 bucket.
	* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)		* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)

			[[Category:BOSS hosting methods]]
			[[Category:Bath Open Source Society]]

Hw2210: Add information about BOSS's kubernetes cluster

2026-06-04T10:20:55Z

Add information about BOSS's kubernetes cluster

New page

To make hosting for free as easy as pie, [[Bath Open Source Society|BOSS]] has its own [[Kubernetes]] cluster. This is hosted on servers donated to us by the Department of Computer Science and is hosted within their server room in 1W. This allows us to host databases and any docker container for free, which all is configured within [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo].

== Deployment ==
If you are wanting to use this cluster for hosting projects, please get in contact with committee. However, this should be part of the process with adding a project.

When deploying, a module and related configuration will be added to [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo], which then is deployed by a sysadmin to the staging cluster. Once we have validated that it works on staging, it will finally be deployed to production.

== Machine Structure ==
{{Note|text=The more technical detail about the cluster is stored within [https://gitlab.bath.ac.uk/cs/boss/int-wiki BOSS's internal wiki in GitLab] and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.}}
Due to me (hw2210) bricking an SSD in 6 months on my own 5 node cluster due to disk intensive operations with [https://longhorn.io/ longhorn] (a storage manager that shares drives across nodes), it was decided that we would only host a 1 node cluster with K3S's basic storage manager to extend the life of our finite disks as much as possible.

From experience, it was also decided that we would host the clusters in VMs with immutable distro's to allow more flexibility with moving VMs between machines and stability (if it continuously crashes, we can get access to the console without plugging into host).

Therefore our final structure was:

* Production host machine running FreeBSD with ZFS (for data protection as ZFS is king)
** Production VM with CoreOS running K3S
* Staging host machine running FreeBSD with ZFS
** Staging VM with CoreOS running K3S

The VM's themselves are then backed up to a backup server allowing for easy whole cluster revertions and recovery without any painful redeployment.

=== CoreOS ===
CoreOS was chosen as it is an immutable distro based off of Fedora with [[SELinux]] support and is extremely lightweight. When first botting up, it uses an ignition file to configure itself (including network access), our ignition files can be found in [https://gitlab.bath.ac.uk/cs/int/ignition our ignition repo]. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.

By immutable, we mean that the root directory cannot be altered, only <code>/var</code> can be altered during the running of the distro. <code>/etc</code> can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in <code>/var/home/<username></code> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now <code>rpm-ostree install ...</code>.

Each alteration to the installed packages or <code>/etc</code> creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow. If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:<syntaxhighlight lang="shell">
sudo ostree admin pin 0
rpm-ostree status # See pinned versions
sudo ostree reset 2 # Reset back to the pinned version
</syntaxhighlight>

=== FreeBSD ===
[[wikipedia:FreeBSD|FreeBSD]] was specifically chosen as it offers two benefits:

* The best support for [[wikipedia:ZFS|ZFS]] on root (Linux doesn't like ZFSes license)
* Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).

[[wikipedia:ZFS|ZFS]] was also chosen specifically because it has world-class data retention and backup tooling for RAID based systems, making it perfect for servers running with multiple large drives. We chose RAIDZ-1, which means one drive can fail and we can still recover all our data.

FreeBSD is like Linux in many ways as it is Unix-like and subsequently many linux tools exist on FreeBSD and can be installed. However there are minor cultural differences, such as <code>sudo</code> and <code>systemd</code> are disliked. Instead <code>doas</code> or <code>mdo</code> is recommended. Additionally VMs/containers are called jails and have many more useful features.

== K3S ==
[https://k3s.io K3S] is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the [https://docs.k3s.io/security/hardening-guide hardened configuration] on a IPv4 only cluster (Bath sadly does not provide an IPv6 support) as well as running with [[SELinux]] enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).

== Helper services ==
Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the [[Kubernetes|Kubernetes page]]. To help with this, we also deploy a range of "helper" applications which help with deployment and add cool features to your own application:

* [https://doc.traefik.io/traefik/ Traefik] - this is our http proxy software, managing routing traffic to the right application. This comes with the ability to [https://doc.traefik.io/traefik/expose/kubernetes/advanced/#create-middlewares create middlewares] to filter traffic based off a range of things, and combined with authelia, allows us to add a login page for applications which don't have one.
* [https://cert-manager.io/ Cert manager] - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.
* [https://www.authelia.com/ Authelia] - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through [https://www.bath.ac.uk/services/group-manager/ Bath's group manager].
* [https://docker-mailserver.github.io/docker-mailserver/latest/ Docker mailserver] - This is configured with SPS, DKIM and DMARC to allow us to send emails under <code>bathcs.com</code> for free and makes sure the emails are likely to actually reach the destination without going into spam.
* [https://cloudnative-pg.io/ Cloud native's Postgres Operator] - This is just a tool to easily deploy postgres databases within the cluster and manage them.
* [https://prometheus-operator.dev/ Prometheus operator] - This allows easy monitoring of the whole cluster and emails us if anything is going wrong
* [https://grafana.com/ Grafana] - cool graphs right?
* [https://github.com/lldap/lldap LLDAP] - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.
* [https://docs.k3s.io/upgrades/automated K3S upgrades] - keeps our cluster up to date
* [https://k8up.io/ k8up] - Automatically backs up our critical PVs to a Scaleway S3 bucket.
* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)

← Older revision		Revision as of 08:06, 26 August 2026
Line 8:		Line 8:
	== Machine Structure ==		== Machine Structure ==
	{{Note\|text=The more technical detail about the cluster is stored within [https://gitlab.bath.ac.uk/cs/boss/int-wiki BOSS's internal wiki in GitLab] and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.}}		{{Note\|text=The more technical detail about the cluster is stored within [https://gitlab.bath.ac.uk/cs/boss/int-wiki BOSS's internal wiki in GitLab] and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.}}
	Due to ~~me (~~hw2210) bricking an SSD in 6 months on ~~my own~~ 5 node cluster due to disk intensive operations with [https://longhorn.io/ longhorn] (a storage manager that shares drives across nodes), it was decided that we would only host a 1 node cluster with K3S's basic storage manager to extend the life of ~~our~~ finite disks as much as possible.		Due to [[User:Hw2210\|hw2210]] bricking an SSD in 6 months on a 5 node cluster due to disk intensive operations with [https://longhorn.io/ longhorn] (a storage manager that shares drives across nodes), it was decided that BOSS would only host a 1 node cluster with K3S's basic storage manager to extend the life of their finite disks as much as possible.

	From experience, it was also decided ~~that we~~ would host the clusters in VMs with immutable ~~distro's~~ to allow more flexibility ~~with~~ moving VMs between machines and stability (if it continuously crashes, we can get access to the console without plugging into host).		From experience, it was also decided to would host the clusters in VMs with immutable distros to allow more flexibility when moving VMs between machines, and stability (if it continuously crashes, you can get access to the console without plugging into host).

	~~Therefore our final structure~~ was:		ZFS was chosen due to extremely good data protection.

	* Production host machine running FreeBSD with ZFS ~~(for data protection as ZFS is king)~~		Therefore the final structure is:

			* Production host machine running FreeBSD with ZFS
	** Production VM with CoreOS running K3S		** Production VM with CoreOS running K3S
	* Staging host machine running FreeBSD with ZFS		* Staging host machine running FreeBSD with ZFS
	** Staging VM with CoreOS running K3S		** Staging VM with CoreOS running K3S

	The ~~VM's~~ themselves are then backed up to a backup server allowing for easy whole cluster ~~revertions~~ and recovery without any painful redeployment.		The VMs themselves are then backed up to a backup server allowing for easy whole cluster reversions and recovery without any painful redeployment.

	=== CoreOS ===		=== CoreOS ===
	CoreOS was chosen as it is an immutable distro based off of Fedora with [[SELinux]] support and is extremely lightweight. When first ~~botting~~ up, it uses an ignition file to configure itself (including network access), ~~our~~ ignition files can be found in [https://gitlab.bath.ac.uk/cs/int/ignition ~~our~~ ignition repo]. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.		CoreOS was chosen as it is an immutable distro based off of Fedora with [[SELinux]] support and is extremely lightweight. When first booting up, it uses an ignition file to configure itself (including network access), the ignition files can be found in [https://gitlab.bath.ac.uk/cs/int/ignition the ignition repo]. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.

	~~By immutable, we mean~~ that the root directory cannot be altered, only <code>/var</code> can be altered during the running of the distro. <code>/etc</code> can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in <code>/var/home/<username></code> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now <code>rpm-ostree install ...</code>.		Immutable means that the root directory cannot be altered, only <code>/var</code> can be altered during the running of the distro. <code>/etc</code> can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in <code>/var/home/<username></code> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now <code>rpm-ostree install ...</code>.

	Each alteration to the installed packages or <code>/etc</code> creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow. If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:<syntaxhighlight lang="shell">		Each alteration to the installed packages or <code>/etc</code> creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow. If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:<syntaxhighlight lang="shell">
Line 35:		Line 37:
	[[wikipedia:FreeBSD\|FreeBSD]] was specifically chosen as it offers two benefits:		[[wikipedia:FreeBSD\|FreeBSD]] was specifically chosen as it offers two benefits:

	* The best support for [[wikipedia:ZFS\|ZFS]] on root (Linux doesn't like ~~ZFSes~~ license)		* The best support for [[wikipedia:ZFS\|ZFS]] on root (Linux doesn't like ZFS's license)
	* Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).		* Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).

Line 43:		Line 45:

	== K3S ==		== K3S ==
	[https://k3s.io K3S] is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the [https://docs.k3s.io/security/hardening-guide hardened configuration] on a IPv4 only cluster (Bath sadly does not provide an IPv6 support) as well as running with [[SELinux]] enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).		[https://k3s.io K3S] is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the [https://docs.k3s.io/security/hardening-guide hardened configuration] on a IPv4 only cluster (Bath sadly does not provide IPv6 support) as well as running with SELinux enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).

	== Helper services ==		== Helper services ==
	Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the [[Kubernetes\|Kubernetes page]]. To help with this, ~~we also deploy~~ a range of "helper" applications which help with deployment and add cool features to your ~~own~~ application:		Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the [[Kubernetes\|Kubernetes page]]. To help with this, the cluster runs a range of "helper" applications which help with deployment and add cool features to your application:

	* [https://doc.traefik.io/traefik/ Traefik] - this is ~~our~~ http proxy software, managing routing traffic to the right application. This comes with the ability to [https://doc.traefik.io/traefik/expose/kubernetes/advanced/#create-middlewares create middlewares] to filter traffic based off a range of things, and combined with authelia, allows ~~us to add~~ a login page for applications which don't have one.		* [https://doc.traefik.io/traefik/ Traefik] - this is the http proxy software, managing routing traffic to the right application. This comes with the ability to [https://doc.traefik.io/traefik/expose/kubernetes/advanced/#create-middlewares create middlewares] to filter traffic based off a range of things, and combined with authelia, allows adding a login page for applications which don't have one.
	* [https://cert-manager.io/ Cert manager] - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.		* [https://cert-manager.io/ Cert manager] - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.
	* [https://www.authelia.com/ Authelia] - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through [https://www.bath.ac.uk/services/group-manager/ Bath's group manager].		* [https://www.authelia.com/ Authelia] - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through [https://www.bath.ac.uk/services/group-manager/ Bath's group manager].
Line 56:		Line 58:
	* [https://grafana.com/ Grafana] - cool graphs right?		* [https://grafana.com/ Grafana] - cool graphs right?
	* [https://github.com/lldap/lldap LLDAP] - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.		* [https://github.com/lldap/lldap LLDAP] - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.
	* [https://docs.k3s.io/upgrades/automated K3S upgrades] - keeps ~~our~~ cluster up to date		* [https://docs.k3s.io/upgrades/automated K3S upgrades] - keeps the cluster up to date
	* [https://k8up.io/ k8up] - Automatically backs up our critical ~~PVs~~ to a Scaleway S3 bucket.		* [https://k8up.io/ k8up] - Automatically backs up our critical Persistent Volumes to a Scaleway S3 bucket.
	* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)		* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)

	[[Category:BOSS hosting methods]]		[[Category:BOSS hosting methods]]
	[[Category:Bath Open Source Society]]		[[Category:Bath Open Source Society]]