Kubernetes

This page is written to be as generic as possible, so if you are just experimenting with kubernetes on a home lab, please feel free to read and hopefully you will learn somethings. There will be multiple references to BOSS's cluster, which has all the security boxes ticked on and so we have to deal with security contexts, network policies and SELinux as they are the bain of all problems.

General knowledge

This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernetes documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

Pod vs Container

A common confusion is that pod's are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

Statefulset vs deployment

Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

If you update any "volume" attribute within a statefulset ALL volumes will be deleted and recreated. To not lose any data, please make sure that the PV's reclaim policy has been set to "Retain". You should generally do this for any data you do not want to lose.

Liveness/Startup probes

Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

Security Context

Within BOSS's kubernetes cluster, we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.

A security context defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

• Run as user
• Don't allow privilege escalation
• Properly define seccomp policy
• Default SELinux container context
• Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:

resource "kubernetes_deployment_v1" "my_deployment" {
  # ...
  spec {
    # ...
    template {
      # ...
      spec {
        container {
          # ...

          security_context {
            run_as_user                = 1000
            run_as_non_root            = true
            allow_privilege_escalation = false
            seccomp_profile {
              type = "RuntimeDefault"
            }
            capabilities {
              drop = ["ALL"]
            }
          }

          # ...
        }
      }
    }
  }
}

See Terraform for more information.

What is a CRD?

A Custom Resource Definition (CRD), allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

Traefik and gateways

A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.

Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the Gateway API, in which traefik implements.

The Gateway API resources are read by traefik, which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of ListenerSets but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (TLSRoute, HTTPRoute, GRPCRoutes and coming in the future TCPRoute and UDPRoute). These routes act as queries to determine when and what traefik should forward to. So for example they act as:

if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service

Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:

Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port

In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

Certificates

The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

• DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
• HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
• Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for Cloudflare proxy-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).

When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.

Cloudflare proxy

Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. k8s.bathcs.com) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

Network Policies

With additional security, comes additional policy managment. Network policies tell kubernetes where a pod is allows to send and receive traffic. However they are a bit confusing at times, and so can cause some headache when trying to debug why your pod cannot communicate with your database.

• By default ALL outgoing traffic is allows.
• By default NO incoming traffic is allows (except from traefik).

So by default, any pod is allows to communicate with any port on the internet, but not allows to communicate with any other pod in the whole cluster. In the futher we hope to disallow both by default, and so you will have to specify exactly what ports (and potentially where) your pod should be communicating.

Within the terraform, we have helper functions built into the utilities for the databases to automatically generate network policies for incoming traffic, relying on the requirement of adding a label to your pod, they are usually outputted by the module under client_labels.

To define your own policy (in terraform ofc), there are two parts ingress (incoming traffic) and egress (outgoing traffic). Both of these can then be a list of rules matching pods that will be allows. They are additive, meaning that all network policies matching the pod will be combined to produce the final ruleset.

The main rules you will want to focus on are:

• ip_block defines a lock of IPs where the traffic originates or is going to
• namespaceSelector defines the labels matching on the namespaces where traffic is allows to/from
• podSelector same as namespaces but specific to pods themselves (e.g. what our databases do)
• ports defines the ports allows by these connections

While defining any egress policies you must remember to include basic services, e.g. DNS and maybe NTP. An example full configuration might look like:

resource "kubernetes_network_policy_v1" "example" {
  metadata {
    name      = "my-policy"
    namespace = var.namespace
  }
  spec {
    egress {
      # Allow pod to use DNS
      ports {
        port     = 53
        protocol = "TCP"
      }
      ports {
        port     = 53
        protocol = "UDP"
      }
      # Allow pod to request its database
      ports {
        port     = 5432
        protocol = "TCP"
      }
    }
    ingress {
      # Allow pods within namespaces that have enabled ldap (with the defined
      # label) to request this pod via the "ldap" port
      ports {
        port     = "ldap"
        protocol = "TCP"
      }
      from {
        namespace_selector {
          match_labels = {
            allow_ldap = true
          }
        }
      }
    }
    # This defines the pod that this network policy will be applied to.
    pod_selector {
      match_labels = {
        app = "affected-pod"
      }
    }
    # We have defined both Ingress and Egress rules for this pod
    policy_types = ["Ingress", "Egress"]
  }
}

This shows an example configuration, allows the defined pod to communicate with a database (note that the database will also be required to have a network policy with an ingress rule allowing the pod to connect) and DNS and allow other pods in the cluster to communciate with it. Note that any http port is missing as traffic is automatically permitted to access any port within the cluster.

Helm

Helm is a tool which allows the deployment of a set of kubernetes resources from a single configuration. So for third party, complicated applications its amazing, and there are a lot hosted on artifacthub and other random places (as you can really easily host a helm repo for free). There are benefits, like being able to rollback a change to a previous version. However, there are a few things to note:

• When configuring in terraform, do NOT add repos! The full url should go in the repo config, or if its an oci url, it should go in the chart attribute.
• Tofu will refuse to deploy a deploying helm chart (you have to rollback first)
• DO NOT STORE SECRETS IN VALUES - values are not stored securely, and so you should never store passwords or api keys directly in the values (this is an easy mistake to make when configuring), all previous sets of values will be stored forever in the cluster. Therefore, if you make this mistake, you will have to rotate the secret or delete the whole helm deployment and start again.
• Helm does not care if resources are changed between deployments - this is both good and bad, it means that you can apply "hacks" to helm charts you know will not change and they will not appear in the terraform plans to be fixed. But again this is not particularly good practice and can result in some confusing behaviour.
• When you delete a deployment, all persistant volumes will get wiped unless they have the "Retain" reclaim policy.

Helm is a great tool for quickly deploying whole clusters of applications, but it should be used with caution, making sure the chart is reputable and well maintained. As we are using terraform for deployments, it should also only be used for third party applications.

Persistant Volumes

Persistant volumes (PVs) and persistant volume claims (PVCs) are one of the more convoluted things in kubernetes and one of the more dangerous as you are handling data.

At a high level, you request storage by creating a persistant volume claim, the storage manager fullfills your claim by creating a persistant volume (which does not have an associated namespace) and then it is assigned to your pod. You should never be creating persistant volumes yourself, and probably want to be creating persistant volumes through the statefulset's template interface.

For us, the storage operator is k3s' built in one, but for clusters with multiple pods, it probably is going to be something like longhorn, which manages keeping replicas of the storage on multiple machines. This brings up a core issue, a persistant volume can only be mounted to a pod on a node that stores the persistant volume, which is why longhorn is necessary on multi-node clusters and is why we decided to only have one node in each of our clusters.

PVs have different types of access modes: ReadWriteOnce, ReadWriteMany, ReadOnlyMany which mostly control how many nodes can read or write to it at once (multiple pods can still mount it, as long as they are all on the same node). So for k3s' storage class does not support ReadWriteMany or ReadOnlyMany, so you should only be setting it to ReadWriteOnce.

Reclaiming PVs is one of the big danger factors. By default a persistant volume will be deleted if there are no longer any claims for it, and the claims will probably get deleted (e.g. in helm and statefulsets). Therefore, if you have important data on a pod, you most likely want to set the reclaim policy, that means the volume cannot be automatically deleted, but it also means, if another PVC comes along and matches the PV, the PV will be assigned to the new claim and subsequently a new pod, which can cause a lot of confusion and headache. But it is more likely that the PVC is the redeployment of the original application and so you want it to be reassigned to the new PVC.

If you delete an application, you MUST delete the PV manually when its reclaim policy is "Retain", otherwise the data will never get deleted and could be reassigned to another claim.

Other volumes

Persistant volumes are probably the easiest to understand, but there are a lot of other types of volumes, most notably empty directorys and ephemoral volumes as well as volumes creating from config maps or secrets. All of these serve different purposes:

• Empty directory - something like /tmp, all data will be stored in RAM if set to writable and will be deleted when the pod restarts or is deleted. You should set a limit to how much RAM is allowed to be stored there, as it could soak up the full resources of the computer.
• Emphemeral volumes - similar to empty directorys, but instead of being stored in RAM, they are stored on disk (through persistant volumes) and will be deleted upon deletion of the pod and so mostly not really useful unless you are expecting to generate a lot of data during the running of the pod and don't want to retain them on reboot.
• Mounting secrets - this is read only storage and just allows you to mount configuration files that store database passwords or similar. Each key within the secret becomes a text file. Note that if its a binary file (e.g. image) you can use the binaryData property and pass in a hex string.
• Mounting config maps - similar to secrets, but just are for files that don't have to be secret and encrypted on the machine.

Deployment with Tofu

Information pertaining to terraform configuration itself can be found on the Terraform page and for specific how to deploy to BOSS' production and staging cluster, please see the project's README and internal wiki for more information. This will mostly focus on the general process of deploying with tofu and the struggles.

Please read the K9s section for information about using k9s for monitoring deployments.

Some resources may also have their own special command set to allow you to perform resource specific actions which is nice (these should be listed at the top right).

Back to deployment, it's mostly:

tofu apply -var-file=./prod.tfvars [-target=module.something]

Using -target can help a lot when the terraform module is huge as it will limit the number of resources it has to check for updates (though this means that you state and configuration can become desynced and so you should still do apply's without the targetting). You can even specify the exact resource you have just edited for super fast (for terraform) iterations.

While it's deploying you cannot hit Ctrl-c or well you can (twice) to force exit the application, but this can result in annoying consequencies:

• Hitting it before you've confirmed the plan (e.g. you accidentally forgot to include target and don't want to wait): The state WILL be locked and will not be unlocked. Therefore you have to run

  tofu force-unlock <uid>
  

  Where the <uid> can be found by trying to run the apply command and it failing.

• Hitting it while it's deploying new resources (e.g. the pods are not deploying and you didn't change the 5 minute timeout): on next run, tofu will try to recreate those resources, and so before running the command, you must open k9s, find the resources and delete them.
• Hitting it while updating helm charts (e.g. you didn't change the 10 minute timeout and it's not working): on the next run, tofu will refuse to deploy it, because the helm deployment is in an invalid state ("deploying") and it will never exit this state. Therefore you must open up k9s type helm <namespace>, wait a year, click r on the effected resource to list all the previous releases, go down to the last successful deployment and click r again to rollback to that release, and again wait a year while k9s refuses to respond.

As you can probably tell deployment takes patience, expecially when you don't alter the timeouts in helm (which default to 10 minutes for creation and 10 minutes for deletion). You don't want to interrupt the flow while tofu is doing it's thing, so make sure you aren't going to have to wait 10 minutes because you made a simple typo causing the pods to crash loop.

You can technically, temporarily edit the resources (with e in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

If updating config maps/secrets, you will have to manually restart affected pods for the changes to apply in some cases!

Retaining PVs

As previously mentioned, whenever deploying new applications, you should mark any persistant volume that stores data you do not want accidentally deleted with a "Retain" reclaim policy. This can easily be done using k9s by typing :pv, finding the pv attached to the claim, pressing e and searching for where the reclaimPolicy is defined. By default it will be "Delete", and you can just replace it with "Retain". k9s will also show you a PV's reclaim policy on the list view which is nice to check if its already been done.

Where is the state stored?

State files store all passwords and secrets generated or collected by terraform unencrypted by default and so you should be extremely cautious with where the files are stored, who has access to them and never commit them.

Usually, if you don't define any backends, it is stored in a .tfstate file within the directory of your folder. If you are just managing your cluster with them, an easy place to store it is within the kubernetes cluster itself, using the kubernetes backend:

terraform {
  # ...

  backend "kubernetes" {
    secret_suffix  = "my-cluster"
    config_paths   = [var.kube_config_path]
    config_context = var.kube_context
  }

  # ...
}

This is BOSS's method of storage as we do not have to trust the GitLab instance with our cluster's life. But others can be chosen, for example within GitLab itself, although this should be done with extreme caution due to the risk of exposing plan files (which contain secrets) to the world.

Managing the state

As we can never write perfect code the first time round, some MRs will result in refactorisation of components into new modules. Therefore state management is important. This annoyingly is painful especially with larger effected packages and so I would recommend considering the effects of destroying all the effected resources and reapplying them (which is what tofu apply ... will do by default). If you cannot afford this (e.g. with persistant volumes) you will have to use tofu state mv which works by moving the specified address to another (make sure there is no typos though!

However this requires the resource type to be the same through the move, which sometimes may not happen. At which point you'll want to delete the state information and re-import it. E.g.

tofu state rm module.example.my_resource.name
# Import the module (Note: the format will differ depending on the provider)
tofu import module.example.module.refractorisation.my_new_resource.name my_resource/name

Updating packages

To make upgrading easier, in submodules, packages are pinned to the nearest major version. You can then update the base package version and run:

tofu init -upgrade

Which will update the package versions on your machine and update the lock file.

If you do not specify -upgrade it will just update the module list (e.g. if you add a new application with a new use of module).

K9s

k9s is recommended for watching deployments and seeing why they failed as its generally just fantastic the more you get used to it. The common shortcuts you need to know are:

• d will describe the currently selected object. If you go to the bottom (Shift-G) you will be able to see events associated with that resource, for pods and statefulsets this is extremely useful as it includes failures for pulling images or security denials.
• l see the logs of all containers associated with the resource.
• x while within a secret/configmap will show you the raw (string) data
• s to shell into a pod (note that some pods do not support this as they do not ship withsh
• r allows you to perform a rolling restart on a statefulset or deployment (meaning there should be no downtime if everything is configured). This is the recommended way to update a pod on production (do not just delete the pod itself).
• <enter> see any subresources (e.g. containers for pods, or pods for deployments/statefulsets)
• Ctrl-d allows you to delete the resource
• Ctrl-f allows you to add a port-forward to your own machine (really useful for debugging if an application is responding but is not available via traefik), or getting direct access to databases.
• :<resource> <namespace|all> will change the list view to the given resource in the namespace
• :<resource> will change the list view for the given resource in the currently selected namespace
• 0-9 are shortcuts to switch between recently selected namespaces (they should show at the top with the current assignment)

Migrating storages

When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so.

All pods using these volumes should be inaccessible during the process of this. This can be done by temporarily deleting any HTTPRoutes to the service.

Is it a database?

Databases have users and passwords associated, which can cause issues if you are redeploying a whole cluster with terraform as new passwords will be generated. You can import new values but that's boring. Instead you can just us pg_dump (or equivalent):

kubectl port-forward -n my_namespace service/my_pod 5432:5432 &
pg_dump $DATABASE_URL > my_pod_bkp.sql
killall kubectl # Stop port-forward
# ...
# Apply the SQL to the volume in the new cluster:
kubectl --context new_cluster port-forward -n my_namespace service/my_port 5432:5432 &
psql -U username -d myDataBase -a -f my_pod_bkp.sql
killall kubectl # Stop port-forward

Anything else

You will need to investigate permissions and such, but kubectl has this cp command which you can use to download whole directories. But before that you should run ls -al while ssh-ed into the machine using the volume to view the permissions of the files.

kubectl cp my_namespace/pod:/where/is/pv/mounted bkp_dir
# ...
kubectl --context new_cluster cp bkp_dir my_namespace/pod:/where/is/pv/mounted

If the pod cannot run without the storage being set. You have two options: set it with basic data to be overridden, or create a dummy pod with the PV mounted and copy it to there.

Remember to mark any new PVs that should retain data with correct reclaim policy.

Multinode clusters

BOSS has specifically chosen not to use multinode clusters and so this is just for the interested people, and hopefully explain why we did not go down this route. Yes, multinode clusters are the whole point of kubernetes, where the program chooses the at least somewhat optimal location to deploy an application and automatically move it to another node if the one its on crashes or restarts for an update. But there are also benefits of kubernetes outside of this feature, such as proper permission system as well as providing a somewhat clean interface for managing deployments.

Additionally, we felt that as a lot of companies are moving towards managing deployments, at least partially, with kubernetes, our use of it will hopefully provide easy place for students to experiment and learn about it (as this is stuff university will never teach you). It should be noted that if companies actually cared about getting the most "bang for your buck", it is worth more to manage deployments yourself and choose which nodes host them, though this can produce more overhead.

Benefits

There are clear benefits of running a multinode cluster, node down? That's fine, all applications can still run just fine and are accessible. Updating your entrypoint? That's fine, just point your assumingly IPv4 to another node, or if you're using IPv6 just setup DNS to point to both nodes. With my multinode cluster, I am writing this currently after one node completely bricked itself and the other node down after I completely bricked it and all applications are still running fine on the three remaining nodes.

Downsides

With all this new power comes significant overhead though. For one, if a node goes down its not always as simple as kubernetes automatically moving things. Firstly, volumes may only be stored on that one node, then they may only be locked to that one node with ReadWriteOnce. So if the machine didn't shutdown properly (or if one just get's disconnected from the control plane) you will have to make intervention steps as soon as possible, though, you don't have to be in a panic to get the node itself back up. If you have correctly setup safe shutdowns and its a routine update though, you probably don't need to touch it.

Power, is another one, the more nodes, the more your control plane has to manage, meaning the more its going to be stuck managing the cluster, and so will not be able to run their own pods. This means that going from a single node cluster to a multinode cluster, you will probably need at least 3-4 additional machines for it to be managed and working properly. Yes you could run a 2 or 3 node cluster, but the benefits really are not there for the massive amount of downtime. On top of this, you probably want at least 1 node of wiggleroom capable of handling your highest power node, this means that you can never reach the full potential of your cluster, as both your control plane and your other nodes must be able to take the strain when that node is disconnected.

Storage is probably the largest issue that you will need to tackle, and is the one that put BOSS of going this route (along with the fact that we wanted to keep some of our servers as hot spares if parts broke). This will be explored more in the "How to manage Storage" section, but the summary is, although there are tools out there, they require a minimum node count of 3 and have significant overhead. Additionally, because of the potential constant movement of data between nodes going down (even for general updates), it causes additional wear and tear on your drives, reducing their life expectancy (I killed a brand-new SSD in 6 months seemingly because my nodes were going down for updates so often).

Networking is a minor one, all nodes need to be connected to control plane and cannot get randomly interrupted or cut off (e.g. you have your control plane in one room hooked up to a UPS and a agent node in another on a UPS, but the router that connects them is not on the UPS). Disconnection at critical times can cause the control node to start moving workloads off the disconnected node onto others, which with statefulsets or ReadWriteMany volumes can cause desynchronisation which is a massive issue and will cause corruption. Finally there is just a minor issue that if you don't setup correctly you will probably be quite confused, Traefik forwarding going between nodes may result in the final application being given the wrong IP address.

Finally, there is also a bit of a weird one, managing SELinux policies and kernel modules across the cluster. I have had multiple issues in the past of minor configuration differences between my nodes resulting in a pod crashlooping on one node and absolutely just fine on another. Therefore this creates additional overhead as you then have to make sure all installs are identical and then potentially install the security profile operator (which is great when it works but not so great when the docker container they use is ancient and doesn't work on the latest systems and k3s accidentally breaks support for it).

And so, in summary, multinode clusters are amazing for chaising that 99.99999999999% uptime and really interesting for learning the difficulties with it and all the solutions out there. But it is really not for the faint of heart, I would recommend trying with a bunch of VMs on a single node just to understand how it works, but if you are just a hobbiest wanting to host some basic applications at home, do not use it. I am planning on deconstructing my cluster and replacing it with a single node similar to BOSS after I leave univesity.

Control plane

The first challenge is the control plane. What's the benefit of a multinode cluster when it relys on a single node for sending control signals. If it goes down, the whole cluster will panic if a pod crashes, it is left in a seriously vulnerable position. Therefore, you instead need to expand your control plane to multiple nodes. K3s offers built in solution for this, utilising etcd by default to share the cluster state between control nodes, or you can just move it off to a separate database cluster (which is what's recommended in production as you probably have another spare 3 machines). Then you must setup a load balancer between the control nodes (which actually helps with not overloading a single control node). But now the issue is that you are relying on a single load balancer. As a hacky solution, I setup a load balancer on all my control nodes which balanced the load between itself and the other nodes. This mean that the DNS record could store all IPs associated with the nodes and hopefully mitigate any communication issues when a single one went down.

However, when setting up etcd, you must remember to point the --server to the load balancer and not a single node (as then the single node becomes the single point of failure).

How to manage Storage

I personally used longhorn, which provided a somewhat easy solution, you just have to change the default storage class and migrate all your PVs. It requires that you have 3 nodes to share data between (as it can then use voting logic to ward off corruption). By default, it then stores volumes on three different nodes at the same time, meaning if one node goes down, it will just make a copy to a new node with the remaining two copies. Additionally, if a pod is assigned to a node without a copy of the data, longhorn can simply make a new copy onto that node (however it is more likely that kubernetes will choose a node that already has the storage).

It then provides a UI where you can easily manually update all the storage volumes after an update of longhorn and also manage automated backups and snapshots. It is actually a really handy tool and generally really cool (though you have to manually label PVs with the types of backups you want).

The issue? It uses around 2GiB of RAM on each individual node. Then if your nodes are constantly going down (e.g. automated updates every week), it will instantly start panicking and moving volumes over which can wear out drives. Additionally, if too many nodes go down, it can be the single source of crashing everything else because of the amount of resources it can use.

Safely shutting down nodes

This is basically a requirement if you actually want to obtain the full benefits of multinode cluster that requires as little manual intervention as possible (which is still somehow more intervention than a single node cluster IMO). There is a great blog on oranki.net on how to set this up, but effectively you need to set that anytime the k3s service is stopped you run:

kubectl drain --ignore-daemonsets --delete-emptydir-data <node>

Then when it comes back up you need to run:

kubectl uncordon <node>

As explained in the blog post, you can do this via a systemctl service.

What this does is makes sure all nodes are safety migrated onto another node before a shutdown can take place, effectively tainting the node (meaning can be assigned to it). Then you must uncordon a node (meaning removing the taint) to tell the control plane that you can now place pods on it again.

Backups

Within kubernetes, I'd say there are 3 types of backups:

• Whole OS/VM backups (really easy if host is running ZFS, but large)
• Resource configuration backup (saves time but if you are using terraform, not particularly useful): velero
• Persistant volume backup (small and application specific): built into some storage classes (e.g. longhorn) or you can use k8up

All of which you may want to use, however because I velero isn't amazing (It fails to back up PVs and the helm chart depends on paid pods, I prefer mixing a whole VM backup with ZFS snapshots and replication, and then do per volume backups with k8up for really important data to offsite cloud S3 bucket.

Alerting

Alerting is really important, for this I would simply just point to the prometheus operator or the kube-prometheus-stack, as these provide really easy methods to automatically monitor the clusters and send emails when something might be going wrong (but it can be extremely noisy) and node exporter requires additional configuration for SELinux enabled systems. This provides a nice standardised system that a lot of helm charts implement (and offer grafana boards you can import).

When you get an email, you should investigate what the error actual means and what are the recommended steps to fix. E.g. if a job fails, you can either delete it or rerun it until it works again. For CPUThrottlingHigh errors, I still have absolutely no clue what you are supposed to do about it other than increasing the permitted resources for a pod (which isn't recommended).