Kubernetes

I (hw2210) have been asked to write down my processes for working with kubernetes and terraform as I am leaving this year, and hopefully it will serve as useful information to any future sysadmin. This page is the output of this and is written to be as generic as possible, so if you are just experimenting with kubernetes on a home lab, please feel free to read and hopefully you will learn somethings. There will be multiple references to BOSS's cluster, which has all the security boxes ticked on and so we have to deal with security contexts, network policies and SELinux as they are the bain of all problems.

General knowledge

This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

Pod vs Container

A common confusion is that pod's are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

Statefulset vs deployment

Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

If you update any "volume" attribute within a statefulset ALL volumes will be deleted and recreated. To not lose any data, please make sure that the PV's reclaim policy has been set to "Retain". You should generally do this for any data you do not want to lose.

Liveness/Startup probes

Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

Security Context

Within BOSS's kubernetes cluster, we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.

A security context defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

• Run as user
• Don't allow privilege escalation
• Properly define seccomp policy
• Default SELinux container context
• Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:

resource "kubernetes_deployment_v1" "my_deployment" {
  # ...
  spec {
    # ...
    template {
      # ...
      spec {
        container {
          # ...

          security_context {
            run_as_user                = 1000
            run_as_non_root            = true
            allow_privilege_escalation = false
            seccomp_profile {
              type = "RuntimeDefault"
            }
            capabilities {
              drop = ["ALL"]
            }
          }

          # ...
        }
      }
    }
  }
}

See Terraform for more information.

What is a CRD?

A Custom Resource Definition (CRD), allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

Traefik and gateways

A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.

Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the Gateway API, in which traefik implements.

The Gateway API resources are read by traefik, which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of ListenerSets but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (TLSRoute, HTTPRoute, GRPCRoutes and coming in the future TCPRoute and UDPRoute). These routes act as queries to determine when and what traefik should forward to. So for example they act as:

if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service

Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:

Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port

In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

Certificates

The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

• DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
• HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
• Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for Cloudflare proxy-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).

When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.

Cloudflare proxy

Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. k8s.bathcs.com) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

Network Policies

With additional security, comes additional policy managment. Network policies tell kubernetes where a pod is allows to send and receive traffic. However they are a bit confusing at times, and so can cause some headache when trying to debug why your pod cannot communicate with your database.

• By default ALL outgoing traffic is allows.
• By default NO incoming traffic is allows (except from traefik).

So by default, any pod is allows to communicate with any port on the internet, but not allows to communicate with any other pod in the whole cluster. In the futher we hope to disallow both by default, and so you will have to specify exactly what ports (and potentially where) your pod should be communicating.

Within the terraform, we have helper functions built into the utilities for the databases to automatically generate network policies for incoming traffic, relying on the requirement of adding a label to your pod, they are usually outputted by the module under client_labels.

To define your own policy (in terraform ofc), there are two parts ingress (incoming traffic) and egress (outgoing traffic). Both of these can then be a list of rules matching pods that will be allows. They are additive, meaning that all network policies matching the pod will be combined to produce the final ruleset.

The main rules you will want to focus on are:

• ip_block defines a lock of IPs where the traffic originates or is going to
• namespaceSelector defines the labels matching on the namespaces where traffic is allows to/from
• podSelector same as namespaces but specific to pods themselves (e.g. what our databases do)
• ports defines the ports allows by these connections

While defining any egress policies you must remember to include basic services, e.g. DNS and maybe NTP. An example full configuration might look like:

resource "kubernetes_network_policy_v1" "example" {
  metadata {
    name      = "my-policy"
    namespace = var.namespace
  }
  spec {
    egress {
      # Allow pod to use DNS
      ports {
        port     = 53
        protocol = "TCP"
      }
      ports {
        port     = 53
        protocol = "UDP"
      }
      # Allow pod to request its database
      ports {
        port     = 5432
        protocol = "TCP"
      }
    }
    ingress {
      # Allow pods within namespaces that have enabled ldap (with the defined
      # label) to request this pod via the "ldap" port
      ports {
        port     = "ldap"
        protocol = "TCP"
      }
      from {
        namespace_selector {
          match_labels = {
            allow_ldap = true
          }
        }
      }
    }
    # This defines the pod that this network policy will be applied to.
    pod_selector {
      match_labels = {
        app = "affected-pod"
      }
    }
    # We have defined both Ingress and Egress rules for this pod
    policy_types = ["Ingress", "Egress"]
  }
}

This shows an example configuration, allows the defined pod to communicate with a database (note that the database will also be required to have a network policy with an ingress rule allowing the pod to connect) and DNS and allow other pods in the cluster to communciate with it. Note that any http port is missing as traffic is automatically permitted to access any port within the cluster.

Helm

Helm is a tool which allows the deployment of a set of kubernetes resources from a single configuration. So for third party, complicated applications its amazing, and there are a lot hosted on artifacthub and other random places (as you can really easily host a helm repo for free). There are benefits, like being able to rollback a change to a previous version. However, there are a few things to note:

• When configuring in terraform, do NOT add repos! The full url should go in the repo config, or if its an oci url, it should go in the chart attribute.
• Tofu will refuse to deploy a deploying helm chart (you have to rollback first)
• DO NOT STORE SECRETS IN VALUES - values are not stored securely, and so you should never store passwords or api keys directly in the values (this is an easy mistake to make when configuring), all previous sets of values will be stored forever in the cluster. Therefore, if you make this mistake, you will have to rotate the secret or delete the whole helm deployment and start again.
• Helm does not care if resources are changed between deployments - this is both good and bad, it means that you can apply "hacks" to helm charts you know will not change and they will not appear in the terraform plans to be fixed. But again this is not particularly good practice and can result in some confusing behaviour.
• When you delete a deployment, all persistant volumes will get wiped unless they have the "Retain" reclaim policy.

Helm is a great tool for quickly deploying whole clusters of applications, but it should be used with caution, making sure the chart is reputable and well maintained. As we are using terraform for deployments, it should also only be used for third party applications.

Persistant Volumes

Persistant volumes (PVs) and persistant volume claims (PVCs) are one of the more convoluted things in kubernetes and one of the more dangerous as you are handling data.

At a high level, you request storage by creating a persistant volume claim, the storage manager fullfills your claim by creating a persistant volume (which does not have an associated namespace) and then it is assigned to your pod. You should never be creating persistant volumes yourself, and probably want to be creating persistant volumes through the statefulset's template interface.

For us, the storage operator is k3s' built in one, but for clusters with multiple pods, it probably is going to be something like longhorn, which manages keeping replicas of the storage on multiple machines. This brings up a core issue, a persistant volume can only be mounted to a pod on a node that stores the persistant volume, which is why longhorn is necessary on multi-node clusters and is why we decided to only have one node in each of our clusters.

PVs have different types of access modes: ReadWriteOnce, ReadWriteMany, ReadOnlyMany which mostly control how many nodes can read or write to it at once (multiple pods can still mount it, as long as they are all on the same node). So for k3s' storage class does not support ReadWriteMany or ReadOnlyMany, so you should only be setting it to ReadWriteOnce.

Reclaiming PVs is one of the big danger factors. By default a persistant volume will be deleted if there are no longer any claims for it, and the claims will probably get deleted (e.g. in helm and statefulsets). Therefore, if you have important data on a pod, you most likely want to set the reclaim policy, that means the volume cannot be automatically deleted, but it also means, if another PVC comes along and matches the PV, the PV will be assigned to the new claim and subsequently a new pod, which can cause a lot of confusion and headache. But it is more likely that the PVC is the redeployment of the original application and so you want it to be reassigned to the new PVC.

If you delete an application, you MUST delete the PV manually when its reclaim policy is "Retain", otherwise the data will never get deleted and could be reassigned to another claim.

Other volumes

Persistant volumes are probably the easiest to understand, but there are a lot of other types of volumes, most notably empty directorys and ephemoral volumes as well as volumes creating from config maps or secrets. All of these serve different purposes:

• Empty directory - something like /tmp, all data will be stored in RAM if set to writable and will be deleted when the pod restarts or is deleted. You should set a limit to how much RAM is allowed to be stored there, as it could soak up the full resources of the computer.
• Emphemeral volumes - similar to empty directorys, but instead of being stored in RAM, they are stored on disk (through persistant volumes) and will be deleted upon deletion of the pod and so mostly not really useful unless you are expecting to generate a lot of data during the running of the pod and don't want to retain them on reboot.
• Mounting secrets - this is read only storage and just allows you to mount configuration files that store database passwords or similar. Each key within the secret becomes a text file. Note that if its a binary file (e.g. image) you can use the binaryData property and pass in a hex string.
• Mounting config maps - similar to secrets, but just are for files that don't have to be secret and encrypted on the machine.

Deployment with Tofu

Information pertaining to terraform configuration itself can be found on the Terraform page and for specific how to deploy to BOSS' production and staging cluster, please see the project's README and internal wiki for more information. This will mostly focus on the general process of deploying with tofu and the struggles.

k9s is recommended for watching deployments and seeing why they failed as its generally just fantastic the more you get used to it. The common shortcuts you need to know are:

• d will describe the currently selected object. If you go to the bottom (Shift-G) you will be able to see events associated with that resource, for pods and statefulsets this is extremely useful as it includes failures for pulling images or security denials.
• l see the logs of all containers associated with the resource.
• x while within a secret/configmap will show you the raw (string) data
• s to shell into a pod (note that some pods do not support this as they do not ship withsh
• r allows you to perform a rolling restart on a statefulset or deployment (meaning there should be no downtime if everything is configured). This is the recommended way to update a pod on production (do not just delete the pod itself).
• <enter> see any subresources (e.g. containers for pods, or pods for deployments/statefulsets)
• Ctrl-d allows you to delete the resource
• Ctrl-f allows you to add a port-forward to your own machine (really useful for debugging if an application is responding but is not available via traefik), or getting direct access to databases.
• :<resource> <namespace|all> will change the list view to the given resource in the namespace
• :<resource> will change the list view for the given resource in the currently selected namespace
• 0-9 are shortcuts to switch between recently selected namespaces (they should show at the top with the current assignment)

Some resources may also have their own special command set to allow you to perform resource specific actions which is nice (these should be listed at the top right).

Back to deployment, it's mostly:

tofu apply -var-file=./prod.tfvars [-target=module.something]

Using -target can help a lot when the terraform module is huge as it will limit the number of resources it has to check for updates (though this means that you state and configuration can become desynced and so you should still do apply's without the targetting). You can even specify the exact resource you have just edited for super fast (for terraform) iterations.

While it's deploying you cannot hit Ctrl-c or well you can (twice) to force exit the application, but this can result in annoying consequencies:

• Hitting it before you've confirmed the plan (e.g. you accidentally forgot to include target and don't want to wait): The state WILL be locked and will not be unlocked. Therefore you have to run

  tofu force-unlock <uid>
  

  Where the <uid> can be found by trying to run the apply command and it failing.

• Hitting it while it's deploying new resources (e.g. the pods are not deploying and you didn't change the 5 minute timeout): on next run, tofu will try to recreate those resources, and so before running the command, you must open k9s, find the resources and delete them.
• Hitting it while updating helm charts (e.g. you didn't change the 10 minute timeout and it's not working): on the next run, tofu will refuse to deploy it, because the helm deployment is in an invalid state ("deploying") and it will never exit this state. Therefore you must open up k9s type helm <namespace>, wait a year, click r on the effected resource to list all the previous releases, go down to the last successful deployment and click r again to rollback to that release, and again wait a year while k9s refuses to respond.

As you can probably tell deployment takes patience, expecially when you don't alter the timeouts in helm (which default to 10 minutes for creation and 10 minutes for deletion). You don't want to interrupt the flow while tofu is doing it's thing, so make sure you aren't going to have to wait 10 minutes because you made a simple typo causing the pods to crash loop.

You can technically, temporarily edit the resources (with e in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

Retaining PVs

As previously mentioned, whenever deploying new applications, you should mark any persistant volume that stores data you do not want accidentally deleted with a "Retain" reclaim policy. This can easily be done using k9s by typing :pv, finding the pv attached to the claim, pressing e and searching for where the reclaimPolicy is defined. By default it will be "Delete", and you can just replace it with "Retain". k9s will also show you a PV's reclaim policy on the list view which is nice to check if its already been done.

Where is the state stored?

State files store all passwords and secrets generated or collected by terraform unencrypted by default and so you should be extremely cautious with where the files are stored, who has access to them and never commit them.

Usually, if you don't define any backends, it is stored in a .tfstate file within the directory of your folder. If you are just managing your cluster with them, an easy place to store it is within the kubernetes cluster itself, using the kubernetes backend:

terraform {
  # ...

  backend "kubernetes" {
    secret_suffix  = "my-cluster"
    config_paths   = [var.kube_config_path]
    config_context = var.kube_context
  }

  # ...
}

This is BOSS's method of storage as we do not have to trust the GitLab instance with our cluster's life. But others can be chosen, for example within GitLab itself, although this should be done with extreme caution due to the risk of exposing plan files (which contain secrets) to the world.