Kubernetes

General knowledge

This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

Pod vs Container

A common confusion is that pod's are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

Statefulset vs deployment

Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

Liveness/Startup probes

Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

Security Context

Within BOSS's kubernetes cluster, we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.

A security context defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

• Run as user
• Don't allow privilege escalation
• Properly define seccomp policy
• Default SELinux container context
• Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:

resource "kubernetes_deployment_v1" "my_deployment" {
  # ...
  spec {
    # ...
    template {
      # ...
      spec {
        container {
          # ...

          security_context {
            run_as_user                = 1000
            run_as_non_root            = true
            allow_privilege_escalation = false
            seccomp_profile {
              type = "RuntimeDefault"
            }
            capabilities {
              drop = ["ALL"]
            }
          }

          # ...
        }
      }
    }
  }
}

See Terraform for more information.

What is a CRD?

A Custom Resource Definition (CRD), allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.