Kubernetes

General knowledge

This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

Pod vs Container

A common confusion is that pod's are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

Statefulset vs deployment

Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

Liveness/Startup probes

Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

Security Context

Within BOSS's kubernetes cluster, we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.

A security context defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

• Run as user
• Don't allow privilege escalation
• Properly define seccomp policy
• Default SELinux container context
• Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:

resource "kubernetes_deployment_v1" "my_deployment" {
  # ...
  spec {
    # ...
    template {
      # ...
      spec {
        container {
          # ...

          security_context {
            run_as_user                = 1000
            run_as_non_root            = true
            allow_privilege_escalation = false
            seccomp_profile {
              type = "RuntimeDefault"
            }
            capabilities {
              drop = ["ALL"]
            }
          }

          # ...
        }
      }
    }
  }
}

See Terraform for more information.

What is a CRD?

A Custom Resource Definition (CRD), allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

Traefik and gateways

A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.

Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the Gateway API, in which traefik implements.

The Gateway API resources are read by traefik, which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of ListenerSets but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (TLSRoute, HTTPRoute, GRPCRoutes and coming in the future TCPRoute and UDPRoute). These routes act as queries to determine when and what traefik should forward to. So for example they act as:

if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service

Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:

Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port

In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

Certificates

The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

• DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
• HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
• Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for Cloudflare proxy-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).

When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.

Cloudflare proxy

Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. k8s.bathcs.com) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".