Bath Wiki - User contributions [en-gb]

Terraform

2026-08-25T20:41:34Z

Hw2210: /* Module structure */ Correct filenames in structure

[[BOSS]] uses terraform (actually [https://opentofu.org/ <code>opentofu</code>]) to deploy all the resources onto [[BOSS/Hosting/Cluster|their kubernetes cluster]], which tries to be the language in which you can deploy anything and everything and if you need to quickly redeploy a whole machine you can with a simple command. In practise it doesn’t work like that, but its good enough for our needs.

You can find the [https://gitlab.bath.ac.uk/cs/int/terraform terraform repo on gitlab], this page was originally taken from the README as it got too long.

== Other resources ==

* [https://developer.hashicorp.com/terraform/tutorials Hashicorp (the maker of the closed source terraform) has some good tutorials on their site]

Basically that’s it really, the rest of this will be looking at how to deploy an application with our configuration.

== Terraform and its woes ==

Before we get into making an app, I must briefly explain terraform and its benefits/issues/confusing behaviours.

This expects you to have a rough idea around how terraform works. But here is a quick explainer: terraform is build around '''resources''', provided by '''providers'''. These resources have a state stored locally in a state file, whether they are deployed, generated values etc. (note that these can literally be anything e.g. from random passwords to HTTP reequests to kubernetes resources to DNS records). These resources can then be organised into modules, which (can) have outputs from values generated by the resources. There are also “data”, but this is basically a reference to another resource which doesn’t have the controls.

When deploying, terraform will then check the state of all the current deployed modules (even pinging servers if needed) and find anything that has changed (e.g. new resources or updated values) and deploy everything.

=== Module structure ===

Modules are the core of terraform and can be a bit tricky to get your head around, as initially they are quite limited (e.g. there is no such thing as a global variable).

But basically a module has a list of inputs as and a list of outputs (and then providers). So it is expected that your module deploys some resources which are then used to output something. E.g. in [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ldap/user|our ldap user module] the module generates a random password, creates the user and assigns them to a group, then outputs the username, email and password to be used later in another resource.

So e.g.

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
// Note passing providers act somewhat like global constants, passing configuration (e.g. what ldap server we mean)
// to the module
lldap = lldap
}
}

// You can then use the email by: module.user.email or password: module.user.password</syntaxhighlight>
'''NOTE''': All files within a module (the folder) are treated as one global space, similar to how Go works. You can reference variables, locals and resources throughout all files within a module which makes it quite difficult to organise nicely.

The modules are usually structure in the way:

* <code>main.tf</code> – Usually where your <code>providers</code> go <s>and if you are lazy (like me), everything else</s>
* <code>variables.tf</code> – Where you put all your variables (even if there are none). As explained in [[#Variable madness|Variable madness]], I really don’t like this but we have decided to comply with terraform😞.
* <code>outputs.tf</code> – All your outputs go here (even if there are none)
* <code>*.tf</code> – Anything else, if you want to split it out nicely into other files

=== Variable madness ===

Terraform variables suck.

Anyway, so basically terraform requires you do a full definition for every variable:

<syntaxhighlight lang="terraform">variable "my_var" {
type = string

description = "Something"
nullable = false

sensitive = false # If something is sensitive MAKE THIS TRUE
}
// You can then later reference it with var.my_var</syntaxhighlight>
This defines something that must be inputed by the user, either through the module or your <code>tfvars</code> file (if in the root directory).

Due to this verbosity, and sometimes complex nature of the interfaces I like to create, I have used the <code>object</code> type e.g.:

<syntaxhighlight lang="terraform">variable "my_var" {
type = object({
l = list(string)
m = map(bool) # string -> bool. Same syntax as object, just more flexible
s = set(string) # Yes this is different to list but using the same []
option = optional(string, "my_default")
})
// ...
}</syntaxhighlight>
But even then you can’t define nice defaults for each sub item and the description is for the whole variable, so it has to be done as so. This is just raw pain and not particularly great syntax in my opinion. Also you cannot enable/disable sensitive nature of values for sub items, this means the whole object must be defined as sensitive if you have one password.

And due to the lack of global constants, you must define every variable in every sub project and duplicate the types (yes there is '''no''' way to define a type to use throughout the project).

It is also recommended that you put all variables in a <code>vars.tf</code> file. Which sure does make sense for small modules, but if its that small I find it easier to just chuck at the top of the <code>init.tf</code> file (as the terraform syntax highlighter is soooo broken). Then if its large, I find it more useful to put the variables where they are actually used – but then again this is confusing because the syntax and tooling is so bad.

Oh yeah sorry and then there are '''locals''' which are constants you can define from resources/variables and will be calculated when the information is ready. E.g.

<syntaxhighlight lang="terraform">locals {
temp_val = "hi"
}
// Then you can reference with local.temp_val</syntaxhighlight>

=== Depends on and its pains ===

The one issue with terraform is that its really slow with large projects with this. But the nature of the design encourages large projects (as you want to reference things throughout the smaller apps).

This is due to it having to create a dependency graph where objects wait on their dependencies. These dependencies can be defined by <code>depends_on</code> in any resource or just referencing a value from another resource.

This is really useful so deployments are not actually deployed until all the secrets are deployed. Due to my perferable of not repeating myself, I heavily use the inferred dependency from the referring to resource names e.g. <code>kubernetes_secret_v1.secret.metadata[0].name</code> (yes this is why I don’t just do the simple thing and use the shorter name, its good to know where the value comes from).

'''BUT''' you cannot rely on this, as some things take time to actually deploy even if it says its successful. Therefore you may need to use timers instead. I usually don’t bother due to the numerous other issues with terraform making it so its not actually perfect so there’s minimal point actually making it easy to deploy from scratch.

=== Timeouts ===

If something goes wrong during deployment, e.g. you make a typo, you will have to wait the FULL timeout time. This is really painful when you typo the hostname to the db causing the pod to crashloop in a helm config and you have to wait 10 minutes for terraform to give up. You can Ctrl-C, Ctrl-C, but this causes more issues as you will have to manually intervene and delete the helm chart/deployment before you run the command again.

Instead I recommend shortening the timeouts for the deployment/helm to one more applicable to the application. A lot of our first-party stuff usually deploys in a few seconds and if it doesn’t, something has gone very wrong.

=== Commas or no commas? ===

The terraform syntax is… interesting. Commas are optional in most cases. So I would recommend, not typing commas where they optional.

'''BUT''' within lists/sets (basically between <code>[]</code>) you have to type commas, and if this is across multiple lines '''PLEASE ADD TRAILING COMMAS'''. The reason? Git histories look sooooooo much better.

=== The recreation footgun ===

If you are new to terraform, you might make the following change without thinking too much:

<syntaxhighlight lang="terraform">
resource "aws_db_instance" "old_name" {
# configuration goes here
}
</syntaxhighlight>

into

<syntaxhighlight lang="terraform">
resource "aws_db_instance" "cool_new_name" {
# EXACT SAME configuration goes here
}
</syntaxhighlight>

This will actually completely wipe that database instance from the face of the planet if you just blindly run <code>tofu plan</code> then <code>tofu apply</code>.

What terraform actually does is:

# Oh, <code>old_name</code> doesn't exist anymore? I'll delete that instance then.
# Ah, a new <code>cool_new_name</code> instance! I'll make that fresh then.

You are now left with no data (unless you made backups).

To avoid this, we have a couple options. The first and generally preferred option is to use a [https://developer.hashicorp.com/terraform/language/block/moved <code>moved</code> block]. For the above example, we can do:

<syntaxhighlight lang="terraform">
moved {
from = aws_db_instance.old_name
to = aws_db_instance.cool_new_name
}

resource "aws_db_instance" "cool_new_name" {
# original configuration goes here
}
</syntaxhighlight>

These may litter the code and so they should be removed after a while and are only really preferred if multiple people are working on the terraform. For example, it shows a reviewer that you've renamed a resource and if they deploy it, they won't mess it up.

The other option is to rename it in the code and just run:

<syntaxhighlight lang="sh">
tofu state mv aws_db_instance.old_name aws_db_instance.cool_new_name
</syntaxhighlight>

Then, when you <code>tofu plan</code> you shouldn't see any deletion for the old resource.

== How to create a basic project ==

=== Creating a namespace ===

There is a handy util module for this:

<syntaxhighlight lang="terraform">module "example_namespace" {
source = "./utils/namespace"

name = "example"

enable_dns = true
enable_mail = true
enable_lldap = true
bkp = {
// ...
}

providers = {
kubernetes = kubernetes
}
}</syntaxhighlight>
This allows you to enable or disable features for your namespace, e.g. if your pods need to communicate with the outside world, enabling the DNS. All of these features are disabled by default and its heavily encouraged to only enable the features if the namespace needs it.

The next thing is to configure backups through this, to reduce our dependence and costs from our s3 provider, it is recommended that backups are disabled for all namespaces whos data can be regenerated (e.g. froom). If you do enable it, it is then encouraged that you disable backups for any database or pvc that you don’t need backing up with the <code>k8up.io/backup=false</code> annotation (you may notice that all <code>valkey</code> instances set this by default if you are using the <code>app/valkey</code> module).

==== Placement ====

Within this repository, it is tradition to put the namespace creation at the highest level, e.g. <code>20_apps.tf</code>. This means that the apps themselves do not control the namespace they are created in. It is mostly just a personal preference from me after years of configuring k8s on terraform.

=== Using Helm ===

Helm is by far the easiest way to deploy third-party tools, and is used throught this repo despite it’s drawbacks when combined with terraform (it’s just so easy).

You just add helm to the providers list (which defines what terraform modules you are integrating with):

<syntaxhighlight lang="terraform">terraform {
required_providers {
helm = {
source = "hashicorp/helm"
version = "~>3.1.1"
}
}
}</syntaxhighlight>
You can then use the <code>helm_release</code> resource, which takes the form:

<syntaxhighlight lang="terraform">resource "helm_release" "my_app" {
name = "my_app"
namespace = var.namespace

repository = "https://charts.example.com"
chart = "the_app"
version = "version"

values = [yamlencode({
// Values go here written within the terraform config language
})]
}</syntaxhighlight>
This deploys all the resources an application needs and manages and restarting if a secret or config map changes, providing all the configuration at your fingertips.

However this comes at a cost:

* '''SECRETS SHOULD NOT GO IN THE HELM CONFIG'''. This is a big one, all values are easily accessible unencrypted on the cluster, therefore any secrets '''MUST''' go in a <code>kubernetes_secrets_v1</code> object and you should use a <code>secretsRef</code> or similar to link it. If the helm chart does not support this '''DO NOT USE IT'''. Helm also stores the history of all values, therefore if you put it temporarily within helm values for testing, you must to a password rotation.
* You have no power over the types of resources and the structure in which it deploys. This means that if a feature or support for our strict network policies are not implemented, you have to either not use the helm chart completely or fork your own (which we definitely don’t want to do).
* If the helm chart gets deleted, all pvc related ''might'' also get deleted (unless they have the <code>Retain</code> policy, which should be the case for everything).
* Sometimes they don’t have the proper security contexts/network policies by default so you will have to add them youself (see the below section)

Overall, helm is pretty good, just use with caution and understand what templates you are inflicting. Note, you will probably have to get pretty good at reading not only default values, but schemas and the templating language of helm itself, as sometimes the charts are not particularly well documented.

=== Using kubernetes ===

For this you need to understand a bit of structure of how kubernetes works. I will assume that you are deploying a pod. If that pod needs storage attached (and not through SQL or Redis), then you will need to use a [https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ <code>StatefulSet</code>]. If you have no storage or are just communicating with a postgres server or redis, you can instead use a [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ <code>Deployment</code>].

The difference between these two concepts are not particularly seen in the world of a single node cluster, but basically deployments are free to spin up another even if the previous one is still terminating or if the node is non responsive. On the other hand, statefulsets must ensure that no two nodes are trying to access the same data, therefore cannot automatically start up if a node goes down.

This also means that it is much easier to scale a deployment to multiple nodes, vs a statefulsets which must have separate volumes per pod.

Anyway, both statefulsets and deployments have a template configuration for creating the pod associated with itself. This pod has a label which is used to monitor and track the associated pods with its parent. So the structure is as follows:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
metadata {
name = "my_deployment"
namespace = var.namespace
}
spec {
replicas = 1
selector {
match_labels = {
app = "the_deployment"
}
}
// This doesn't really matter in a one node cluster with a replicas = 1
strategy {
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "the_deployment"
}
}
spec {
container {
name = "my_deployment"
image = "bathbcss/my_image:latest"
image_pull_policy = "Always" // Should only be set if the above is "latest"

// This should be the default security context to comply with our pod security policies
security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// This is how the pod is checked its alive, so if something happens,
// e.g. job which causes it to become unresponsive, it will be automatically killed off and replaced
liveness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

// This allows the tracking of when the pod starts, so we wait until the pod is ready to receive requests
startup_probe {
http_get {
path = "/healthz"
port = 8080
}
// 3 * 30 = 90 seconds to start
failure_threshold = 30
// If it takes a while to startup, increase this time
period_seconds = 3
}

// If exposing a port the port to expose
port {
container_port = 8080
// Make sure to give it a name so we can use the name in services
name = "web"
}

// env and env_from definitions
}
}
}
}
}</syntaxhighlight>
''Quick side note: <code>kubernetes_\*_v1</code>is the preferred resouce name, any resource that does not have<code>\_v1</code> on the end is deprecated and should not be used.''

It is recommended that the version of the image is actually set and <code>latest</code> is not used, however to reduce the admin overhead, for internal projects it can be easier to set to <code>latest</code> with an image pull policy of <code>Always</code>. However this means if you want the latest version, you must have access to the cluster to restart a pod.

If you are exposing pods, this should be tied with a <code>Service</code>, as seen below.

==== Security context ====

As you will notice in the example above, we have a security context set. This is '''required''' by the pod security contenxt, otherwise it will not deploy. In most cases you can copy either of the two following policies, depending on whether it is within a kubernetes resource or helm/kubernetes manifest resource:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

securityContext = {
runAsUser = 1000
runAsNonRoot = true
allowPrivilegeEscalation = false
seccompProfile = {
type = "RuntimeDefault"
}
capabilities = {
drop = ["ALL"]
}
}

// ....
}
}</syntaxhighlight>
Note that we are running as a user (not root), setting the default seccompProfile (you ''should'' only need the default unless you are doing weird things with the host machine) as well as dropping all capabilities (you may need to add some back in but I will leave to you as you probably know more than me – NOTE: Some are disabled by our pod security policy but can be override with <code>baseline</code>).

==== Liveness and startup probe ====

The liveness and startup probes are not necessary, but is a nice to have. The liveness probe allows the cluster to detect if a pod becomes unresponsive and is then able to kill it if that is the case. Whereas a startup probe makes it so the cluster knows exactly when the pod is able to receive responses.

Please see the [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ kubernetes docs on probes] for more information on the options. But in most cases the HTTP get option should suffice, which just looks for a status 2xx code.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

liveness_probe {
http_get {
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

startup_probe {
http_get {
port = 8080
}
failure_threshold = 30
period_seconds = 3
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

livenessProbe = {
httpGet = {
port = 8080
}
initialDelaySeconds = 5
periodSeconds = 10
}

startupProbe = {
httpGet = {
port = 8080
}
failureThreshold = 30
periodSeconds = 3
}

// ....
}
}</syntaxhighlight>
==== Environmental variables ====

When configuring deployments, you will want to set environmental variables. There are a few ways to do it, but note '''ANY PASSWORDS/API KEYS GO IN SECRETS''' not the environmental variables. As you will see I will example how to do this.

By default the <code>env</code> list can be used to set a single environmental variable e.g.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "TEST"
value = "my_value"
}

// ....
}</syntaxhighlight>
However, this is quite verbose and takes a lot of space, so if you are configuring a lot of variables or have secrets, you will want to use the <code>env_from</code> list. This allows you to reference a config map or secrets (this is the most basic form).

These look like:

<syntaxhighlight lang="terraform">resource "kubernetes_secret_v1" "module" {
metadata {
name = "deployment-db"
namespace = var.namespace
}
data = {
DATABASE_URL = module.database.url
}
type = "Opaque"
}

resource "kubernetes_config_map_v1" "module" {
metadata {
name = "deployment-config"
namespace = var.namespace
}
data = {
PUBLIC_VALUE = "yoooo"
ROCKET_ADDRESS = "::"
}
}

resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env_from {
secret_ref {
name = kubernetes_secret_v1.module.metadata[0].name
}
}
env_from {
config_map_ref {
name = kubernetes_config_map_v1.module.metadata[0].name
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

envFrom = [
{
secretRef = {
name = kubernetes_secret_v1.module.metadata[0].name
}
},
{
configMapRef = {
name = kubernetes_config_map_v1.module.metadata[0].name
}
},
]

// ....
}
}</syntaxhighlight>
Note that you can set the value of a environmental variable from a secret on an individual basis, which can be useful if you are storing environmental variables as well as files inside your secret. E.g:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "DB_PASSWORD"
value_from {
secret_key_ref {
name = kubernetes_secret_v1.module.metadata[0].name
key = "password"
}
}
}

// ....
}</syntaxhighlight>
==== Third-party CRDs ====

Now this is is where terraform becomes less good. Basically when deploying using the kubernetes API, the checks will make sure the CRDs (so the like api and kind are installed and supported on the kubernetes cluster). This means that you won’t even be able to run.

Basically it means that you need to comment out manifests that reference these resources until the CRDs are deployed (usually though a helm chart or something).

==== PVCs ====

Please remember '''ANY PASSWORDS/API KEYS/CERTIFICATES GO IN SECRETS''' not in the storage (this also means its configurable by us and yes they can be mounted as read only volumes).

So, kubernetes storage works around persistant volumes which are requested by persistant volume claims. On our k3s single node, we are just using the k3s filesystem class. This means it’s a bit basic but does the job.

Things to note:

* You probably should be manually creating persistant volume claims (and definitely not persistant volumes), instead using <code>statefulsets</code>
* It’s really hard to change persistant volumes post fact, so please go through testing phase if you are unsure about anything.
* K3s does not support the storage limit, so please '''DON’T RELY ON IT''' to stop abusive behaviour.
* If you are defining yourself, do not accidentally make your deployment depend on the persistant volume claim, as the pvc will not be created until it is used in something. See [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/services/mail/mailserver.tf|the mailserver module] of how to handle it. '''NOTE''': If you reference the pvc config in your deployment, terraform will add that automatically to the <code>depends_on</code> list.
* If it is critical data you will need to '''manually update the pv to “retain” its data''' if the pvc gets deleted. This just adds a bit of safety if you mess up a deployment. This can be done in <code>k9s</code> by an admin, updating the <code>persistentVolumeReclaimPolicy</code> to <code>Retain</code> instead of <code>Delete</code>.

So how you should be using pvc, in statefulsets:

<syntaxhighlight lang="terraform">resource "kubernetes_stateful_set_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
service_name = var.name
replicas = 1
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
}
}
spec {
container {
name = "my_app"
image = "bathbcss/my_app:1.0.0"

// ...

volume_mount {
name = "data"
mount_path = "/data"
}
}
}
}
volume_claim_template {
metadata {
name = "data"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
}
}
}
}</syntaxhighlight>
==== Services ====

Servies are the first way to adding an ingress, and helps give a common endpoint if you have multiple pods running (though we don’t do this). If using within the cluster you can access a service with the address <code>service_name.namespace.svc.cluster.local</code>. Note that if you are accessing it from within the same namespace you can just use the <code>service_name</code> as the hostname (and it is better on a network policy basis).

For a namespace you just need a label to select on (note we have to define an <code>app</code> label for the statefulset/deployment anyway so you can just use this).

So it should look like this:

<syntaxhighlight lang="terraform">resource "kubernetes_service_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
port {
port = 8080
name = "web"
}
selector = {
app = var.name
}
type = "ClusterIP"
}
}</syntaxhighlight>
And then you can just access the port by <code>service_name:8080</code>. Note that you can also set a <code>target_port</code> if you want to change the port from the deployment and service (but like why?).

Also giving it a <code>name</code> is important, so in the ingress we can just use the name instead of the port number itself, increasing readability.

==== Network policies ====

Now comes the pain. If a helm chart has a network policy, use that (but please actually read what permissions it gives).

Network policies, either grant, or block network “ingress” (stuff going into the pod) and “egress” (stuff going out of the pod). By default, due to our security, ingress to our pods is denied (so anything in the cluster), but egress to the outside world is allowed.

You then use either pod selectors or namespace selectors to allow traffic from a pod (which is how <code>enable_dns,lldap,mail</code> works). In most cases egress can be left alone, unless you want to block a node from doing something in particular.

As talked about in the next section, for postgres and valkey, we create these policies, limited to pods with the <code>${name}-${service}-client=true</code> label allowing only inter-namespace communication.

If you do have to write one, which I really hope you don’t, please read [https://kubernetes.io/docs/concepts/services-networking/network-policies/ kubernetes documentation on Network Policies].

=== Postgres and Redis/Valkey ===

We have utilities for [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/postgres|postgres] and redis (through [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/valkey|valkey] due to redis being really hard to run).

If you require these, we highly recommend the above (though postgres needs moving away from bitnami due to the requirement of money for stability). These setup the network policies you need as well as generating secure passwords and you can look at the <code>outputs.tf</code> file to see the outputs you can use from a module.

'''NOTE:''' If using this, for any pod accessing the data, it must be in the same namespace and have the labels <code>${name}-postgresql-client=true</code> and/or <code>${name}-valkey-client=true</code>, otherwise the traffic will be denied by the network policies.

E.g.

<syntaxhighlight lang="terraform">module "database" {
source = "../postgres"

namespace = var.namespace
prefix = var.name
username = "user"
database = "db_name"
size = "250Mi"

providers = {
helm = helm
}
}

// Create a secret with module.database.url

resource "kubernetes_deployment_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
// This **MUST** be defined otherwise you'll get weird errors
froom-pg-postgresql-client = true
}
}
// ...
}
}
}</syntaxhighlight>
Side note: the namespace will also need to have <code>enable_dns</code> to be able to access it

=== Adding ingress ===

Adding ingress to service basically means that you are making it accessible to the outside world.

If you are doing this, please consider security heavily:

* Can any user alter the DB? Are you doing proper type checking on the inputs?
* Do you make sure to not expose any secrets e.g. db urls
* Do you really need to expose this pod? Or can you leave it to k9s port forwarding?

Then you need to ask:

* Just expose it to people within the bath network e.g. on <code>*.k8s.bathcs.com</code>
* Expose it to the whole world <code>*.bathcs.com</code> (note coordination with backstage to get them to update their traefik will be necessary)

Please expose to the utter minimum people.

If exposing to the full network please note the flow is:

* <code>app.bathcs.com</code> <- You should accept this domain in the ingress
* <code>app.bcss.su.bath.ac.uk</code> <- You should accept this domain in the ingress
* <code>app.k8s.bathcs.com</code> <- This is what the certificate you should be giving (due to this is hostname that backstage is requesting)

Depending on the chosen level, please look at the [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ingress|<code>utils/ingress/*</code> modules] as these handle this most of this flow for you.

E.g. making it internal to bath uni only:

<syntaxhighlight lang="terraform">module "dns" {
source = "../../utils/ingress/dns_flow"

subdomain = var.subdomain
# We don't want public one, so don't specify cloudflare_zone_id
k8s_cloudflare_zone_id = var.cloudflare_zone_id

# Expanded for effect
domains = {
uni = var.domains.uni
k8s = var.domains.k8s
}

providers = {
cloudflare = cloudflare,
}
}

module "ingress" {
source = "../../utils/ingress/tls"

name = var.subdomain
entry_points = ["websecure"]
namespace = var.namespace
host = module.dns.hosts.k8s
additional_ingress_hosts = [module.dns.hosts.uni]
service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "http"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
}
}</syntaxhighlight>
If using a helm repo you may have to manually define the certificate, which is quite easy:

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "cert" {
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "Certificate"
metadata = {
name = "${var.name}-cert"
namespace = var.namespace
}
spec = {
secretName = "${var.name}-cert-secret"
issuerRef = var.cert_issuer
dnsNames = [var.host]
}
}
}</syntaxhighlight>
Which outputs the cert to the secret with name <code>${var.name}-cert-secret</code>. You can also use <code>kubernetes_manifest.cert.manifest.spec.secretName</code> (you can guess what I prefer).

If you are making it publically accessible, you can use the full flow which does both dns and ingress records:

<syntaxhighlight lang="terraform">module "ingress" {
source = "../../utils/ingress/full_flow"

cloudflare_zone_id = var.cloudflare_zone_id
namespace = var.namespace
domains = var.domains
subdomain = var.subdomain

service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "web"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
cloudflare = cloudflare
}
}</syntaxhighlight>
==== Putting it behind authelia ====

If the application doesn’t have oauth integration build in and you are wanting to protect it behind authelia you can add middleware of <code>auth-forwardauth-authelia@kubernetescrd</code> to require them to go through authelia.

You will then have to add an access control rule to authelia e.g.

<pre>rules = [
{
domain = "app.k8s.bathcs.com"
subject = ["group:boss-example-group", "user:hw2210"]
},
]</pre>
'''NOTE''': you can create and manage groups on [https://www.bath.ac.uk/groupmanager/ Bath's group manager], which creates a unix group that can be seen in the LDAP server.

=== Cloudflare dns records ===

Cloudflare provider is an actual pain. You would’ve hoped it would be good, but its not amazing (has previously caused updates on every <code>apply</code>). Basically when creating a dns record you should use the whole address e.g. <code>app.bathcs.com</code> instead of <code>app</code>. This is because it will cause an update on the second application, changing the <code>name</code> value (which we use to get the full domain).

It sucks because theres an additional useless variable ontop of the <code>cloudflare_zone_id</code> (yes you can use the “data” thing to solve this but still).

Additionally for long TXT records, it will add additional quotes, and terraform will continually think you need to change that if you don’t yourself add the quotes into the data.

=== Sending mail ===

Sending mail is somewhat weird.

Basically just because I can, the authentication is managed by <code>lldap</code> (a really fast ldap implementation). This should not be confused with the ldap service that Authelia is hooked up with, as it is not.

Once you’ve enabled mail in the namespace you then have to create an ldap user:

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
lldap = lldap
}
}</syntaxhighlight>
You can then use this to access the mail with <code>module.user.name</code>, <code>module.user.password</code> and <code>module.user.email</code>. '''NOTE''' due to it not being exposed, we must use the kubernetes domain certificate, which means that you normally have to disable tls verification it or set the expected domain to <code>bathcs.com</code> (as seen in authelia).

=== Creating an oauth client ===

I have created a [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/oauth/client|OAuth client module] for generating all the secret data you need and generate the client config (under the <code>config</code> output) you can then pass up and then into the authelia module (see <code>grafana</code> as an example).

But basically please read [https://www.authelia.com/integration/openid-connect/introduction/ the authelia oidc docs] for a full explanation of how it works.

Bath Open Source Society

2026-06-11T18:16:51Z

Hw2210: /* Get Started */

Bath Open Source Society (BOSS), is a society focused on providing computer services to students and societies. Their members and committee build and provide websites and tools which students and societies can use.

All services are hosted under [https://bathcs.com bathcs.com], which is a domain owned by the society, as well as [https://bcss.su.bath.ac.uk bcss.su.bath.ac.uk].

The committee's contact details can be found on their [https://thesubath.com/boss SU page].

== Services ==
There are many services maintained by BOSS, students are free to use most of them, with only a few requiring permission.

=== General ===

* [https://froom.bathcs.com Froom], a tool to help you find free unbookable rooms on campus if you just need a room for brief meetings or some quite time.
* [[Main Page|Wiki]], this mediawiki instance, where students can store general knowledge and information about university life

=== Society tools ===

* [https://vault.bathcs.com VaultTub], a hosted password manager compatible with [https://bitwarden.com/ Bitwarden], allows societies to easily share password's with other committee members and perform handovers relatively simply. If your society wants to use this, please email [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk] and we can set you up with an organisation.

* [https://signin.bathcs.com Sign me in!], a tool created for societies to check and track people signed up for a ticketed event and is compatible with the SU's library card reader for even faster signin process.
* [https://gitlab.bath.ac.uk/cs/services/timetable-heater Timetable Heatmap], a tool to find the best times for weekly events with a chosen audience. Please note that this does require some programming knowledge to be able to use.

=== Departmental tools ===
We also have tools for general departmental student activity, if departments want to use these tools, please contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk] and we can set you up with it.

* [https://gitlab.bath.ac.uk/cs/pm/peer-mentor-website Peer Mentor Grouper], this is a basic site that allows freshers to choose preferred peer mentors and then assigns the best combination to achieve most people's preferred choice.

=== Custom ===
If a society wants a custom website or tool for an event they are running, please get in contact with [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk]. Though your dream may not be completely achieved, BOSS will try to fit your needs.

With this, we are able to create and host services such as:

* Basic webpage for advertising your events or just your society
* Websites with integration with [https://auth.bath.ac.uk auth.bath.ac.uk] to provide minimal account system (as long as it complies with GDPR).
* Provide graphs which track website performance
* Send emails from <code>bathcs.com</code> domain, with reduced likelihood of going into spam.
* Websites with restricted user access (e.g. only committee members)

Websites we have previously helped:

* Bath Hack 24-26
* Witathon 25-26

Although, we do have limited resources and so we can only build a few projects from scratch each year, but if you are somewhat technical and have already built a website, we are happy to help you host it.

=== Future ===
BOSS tracks all of its potential future projects on its [https://gitlab.bath.ac.uk/cs/planning/ planning board], which any student can contribute to or discuss the projects.

== Contributing ==
As a society, BOSS members can contribute to any project to help them improve and grow, there are many ways you can contribute, even without any technical knowledge. We track all our issues and store all our code on [https://gitlab.bath.ac.uk/cs Bath's GitLab instance], under the <code>cs</code> organisation and all conversation is done through our hosted Matrix instance.

=== Get Started ===


=== Tester/stakeholder ===
If you want to provide invaluable source of information about up and coming projects, you can get involved by joining our group chats and respond to questions around design. You can checkout current project ideas on our [https://gitlab.bath.ac.uk/cs/planning/ planning board].

This makes sure our new projects are useful and relavent to you as a student.

=== Bug reporter/Feature requester ===
If you find a bug or issue with a service we provide, or even just want to see a specific feature added, please see [[Creating a work item]]. This provides tasks for other students to complete and makes sure that our services are of the highest quality. Please be patient though, development takes time, especially when no one is paid to work on any specific feature.

=== Designer ===
If you are good at art (unlike a lot of Computer Scientists), you can get involved by creating logos or even working with a developer to improve the style of our websites, through CSS and HTML. We are always looking to making our sites more intuitive and this is an invaluable source of help with that.

=== Documenter ===
Computer scientists commonly struggle at writing long essays, though they love talking about their inventions. So you can help out by creating wiki pages about how to use a piece of software, or even read some code that is created and add documentation to the functions explaining what they do and how they work. As students leave every year, we need to make sure our code is as readable as possible.

=== Developer ===
The core of our members, those who are just learning to code for a bit of fun, or wanting to get some practice before going to work at a real company. We work in a range of languages as we believe you should choose the right language for the job, but we tend to use Rust and TypeScript as they are the fun cool languages on the block.

Before you contribute to any of our projects however, you must agree to our [https://boss.bathcs.com/policies/licenses/#developers-certificate-of-origin Developer Certificate of Origin] and understand our [https://boss.bathcs.com/policies/genai/ Generate AI Policy]. But you are free to fork any of our projects, make your changes and then submit an MR. If you are wanting to fix an issue, please send a message on that specific work item to state you are working on it, and make sure no one else is doing the same thing.

If it is a large feature that is not an issue, please get in touch with the project owner or committee to discuss if it is wanted, as we do not want to waste your time.

=== Project Owner ===
If you are assigned this role, you will be responsible for managing work items, reviewing merge requests and publishing releases of your projects. This role is not elected as we want the people with the most knowledge and understanding of the project.

Project owners will be chosen each year and trained up before the previous owner leaves.

=== Sysadmin ===
If you much prefer configuring systems, managing deployments and putting out fires (hopefully not literally), you can apply to become a system administrator. We aim to make this role as easy as possible, relying on automated deployments where ever possible. Although, many other factors can go wrong with 4 servers, and so we need people regularly checking in on them and making sure they have not been hacked or something.

You will need specific knowledge of linux commands, ssh and maybe even kubernetes and FreeBSD (but that's generally easy to learn and we offer training). You will get varying levels of access to our production and staging kubernetes clusters as well as the machines that run them.

Kubernetes

2026-06-10T09:09:56Z

Hw2210: /* Deployment with Tofu */ Add note about configmaps requiring deployments to manually be restarted

I ([[User:Hw2210|hw2210]]) have been asked to write down my processes for working with kubernetes and terraform as I am leaving this year, and hopefully it will serve as useful information to any future sysadmin. This page is the output of this and is written to be as generic as possible, so if you are just experimenting with kubernetes on a home lab, please feel free to read and hopefully you will learn somethings. There will be multiple references to [[BOSS/Hosting/Cluster|BOSS's cluster]], which has all the security boxes ticked on and so we have to deal with security contexts, network policies and [[SELinux]] as they are the bain of all problems.

== General knowledge ==
This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

=== Pod vs Container ===
A common confusion is that [https://kubernetes.io/docs/concepts/workloads/pods/ pod]'s are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

=== Statefulset vs deployment ===
Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

{{Note|text=If you update any "volume" attribute within a statefulset ALL volumes will be deleted and recreated. To not lose any data, please make sure that the PV's reclaim policy has been set to "Retain". You should generally do this for any data you do not want to lose.|type=warn}}

==== Liveness/Startup probes ====
Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

==== Security Context ====
{{Note|text=Within [[BOSS/Hosting/Cluster|BOSS's kubernetes cluster]], we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.}}
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ security context] defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

* Run as user
* Don't allow privilege escalation
* Properly define seccomp policy
* Default SELinux container context
* Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:<syntaxhighlight lang="terraform">
resource "kubernetes_deployment_v1" "my_deployment" {
# ...
spec {
# ...
template {
# ...
spec {
container {
# ...

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

# ...
}
}
}
}
}
</syntaxhighlight>See [[Terraform]] for more information.

=== What is a CRD? ===
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ Custom Resource Definition (CRD)], allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

=== Traefik and gateways ===
[[File:Gateway diagram.svg|thumb|368x368px|A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.]]
Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the [https://kubernetes.io/docs/concepts/services-networking/gateway/ Gateway API], in which traefik implements.

The Gateway API resources are read by [https://doc.traefik.io/traefik/ traefik], which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of <code>ListenerSets</code> but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (<code>TLSRoute</code>, <code>HTTPRoute</code>, <code>GRPCRoutes</code> and coming in the future <code>TCPRoute</code> and <code>UDPRoute</code>). These routes act as queries to determine when and what traefik should forward to. So for example they act as:<syntaxhighlight lang="text">
if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service
</syntaxhighlight>Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:
Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port
In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

==== Certificates ====
The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

* DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
* HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
* Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for [[Kubernetes#Cloudflare proxy|Cloudflare proxy]]-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).
{{Note|text=When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.|type=reminder}}

==== Cloudflare proxy ====
Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. <code>k8s.bathcs.com</code>) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

=== Network Policies ===
With additional security, comes additional policy managment. [https://kubernetes.io/docs/concepts/services-networking/network-policies/ Network policies] tell kubernetes where a pod is allows to send and receive traffic. However they are a bit confusing at times, and so can cause some headache when trying to debug why your pod cannot communicate with your database.

* By default ALL outgoing traffic is allows.
* By default NO incoming traffic is allows (except from traefik).

So by default, any pod is allows to communicate with any port on the internet, but not allows to communicate with any other pod in the whole cluster. In the futher we hope to disallow both by default, and so you will have to specify exactly what ports (and potentially where) your pod should be communicating.

Within the terraform, we have helper functions built into the utilities for the databases to automatically generate network policies for incoming traffic, relying on the requirement of adding a label to your pod, they are usually outputted by the module under <code>client_labels</code>.

To define your own policy (in terraform ofc), there are two parts <code>ingress</code> (incoming traffic) and <code>egress</code> (outgoing traffic). Both of these can then be a list of rules matching pods that will be allows. They are additive, meaning that all network policies matching the pod will be combined to produce the final ruleset.

The main rules you will want to focus on are:

* <code>ip_block</code> defines a lock of IPs where the traffic originates or is going to
* <code>namespaceSelector</code> defines the labels matching on the namespaces where traffic is allows to/from
* <code>podSelector</code> same as namespaces but specific to pods themselves (e.g. what our databases do)
* <code>ports</code> defines the ports allows by these connections

While defining any egress policies you must remember to include basic services, e.g. DNS and maybe NTP. An example full configuration might look like:<syntaxhighlight lang="terraform">
resource "kubernetes_network_policy_v1" "example" {
metadata {
name = "my-policy"
namespace = var.namespace
}
spec {
egress {
# Allow pod to use DNS
ports {
port = 53
protocol = "TCP"
}
ports {
port = 53
protocol = "UDP"
}
# Allow pod to request its database
ports {
port = 5432
protocol = "TCP"
}
}
ingress {
# Allow pods within namespaces that have enabled ldap (with the defined
# label) to request this pod via the "ldap" port
ports {
port = "ldap"
protocol = "TCP"
}
from {
namespace_selector {
match_labels = {
allow_ldap = true
}
}
}
}
# This defines the pod that this network policy will be applied to.
pod_selector {
match_labels = {
app = "affected-pod"
}
}
# We have defined both Ingress and Egress rules for this pod
policy_types = ["Ingress", "Egress"]
}
}

</syntaxhighlight>This shows an example configuration, allows the defined pod to communicate with a database (note that the database will also be required to have a network policy with an ingress rule allowing the pod to connect) and DNS and allow other pods in the cluster to communciate with it. Note that any http port is missing as traffic is automatically permitted to access any port within the cluster.

=== Helm ===
[https://helm.sh/ Helm] is a tool which allows the deployment of a set of kubernetes resources from a single configuration. So for third party, complicated applications its amazing, and there are a lot hosted on [https://artifacthub.io/ artifacthub] and other random places (as you can really easily host a helm repo for free). There are benefits, like being able to rollback a change to a previous version. However, there are a few things to note:

* When configuring in terraform, do NOT add repos! The full url should go in the <code>repo</code> config, or if its an oci url, it should go in the <code>chart</code> attribute.
* Tofu will refuse to deploy a deploying helm chart (you have to rollback first)
* '''DO NOT STORE SECRETS IN VALUES''' - values are not stored securely, and so you should never store passwords or api keys directly in the values (this is an easy mistake to make when configuring), all previous sets of values will be stored forever in the cluster. Therefore, if you make this mistake, you will have to rotate the secret or delete the whole helm deployment and start again.
* Helm does not care if resources are changed between deployments - this is both good and bad, it means that you can apply "hacks" to helm charts you know will not change and they will not appear in the terraform plans to be fixed. But again this is not particularly good practice and can result in some confusing behaviour.
* When you delete a deployment, all persistant volumes will get wiped unless they have the "Retain" reclaim policy.

Helm is a great tool for quickly deploying whole clusters of applications, but it should be used with caution, making sure the chart is reputable and well maintained. As we are using terraform for deployments, it should also only be used for third party applications.

=== Persistant Volumes ===
[https://kubernetes.io/docs/concepts/storage/persistent-volumes/ Persistant volumes] (PVs) and persistant volume claims (PVCs) are one of the more convoluted things in kubernetes and one of the more dangerous as you are handling data.

At a high level, you request storage by creating a persistant volume claim, the storage manager fullfills your claim by creating a persistant volume (which does not have an associated namespace) and then it is assigned to your pod. You should never be creating persistant volumes yourself, and probably want to be creating persistant volumes through the statefulset's template interface.

For us, the storage operator is k3s' built in one, but for clusters with multiple pods, it probably is going to be something like [https://longhorn.io/ longhorn], which manages keeping replicas of the storage on multiple machines. This brings up a core issue, a persistant volume can only be mounted to a pod on a node that stores the persistant volume, which is why longhorn is necessary on multi-node clusters and is why we decided to only have one node in each of our clusters.

PVs have different types of [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes access modes]: <code>ReadWriteOnce</code>, <code>ReadWriteMany</code>, <code>ReadOnlyMany</code> which mostly control how many nodes can read or write to it at once (multiple pods can still mount it, as long as they are all on the same node). So for k3s' storage class does not support <code>ReadWriteMany</code> or <code>ReadOnlyMany</code>, so you should only be setting it to <code>ReadWriteOnce</code>.

Reclaiming PVs is one of the big danger factors. By default a persistant volume will be deleted if there are no longer any claims for it, and the claims will probably get deleted (e.g. in helm and statefulsets). Therefore, if you have important data on a pod, you most likely want to set the [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy reclaim policy], that means the volume cannot be automatically deleted, but it also means, if another PVC comes along and matches the PV, the PV will be assigned to the new claim and subsequently a new pod, which can cause a lot of confusion and headache. But it is more likely that the PVC is the redeployment of the original application and so you want it to be reassigned to the new PVC.
{{Note|text=If you delete an application, you MUST delete the PV manually when its reclaim policy is "Retain", otherwise the data will never get deleted and could be reassigned to another claim.|type=warn}}

===== Other volumes =====
Persistant volumes are probably the easiest to understand, but there are a lot of other types of volumes, most notably empty directorys and ephemoral volumes as well as volumes creating from config maps or secrets. All of these serve different purposes:

* Empty directory - something like <code>/tmp</code>, all data will be stored in RAM if set to writable and will be deleted when the pod restarts or is deleted. You should set a limit to how much RAM is allowed to be stored there, as it could soak up the full resources of the computer.
* [https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/ Emphemeral volumes] - similar to empty directorys, but instead of being stored in RAM, they are stored on disk (through persistant volumes) and will be deleted upon deletion of the pod and so mostly not really useful unless you are expecting to generate a lot of data during the running of the pod and don't want to retain them on reboot.
* Mounting secrets - this is read only storage and just allows you to mount configuration files that store database passwords or similar. Each key within the secret becomes a text file. Note that if its a binary file (e.g. image) you can use the <code>binaryData</code> property and pass in a hex string.
* Mounting config maps - similar to secrets, but just are for files that don't have to be secret and encrypted on the machine.

=== Cronjobs ===

=== K3S ===

== Deployment with Tofu ==
Information pertaining to terraform configuration itself can be found on [[Terraform|the Terraform page]] and for specific how to deploy to BOSS' production and staging cluster, please see the [https://gitlab.bath.ac.uk/cs/int/terraform project's README] and [https://gitlab.bath.ac.uk/cs/boss/int-wiki internal wiki] for more information. This will mostly focus on the general process of deploying with tofu and the struggles.

Please read the [[Kubernetes#K9s|K9s section]] for information about using k9s for monitoring deployments.

Some resources may also have their own special command set to allow you to perform resource specific actions which is nice (these should be listed at the top right).

Back to deployment, it's mostly:<syntaxhighlight lang="sh">
tofu apply -var-file=./prod.tfvars [-target=module.something]
</syntaxhighlight>Using <code>-target</code> can help a lot when the terraform module is huge as it will limit the number of resources it has to check for updates (though this means that you state and configuration can become desynced and so you should still do apply's without the targetting). You can even specify the exact resource you have just edited for super fast (for terraform) iterations.

While it's deploying you cannot hit <code>Ctrl-c</code> or well you can (twice) to force exit the application, but this can result in annoying consequencies:

* Hitting it before you've confirmed the plan (e.g. you accidentally forgot to include target and don't want to wait): The state '''WILL''' be locked and will not be unlocked. Therefore you have to run<syntaxhighlight lang="sh">
tofu force-unlock <uid>
</syntaxhighlight>Where the <code><uid></code> can be found by trying to run the apply command and it failing.

* Hitting it while it's deploying new resources (e.g. the pods are not deploying and you didn't change the 5 minute timeout): on next run, tofu will try to recreate those resources, and so before running the command, you must open k9s, find the resources and delete them.
* Hitting it while updating helm charts (e.g. you didn't change the 10 minute timeout and it's not working): on the next run, tofu will refuse to deploy it, because the helm deployment is in an invalid state ("deploying") and it will never exit this state. Therefore you must open up <code>k9s</code> type <code>helm <namespace></code>, wait a year, click <code>r</code> on the effected resource to list all the previous releases, go down to the last successful deployment and click <code>r</code> again to rollback to that release, and again wait a year while k9s refuses to respond.

As you can probably tell deployment takes patience, expecially when you don't alter the timeouts in helm (which default to 10 minutes for creation and 10 minutes for deletion). You don't want to interrupt the flow while tofu is doing it's thing, so make sure you aren't going to have to wait 10 minutes because you made a simple typo causing the pods to crash loop.

You can technically, temporarily edit the resources (with <code>e</code> in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

{{Note|text=If updating config maps/secrets, you will have to manually restart affected pods for the changes to apply in some cases!}}

=== Retaining PVs ===
As previously mentioned, whenever deploying new applications, you should mark any persistant volume that stores data you do not want accidentally deleted with a "Retain" reclaim policy. This can easily be done using <code>k9s</code> by typing <code>:pv</code>, finding the pv attached to the claim, pressing <code>e</code> and searching for where the <code>reclaimPolicy</code> is defined. By default it will be "Delete", and you can just replace it with "Retain". <code>k9s</code> will also show you a PV's reclaim policy on the list view which is nice to check if its already been done.

=== Where is the state stored? ===
{{Note|text=State files store all passwords and secrets generated or collected by terraform unencrypted by default and so you should be extremely cautious with where the files are stored, who has access to them and never commit them.|type=warn}}
Usually, if you don't define any backends, it is stored in a <code>.tfstate</code> file within the directory of your folder. If you are just managing your cluster with them, an easy place to store it is within the kubernetes cluster itself, using the kubernetes backend:<syntaxhighlight lang="terraform">
terraform {
# ...

backend "kubernetes" {
secret_suffix = "my-cluster"
config_paths = [var.kube_config_path]
config_context = var.kube_context
}

# ...
}

</syntaxhighlight>This is BOSS's method of storage as we do not have to trust the GitLab instance with our cluster's life. But others can be chosen, for example [https://docs.gitlab.com/user/infrastructure/iac/terraform_state/ within GitLab itself], although this should be done with extreme caution due to the risk of exposing plan files (which contain secrets) to the world.

=== Managing the state ===
As we can never write perfect code the first time round, some MRs will result in refactorisation of components into new modules. Therefore state management is important. This annoyingly is painful especially with larger effected packages and so I would recommend considering the effects of destroying all the effected resources and reapplying them (which is what <code>tofu apply ...</code> will do by default). If you cannot afford this (e.g. with persistant volumes) you will have to use <code>tofu state mv</code> which works by moving the specified address to another (make sure there is no typos though!

However this requires the resource type to be the same through the move, which sometimes may not happen. At which point you'll want to delete the state information and re-import it. E.g.<syntaxhighlight lang="sh">
tofu state rm module.example.my_resource.name
# Import the module (Note: the format will differ depending on the provider)
tofu import module.example.module.refractorisation.my_new_resource.name my_resource/name
</syntaxhighlight>

=== Updating packages ===
To make upgrading easier, in submodules, packages are pinned to the nearest major version. You can then update the base package version and run:<syntaxhighlight lang="sh">
tofu init -upgrade
</syntaxhighlight>Which will update the package versions on your machine and update the lock file.

If you do not specify <code>-upgrade</code> it will just update the module list (e.g. if you add a new application with a new use of <code>module</code>).

== K9s ==
[https://k9scli.io/ k9s] is recommended for watching deployments and seeing why they failed as its generally just fantastic the more you get used to it. The common shortcuts you need to know are:

* <code>d</code> will describe the currently selected object. If you go to the bottom (<code>Shift-G</code>) you will be able to see events associated with that resource, for pods and statefulsets this is '''extremely useful''' as it includes failures for pulling images or security denials.
* <code>l</code> see the logs of all containers associated with the resource.
* <code>x</code> while within a secret/configmap will show you the raw (string) data
* <code>s</code> to shell into a pod (note that some pods do not support this as they do not ship withsh
* <code>r</code> allows you to perform a rolling restart on a statefulset or deployment (meaning there should be no downtime if everything is configured). This is the recommended way to update a pod on production ('''do not''' just delete the pod itself).
* <code><enter></code> see any subresources (e.g. containers for pods, or pods for deployments/statefulsets)
* <code>Ctrl-d</code> allows you to delete the resource
* <code>Ctrl-f</code> allows you to add a port-forward to your own machine (really useful for debugging if an application is responding but is not available via traefik), or getting direct access to databases.
* <code>:<resource> <namespace|all></code> will change the list view to the given resource in the namespace
* <code>:<resource></code> will change the list view for the given resource in the currently selected namespace
* <code>0-9</code> are shortcuts to switch between recently selected namespaces (they should show at the top with the current assignment)

== Migrating storages ==
When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so.

{{Note|text=All pods using these volumes should be inaccessible during the process of this. This can be done by temporarily deleting any <code>HTTPRoute</code>s to the service.}}

=== Is it a database? ===
Databases have users and passwords associated, which can cause issues if you are redeploying a whole cluster with terraform as new passwords will be generated. You can import new values but that's boring. Instead you can just us <code>pg_dump</code> (or equivalent):<syntaxhighlight lang="sh">
kubectl port-forward -n my_namespace service/my_pod 5432:5432 &
pg_dump $DATABASE_URL > my_pod_bkp.sql
killall kubectl # Stop port-forward
# ...
# Apply the SQL to the volume in the new cluster:
kubectl --context new_cluster port-forward -n my_namespace service/my_port 5432:5432 &
psql -U username -d myDataBase -a -f my_pod_bkp.sql
killall kubectl # Stop port-forward
</syntaxhighlight>

=== Anything else ===
You will need to investigate permissions and such, but kubectl has this [https://kubernetes.io/docs/reference/kubectl/generated/kubectl_cp/ <code>cp</code> command] which you can use to download whole directories. But before that you should run <code>ls -al</code> while ssh-ed into the machine using the volume to view the permissions of the files.<syntaxhighlight lang="sh">
kubectl cp my_namespace/pod:/where/is/pv/mounted bkp_dir
# ...
kubectl --context new_cluster cp bkp_dir my_namespace/pod:/where/is/pv/mounted
</syntaxhighlight>If the pod cannot run without the storage being set. You have two options: set it with basic data to be overridden, or create a dummy pod with the PV mounted and copy it to there.

Remember to mark any new PVs that should retain data [[Kubernetes#Retaining PVs|with correct reclaim policy]].

== Multinode clusters ==
[[Bath Open Source Society|BOSS]] has specifically chosen not to use multinode clusters and so this is just for the interested people, and hopefully explain why we did not go down this route. Yes, multinode clusters are the whole point of kubernetes, where the program chooses the at least somewhat optimal location to deploy an application and automatically move it to another node if the one its on crashes or restarts for an update. But there are also benefits of kubernetes outside of this feature, such as proper permission system as well as providing a somewhat clean interface for managing deployments.

Additionally, we felt that as a lot of companies are moving towards managing deployments, at least partially, with kubernetes, our use of it will hopefully provide easy place for students to experiment and learn about it (as this is stuff university will never teach you). It should be noted that if companies actually cared about getting the most "bang for your buck", it is worth more to manage deployments yourself and choose which nodes host them, though this can produce more overhead.

=== Benefits ===
There are clear benefits of running a multinode cluster, node down? That's fine, all applications can still run just fine and are accessible. Updating your entrypoint? That's fine, just point your assumingly IPv4 to another node, or if you're using IPv6 just setup DNS to point to both nodes. With my multinode cluster, I am writing this currently after one node completely bricked itself and the other node down after I completely bricked it and all applications are still running fine on the three remaining nodes.

=== Downsides ===
With all this new power comes significant overhead though. For one, if a node goes down its not always as simple as kubernetes automatically moving things. Firstly, volumes may only be stored on that one node, then they may only be locked to that one node with <code>ReadWriteOnce</code>. So if the machine didn't shutdown properly (or if one just get's disconnected from the control plane) you will have to make intervention steps as soon as possible, though, you don't have to be in a panic to get the node itself back up. If you have correctly [[Kubernetes#Safely shutting down nodes|setup safe shutdowns]] and its a routine update though, you probably don't need to touch it.

Power, is another one, the more nodes, the more your [[Kubernetes#Control plane|control plane]] has to manage, meaning the more its going to be stuck managing the cluster, and so will not be able to run their own pods. This means that going from a single node cluster to a multinode cluster, you will probably need at least 3-4 additional machines for it to be managed and working properly. Yes you could run a 2 or 3 node cluster, but the benefits really are not there for the massive amount of downtime. On top of this, you probably want at least 1 node of wiggleroom capable of handling your highest power node, this means that you can never reach the full potential of your cluster, as both your control plane and your other nodes must be able to take the strain when that node is disconnected.

Storage is probably the largest issue that you will need to tackle, and is the one that put BOSS of going this route (along with the fact that we wanted to keep some of our servers as hot spares if parts broke). This will be explored more in the [[Kubernetes#How to manage Storage|"How to manage Storage" section]], but the summary is, although there are tools out there, they require a minimum node count of 3 and have significant overhead. Additionally, because of the potential constant movement of data between nodes going down (even for general updates), it causes additional wear and tear on your drives, reducing their life expectancy (I killed a brand-new SSD in 6 months seemingly because my nodes were going down for updates so often).

Networking is a minor one, all nodes need to be connected to control plane and cannot get randomly interrupted or cut off (e.g. you have your control plane in one room hooked up to a UPS and a agent node in another on a UPS, but the router that connects them is not on the UPS). Disconnection at critical times can cause the control node to start moving workloads off the disconnected node onto others, which with statefulsets or <code>ReadWriteMany</code> volumes can cause desynchronisation which is a massive issue and will cause corruption. Finally there is just a minor issue that if you don't setup correctly you will probably be quite confused, Traefik forwarding going between nodes may result in the final application being given the wrong IP address.

Finally, there is also a bit of a weird one, managing SELinux policies and kernel modules across the cluster. I have had multiple issues in the past of minor configuration differences between my nodes resulting in a pod crashlooping on one node and absolutely just fine on another. Therefore this creates additional overhead as you then have to make sure all installs are identical and then potentially install the security profile operator (which is great when it works but not so great when the docker container they use is ancient and doesn't work on the latest systems and k3s accidentally breaks support for it).

And so, in summary, multinode clusters are amazing for chaising that 99.99999999999% uptime and really interesting for learning the difficulties with it and all the solutions out there. But it is really not for the faint of heart, I would recommend trying with a bunch of VMs on a single node just to understand how it works, but if you are just a hobbiest wanting to host some basic applications at home, do not use it. I am planning on deconstructing my cluster and replacing it with a single node similar to [[Bath Open Source Society|BOSS]] after I leave univesity.

=== Control plane ===
The first challenge is the control plane. What's the benefit of a multinode cluster when it relys on a single node for sending control signals. If it goes down, the whole cluster will panic if a pod crashes, it is left in a seriously vulnerable position. Therefore, you instead need to expand your control plane to multiple nodes. K3s offers built in solution for this, [https://docs.k3s.io/datastore/ha-embedded?_highlight=etcd utilising <code>etcd</code>] by default to share the cluster state between control nodes, or you can just move it off to [https://docs.k3s.io/datastore/ha a separate database cluster] (which is what's recommended in production as you probably have another spare 3 machines). Then you must setup a load balancer between the control nodes (which actually helps with not overloading a single control node). But now the issue is that you are relying on a single load balancer. As a hacky solution, I setup a load balancer on all my control nodes which balanced the load between itself and the other nodes. This mean that the DNS record could store all IPs associated with the nodes and hopefully mitigate any communication issues when a single one went down.

However, when setting up etcd, you must remember to point the <code>--server</code> to the load balancer and not a single node (as then the single node becomes the single point of failure).

=== How to manage Storage ===
I personally used [https://longhorn.io/ longhorn], which provided a somewhat easy solution, you just have to change the default storage class and migrate all your PVs. It requires that you have 3 nodes to share data between (as it can then use voting logic to ward off corruption). By default, it then stores volumes on three different nodes at the same time, meaning if one node goes down, it will just make a copy to a new node with the remaining two copies. Additionally, if a pod is assigned to a node without a copy of the data, longhorn can simply make a new copy onto that node (however it is more likely that kubernetes will choose a node that already has the storage).

It then provides a UI where you can easily manually update all the storage volumes after an update of longhorn and also manage automated backups and snapshots. It is actually a really handy tool and generally really cool (though you have to manually label PVs with the types of backups you want).

The issue? It uses around 2GiB of RAM on each individual node. Then if your nodes are constantly going down (e.g. automated updates every week), it will instantly start panicking and moving volumes over which can wear out drives. Additionally, if too many nodes go down, it can be the single source of crashing everything else because of the amount of resources it can use.

=== Safely shutting down nodes ===
This is basically a requirement if you actually want to obtain the full benefits of multinode cluster that requires as little manual intervention as possible (which is still somehow more intervention than a single node cluster IMO). There is a great [https://oranki.net/posts/2025-01-09-graceful-k3s-shutdown/ blog on <code>oranki.net</code>] on how to set this up, but effectively you need to set that anytime the <code>k3s</code> service is stopped you run:<syntaxhighlight lang="sh">
kubectl drain --ignore-daemonsets --delete-emptydir-data <node>
</syntaxhighlight>Then when it comes back up you need to run:<syntaxhighlight lang="sh">
kubectl uncordon <node>
</syntaxhighlight>As explained in the blog post, you can do this via a systemctl service.

What this does is makes sure all nodes are safety migrated onto another node before a shutdown can take place, effectively tainting the node (meaning can be assigned to it). Then you must <code>uncordon</code> a node (meaning removing the taint) to tell the control plane that you can now place pods on it again.

== Backups ==
Within kubernetes, I'd say there are 3 types of backups:

* Whole OS/VM backups (really easy if host is running ZFS, but large)
* Resource configuration backup (saves time but if you are using terraform, not particularly useful): [https://velero.io/ velero]
* Persistant volume backup (small and application specific): built into some storage classes (e.g. [https://longhorn.io longhorn]) or you can use [https://k8up.io/ k8up]

All of which you may want to use, however because I velero isn't amazing (It fails to back up PVs and the helm chart depends on paid pods, I prefer mixing a whole VM backup with ZFS snapshots and replication, and then do per volume backups with k8up for really important data to offsite cloud S3 bucket.

== Alerting ==
Alerting is really important, for this I would simply just point to the [https://prometheus-operator.dev/ prometheus operator] or the [https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack kube-prometheus-stack], as these provide really easy methods to automatically monitor the clusters and send emails when something might be going wrong (but it can be extremely noisy) and node exporter requires additional configuration for SELinux enabled systems. This provides a nice standardised system that a lot of helm charts implement (and offer grafana boards you can import).

When you get an email, you should investigate what the error actual means and what are the recommended steps to fix. E.g. if a job fails, you can either delete it or rerun it until it works again. For CPUThrottlingHigh errors, I still have absolutely no clue what you are supposed to do about it other than increasing the permitted resources for a pod (which isn't recommended).

== Troubleshooting ==

=== Kubernetes event log ===

=== Traefik dashboard ===

=== Shell-ing into pods ===

=== K3S Log ===

=== Emergency Debug pods ===

=== Cloudflare API tokens ===

=== SELinux ===

=== Tofu ===

=== Scenarios ===

==== Deployment/statefulset created but pod not creating ====

==== Pod in cash loop ====

==== Pod cannot request any website ====

==== Pod can access the internet but everything returns self-signed certificate ====

==== Deployment can't access its database/valkey ====

==== Pod is requesting denied system privileges ====

==== Pod is up, but I get a 404 when requesting visiting the page ====

==== Pod is up, but I get a 403 when requesting visiting the page ====

==== Authelia refuses to start up ====

==== Node is crashing often after running out of RAM ====

==== On IPV6 cluster and requests randomly timeout or return 404 ====

==== Statefulset refusing to start pod (PVC) ====

==== A node just crashed and went offline ====

=== Cleaning up ===

Git/GitLab

2026-06-08T19:17:04Z

Hw2210: Resize images

We require that the websites hosted on bathcs.com should be stored in [https://gitlab.bath.ac.uk/cs the cs organisation on Bath’s GitLab instance]. Any project used by student-led initiatives can also be stored on here.

GitLab was chosen because it has a lot more (useful) features than GitHub. However this does mean that it is aimed at people who has a basic knowledge of how to properly manage git projects.

Therefore you will need to be able to use GitLab.

{{Note|type=warn|text=
You cannot push or pull to the GitLab outside of the VPN, even though you can access the page.
}}

== Setup ==

You might have set this up when handing over, if you haven’t, here are the instructions. Note you will have to ask an organisation manager to add you to the organisation if you are going to be editing any current projects.

* Go to https://gitlab.bath.ac.uk
* Login with your University credentials
* Setup TOTP as your 2FA (one time codes). See [[2FA#TOTP|our 2FA section]] for more information about what this is and how to add one (you can skip to the section where it says “Once you see a QR code”)

Once you have logged in, your account will be created and so can be added to organisations or tagged (using @username).

=== SSH Keys ===

If you want to commit or clone repos, you will also need to setup an SSH key:

# Create an SSH key: see [[SSH keys|our documentation]] for information on how to set one up.
# Go to your preferences (click profile icon then click preferences).
# Click “SSH Keys”.
# Copy and paste the public key into the “Key” section.
# Give it a title.
# Change usage type to “Authentication only”.
# Click “Add key”.

By default this will expire in a year, after which you will have to add another key in the same way.

=== Extra Emails ===

You may have already set up git with your personal email address or you have a git signing key with a specific email. You probably don’t want to edit the configuration on a per project basis.

To solve this, GitLab and GitHub allow you to have multiple email addresses linked to your account, this can be found by going to preferences (click the profile icon then preferences) then clicking “Emails”.

You can then type in another email into the box and click “Add email address”. You will then have to verify the email by clicking the link it gives you.

== The CS Organisation ==

All Student Led Initiative projects are stored on the [https://gitlab.bath.ac.uk/cs Computer Science SLI organisation]. If you are a committee member you should have been given access to it. If not please contact the current owners, who can be found by going “Group information > Members”.

[[File:git-gitlab_members.gif|Members page on GitLab showing the current owners and maintainers found by going to Members in the navigation group “Group Information”|764x764px]]

The organisation or group has subgroups for all societies or student-led initiatives. You should have developer rights on all initiatives which you are a part of, which allows you to create branches on the existing projects and push to them. If you ask nicely you may get maintainer rights, which will allow you to create new projects and merge MRs into main (however you still can’t push to the <code>main</code> branch).

There are some projects which live outside the subgroups (e.g. this wiki) as they don’t have a organisation apart of. You will have to specifically ask permission to access individual projects for security reasons.

== Issues ==

=== Creating ===

[[File:git-create_issue.gif|gif showing the steps of going to the issues menu under pinned and clicking new issue button|728x728px]]

Creating an issue is easy, you can click on the “Issues” tab under “pinned” and click “New Issue” and start typing. The description is written in markdown. You can [https://docs.gitlab.com/ee/user/markdown.html learn markdown on the GitLab website].

If the project is configured with templates, there should be a dropdown for you to choose the one most relavent to your issue type.

[[File:git-mr_templates.gif|gif showing the dropdown to select the template to use above the description|491x491px]]

You can then assign someone an issue from the menu if you know who will be working on it. Labels are also available, but these are mostly for the maintainers of the project to use.

Once the issue is created, it will assign the issue a number (you can see this in the URL), which you can use to cross-reference issues or MRs by putting a <code>#</code> before it. GitLab will pick this up and turn it into a link (you can also just put the link and it will be sortened). Note that issues on other projects can also be referenced.

=== Managing Issues ===

There is not much to say about managing issues. All conversations about the issue should be done on the issue itself.

Once someone is working on it, they should be assigned the issue and it should be referenced in the MR which fixes it.

If “Closes” is put before the link to the issue on an MR, once it is merged, the issue should automatically be closed (but please always do check all the related issues are closed).

== Forking + Branches ==

If you are a developer or maintainer on a project, you should be able to create a branch on the project (but just not push directly main). Branches should be under the format <code><username>/<fix_summary></code>.

If you are not a developer but would like to contribute to a project, you should be able to fork the repository into your own space to create make any edits you wish.

== Merge Requests ==

=== Creating ===

[[File:git-create_mr.gif|gif showing the steps of going to the merge requests menu under pinned and clicking new merge request button and choosing a branch to merge into main|820x820px]]

Once you have created a fork or branch, you can create a merge request, either by clicking the link which is shown after pushing, or going onto the project and clicking “Merge requests” at the left and choosing “New merge request”. You can then choose the branch you want to merge from and the one you are merging to (most likely <code>main</code>).

The interface is very similar to issues, with the summary written in markdown and templates available.

If the Merge Request is not finished, and so still a work in progress, you can click the “Mark Draft” button or put “DRAFT:” at the start of the summary.

You can then assign the MR to yourself (as you will be managing it). You then also want to set the reviewer to a maintainer who can then merge the issue.

If the MR depends on another MR to be merged beforehand, please add the “blocked” label and mention what its blocked on in a comment or the issue directly.

=== Managing ===

Sadly as the university does not pay for premium GitLab, we have to do most security practices as manual steps. So please do not go against these procedures, we beg of you.

If you are the reviewer of a MR, you can go onto the “Changes” tab where you can see all the differences between this and the main branch. There is a settings button where you can change parts of the look. I recommend enabling “show whitespace changes” to help find where people have accidentally left trailing whitespace.

You can then add a comment on a line by hovering over the line and clicking the comment button on the left hand side. You can also do this on images if you wish.

When adding a comment you have the option to “start a review” which is recommended if you are going to be making multiple comments as it send them out all at once instead of having a separate email for each comment (which is a lot).

If you drag the comment, you can also select multiple lines which is useful for segments of code which need changing.

[[File:git-start_review.gif|gif showing going to commits tab and adding a comment to the review, selecting multiple lines|524x524px]]

It is expected that the assignee will then resolve all of these issues so it is not up to the reviewer (but as we are students they may need chasing).

Once you are happy with the MR, you can then approve the merge request before merging it.

Please make sure that:

* You don’t approve your own MR
* Don’t merge something without approval (if it is going into <code>main</code>)
* Don’t approve without actually reading all the changes

== Creating a new Project ==

Over time we may want to add more projects to the organisation. This page focuses on creating projects from scratch. If you are just wanting to mirror, see the [[Git/Mirrors|page about mirrors]].

When creating a new repo or project, you need to decide which subgroup it should go under. This should be quite simple, however you may decide a new subgroup is required.

If you are not a maintainer in a repository, you will have to find the maintainers of the subgroup and ask one of them to set up the repo for you and then add you to the access list.

Other things to consider when creating a project:

* Name (please make this understandable about what it is for)
* Visibility level:
** '''Private''': only people with access can see it (preferably not), however is understandable if you are just testing out a project.
** '''Internal''': Only bath students can see it when logged in, this is useful for things which interact with University of Bath services which they may not want known to the public.
** '''Public''': This should be used for most projects to encourage the open source nature of the community.

Once a repository is created you can add it as a remote to your repo or clone it.

You should then:

* Go to “Settings > Repository > Protected branches” and change “Allowed to push and merge” to be “No one” and make sure “Merge MRs” is set to “Maintainers only”.
* Add a description or logo in the “Settings > General” tab
* Add a license as explained [[Git/Special files#license|here]].

[[File:git-setup_new_project.gif|gif showing the process of editing the protected branches to only allow maintainers to merge to main and no one to push to the main|627x627px]]

=== Manage Access ===

You can choose who can access the repository in “Manage > Members” if you are a Maintainer or Owner.

To add new person you can click “Invite members” and type in their username (they will need to have signed into GitLab for their username to appear).

You then need to choose their role. It is recommended to put them as “Developer” unless you want them to be able to merge requests into <code>main</code>, then put them as “Maintainer”.

=== Labels ===

Under “Manage > Labels” you can add or create labels which can be used on merge requests or issues. You should inherit any labels from the parent group, and preferably labels should be group wide. But you can create a label and ask an Owner to upgrade the label to a group label by clicking the 3 dots.

=== Issue + MR Templates ===

In your project, you may want to have templates for people creating issues or MRs to save them time and to standardise them.

This is quite simple to do in GitLab as explained [https://docs.gitlab.com/ee/user/project/description_templates.html here].

In short, you need to add markdown files to <code>.gitlab/issue_templates/</code> or <code>.gitlab/merge_request_templates/</code>. The default should be named <code>default.md</code>.

Examples:

<ul>
<li><code>.gitlab/issue_templates/default.md</code>
<pre class="md">### Summary



### Steps to reproduce

</pre></li>
<li><code>.gitlab/merge_request_templates/default.md</code>
<pre class="md">### Related Issue



Closes #

### Summary



### Testing

</pre></li>
<li><code>.gitlab/merge_request_templates/bug_fix.md</code>
<pre class="md">### Summary



### Testing

</pre></li></ul>

=== CI Pipelines ===

GitLab pipelines are super powerful and I don’t think I could write better documentation than GitLab themselves (which can be found [https://docs.gitlab.com/ee/ci/pipelines/ here]).

This is just here to say they do exist and they should be used for automated testing or even deployment.

The configuration can be found in the <code>.gitlab-ci.yml</code> file.

== See also ==

{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Kubernetes

2026-06-07T13:22:10Z

Hw2210: Add info about migration, backups and alerting

I ([[User:Hw2210|hw2210]]) have been asked to write down my processes for working with kubernetes and terraform as I am leaving this year, and hopefully it will serve as useful information to any future sysadmin. This page is the output of this and is written to be as generic as possible, so if you are just experimenting with kubernetes on a home lab, please feel free to read and hopefully you will learn somethings. There will be multiple references to [[BOSS/Hosting/Cluster|BOSS's cluster]], which has all the security boxes ticked on and so we have to deal with security contexts, network policies and [[SELinux]] as they are the bain of all problems.

== General knowledge ==
This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

=== Pod vs Container ===
A common confusion is that [https://kubernetes.io/docs/concepts/workloads/pods/ pod]'s are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

=== Statefulset vs deployment ===
Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

{{Note|text=If you update any "volume" attribute within a statefulset ALL volumes will be deleted and recreated. To not lose any data, please make sure that the PV's reclaim policy has been set to "Retain". You should generally do this for any data you do not want to lose.|type=warn}}

==== Liveness/Startup probes ====
Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

==== Security Context ====
{{Note|text=Within [[BOSS/Hosting/Cluster|BOSS's kubernetes cluster]], we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.}}
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ security context] defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

* Run as user
* Don't allow privilege escalation
* Properly define seccomp policy
* Default SELinux container context
* Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:<syntaxhighlight lang="terraform">
resource "kubernetes_deployment_v1" "my_deployment" {
# ...
spec {
# ...
template {
# ...
spec {
container {
# ...

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

# ...
}
}
}
}
}
</syntaxhighlight>See [[Terraform]] for more information.

=== What is a CRD? ===
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ Custom Resource Definition (CRD)], allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

=== Traefik and gateways ===
[[File:Gateway diagram.svg|thumb|368x368px|A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.]]
Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the [https://kubernetes.io/docs/concepts/services-networking/gateway/ Gateway API], in which traefik implements.

The Gateway API resources are read by [https://doc.traefik.io/traefik/ traefik], which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of <code>ListenerSets</code> but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (<code>TLSRoute</code>, <code>HTTPRoute</code>, <code>GRPCRoutes</code> and coming in the future <code>TCPRoute</code> and <code>UDPRoute</code>). These routes act as queries to determine when and what traefik should forward to. So for example they act as:<syntaxhighlight lang="text">
if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service
</syntaxhighlight>Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:
Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port
In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

==== Certificates ====
The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

* DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
* HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
* Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for [[Kubernetes#Cloudflare proxy|Cloudflare proxy]]-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).
{{Note|text=When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.|type=reminder}}

==== Cloudflare proxy ====
Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. <code>k8s.bathcs.com</code>) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

=== Network Policies ===
With additional security, comes additional policy managment. [https://kubernetes.io/docs/concepts/services-networking/network-policies/ Network policies] tell kubernetes where a pod is allows to send and receive traffic. However they are a bit confusing at times, and so can cause some headache when trying to debug why your pod cannot communicate with your database.

* By default ALL outgoing traffic is allows.
* By default NO incoming traffic is allows (except from traefik).

So by default, any pod is allows to communicate with any port on the internet, but not allows to communicate with any other pod in the whole cluster. In the futher we hope to disallow both by default, and so you will have to specify exactly what ports (and potentially where) your pod should be communicating.

Within the terraform, we have helper functions built into the utilities for the databases to automatically generate network policies for incoming traffic, relying on the requirement of adding a label to your pod, they are usually outputted by the module under <code>client_labels</code>.

To define your own policy (in terraform ofc), there are two parts <code>ingress</code> (incoming traffic) and <code>egress</code> (outgoing traffic). Both of these can then be a list of rules matching pods that will be allows. They are additive, meaning that all network policies matching the pod will be combined to produce the final ruleset.

The main rules you will want to focus on are:

* <code>ip_block</code> defines a lock of IPs where the traffic originates or is going to
* <code>namespaceSelector</code> defines the labels matching on the namespaces where traffic is allows to/from
* <code>podSelector</code> same as namespaces but specific to pods themselves (e.g. what our databases do)
* <code>ports</code> defines the ports allows by these connections

While defining any egress policies you must remember to include basic services, e.g. DNS and maybe NTP. An example full configuration might look like:<syntaxhighlight lang="terraform">
resource "kubernetes_network_policy_v1" "example" {
metadata {
name = "my-policy"
namespace = var.namespace
}
spec {
egress {
# Allow pod to use DNS
ports {
port = 53
protocol = "TCP"
}
ports {
port = 53
protocol = "UDP"
}
# Allow pod to request its database
ports {
port = 5432
protocol = "TCP"
}
}
ingress {
# Allow pods within namespaces that have enabled ldap (with the defined
# label) to request this pod via the "ldap" port
ports {
port = "ldap"
protocol = "TCP"
}
from {
namespace_selector {
match_labels = {
allow_ldap = true
}
}
}
}
# This defines the pod that this network policy will be applied to.
pod_selector {
match_labels = {
app = "affected-pod"
}
}
# We have defined both Ingress and Egress rules for this pod
policy_types = ["Ingress", "Egress"]
}
}

</syntaxhighlight>This shows an example configuration, allows the defined pod to communicate with a database (note that the database will also be required to have a network policy with an ingress rule allowing the pod to connect) and DNS and allow other pods in the cluster to communciate with it. Note that any http port is missing as traffic is automatically permitted to access any port within the cluster.

=== Helm ===
[https://helm.sh/ Helm] is a tool which allows the deployment of a set of kubernetes resources from a single configuration. So for third party, complicated applications its amazing, and there are a lot hosted on [https://artifacthub.io/ artifacthub] and other random places (as you can really easily host a helm repo for free). There are benefits, like being able to rollback a change to a previous version. However, there are a few things to note:

* When configuring in terraform, do NOT add repos! The full url should go in the <code>repo</code> config, or if its an oci url, it should go in the <code>chart</code> attribute.
* Tofu will refuse to deploy a deploying helm chart (you have to rollback first)
* '''DO NOT STORE SECRETS IN VALUES''' - values are not stored securely, and so you should never store passwords or api keys directly in the values (this is an easy mistake to make when configuring), all previous sets of values will be stored forever in the cluster. Therefore, if you make this mistake, you will have to rotate the secret or delete the whole helm deployment and start again.
* Helm does not care if resources are changed between deployments - this is both good and bad, it means that you can apply "hacks" to helm charts you know will not change and they will not appear in the terraform plans to be fixed. But again this is not particularly good practice and can result in some confusing behaviour.
* When you delete a deployment, all persistant volumes will get wiped unless they have the "Retain" reclaim policy.

Helm is a great tool for quickly deploying whole clusters of applications, but it should be used with caution, making sure the chart is reputable and well maintained. As we are using terraform for deployments, it should also only be used for third party applications.

=== Persistant Volumes ===
[https://kubernetes.io/docs/concepts/storage/persistent-volumes/ Persistant volumes] (PVs) and persistant volume claims (PVCs) are one of the more convoluted things in kubernetes and one of the more dangerous as you are handling data.

At a high level, you request storage by creating a persistant volume claim, the storage manager fullfills your claim by creating a persistant volume (which does not have an associated namespace) and then it is assigned to your pod. You should never be creating persistant volumes yourself, and probably want to be creating persistant volumes through the statefulset's template interface.

For us, the storage operator is k3s' built in one, but for clusters with multiple pods, it probably is going to be something like [https://longhorn.io/ longhorn], which manages keeping replicas of the storage on multiple machines. This brings up a core issue, a persistant volume can only be mounted to a pod on a node that stores the persistant volume, which is why longhorn is necessary on multi-node clusters and is why we decided to only have one node in each of our clusters.

PVs have different types of [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes access modes]: <code>ReadWriteOnce</code>, <code>ReadWriteMany</code>, <code>ReadOnlyMany</code> which mostly control how many nodes can read or write to it at once (multiple pods can still mount it, as long as they are all on the same node). So for k3s' storage class does not support <code>ReadWriteMany</code> or <code>ReadOnlyMany</code>, so you should only be setting it to <code>ReadWriteOnce</code>.

Reclaiming PVs is one of the big danger factors. By default a persistant volume will be deleted if there are no longer any claims for it, and the claims will probably get deleted (e.g. in helm and statefulsets). Therefore, if you have important data on a pod, you most likely want to set the [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy reclaim policy], that means the volume cannot be automatically deleted, but it also means, if another PVC comes along and matches the PV, the PV will be assigned to the new claim and subsequently a new pod, which can cause a lot of confusion and headache. But it is more likely that the PVC is the redeployment of the original application and so you want it to be reassigned to the new PVC.
{{Note|text=If you delete an application, you MUST delete the PV manually when its reclaim policy is "Retain", otherwise the data will never get deleted and could be reassigned to another claim.|type=warn}}

===== Other volumes =====
Persistant volumes are probably the easiest to understand, but there are a lot of other types of volumes, most notably empty directorys and ephemoral volumes as well as volumes creating from config maps or secrets. All of these serve different purposes:

* Empty directory - something like <code>/tmp</code>, all data will be stored in RAM if set to writable and will be deleted when the pod restarts or is deleted. You should set a limit to how much RAM is allowed to be stored there, as it could soak up the full resources of the computer.
* [https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/ Emphemeral volumes] - similar to empty directorys, but instead of being stored in RAM, they are stored on disk (through persistant volumes) and will be deleted upon deletion of the pod and so mostly not really useful unless you are expecting to generate a lot of data during the running of the pod and don't want to retain them on reboot.
* Mounting secrets - this is read only storage and just allows you to mount configuration files that store database passwords or similar. Each key within the secret becomes a text file. Note that if its a binary file (e.g. image) you can use the <code>binaryData</code> property and pass in a hex string.
* Mounting config maps - similar to secrets, but just are for files that don't have to be secret and encrypted on the machine.

=== Cronjobs ===

=== K3S ===

== Deployment with Tofu ==
Information pertaining to terraform configuration itself can be found on [[Terraform|the Terraform page]] and for specific how to deploy to BOSS' production and staging cluster, please see the [https://gitlab.bath.ac.uk/cs/int/terraform project's README] and [https://gitlab.bath.ac.uk/cs/boss/int-wiki internal wiki] for more information. This will mostly focus on the general process of deploying with tofu and the struggles.

Please read the [[Kubernetes#K9s|K9s section]] for information about using k9s for monitoring deployments.

Some resources may also have their own special command set to allow you to perform resource specific actions which is nice (these should be listed at the top right).

Back to deployment, it's mostly:<syntaxhighlight lang="sh">
tofu apply -var-file=./prod.tfvars [-target=module.something]
</syntaxhighlight>Using <code>-target</code> can help a lot when the terraform module is huge as it will limit the number of resources it has to check for updates (though this means that you state and configuration can become desynced and so you should still do apply's without the targetting). You can even specify the exact resource you have just edited for super fast (for terraform) iterations.

While it's deploying you cannot hit <code>Ctrl-c</code> or well you can (twice) to force exit the application, but this can result in annoying consequencies:

* Hitting it before you've confirmed the plan (e.g. you accidentally forgot to include target and don't want to wait): The state '''WILL''' be locked and will not be unlocked. Therefore you have to run<syntaxhighlight lang="sh">
tofu force-unlock <uid>
</syntaxhighlight>Where the <code><uid></code> can be found by trying to run the apply command and it failing.

* Hitting it while it's deploying new resources (e.g. the pods are not deploying and you didn't change the 5 minute timeout): on next run, tofu will try to recreate those resources, and so before running the command, you must open k9s, find the resources and delete them.
* Hitting it while updating helm charts (e.g. you didn't change the 10 minute timeout and it's not working): on the next run, tofu will refuse to deploy it, because the helm deployment is in an invalid state ("deploying") and it will never exit this state. Therefore you must open up <code>k9s</code> type <code>helm <namespace></code>, wait a year, click <code>r</code> on the effected resource to list all the previous releases, go down to the last successful deployment and click <code>r</code> again to rollback to that release, and again wait a year while k9s refuses to respond.

As you can probably tell deployment takes patience, expecially when you don't alter the timeouts in helm (which default to 10 minutes for creation and 10 minutes for deletion). You don't want to interrupt the flow while tofu is doing it's thing, so make sure you aren't going to have to wait 10 minutes because you made a simple typo causing the pods to crash loop.

You can technically, temporarily edit the resources (with <code>e</code> in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

=== Retaining PVs ===
As previously mentioned, whenever deploying new applications, you should mark any persistant volume that stores data you do not want accidentally deleted with a "Retain" reclaim policy. This can easily be done using <code>k9s</code> by typing <code>:pv</code>, finding the pv attached to the claim, pressing <code>e</code> and searching for where the <code>reclaimPolicy</code> is defined. By default it will be "Delete", and you can just replace it with "Retain". <code>k9s</code> will also show you a PV's reclaim policy on the list view which is nice to check if its already been done.

=== Where is the state stored? ===
{{Note|text=State files store all passwords and secrets generated or collected by terraform unencrypted by default and so you should be extremely cautious with where the files are stored, who has access to them and never commit them.|type=warn}}
Usually, if you don't define any backends, it is stored in a <code>.tfstate</code> file within the directory of your folder. If you are just managing your cluster with them, an easy place to store it is within the kubernetes cluster itself, using the kubernetes backend:<syntaxhighlight lang="terraform">
terraform {
# ...

backend "kubernetes" {
secret_suffix = "my-cluster"
config_paths = [var.kube_config_path]
config_context = var.kube_context
}

# ...
}

</syntaxhighlight>This is BOSS's method of storage as we do not have to trust the GitLab instance with our cluster's life. But others can be chosen, for example [https://docs.gitlab.com/user/infrastructure/iac/terraform_state/ within GitLab itself], although this should be done with extreme caution due to the risk of exposing plan files (which contain secrets) to the world.

=== Managing the state ===
As we can never write perfect code the first time round, some MRs will result in refactorisation of components into new modules. Therefore state management is important. This annoyingly is painful especially with larger effected packages and so I would recommend considering the effects of destroying all the effected resources and reapplying them (which is what <code>tofu apply ...</code> will do by default). If you cannot afford this (e.g. with persistant volumes) you will have to use <code>tofu state mv</code> which works by moving the specified address to another (make sure there is no typos though!

However this requires the resource type to be the same through the move, which sometimes may not happen. At which point you'll want to delete the state information and re-import it. E.g.<syntaxhighlight lang="sh">
tofu state rm module.example.my_resource.name
# Import the module (Note: the format will differ depending on the provider)
tofu import module.example.module.refractorisation.my_new_resource.name my_resource/name
</syntaxhighlight>

=== Updating packages ===
To make upgrading easier, in submodules, packages are pinned to the nearest major version. You can then update the base package version and run:<syntaxhighlight lang="sh">
tofu init -upgrade
</syntaxhighlight>Which will update the package versions on your machine and update the lock file.

If you do not specify <code>-upgrade</code> it will just update the module list (e.g. if you add a new application with a new use of <code>module</code>).

== K9s ==
[https://k9scli.io/ k9s] is recommended for watching deployments and seeing why they failed as its generally just fantastic the more you get used to it. The common shortcuts you need to know are:

* <code>d</code> will describe the currently selected object. If you go to the bottom (<code>Shift-G</code>) you will be able to see events associated with that resource, for pods and statefulsets this is '''extremely useful''' as it includes failures for pulling images or security denials.
* <code>l</code> see the logs of all containers associated with the resource.
* <code>x</code> while within a secret/configmap will show you the raw (string) data
* <code>s</code> to shell into a pod (note that some pods do not support this as they do not ship withsh
* <code>r</code> allows you to perform a rolling restart on a statefulset or deployment (meaning there should be no downtime if everything is configured). This is the recommended way to update a pod on production ('''do not''' just delete the pod itself).
* <code><enter></code> see any subresources (e.g. containers for pods, or pods for deployments/statefulsets)
* <code>Ctrl-d</code> allows you to delete the resource
* <code>Ctrl-f</code> allows you to add a port-forward to your own machine (really useful for debugging if an application is responding but is not available via traefik), or getting direct access to databases.
* <code>:<resource> <namespace|all></code> will change the list view to the given resource in the namespace
* <code>:<resource></code> will change the list view for the given resource in the currently selected namespace
* <code>0-9</code> are shortcuts to switch between recently selected namespaces (they should show at the top with the current assignment)

== Migrating storages ==
When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so.

{{Note|text=All pods using these volumes should be inaccessible during the process of this. This can be done by temporarily deleting any <code>HTTPRoute</code>s to the service.}}

=== Is it a database? ===
Databases have users and passwords associated, which can cause issues if you are redeploying a whole cluster with terraform as new passwords will be generated. You can import new values but that's boring. Instead you can just us <code>pg_dump</code> (or equivalent):<syntaxhighlight lang="sh">
kubectl port-forward -n my_namespace service/my_pod 5432:5432 &
pg_dump $DATABASE_URL > my_pod_bkp.sql
killall kubectl # Stop port-forward
# ...
# Apply the SQL to the volume in the new cluster:
kubectl --context new_cluster port-forward -n my_namespace service/my_port 5432:5432 &
psql -U username -d myDataBase -a -f my_pod_bkp.sql
killall kubectl # Stop port-forward
</syntaxhighlight>

=== Anything else ===
You will need to investigate permissions and such, but kubectl has this [https://kubernetes.io/docs/reference/kubectl/generated/kubectl_cp/ <code>cp</code> command] which you can use to download whole directories. But before that you should run <code>ls -al</code> while ssh-ed into the machine using the volume to view the permissions of the files.<syntaxhighlight lang="sh">
kubectl cp my_namespace/pod:/where/is/pv/mounted bkp_dir
# ...
kubectl --context new_cluster cp bkp_dir my_namespace/pod:/where/is/pv/mounted
</syntaxhighlight>If the pod cannot run without the storage being set. You have two options: set it with basic data to be overridden, or create a dummy pod with the PV mounted and copy it to there.

Remember to mark any new PVs that should retain data [[Kubernetes#Retaining PVs|with correct reclaim policy]].

== Multinode clusters ==
[[Bath Open Source Society|BOSS]] has specifically chosen not to use multinode clusters and so this is just for the interested people, and hopefully explain why we did not go down this route. Yes, multinode clusters are the whole point of kubernetes, where the program chooses the at least somewhat optimal location to deploy an application and automatically move it to another node if the one its on crashes or restarts for an update. But there are also benefits of kubernetes outside of this feature, such as proper permission system as well as providing a somewhat clean interface for managing deployments.

Additionally, we felt that as a lot of companies are moving towards managing deployments, at least partially, with kubernetes, our use of it will hopefully provide easy place for students to experiment and learn about it (as this is stuff university will never teach you). It should be noted that if companies actually cared about getting the most "bang for your buck", it is worth more to manage deployments yourself and choose which nodes host them, though this can produce more overhead.

=== Benefits ===
There are clear benefits of running a multinode cluster, node down? That's fine, all applications can still run just fine and are accessible. Updating your entrypoint? That's fine, just point your assumingly IPv4 to another node, or if you're using IPv6 just setup DNS to point to both nodes. With my multinode cluster, I am writing this currently after one node completely bricked itself and the other node down after I completely bricked it and all applications are still running fine on the three remaining nodes.

=== Downsides ===
With all this new power comes significant overhead though. For one, if a node goes down its not always as simple as kubernetes automatically moving things. Firstly, volumes may only be stored on that one node, then they may only be locked to that one node with <code>ReadWriteOnce</code>. So if the machine didn't shutdown properly (or if one just get's disconnected from the control plane) you will have to make intervention steps as soon as possible, though, you don't have to be in a panic to get the node itself back up. If you have correctly [[Kubernetes#Safely shutting down nodes|setup safe shutdowns]] and its a routine update though, you probably don't need to touch it.

Power, is another one, the more nodes, the more your [[Kubernetes#Control plane|control plane]] has to manage, meaning the more its going to be stuck managing the cluster, and so will not be able to run their own pods. This means that going from a single node cluster to a multinode cluster, you will probably need at least 3-4 additional machines for it to be managed and working properly. Yes you could run a 2 or 3 node cluster, but the benefits really are not there for the massive amount of downtime. On top of this, you probably want at least 1 node of wiggleroom capable of handling your highest power node, this means that you can never reach the full potential of your cluster, as both your control plane and your other nodes must be able to take the strain when that node is disconnected.

Storage is probably the largest issue that you will need to tackle, and is the one that put BOSS of going this route (along with the fact that we wanted to keep some of our servers as hot spares if parts broke). This will be explored more in the [[Kubernetes#How to manage Storage|"How to manage Storage" section]], but the summary is, although there are tools out there, they require a minimum node count of 3 and have significant overhead. Additionally, because of the potential constant movement of data between nodes going down (even for general updates), it causes additional wear and tear on your drives, reducing their life expectancy (I killed a brand-new SSD in 6 months seemingly because my nodes were going down for updates so often).

Networking is a minor one, all nodes need to be connected to control plane and cannot get randomly interrupted or cut off (e.g. you have your control plane in one room hooked up to a UPS and a agent node in another on a UPS, but the router that connects them is not on the UPS). Disconnection at critical times can cause the control node to start moving workloads off the disconnected node onto others, which with statefulsets or <code>ReadWriteMany</code> volumes can cause desynchronisation which is a massive issue and will cause corruption. Finally there is just a minor issue that if you don't setup correctly you will probably be quite confused, Traefik forwarding going between nodes may result in the final application being given the wrong IP address.

Finally, there is also a bit of a weird one, managing SELinux policies and kernel modules across the cluster. I have had multiple issues in the past of minor configuration differences between my nodes resulting in a pod crashlooping on one node and absolutely just fine on another. Therefore this creates additional overhead as you then have to make sure all installs are identical and then potentially install the security profile operator (which is great when it works but not so great when the docker container they use is ancient and doesn't work on the latest systems and k3s accidentally breaks support for it).

And so, in summary, multinode clusters are amazing for chaising that 99.99999999999% uptime and really interesting for learning the difficulties with it and all the solutions out there. But it is really not for the faint of heart, I would recommend trying with a bunch of VMs on a single node just to understand how it works, but if you are just a hobbiest wanting to host some basic applications at home, do not use it. I am planning on deconstructing my cluster and replacing it with a single node similar to [[Bath Open Source Society|BOSS]] after I leave univesity.

=== Control plane ===
The first challenge is the control plane. What's the benefit of a multinode cluster when it relys on a single node for sending control signals. If it goes down, the whole cluster will panic if a pod crashes, it is left in a seriously vulnerable position. Therefore, you instead need to expand your control plane to multiple nodes. K3s offers built in solution for this, [https://docs.k3s.io/datastore/ha-embedded?_highlight=etcd utilising <code>etcd</code>] by default to share the cluster state between control nodes, or you can just move it off to [https://docs.k3s.io/datastore/ha a separate database cluster] (which is what's recommended in production as you probably have another spare 3 machines). Then you must setup a load balancer between the control nodes (which actually helps with not overloading a single control node). But now the issue is that you are relying on a single load balancer. As a hacky solution, I setup a load balancer on all my control nodes which balanced the load between itself and the other nodes. This mean that the DNS record could store all IPs associated with the nodes and hopefully mitigate any communication issues when a single one went down.

However, when setting up etcd, you must remember to point the <code>--server</code> to the load balancer and not a single node (as then the single node becomes the single point of failure).

=== How to manage Storage ===
I personally used [https://longhorn.io/ longhorn], which provided a somewhat easy solution, you just have to change the default storage class and migrate all your PVs. It requires that you have 3 nodes to share data between (as it can then use voting logic to ward off corruption). By default, it then stores volumes on three different nodes at the same time, meaning if one node goes down, it will just make a copy to a new node with the remaining two copies. Additionally, if a pod is assigned to a node without a copy of the data, longhorn can simply make a new copy onto that node (however it is more likely that kubernetes will choose a node that already has the storage).

It then provides a UI where you can easily manually update all the storage volumes after an update of longhorn and also manage automated backups and snapshots. It is actually a really handy tool and generally really cool (though you have to manually label PVs with the types of backups you want).

The issue? It uses around 2GiB of RAM on each individual node. Then if your nodes are constantly going down (e.g. automated updates every week), it will instantly start panicking and moving volumes over which can wear out drives. Additionally, if too many nodes go down, it can be the single source of crashing everything else because of the amount of resources it can use.

=== Safely shutting down nodes ===
This is basically a requirement if you actually want to obtain the full benefits of multinode cluster that requires as little manual intervention as possible (which is still somehow more intervention than a single node cluster IMO). There is a great [https://oranki.net/posts/2025-01-09-graceful-k3s-shutdown/ blog on <code>oranki.net</code>] on how to set this up, but effectively you need to set that anytime the <code>k3s</code> service is stopped you run:<syntaxhighlight lang="sh">
kubectl drain --ignore-daemonsets --delete-emptydir-data <node>
</syntaxhighlight>Then when it comes back up you need to run:<syntaxhighlight lang="sh">
kubectl uncordon <node>
</syntaxhighlight>As explained in the blog post, you can do this via a systemctl service.

What this does is makes sure all nodes are safety migrated onto another node before a shutdown can take place, effectively tainting the node (meaning can be assigned to it). Then you must <code>uncordon</code> a node (meaning removing the taint) to tell the control plane that you can now place pods on it again.

== Backups ==
Within kubernetes, I'd say there are 3 types of backups:

* Whole OS/VM backups (really easy if host is running ZFS, but large)
* Resource configuration backup (saves time but if you are using terraform, not particularly useful): [https://velero.io/ velero]
* Persistant volume backup (small and application specific): built into some storage classes (e.g. [https://longhorn.io longhorn]) or you can use [https://k8up.io/ k8up]

All of which you may want to use, however because I velero isn't amazing (It fails to back up PVs and the helm chart depends on paid pods, I prefer mixing a whole VM backup with ZFS snapshots and replication, and then do per volume backups with k8up for really important data to offsite cloud S3 bucket.

== Alerting ==
Alerting is really important, for this I would simply just point to the [https://prometheus-operator.dev/ prometheus operator] or the [https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack kube-prometheus-stack], as these provide really easy methods to automatically monitor the clusters and send emails when something might be going wrong (but it can be extremely noisy) and node exporter requires additional configuration for SELinux enabled systems. This provides a nice standardised system that a lot of helm charts implement (and offer grafana boards you can import).

When you get an email, you should investigate what the error actual means and what are the recommended steps to fix. E.g. if a job fails, you can either delete it or rerun it until it works again. For CPUThrottlingHigh errors, I still have absolutely no clue what you are supposed to do about it other than increasing the permitted resources for a pod (which isn't recommended).

== Troubleshooting ==

=== Kubernetes event log ===

=== Traefik dashboard ===

=== Shell-ing into pods ===

=== K3S Log ===

=== Emergency Debug pods ===

=== Cloudflare API tokens ===

=== SELinux ===

=== Tofu ===

=== Scenarios ===

==== Deployment/statefulset created but pod not creating ====

==== Pod in cash loop ====

==== Pod cannot request any website ====

==== Pod can access the internet but everything returns self-signed certificate ====

==== Deployment can't access its database/valkey ====

==== Pod is requesting denied system privileges ====

==== Pod is up, but I get a 404 when requesting visiting the page ====

==== Pod is up, but I get a 403 when requesting visiting the page ====

==== Authelia refuses to start up ====

==== Node is crashing often after running out of RAM ====

==== On IPV6 cluster and requests randomly timeout or return 404 ====

==== Statefulset refusing to start pod (PVC) ====

==== A node just crashed and went offline ====

=== Cleaning up ===

Kubernetes

2026-06-07T12:11:55Z

Hw2210: Add multinode cluster info and some stuff on migrating storage

I ([[User:Hw2210|hw2210]]) have been asked to write down my processes for working with kubernetes and terraform as I am leaving this year, and hopefully it will serve as useful information to any future sysadmin. This page is the output of this and is written to be as generic as possible, so if you are just experimenting with kubernetes on a home lab, please feel free to read and hopefully you will learn somethings. There will be multiple references to [[BOSS/Hosting/Cluster|BOSS's cluster]], which has all the security boxes ticked on and so we have to deal with security contexts, network policies and [[SELinux]] as they are the bain of all problems.

== General knowledge ==
This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

=== Pod vs Container ===
A common confusion is that [https://kubernetes.io/docs/concepts/workloads/pods/ pod]'s are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

=== Statefulset vs deployment ===
Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

{{Note|text=If you update any "volume" attribute within a statefulset ALL volumes will be deleted and recreated. To not lose any data, please make sure that the PV's reclaim policy has been set to "Retain". You should generally do this for any data you do not want to lose.|type=warn}}

==== Liveness/Startup probes ====
Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

==== Security Context ====
{{Note|text=Within [[BOSS/Hosting/Cluster|BOSS's kubernetes cluster]], we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.}}
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ security context] defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

* Run as user
* Don't allow privilege escalation
* Properly define seccomp policy
* Default SELinux container context
* Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:<syntaxhighlight lang="terraform">
resource "kubernetes_deployment_v1" "my_deployment" {
# ...
spec {
# ...
template {
# ...
spec {
container {
# ...

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

# ...
}
}
}
}
}
</syntaxhighlight>See [[Terraform]] for more information.

=== What is a CRD? ===
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ Custom Resource Definition (CRD)], allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

=== Traefik and gateways ===
[[File:Gateway diagram.svg|thumb|368x368px|A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.]]
Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the [https://kubernetes.io/docs/concepts/services-networking/gateway/ Gateway API], in which traefik implements.

The Gateway API resources are read by [https://doc.traefik.io/traefik/ traefik], which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of <code>ListenerSets</code> but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (<code>TLSRoute</code>, <code>HTTPRoute</code>, <code>GRPCRoutes</code> and coming in the future <code>TCPRoute</code> and <code>UDPRoute</code>). These routes act as queries to determine when and what traefik should forward to. So for example they act as:<syntaxhighlight lang="text">
if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service
</syntaxhighlight>Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:
Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port
In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

==== Certificates ====
The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

* DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
* HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
* Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for [[Kubernetes#Cloudflare proxy|Cloudflare proxy]]-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).
{{Note|text=When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.|type=reminder}}

==== Cloudflare proxy ====
Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. <code>k8s.bathcs.com</code>) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

=== Network Policies ===
With additional security, comes additional policy managment. [https://kubernetes.io/docs/concepts/services-networking/network-policies/ Network policies] tell kubernetes where a pod is allows to send and receive traffic. However they are a bit confusing at times, and so can cause some headache when trying to debug why your pod cannot communicate with your database.

* By default ALL outgoing traffic is allows.
* By default NO incoming traffic is allows (except from traefik).

So by default, any pod is allows to communicate with any port on the internet, but not allows to communicate with any other pod in the whole cluster. In the futher we hope to disallow both by default, and so you will have to specify exactly what ports (and potentially where) your pod should be communicating.

Within the terraform, we have helper functions built into the utilities for the databases to automatically generate network policies for incoming traffic, relying on the requirement of adding a label to your pod, they are usually outputted by the module under <code>client_labels</code>.

To define your own policy (in terraform ofc), there are two parts <code>ingress</code> (incoming traffic) and <code>egress</code> (outgoing traffic). Both of these can then be a list of rules matching pods that will be allows. They are additive, meaning that all network policies matching the pod will be combined to produce the final ruleset.

The main rules you will want to focus on are:

* <code>ip_block</code> defines a lock of IPs where the traffic originates or is going to
* <code>namespaceSelector</code> defines the labels matching on the namespaces where traffic is allows to/from
* <code>podSelector</code> same as namespaces but specific to pods themselves (e.g. what our databases do)
* <code>ports</code> defines the ports allows by these connections

While defining any egress policies you must remember to include basic services, e.g. DNS and maybe NTP. An example full configuration might look like:<syntaxhighlight lang="terraform">
resource "kubernetes_network_policy_v1" "example" {
metadata {
name = "my-policy"
namespace = var.namespace
}
spec {
egress {
# Allow pod to use DNS
ports {
port = 53
protocol = "TCP"
}
ports {
port = 53
protocol = "UDP"
}
# Allow pod to request its database
ports {
port = 5432
protocol = "TCP"
}
}
ingress {
# Allow pods within namespaces that have enabled ldap (with the defined
# label) to request this pod via the "ldap" port
ports {
port = "ldap"
protocol = "TCP"
}
from {
namespace_selector {
match_labels = {
allow_ldap = true
}
}
}
}
# This defines the pod that this network policy will be applied to.
pod_selector {
match_labels = {
app = "affected-pod"
}
}
# We have defined both Ingress and Egress rules for this pod
policy_types = ["Ingress", "Egress"]
}
}

</syntaxhighlight>This shows an example configuration, allows the defined pod to communicate with a database (note that the database will also be required to have a network policy with an ingress rule allowing the pod to connect) and DNS and allow other pods in the cluster to communciate with it. Note that any http port is missing as traffic is automatically permitted to access any port within the cluster.

=== Helm ===
[https://helm.sh/ Helm] is a tool which allows the deployment of a set of kubernetes resources from a single configuration. So for third party, complicated applications its amazing, and there are a lot hosted on [https://artifacthub.io/ artifacthub] and other random places (as you can really easily host a helm repo for free). There are benefits, like being able to rollback a change to a previous version. However, there are a few things to note:

* When configuring in terraform, do NOT add repos! The full url should go in the <code>repo</code> config, or if its an oci url, it should go in the <code>chart</code> attribute.
* Tofu will refuse to deploy a deploying helm chart (you have to rollback first)
* '''DO NOT STORE SECRETS IN VALUES''' - values are not stored securely, and so you should never store passwords or api keys directly in the values (this is an easy mistake to make when configuring), all previous sets of values will be stored forever in the cluster. Therefore, if you make this mistake, you will have to rotate the secret or delete the whole helm deployment and start again.
* Helm does not care if resources are changed between deployments - this is both good and bad, it means that you can apply "hacks" to helm charts you know will not change and they will not appear in the terraform plans to be fixed. But again this is not particularly good practice and can result in some confusing behaviour.
* When you delete a deployment, all persistant volumes will get wiped unless they have the "Retain" reclaim policy.

Helm is a great tool for quickly deploying whole clusters of applications, but it should be used with caution, making sure the chart is reputable and well maintained. As we are using terraform for deployments, it should also only be used for third party applications.

=== Persistant Volumes ===
[https://kubernetes.io/docs/concepts/storage/persistent-volumes/ Persistant volumes] (PVs) and persistant volume claims (PVCs) are one of the more convoluted things in kubernetes and one of the more dangerous as you are handling data.

At a high level, you request storage by creating a persistant volume claim, the storage manager fullfills your claim by creating a persistant volume (which does not have an associated namespace) and then it is assigned to your pod. You should never be creating persistant volumes yourself, and probably want to be creating persistant volumes through the statefulset's template interface.

For us, the storage operator is k3s' built in one, but for clusters with multiple pods, it probably is going to be something like [https://longhorn.io/ longhorn], which manages keeping replicas of the storage on multiple machines. This brings up a core issue, a persistant volume can only be mounted to a pod on a node that stores the persistant volume, which is why longhorn is necessary on multi-node clusters and is why we decided to only have one node in each of our clusters.

PVs have different types of [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes access modes]: <code>ReadWriteOnce</code>, <code>ReadWriteMany</code>, <code>ReadOnlyMany</code> which mostly control how many nodes can read or write to it at once (multiple pods can still mount it, as long as they are all on the same node). So for k3s' storage class does not support <code>ReadWriteMany</code> or <code>ReadOnlyMany</code>, so you should only be setting it to <code>ReadWriteOnce</code>.

Reclaiming PVs is one of the big danger factors. By default a persistant volume will be deleted if there are no longer any claims for it, and the claims will probably get deleted (e.g. in helm and statefulsets). Therefore, if you have important data on a pod, you most likely want to set the [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy reclaim policy], that means the volume cannot be automatically deleted, but it also means, if another PVC comes along and matches the PV, the PV will be assigned to the new claim and subsequently a new pod, which can cause a lot of confusion and headache. But it is more likely that the PVC is the redeployment of the original application and so you want it to be reassigned to the new PVC.
{{Note|text=If you delete an application, you MUST delete the PV manually when its reclaim policy is "Retain", otherwise the data will never get deleted and could be reassigned to another claim.|type=warn}}

===== Other volumes =====
Persistant volumes are probably the easiest to understand, but there are a lot of other types of volumes, most notably empty directorys and ephemoral volumes as well as volumes creating from config maps or secrets. All of these serve different purposes:

* Empty directory - something like <code>/tmp</code>, all data will be stored in RAM if set to writable and will be deleted when the pod restarts or is deleted. You should set a limit to how much RAM is allowed to be stored there, as it could soak up the full resources of the computer.
* [https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/ Emphemeral volumes] - similar to empty directorys, but instead of being stored in RAM, they are stored on disk (through persistant volumes) and will be deleted upon deletion of the pod and so mostly not really useful unless you are expecting to generate a lot of data during the running of the pod and don't want to retain them on reboot.
* Mounting secrets - this is read only storage and just allows you to mount configuration files that store database passwords or similar. Each key within the secret becomes a text file. Note that if its a binary file (e.g. image) you can use the <code>binaryData</code> property and pass in a hex string.
* Mounting config maps - similar to secrets, but just are for files that don't have to be secret and encrypted on the machine.

=== Cronjobs ===

=== K3S ===

== Deployment with Tofu ==
Information pertaining to terraform configuration itself can be found on [[Terraform|the Terraform page]] and for specific how to deploy to BOSS' production and staging cluster, please see the [https://gitlab.bath.ac.uk/cs/int/terraform project's README] and [https://gitlab.bath.ac.uk/cs/boss/int-wiki internal wiki] for more information. This will mostly focus on the general process of deploying with tofu and the struggles.

Please read the [[Kubernetes#K9s|K9s section]] for information about using k9s for monitoring deployments.

Some resources may also have their own special command set to allow you to perform resource specific actions which is nice (these should be listed at the top right).

Back to deployment, it's mostly:<syntaxhighlight lang="sh">
tofu apply -var-file=./prod.tfvars [-target=module.something]
</syntaxhighlight>Using <code>-target</code> can help a lot when the terraform module is huge as it will limit the number of resources it has to check for updates (though this means that you state and configuration can become desynced and so you should still do apply's without the targetting). You can even specify the exact resource you have just edited for super fast (for terraform) iterations.

While it's deploying you cannot hit <code>Ctrl-c</code> or well you can (twice) to force exit the application, but this can result in annoying consequencies:

* Hitting it before you've confirmed the plan (e.g. you accidentally forgot to include target and don't want to wait): The state '''WILL''' be locked and will not be unlocked. Therefore you have to run<syntaxhighlight lang="sh">
tofu force-unlock <uid>
</syntaxhighlight>Where the <code><uid></code> can be found by trying to run the apply command and it failing.

* Hitting it while it's deploying new resources (e.g. the pods are not deploying and you didn't change the 5 minute timeout): on next run, tofu will try to recreate those resources, and so before running the command, you must open k9s, find the resources and delete them.
* Hitting it while updating helm charts (e.g. you didn't change the 10 minute timeout and it's not working): on the next run, tofu will refuse to deploy it, because the helm deployment is in an invalid state ("deploying") and it will never exit this state. Therefore you must open up <code>k9s</code> type <code>helm <namespace></code>, wait a year, click <code>r</code> on the effected resource to list all the previous releases, go down to the last successful deployment and click <code>r</code> again to rollback to that release, and again wait a year while k9s refuses to respond.

As you can probably tell deployment takes patience, expecially when you don't alter the timeouts in helm (which default to 10 minutes for creation and 10 minutes for deletion). You don't want to interrupt the flow while tofu is doing it's thing, so make sure you aren't going to have to wait 10 minutes because you made a simple typo causing the pods to crash loop.

You can technically, temporarily edit the resources (with <code>e</code> in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

=== Retaining PVs ===
As previously mentioned, whenever deploying new applications, you should mark any persistant volume that stores data you do not want accidentally deleted with a "Retain" reclaim policy. This can easily be done using <code>k9s</code> by typing <code>:pv</code>, finding the pv attached to the claim, pressing <code>e</code> and searching for where the <code>reclaimPolicy</code> is defined. By default it will be "Delete", and you can just replace it with "Retain". <code>k9s</code> will also show you a PV's reclaim policy on the list view which is nice to check if its already been done.

=== Where is the state stored? ===
{{Note|text=State files store all passwords and secrets generated or collected by terraform unencrypted by default and so you should be extremely cautious with where the files are stored, who has access to them and never commit them.|type=warn}}
Usually, if you don't define any backends, it is stored in a <code>.tfstate</code> file within the directory of your folder. If you are just managing your cluster with them, an easy place to store it is within the kubernetes cluster itself, using the kubernetes backend:<syntaxhighlight lang="terraform">
terraform {
# ...

backend "kubernetes" {
secret_suffix = "my-cluster"
config_paths = [var.kube_config_path]
config_context = var.kube_context
}

# ...
}

</syntaxhighlight>This is BOSS's method of storage as we do not have to trust the GitLab instance with our cluster's life. But others can be chosen, for example [https://docs.gitlab.com/user/infrastructure/iac/terraform_state/ within GitLab itself], although this should be done with extreme caution due to the risk of exposing plan files (which contain secrets) to the world.

=== Managing the state ===
As we can never write perfect code the first time round, some MRs will result in refactorisation of components into new modules. Therefore state management is important. This annoyingly is painful especially with larger effected packages and so I would recommend considering the effects of destroying all the effected resources and reapplying them (which is what <code>tofu apply ...</code> will do by default). If you cannot afford this (e.g. with persistant volumes) you will have to use <code>tofu state mv</code> which works by moving the specified address to another (make sure there is no typos though!

However this requires the resource type to be the same through the move, which sometimes may not happen. At which point you'll want to delete the state information and re-import it. E.g.<syntaxhighlight lang="sh">
tofu state rm module.example.my_resource.name
# Import the module (Note: the format will differ depending on the provider)
tofu import module.example.module.refractorisation.my_new_resource.name my_resource/name
</syntaxhighlight>

=== Updating packages ===
To make upgrading easier, in submodules, packages are pinned to the nearest major version. You can then update the base package version and run:<syntaxhighlight lang="sh">
tofu init -upgrade
</syntaxhighlight>Which will update the package versions on your machine and update the lock file.

If you do not specify <code>-upgrade</code> it will just update the module list (e.g. if you add a new application with a new use of <code>module</code>).

== K9s ==
[https://k9scli.io/ k9s] is recommended for watching deployments and seeing why they failed as its generally just fantastic the more you get used to it. The common shortcuts you need to know are:

* <code>d</code> will describe the currently selected object. If you go to the bottom (<code>Shift-G</code>) you will be able to see events associated with that resource, for pods and statefulsets this is '''extremely useful''' as it includes failures for pulling images or security denials.
* <code>l</code> see the logs of all containers associated with the resource.
* <code>x</code> while within a secret/configmap will show you the raw (string) data
* <code>s</code> to shell into a pod (note that some pods do not support this as they do not ship withsh
* <code>r</code> allows you to perform a rolling restart on a statefulset or deployment (meaning there should be no downtime if everything is configured). This is the recommended way to update a pod on production ('''do not''' just delete the pod itself).
* <code><enter></code> see any subresources (e.g. containers for pods, or pods for deployments/statefulsets)
* <code>Ctrl-d</code> allows you to delete the resource
* <code>Ctrl-f</code> allows you to add a port-forward to your own machine (really useful for debugging if an application is responding but is not available via traefik), or getting direct access to databases.
* <code>:<resource> <namespace|all></code> will change the list view to the given resource in the namespace
* <code>:<resource></code> will change the list view for the given resource in the currently selected namespace
* <code>0-9</code> are shortcuts to switch between recently selected namespaces (they should show at the top with the current assignment)

== Migrating storages ==
When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so:

=== Is it a database? ===
Databases have users and passwords associated, which can cause issues if you are redeploying a whole cluster with terraform as new passwords will be generated. You can import new values but that's boring. Instead you can just us <code>pg_dump</code> (or equivalent):<syntaxhighlight lang="sh">
kubectl port-forward -n my_namespace service/my_pod 5432:5432 &
pg_dump $DATABASE_URL > my_pod_bkp.sql
killall kubectl # Stop port-forward
# ...
# Apply the SQL to the volume in the new cluster:
kubectl --context new_cluster port-forward -n my_namespace service/my_port 5432:5432 &
psql -U username -d myDataBase -a -f my_pod_bkp.sql
killall kubectl # Stop port-forward
</syntaxhighlight>

== Multinode clusters ==
[[Bath Open Source Society|BOSS]] has specifically chosen not to use multinode clusters and so this is just for the interested people, and hopefully explain why we did not go down this route. Yes, multinode clusters are the whole point of kubernetes, where the program chooses the at least somewhat optimal location to deploy an application and automatically move it to another node if the one its on crashes or restarts for an update. But there are also benefits of kubernetes outside of this feature, such as proper permission system as well as providing a somewhat clean interface for managing deployments.

Additionally, we felt that as a lot of companies are moving towards managing deployments, at least partially, with kubernetes, our use of it will hopefully provide easy place for students to experiment and learn about it (as this is stuff university will never teach you). It should be noted that if companies actually cared about getting the most "bang for your buck", it is worth more to manage deployments yourself and choose which nodes host them, though this can produce more overhead.

=== Benefits ===
There are clear benefits of running a multinode cluster, node down? That's fine, all applications can still run just fine and are accessible. Updating your entrypoint? That's fine, just point your assumingly IPv4 to another node, or if you're using IPv6 just setup DNS to point to both nodes. With my multinode cluster, I am writing this currently after one node completely bricked itself and the other node down after I completely bricked it and all applications are still running fine on the three remaining nodes.

=== Downsides ===
With all this new power comes significant overhead though. For one, if a node goes down its not always as simple as kubernetes automatically moving things. Firstly, volumes may only be stored on that one node, then they may only be locked to that one node with <code>ReadWriteOnce</code>. So if the machine didn't shutdown properly (or if one just get's disconnected from the control plane) you will have to make intervention steps as soon as possible, though, you don't have to be in a panic to get the node itself back up. If you have correctly [[Kubernetes#Safely shutting down nodes|setup safe shutdowns]] and its a routine update though, you probably don't need to touch it.

Power, is another one, the more nodes, the more your [[Kubernetes#Control plane|control plane]] has to manage, meaning the more its going to be stuck managing the cluster, and so will not be able to run their own pods. This means that going from a single node cluster to a multinode cluster, you will probably need at least 3-4 additional machines for it to be managed and working properly. Yes you could run a 2 or 3 node cluster, but the benefits really are not there for the massive amount of downtime. On top of this, you probably want at least 1 node of wiggleroom capable of handling your highest power node, this means that you can never reach the full potential of your cluster, as both your control plane and your other nodes must be able to take the strain when that node is disconnected.

Storage is probably the largest issue that you will need to tackle, and is the one that put BOSS of going this route (along with the fact that we wanted to keep some of our servers as hot spares if parts broke). This will be explored more in the [[Kubernetes#How to manage Storage|"How to manage Storage" section]], but the summary is, although there are tools out there, they require a minimum node count of 3 and have significant overhead. Additionally, because of the potential constant movement of data between nodes going down (even for general updates), it causes additional wear and tear on your drives, reducing their life expectancy (I killed a brand-new SSD in 6 months seemingly because my nodes were going down for updates so often).

Networking is a minor one, all nodes need to be connected to control plane and cannot get randomly interrupted or cut off (e.g. you have your control plane in one room hooked up to a UPS and a agent node in another on a UPS, but the router that connects them is not on the UPS). Disconnection at critical times can cause the control node to start moving workloads off the disconnected node onto others, which with statefulsets or <code>ReadWriteMany</code> volumes can cause desynchronisation which is a massive issue and will cause corruption. Finally there is just a minor issue that if you don't setup correctly you will probably be quite confused, Traefik forwarding going between nodes may result in the final application being given the wrong IP address.

Finally, there is also a bit of a weird one, managing SELinux policies and kernel modules across the cluster. I have had multiple issues in the past of minor configuration differences between my nodes resulting in a pod crashlooping on one node and absolutely just fine on another. Therefore this creates additional overhead as you then have to make sure all installs are identical and then potentially install the security profile operator (which is great when it works but not so great when the docker container they use is ancient and doesn't work on the latest systems and k3s accidentally breaks support for it).

And so, in summary, multinode clusters are amazing for chaising that 99.99999999999% uptime and really interesting for learning the difficulties with it and all the solutions out there. But it is really not for the faint of heart, I would recommend trying with a bunch of VMs on a single node just to understand how it works, but if you are just a hobbiest wanting to host some basic applications at home, do not use it. I am planning on deconstructing my cluster and replacing it with a single node similar to [[Bath Open Source Society|BOSS]] after I leave univesity.

=== Control plane ===
The first challenge is the control plane. What's the benefit of a multinode cluster when it relys on a single node for sending control signals. If it goes down, the whole cluster will panic if a pod crashes, it is left in a seriously vulnerable position. Therefore, you instead need to expand your control plane to multiple nodes. K3s offers built in solution for this, [https://docs.k3s.io/datastore/ha-embedded?_highlight=etcd utilising <code>etcd</code>] by default to share the cluster state between control nodes, or you can just move it off to [https://docs.k3s.io/datastore/ha a separate database cluster] (which is what's recommended in production as you probably have another spare 3 machines). Then you must setup a load balancer between the control nodes (which actually helps with not overloading a single control node). But now the issue is that you are relying on a single load balancer. As a hacky solution, I setup a load balancer on all my control nodes which balanced the load between itself and the other nodes. This mean that the DNS record could store all IPs associated with the nodes and hopefully mitigate any communication issues when a single one went down.

However, when setting up etcd, you must remember to point the <code>--server</code> to the load balancer and not a single node (as then the single node becomes the single point of failure).

=== How to manage Storage ===
I personally used [https://longhorn.io/ longhorn], which provided a somewhat easy solution, you just have to change the default storage class and migrate all your PVs. It requires that you have 3 nodes to share data between (as it can then use voting logic to ward off corruption). By default, it then stores volumes on three different nodes at the same time, meaning if one node goes down, it will just make a copy to a new node with the remaining two copies. Additionally, if a pod is assigned to a node without a copy of the data, longhorn can simply make a new copy onto that node (however it is more likely that kubernetes will choose a node that already has the storage).

It then provides a UI where you can easily manually update all the storage volumes after an update of longhorn and also manage automated backups and snapshots. It is actually a really handy tool and generally really cool (though you have to manually label PVs with the types of backups you want).

The issue? It uses around 2GiB of RAM on each individual node. Then if your nodes are constantly going down (e.g. automated updates every week), it will instantly start panicking and moving volumes over which can wear out drives. Additionally, if too many nodes go down, it can be the single source of crashing everything else because of the amount of resources it can use.

=== Safely shutting down nodes ===
This is basically a requirement if you actually want to obtain the full benefits of multinode cluster that requires as little manual intervention as possible (which is still somehow more intervention than a single node cluster IMO). There is a great [https://oranki.net/posts/2025-01-09-graceful-k3s-shutdown/ blog on <code>oranki.net</code>] on how to set this up, but effectively you need to set that anytime the <code>k3s</code> service is stopped you run:<syntaxhighlight lang="sh">
kubectl drain --ignore-daemonsets --delete-emptydir-data <node>
</syntaxhighlight>Then when it comes back up you need to run:<syntaxhighlight lang="sh">
kubectl uncordon <node>
</syntaxhighlight>As explained in the blog post, you can do this via a systemctl service.

What this does is makes sure all nodes are safety migrated onto another node before a shutdown can take place, effectively tainting the node (meaning can be assigned to it). Then you must <code>uncordon</code> a node (meaning removing the taint) to tell the control plane that you can now place pods on it again.

== Backups ==

== Alerting ==

== Troubleshooting ==

=== Kubernetes event log ===

=== Traefik dashboard ===

=== Shell-ing into pods ===

=== K3S Log ===

=== Emergency Debug pods ===

=== Cloudflare API tokens ===

=== SELinux ===

=== Tofu ===

=== Scenarios ===

==== Deployment/statefulset created but pod not creating ====

==== Pod in cash loop ====

==== Pod cannot request any website ====

==== Pod can access the internet but everything returns self-signed certificate ====

==== Deployment can't access its database/valkey ====

==== Pod is requesting denied system privileges ====

==== Pod is up, but I get a 404 when requesting visiting the page ====

==== Pod is up, but I get a 403 when requesting visiting the page ====

==== Authelia refuses to start up ====

==== Node is crashing often after running out of RAM ====

==== On IPV6 cluster and requests randomly timeout or return 404 ====

==== Statefulset refusing to start pod (PVC) ====

==== A node just crashed and went offline ====

=== Cleaning up ===

User:Hw2210

2026-06-05T18:50:30Z

Hw2210: Some basic text I guess

I am Hugo, ex-Computer Science student and am the initial fouder of bathcs.com along with Pal Kerecsenyi and Penn Mackintosh and previously hosted froom and other projects on my homelab. [[Bath Open Source Society|BOSS]] was founded to replace me and have a sustainable hosting.

Initially the predecessor to BOSS was founded to provide a coordinated resources for the expanding number of computer science soceities as well as provide a service for storing your password (which is where [[VaultTub]] was born). From this we created the [https://gitlab.bath.ac.uk/cs "cs" organisation on Bath's GitLab] to store all code written so we can share common creations (such as the peer mentor website). Then we created a wiki to provide information about hosting stuff for free.

== Links ==

* LinkedIn: [https://www.linkedin.com/in/hugo-whittome-69053924a/ Hugo Whittome]
* Website: [https://wilfsilver.co.uk wilfsilver.co.uk]
* Mastodon: [https://mstdn.social/@wilfsilver @wilfsilver@mstdn.social]

Kubernetes

2026-06-05T17:37:22Z

Hw2210: Work on writing the deployment section.

Git/Advanced

2026-06-05T13:56:41Z

Hw2210: /* See also */ Include Git in pages

Git is a really powerful command line tool when you start looking into the advanced things you can do, but when maintaining or developing a project, you are probably not going to touch most of them. This is a list of ones which I use on a regular basis.

It’s important to note that the more advanced commands can have quite powerful functions which can lose code if used incorrectly. However, in most cases git caches a significant amount, so if you lose a commit or some changes, '''don’t panic''' and look up your issue (someone has done this before and found a way to get the changes back).

== More on Commit IDs ==

Where it says to use the <code>commit-id</code>, there are different inputs you can use to specify multiple commits easily.

Let A and B be the ids for two different commits. To reference a range you can do:

* A to B (including A): <code>A^..B</code>
* A to B (excluding A): <code>A..B</code>

Or you can specify the last <code>X</code> commits:

* <code>HEAD~X</code> e.g. <code>git reset --soft HEAD~4</code>

== Stashing ==

Stashing is used when you want to record the current state of the working directory and the index, but want to go back to a clean working directory.

It allows you to set aside some changes for later or just never commit them (without altering the [[Git/Special files#Gitignore|<code>.gitignore</code> file]]).

By stashing your changes, they will be removed so you can no longer see them, but you can always <code>pop</code> or <code>apply</code> the stash to get the back at any point.

=== Commands ===

<syntaxhighlight lang="sh">git stash # Stash current unstaged (but tracked) changes
git stash apply # Restore the last stash without deleting it
git stash pop # Restore the last stash and delete it
git stash list # List the stashed changes
git stash show # Inspect the stashed changes</syntaxhighlight>

{{Note|type=reminder|text=
GUI tools e.g. lazygit or VSCode (with git extension pack) normally have a better interface for adding and removing stashes - so probably use that instead.
}}This is usually used with <code>git switch</code> or <code>git checkout</code> to allow you to switch branches when you still have uncommitted changes that may conflict.

== Reverting vs Resetting ==

To either reset or revert, you need the git commit hash id (or the start of it), this can be done via <code>git log</code>.

Once you have that, the interface is quite similar:

<syntaxhighlight lang="sh">git reset commit-id
git revert commit-id</syntaxhighlight>
They both do the same thing of reverting the changes from the given commit. But the difference is, <code>revert</code> will create a new commit (therefore you can just run <code>git push</code> or merging without any issues), whereas <code>reset</code> will remove those commits outright.

When resetting, the tree (simply, the list of commits) itself is altered. This means that when pushing to a repo which already contains that commit, you have to use <code>git push --force</code>. This will cause anyone else working on the same branch to lose their changes, as they are forced to run <code>git pull --force</code>.

This command is dangerous so make sure you check everything before using it as it forces the upstream to be exactly like your local branch. So if you accidentally removed the wrong commit, you cannot easily get it back.

=== Extra options ===

With <code>reset</code> you also have extra options which you may find useful. You can either use <code>--soft</code> (Put all the changes of the commit in staged) or <code>--hard</code> (which is the default, just forget all changes).

With <code>revert</code> you can use <code>--no-commit</code> which will put the inverse of the changes in staged (and not create a new commit). This allows you to add multiple reverts or more changes in it.

== Rebasing ==

Rebasing is used when commits have been added to <code>main</code> and you want to bring them to your branch (with was branched off of <code>main</code>).

To rebase off of main (when you are in your branch), you can run:

<syntaxhighlight lang="sh">git rebase main</syntaxhighlight>
{{Note|text=
As we are changing the structure of the commit tree, you must then run <code>git push --force</code> once you are 100% happy none of your changes have been lost.

As usual, if there is anyone else using your current branch, you shouldn’t do this.
}}

The steps it is takes are:

* Temporarily reset all your commits which you added to this branch
* Apply all the new commits in <code>main</code> (or the given branch)
* Re-apply all of your commits

=== Dealing with Conflicts ===

Sometimes there may be conflicting changes from the changes added to main (e.g. you’ve changed the same line as another change).

This is when it gets quite confusing and dangerous.

If this happens, git will print out an error, saying what files are are in conflict and where to find them.

If you run <code>git status</code> you will see that some files are staged and some are not. The idea for these are:

* '''Staged files''': These are the files which are not conflicting and will be committed on <code>git rebase --continue</code>.
* '''Unstaged files''': These are the files with conflicts

If you open one of the conflicting files you will find that git has altered it where the conflicts are.

The format is:

<pre class="txt"><<<<<<< HEAD
- Auto writing README
- A cool logo - hw
- who's the above guy?
=======
- something
- A cool logo - hw
- other
>>>>>>> 8f309e1 (Test commit)</pre>
{{Note|text= class="note">
VSCode has its own custom interface for this [https://code.visualstudio.com/docs/sourcecontrol/overview#_3way-merge-editor read more here]
}}

You will see the changes you made are below the <code>=======</code> and the (updated) upstream code in <code>main</code> is above it.

What you have to do is, for each of these conflicts, to choose which one to keep (or create a mixture). To do this, you just remove everything that shouldn’t be there (which includes <code><<<<<<< HEAD</code>, <code>=======</code>, etc). For example, in this case it should result in:

<pre class="txt">- Auto writing README
- something
- A cool logo - hw
- other
- who's the above guy?</pre>
Once you are happy an entire file is now conflict-free and correct, you can stage it.

Then once there are no unstaged files left you can run <code>git rebase --continue</code>, which will save the changes under the original commit (sometimes it will ask you to confirm the commit message with your editor, you can just save and exit it).

Once you have finished rebasing, make sure to test that your code still works. It’s common for code to break after rebasing due to unexpected changes made by someone else.

=== Interactive Rebase ===

Interactive rebasing is one of the most powerful and fun commands ever. However it comes with the downside of it being quite dangerous.

It is used for reorganising a merge request and managing commits to clean up the git log.

To use it:

<syntaxhighlight lang="sh">git rebase -i HEAD~X
# Or interactive rebase off of main
git rebase -i main</syntaxhighlight>
This should bring up your default text editor with a list of commits with a list of commands at the end, which looks like:

<pre>pick af44004 Annoying rebasing commit
pick ceb4454 Add quote from Varnie
pick 1f156e8 Test commit

# Rebase 315c076..1f156e8 onto 315c076 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
# commit's log message, unless -C is used, in which case
# keep only this commit's message; -c is same as -C but
# opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# create a merge commit using the original merge commit's
# message (or the oneline, if no original merge commit was
# specified); use -c <commit> to reword the commit message
# u, update-ref <ref> = track a placeholder for the <ref> to be updated
# to this position in the new commits. The <ref> is
# updated at the end of the rebase
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#</pre>
You can see all the commits which will be re-applied after the branch has been rolled back and commits from main been added.

These all have the word <code>pick</code> infront of them, meaning they will be committed as is, with the same message, and no editing happening.

You can look over the other commands to see what you can do, but how it works is by replacing <code>pick</code> with something else, e.g. <code>fixup</code> or <code>reword</code>.

=== The Fixup Commit ===

This is a weird option when committing, that can be useful when you have an MR and you are wanting to keep a neat commit log, but are responding to review feedback.

How this works, is lets say you’ve fixed something with a previous commit in your MR and were wanting to (when merging) squash this fix into that commit. But you want to have it as a separate commit to help the reviewer to see that you have fixed it.

To do this you simply run:

<syntaxhighlight lang="sh">git commit --fixup="amend:<git_commit_id>"</syntaxhighlight>
Which will create a new commit with the message:

<pre>amend! Add information for publicising events from BCSS and where to go

> Add information for publicising events from BCSS and where to go</pre>
Which you can then push as its own commit.

Then when you come to merge you can run:

<syntaxhighlight lang="sh">git rebase -i main --autosquash</syntaxhighlight>
Which will automatically move your amend commit to be <code>fixup</code> above the commit you were amending.

== Merging ==

Normally merging is done via the interface on the remote repository system you are using. However it can be done in the command line as well (it can also be used in place of <code>rebase</code> so you don’t have to use the dangerous <code>git push --force</code>).

It copies all the commits which have been added on a branch to the branch you are currently on (e.g. <code>main</code>).

So the process to merge a branch into main is:

<syntaxhighlight lang="sh">git checkout main
git merge branch-name
git branch -d branch-name # Deletes the branch If you don't need the branch any
# more
git push</syntaxhighlight>
Conflicts can still happen, see [[Git/Advanced#dealing-with-conflicts|above]] for more information about how to manage them.

{{Note|text=
When merging, cherry-picking or other, and you are dealing with conflicts you need to use the subcommand with <code>--continue</code>, for example <code>git merge --continue</code> or <code>git cherry-pick --continue</code>.
}}

== Cherry-picking ==

Cherry picking allows you to bring a single commit (or multiple, see [[#more-on-commit-ids|here]]) from another branch to your current one.

To do this, it is as simple as:

<syntaxhighlight lang="sh">git cherry-pick commit-id
git push</syntaxhighlight>
== Submodules ==

Submodules are normally used in what is called a “monorepo”, a repo which stores multiple difference projects or git repositories.

It is also useful for refactoring some files. E.g. if you need the same files in multiple different projects (e.g. standardised tests or config files), it is common to add a “meta” repo which stores these projects, then add this as a submodule to each project which uses it.

To add a submodule you can run:

<syntaxhighlight lang="sh">git submodule add repo-url folder/to/store</syntaxhighlight>
This will clone the module inside the folder <code>folder/to/store</code> and will add a <code>.gitmodule</code> file in the base of the repo.

When cloning the repo on other devices, you must remember to recursively clone all the submodules as well via:

<syntaxhighlight lang="sh">git clone --recurse-submodules -j8 project-url</syntaxhighlight>

{{Note|text=
<code>-j8</code> is an performance optimisation see [https://stackoverflow.com/questions/3796927/how-do-i-git-clone-a-repo-including-its-submodules#4438292 this Stack Overflow answer]
}}

Or if you have already cloned the repo and were wanting to update all the submodules, you can run:

<syntaxhighlight lang="sh">git submodule update --init --recursive</syntaxhighlight>

=== Managing submodules ===

Once you have cloned a submodule, you will note that any time you pull the latest changes to it, you need to make another commit in the base repo with the update.

This is so that the submodules are locked on specific commits until you specifically say “yes this next commit is fine”.

== Extra Configuration ==

Sometimes there are configuration options that git will recommend when they become a problem, for example:

<syntaxhighlight lang="sh">git config --global pull.rebase true</syntaxhighlight>
This means that when pulling from a branch which has new changes, it will rebase instead of merging the new commits.

=== Commit Signing ===

Commit signing is used to verify if you are who you say you are when committing (e.g. with your email address).

I won’t go into much depth on this, instead just know it exists and is quite good practice to have but not necessary.

You can read more [https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits here].

Be aware that there are some consequences which come along with this:

* If you have setup a strict mode with signing, you cannot commit if you loose access to your signing key
* When rebasing your changes, you have to resign all the commits (meaning you are the only one who can do it)

== See also ==
{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Git/Special files

2026-06-05T13:56:16Z

Hw2210: /* See also */ Don't show redirects

In git there are some files which have some importance, either on your remote repository or just in general.

== README ==

File location: <code>README.md</code>.

When creating a repository on GitHub or GitLab, it will ask if you want to initialise it with a README. This is a file written in [https://confluence.atlassian.com/bitbucketserver/markdown-syntax-guide-776639995.html markdown], which just contains a description of what the project is.

If you visit a project e.g. [https://gitlab.bath.ac.uk/cs/services/froom froom], you will see a description rendered below all the files. This is the README, which both GitLab and GitHub will render on the home directory of your project.

== LICENSE ==

File location: <code>LICENSE</code> or <code>LICENSE.txt</code>.

This stores how a project can be used and altered, if none is given it can be assumed that all rights are reserved. There are strict laws protecting these statuses and so when copying code from open sources projects, please make sure you understand how you can use that code and what the license permits

For [[BOSS]] projects, we have our own [https://boss.bathcs.com/policies/licenses/ License policy] which states all our projects should use the [https://spdx.org/licenses/LLVM-exception.html <code>Apache-2.0 WITH LLVM-exception</code>] license If you are contributing to BOSS, please read the [https://boss.bathcs.com/policies/licenses/|license policy] for the full list of requirements.

== Gitignore ==

File location: <code>.gitignore</code>

This file stores a list of files or folders which should never be included in your git commits.

These will usually include caching files or output directories which should be generated by the developer on their local machine.

An example for a project using <code>deno</code>, <code>yarn</code> and <code>npm</code> is:

<pre># build output
dist/

node_modules
.yarn
!.yarn/releases
!.yarn/plugins
.pnp.*

.vscode
!.vscode/extensions.json
!.vscode/launch.json

# logs
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*

# environment variables
.env
.env.production

# macOS-specific files
.DS_Store
</pre>

But these will vary depending on the tools and language you are using in a given project.

[https://github.com/github/gitignore Templates] are available online or you can use [https://gitignore.io generate a gitignore] specific to your development setup.

== Gitmodules ==

File location <code>.gitmodules</code>

In most cases, this is not required and so will not be in a repository, this is a special file which is normally autogenerated by the [[Git/Advanced#Submodules|<code>submodule</code> command]] in git (and so probably shouldn’t be edited manually).

But this stores all the locations and repos of the submodules you have added.

== GitLab CI ==

File location <code>.gitlab-ci.yml</code>

This file controls the pipelines on the project, [[Git/GitLab#CI Pipelines|see here for more information]].

== MR/Issue Templates ==

File location <code>.gitlab/merge_request_templates/*.md</code> and <code>.gitlab/issue_templates/*.md</code>.

This is better explained [[Git/GitLab#MR Templates|here]].

== See also ==

{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Git/Mirrors

2026-06-05T13:56:00Z

Hw2210: /* See also */ Don't show redirects

Some projects could be a student’s heart and soul. We do not want to take this away from them, instead we just want to store a copy of the current version on bathcs.com.

So the solution to this is to have a mirror repo.

There are multiple ways to solve this:

== GitLab Mirror ==

In GitLab go to “Settings > Repository > Mirroring Repositories”, you can see there is an option to “push” to another repo.

This means that if the student has their own GitLab instance where they store their own version, they can setup this pushing feature to push to a repo on the GitLab.

However, due to security restrictions, this only works when the other server is connected to the University Intranet, so it is not feasible.

== Just do it manually ==

When doing it manually, you can just push to two separate repositories when making changes locally, by going:

<syntaxhighlight lang="sh">git add remote bath git@gitlab.bath.ac.uk:cs/wiki
git push
git push bath</syntaxhighlight>
If you were wanting to have a specialised branch which has some extra features, you can do something like:

<syntaxhighlight lang="sh">git checkout branch-name
git push bath HEAD:main</syntaxhighlight>

== See also ==

{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Git/GitLab

2026-06-05T13:55:45Z

Hw2210: /* See also */ Don't show redirects

We require that the websites hosted on bathcs.com should be stored in [https://gitlab.bath.ac.uk/cs the cs organisation on Bath’s GitLab instance]. Any project used by student-led initiatives can also be stored on here.

GitLab was chosen because it has a lot more (useful) features than GitHub. However this does mean that it is aimed at people who has a basic knowledge of how to properly manage git projects.

Therefore you will need to be able to use GitLab.

{{Note|type=warn|text=
You cannot push or pull to the GitLab outside of the VPN, even though you can access the page.
}}

== Setup ==

You might have set this up when handing over, if you haven’t, here are the instructions. Note you will have to ask an organisation manager to add you to the organisation if you are going to be editing any current projects.

* Go to https://gitlab.bath.ac.uk
* Login with your University credentials
* Setup TOTP as your 2FA (one time codes). See [[2FA#TOTP|our 2FA section]] for more information about what this is and how to add one (you can skip to the section where it says “Once you see a QR code”)

Once you have logged in, your account will be created and so can be added to organisations or tagged (using @username).

=== SSH Keys ===

If you want to commit or clone repos, you will also need to setup an SSH key:

# Create an SSH key: see [[SSH keys|our documentation]] for information on how to set one up.
# Go to your preferences (click profile icon then click preferences).
# Click “SSH Keys”.
# Copy and paste the public key into the “Key” section.
# Give it a title.
# Change usage type to “Authentication only”.
# Click “Add key”.

By default this will expire in a year, after which you will have to add another key in the same way.

=== Extra Emails ===

You may have already set up git with your personal email address or you have a git signing key with a specific email. You probably don’t want to edit the configuration on a per project basis.

To solve this, GitLab and GitHub allow you to have multiple email addresses linked to your account, this can be found by going to preferences (click the profile icon then preferences) then clicking “Emails”.

You can then type in another email into the box and click “Add email address”. You will then have to verify the email by clicking the link it gives you.

== The CS Organisation ==

All Student Led Initiative projects are stored on the [https://gitlab.bath.ac.uk/cs Computer Science SLI organisation]. If you are a committee member you should have been given access to it. If not please contact the current owners, who can be found by going “Group information > Members”.

[[File:git-gitlab_members.gif|Members page on GitLab showing the current owners and maintainers found by going to Members in the navigation group “Group Information”]]

The organisation or group has subgroups for all societies or student-led initiatives. You should have developer rights on all initiatives which you are a part of, which allows you to create branches on the existing projects and push to them. If you ask nicely you may get maintainer rights, which will allow you to create new projects and merge MRs into main (however you still can’t push to the <code>main</code> branch).

There are some projects which live outside the subgroups (e.g. this wiki) as they don’t have a organisation apart of. You will have to specifically ask permission to access individual projects for security reasons.

== Issues ==

=== Creating ===

[[File:git-create_issue.gif|gif showing the steps of going to the issues menu under pinned and clicking new issue button]]

Creating an issue is easy, you can click on the “Issues” tab under “pinned” and click “New Issue” and start typing. The description is written in markdown. You can [https://docs.gitlab.com/ee/user/markdown.html learn markdown on the GitLab website].

If the project is configured with templates, there should be a dropdown for you to choose the one most relavent to your issue type.

[[File:git-mr_templates.gif|gif showing the dropdown to select the template to use above the description]]

You can then assign someone an issue from the menu if you know who will be working on it. Labels are also available, but these are mostly for the maintainers of the project to use.

Once the issue is created, it will assign the issue a number (you can see this in the URL), which you can use to cross-reference issues or MRs by putting a <code>#</code> before it. GitLab will pick this up and turn it into a link (you can also just put the link and it will be sortened). Note that issues on other projects can also be referenced.

=== Managing Issues ===

There is not much to say about managing issues. All conversations about the issue should be done on the issue itself.

Once someone is working on it, they should be assigned the issue and it should be referenced in the MR which fixes it.

If “Closes” is put before the link to the issue on an MR, once it is merged, the issue should automatically be closed (but please always do check all the related issues are closed).

== Forking + Branches ==

If you are a developer or maintainer on a project, you should be able to create a branch on the project (but just not push directly main). Branches should be under the format <code><username>/<fix_summary></code>.

If you are not a developer but would like to contribute to a project, you should be able to fork the repository into your own space to create make any edits you wish.

== Merge Requests ==

=== Creating ===

[[File:git-create_mr.gif|gif showing the steps of going to the merge requests menu under pinned and clicking new merge request button and choosing a branch to merge into main]]

Once you have created a fork or branch, you can create a merge request, either by clicking the link which is shown after pushing, or going onto the project and clicking “Merge requests” at the left and choosing “New merge request”. You can then choose the branch you want to merge from and the one you are merging to (most likely <code>main</code>).

The interface is very similar to issues, with the summary written in markdown and templates available.

If the Merge Request is not finished, and so still a work in progress, you can click the “Mark Draft” button or put “DRAFT:” at the start of the summary.

You can then assign the MR to yourself (as you will be managing it). You then also want to set the reviewer to a maintainer who can then merge the issue.

If the MR depends on another MR to be merged beforehand, please add the “blocked” label and mention what its blocked on in a comment or the issue directly.

=== Managing ===

Sadly as the university does not pay for premium GitLab, we have to do most security practices as manual steps. So please do not go against these procedures, we beg of you.

If you are the reviewer of a MR, you can go onto the “Changes” tab where you can see all the differences between this and the main branch. There is a settings button where you can change parts of the look. I recommend enabling “show whitespace changes” to help find where people have accidentally left trailing whitespace.

You can then add a comment on a line by hovering over the line and clicking the comment button on the left hand side. You can also do this on images if you wish.

When adding a comment you have the option to “start a review” which is recommended if you are going to be making multiple comments as it send them out all at once instead of having a separate email for each comment (which is a lot).

If you drag the comment, you can also select multiple lines which is useful for segments of code which need changing.

[[File:git-start_review.gif|gif showing going to commits tab and adding a comment to the review, selecting multiple lines]]

It is expected that the assignee will then resolve all of these issues so it is not up to the reviewer (but as we are students they may need chasing).

Once you are happy with the MR, you can then approve the merge request before merging it.

Please make sure that:

* You don’t approve your own MR
* Don’t merge something without approval (if it is going into <code>main</code>)
* Don’t approve without actually reading all the changes

== Creating a new Project ==

Over time we may want to add more projects to the organisation. This page focuses on creating projects from scratch. If you are just wanting to mirror, see the [[Git/Mirrors|page about mirrors]].

When creating a new repo or project, you need to decide which subgroup it should go under. This should be quite simple, however you may decide a new subgroup is required.

If you are not a maintainer in a repository, you will have to find the maintainers of the subgroup and ask one of them to set up the repo for you and then add you to the access list.

Other things to consider when creating a project:

* Name (please make this understandable about what it is for)
* Visibility level:
** '''Private''': only people with access can see it (preferably not), however is understandable if you are just testing out a project.
** '''Internal''': Only bath students can see it when logged in, this is useful for things which interact with University of Bath services which they may not want known to the public.
** '''Public''': This should be used for most projects to encourage the open source nature of the community.

Once a repository is created you can add it as a remote to your repo or clone it.

You should then:

* Go to “Settings > Repository > Protected branches” and change “Allowed to push and merge” to be “No one” and make sure “Merge MRs” is set to “Maintainers only”.
* Add a description or logo in the “Settings > General” tab
* Add a license as explained [[Git/Special files#license|here]].

[[File:git-setup_new_project.gif|gif showing the process of editing the protected branches to only allow maintainers to merge to main and no one to push to the main]]

=== Manage Access ===

You can choose who can access the repository in “Manage > Members” if you are a Maintainer or Owner.

To add new person you can click “Invite members” and type in their username (they will need to have signed into GitLab for their username to appear).

You then need to choose their role. It is recommended to put them as “Developer” unless you want them to be able to merge requests into <code>main</code>, then put them as “Maintainer”.

=== Labels ===

Under “Manage > Labels” you can add or create labels which can be used on merge requests or issues. You should inherit any labels from the parent group, and preferably labels should be group wide. But you can create a label and ask an Owner to upgrade the label to a group label by clicking the 3 dots.

=== Issue + MR Templates ===

In your project, you may want to have templates for people creating issues or MRs to save them time and to standardise them.

This is quite simple to do in GitLab as explained [https://docs.gitlab.com/ee/user/project/description_templates.html here].

In short, you need to add markdown files to <code>.gitlab/issue_templates/</code> or <code>.gitlab/merge_request_templates/</code>. The default should be named <code>default.md</code>.

Examples:

<ul>
<li><code>.gitlab/issue_templates/default.md</code>
<pre class="md">### Summary



### Steps to reproduce

</pre></li>
<li><code>.gitlab/merge_request_templates/default.md</code>
<pre class="md">### Related Issue



Closes #

### Summary



### Testing

</pre></li>
<li><code>.gitlab/merge_request_templates/bug_fix.md</code>
<pre class="md">### Summary



### Testing

</pre></li></ul>

=== CI Pipelines ===

GitLab pipelines are super powerful and I don’t think I could write better documentation than GitLab themselves (which can be found [https://docs.gitlab.com/ee/ci/pipelines/ here]).

This is just here to say they do exist and they should be used for automated testing or even deployment.

The configuration can be found in the <code>.gitlab-ci.yml</code> file.

== See also ==

{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Git/Basic Components

2026-06-05T13:55:04Z

Hw2210: /* See also */ Don't show redirects

Git is split up into multiple different subcommands. Here are a few essential ones to understand to use git.

== Initialisation ==

This is not entirely necessary, however it helps to understand what is happening when you clone a repository and the fact that a folder can be initialised with git without being reliant on a remote repository.

When creating a new project locally, you can make the repository be version-controlled by running

<syntaxhighlight lang="sh">git init</syntaxhighlight>
If you then run <code>git status</code> you will see something like:

<pre class="txt">On branch main
nothing to commit, working tree clean</pre>

[[File:git-git_init.gif|gif showing what is explained in this section of initialising a git repo]]

=== Adding a remote ===

Adding a remote repository is not necessary unless you want to be pushing to one as some sort of backup or if you just want to share your code.

To add one (you can also have multiple), you simply run:

<syntaxhighlight lang="sh">git remote add origin <url></syntaxhighlight>

{{Note|text=
Here we are creating a remote called <code>origin</code>, you can change this name if you have multiple. The default name is <code>origin</code> and is what we will be using later on.
}}

[[File:git-git_remote_add.gif|gif showing the process of adding a remote repo and pulling from it]]

== Cloning ==

Cloning basically does everything we explained above automatically for you and then pulls the latest commits to your folder.

The command to clone a repository is:

<syntaxhighlight lang="sh">git clone <url></syntaxhighlight>
This clones the repo into the folder with the same name as the project. E.g. if I were to clone <code>git@gitlab.bath.ac.uk:cs/wiki</code>, it would create a folder called <code>wiki</code>.

If you want a custom folder name, you just simply add another parameter e.g:

<syntaxhighlight lang="sh">git clone git@gitlab.bath.ac.uk:cs/wiki bathcs_wiki</syntaxhighlight>

[[File:git-git_clone.gif|gif showing the process of cloning a repository]]

== Staging and Committing ==

Staging and committing are essential parts of git. The process goes:

* You make some changes
* You stage those changes (basically saying “yes I want these changes to be in the next commit”)
* You commit those changes

By commit, we mean creating a group of changes which gets given an identifier and a summary. We can then use this ID to revert or do something else with those changes later on.

The summary or description just helps you understand what the changes are doing, so you can easily find a commit later on.

=== Staging ===

To stage something you can:

<syntaxhighlight lang="sh">git add file/to/add # To stage one file at a time
git add . # To stage all changes in current directory</syntaxhighlight>
You can then run:

<syntaxhighlight lang="sh">git status</syntaxhighlight>
To see the current unstaged and staged changes.

If you were wanting to unstage all the changes, you can simply go:

<syntaxhighlight lang="sh">git reset # to unstage all staged changes
git reset file/to/unstage # to unstage a particular file</syntaxhighlight>
{{Note|text=
Unstaging does not delete the changes, it just removes it from the “staged” list (which is used when you run <code>commit</code>)
}}

=== Committing ===

Once you have staged the changes you want in the next commit, you can run:

<syntaxhighlight lang="sh">git commit -m "Add setup to README.md"</syntaxhighlight>
Which will commit the staged changes with the message “Add setup to README.md”. If you don’t specify a message, git will open your preferred text editor where you will be forced to add one.

{{Note|text=
All commit summaries should be written in '''the present tense''' and should explain what the changes do.
}}

Options:

* Specify <code>-a</code> to include all unstaged changes as well (except for new files)
* If you add a new line after the summary, you can add a more in-depth description.

[[File:git-git_stage.gif|gif showing the process of adding a file and then staging and committing it]]

=== Amending a commit ===

This is more of an advanced command, but it can be helpful. You probably shouldn’t use this, because it will break things if other people have cloned the repository. If you want to add some changes to your previous commit, there is an <code>--amend</code> option which will add all staged changes to the previous commit. It will also give you a chance to edit the commit message.

<syntaxhighlight lang="sh">git commit --amend
git commit --amend --no-edit # If you want to keep the same commit message</syntaxhighlight>
As this changes the commit itself, if you have previously pushed that commit, you must then run <code>git push --force</code> (warning: this is a dangerous command).

{{Note|text=
Generally, you should create a new commit rather than amending an existing one.

Amending is only acceptable when you are working on your own personal branch, or if you have not yet pushed to the remote.
}}

== History ==

If you were wanting to get the id of a commit or just see the history of changes, you can look at the log via going:

<syntaxhighlight lang="sh">git log --oneline</syntaxhighlight>
{{Note|text=
If <code>--oneline</code> is not specified, it will show all the information about each commit on multiple lines (which is normally a lot harder to read)
}}

This will show you the start of the commit hash (or id) and the summary next to it, including what branch is at each commit.

[[File:git-git_log.gif|gif showing the output of running git log]]

== Branching ==

Branching is another key mechanic in git. Allowing multiple people to work on multiple different new features without getting in each others’ way.

It basically creates a parallel copy of the codebase: a “branch”. You can then add commits to this copy without affecting the original branch. Once you are happy with the changes you can create a [[#merge-or-pull-request|merge or pull request]] to then merge it back into the original branch.

Therefore the process for adding a feature on production is:

* Create a branch from <code>main</code> (normally named something like <code><username>/<description></code> in <code>kebab-case</code>)
* Add commits to fix the bug or add a feature
* Create a MR (or PR)
* Merge the branch back into <code>main</code>

{{Note|text=
Only maintainers can merge a branch back into <code>main</code> for security reasons and so you normally get one to “review” your MR.
}}

=== Commands ===

By default the initial branch should be <code>main</code> (you may see <code>master</code> being used which is another option for the default name). This branch normally stores the stable release of the project.

<syntaxhighlight lang="sh">git branch # List all local branch names
git branch branch-name # Create a new branch from the current
git switch branch-name # Switch to the branch "branch-name"
git switch -c branch-name # Create and switch to the branch "branch-name"</syntaxhighlight>

[[File:git-switch_branch.gif|gif showing the process of creating and switching to a new branch]]

Previously switching branches was incorporated into <code>checkout</code> (which does a lot of other things), however <code>switch</code> was designed to provide a cleaner interface for just switching branches and so we recommend using this instead.

== Pushing & Pulling ==

If you have setup a remote repo, you will probably want to push and pull to it. Meaning: get (“pull”) the latest commits from it and send (“push”) your local commits to it.

To push it is as simple as:

<syntaxhighlight lang="sh">git push</syntaxhighlight>
However, when pushing a branch for the first time (after its creation not cloning) you must set its upstream:

<syntaxhighlight lang="sh">git push --set-upstream origin branch-name
# Or the shorthand:
git push -u origin branch-name</syntaxhighlight>

[[File:git-git_push.gif|gif showing the process of pushing the new branch to the remote repo]]

This tells git that when running <code>git push</code> you want to push to the remote named <code>origin</code> by default.

For pulling, its even simpler, just:

<syntaxhighlight lang="sh">git pull</syntaxhighlight>

== Merge or Pull Request ==

“Merge Requests” is what GitLab calls it and “Pull Requests” is what GitHub calls it.

These are just items or “requests” where a user wants their changes merged into the <code>main</code> branch (or any branch for that matter).

These are usually created and managed via GitHub or GitLab and allow another person to review your changes and merge them (if it is what they want).

To see how to create an MR in GitLab see [[Git/GitLab#Merge Requests|the instructions here]].

[[Category:Tutorial]]

== See also ==

{{Special:PrefixIndex|prefix=Git|namespace=0|hideredirects=1}}

Git/Advanced

2026-06-05T13:54:26Z

Hw2210: /* See also */ Don't show redirects

Git is a really powerful command line tool when you start looking into the advanced things you can do, but when maintaining or developing a project, you are probably not going to touch most of them. This is a list of ones which I use on a regular basis.

It’s important to note that the more advanced commands can have quite powerful functions which can lose code if used incorrectly. However, in most cases git caches a significant amount, so if you lose a commit or some changes, '''don’t panic''' and look up your issue (someone has done this before and found a way to get the changes back).

== More on Commit IDs ==

Where it says to use the <code>commit-id</code>, there are different inputs you can use to specify multiple commits easily.

Let A and B be the ids for two different commits. To reference a range you can do:

* A to B (including A): <code>A^..B</code>
* A to B (excluding A): <code>A..B</code>

Or you can specify the last <code>X</code> commits:

* <code>HEAD~X</code> e.g. <code>git reset --soft HEAD~4</code>

== Stashing ==

Stashing is used when you want to record the current state of the working directory and the index, but want to go back to a clean working directory.

It allows you to set aside some changes for later or just never commit them (without altering the [[Git/Special files#Gitignore|<code>.gitignore</code> file]]).

By stashing your changes, they will be removed so you can no longer see them, but you can always <code>pop</code> or <code>apply</code> the stash to get the back at any point.

=== Commands ===

<syntaxhighlight lang="sh">git stash # Stash current unstaged (but tracked) changes
git stash apply # Restore the last stash without deleting it
git stash pop # Restore the last stash and delete it
git stash list # List the stashed changes
git stash show # Inspect the stashed changes</syntaxhighlight>

{{Note|type=reminder|text=
GUI tools e.g. lazygit or VSCode (with git extension pack) normally have a better interface for adding and removing stashes - so probably use that instead.
}}This is usually used with <code>git switch</code> or <code>git checkout</code> to allow you to switch branches when you still have uncommitted changes that may conflict.

== Reverting vs Resetting ==

To either reset or revert, you need the git commit hash id (or the start of it), this can be done via <code>git log</code>.

Once you have that, the interface is quite similar:

<syntaxhighlight lang="sh">git reset commit-id
git revert commit-id</syntaxhighlight>
They both do the same thing of reverting the changes from the given commit. But the difference is, <code>revert</code> will create a new commit (therefore you can just run <code>git push</code> or merging without any issues), whereas <code>reset</code> will remove those commits outright.

When resetting, the tree (simply, the list of commits) itself is altered. This means that when pushing to a repo which already contains that commit, you have to use <code>git push --force</code>. This will cause anyone else working on the same branch to lose their changes, as they are forced to run <code>git pull --force</code>.

This command is dangerous so make sure you check everything before using it as it forces the upstream to be exactly like your local branch. So if you accidentally removed the wrong commit, you cannot easily get it back.

=== Extra options ===

With <code>reset</code> you also have extra options which you may find useful. You can either use <code>--soft</code> (Put all the changes of the commit in staged) or <code>--hard</code> (which is the default, just forget all changes).

With <code>revert</code> you can use <code>--no-commit</code> which will put the inverse of the changes in staged (and not create a new commit). This allows you to add multiple reverts or more changes in it.

== Rebasing ==

Rebasing is used when commits have been added to <code>main</code> and you want to bring them to your branch (with was branched off of <code>main</code>).

To rebase off of main (when you are in your branch), you can run:

<syntaxhighlight lang="sh">git rebase main</syntaxhighlight>
{{Note|text=
As we are changing the structure of the commit tree, you must then run <code>git push --force</code> once you are 100% happy none of your changes have been lost.

As usual, if there is anyone else using your current branch, you shouldn’t do this.
}}

The steps it is takes are:

* Temporarily reset all your commits which you added to this branch
* Apply all the new commits in <code>main</code> (or the given branch)
* Re-apply all of your commits

=== Dealing with Conflicts ===

Sometimes there may be conflicting changes from the changes added to main (e.g. you’ve changed the same line as another change).

This is when it gets quite confusing and dangerous.

If this happens, git will print out an error, saying what files are are in conflict and where to find them.

If you run <code>git status</code> you will see that some files are staged and some are not. The idea for these are:

* '''Staged files''': These are the files which are not conflicting and will be committed on <code>git rebase --continue</code>.
* '''Unstaged files''': These are the files with conflicts

If you open one of the conflicting files you will find that git has altered it where the conflicts are.

The format is:

<pre class="txt"><<<<<<< HEAD
- Auto writing README
- A cool logo - hw
- who's the above guy?
=======
- something
- A cool logo - hw
- other
>>>>>>> 8f309e1 (Test commit)</pre>
{{Note|text= class="note">
VSCode has its own custom interface for this [https://code.visualstudio.com/docs/sourcecontrol/overview#_3way-merge-editor read more here]
}}

You will see the changes you made are below the <code>=======</code> and the (updated) upstream code in <code>main</code> is above it.

What you have to do is, for each of these conflicts, to choose which one to keep (or create a mixture). To do this, you just remove everything that shouldn’t be there (which includes <code><<<<<<< HEAD</code>, <code>=======</code>, etc). For example, in this case it should result in:

<pre class="txt">- Auto writing README
- something
- A cool logo - hw
- other
- who's the above guy?</pre>
Once you are happy an entire file is now conflict-free and correct, you can stage it.

Then once there are no unstaged files left you can run <code>git rebase --continue</code>, which will save the changes under the original commit (sometimes it will ask you to confirm the commit message with your editor, you can just save and exit it).

Once you have finished rebasing, make sure to test that your code still works. It’s common for code to break after rebasing due to unexpected changes made by someone else.

=== Interactive Rebase ===

Interactive rebasing is one of the most powerful and fun commands ever. However it comes with the downside of it being quite dangerous.

It is used for reorganising a merge request and managing commits to clean up the git log.

To use it:

<syntaxhighlight lang="sh">git rebase -i HEAD~X
# Or interactive rebase off of main
git rebase -i main</syntaxhighlight>
This should bring up your default text editor with a list of commits with a list of commands at the end, which looks like:

<pre>pick af44004 Annoying rebasing commit
pick ceb4454 Add quote from Varnie
pick 1f156e8 Test commit

# Rebase 315c076..1f156e8 onto 315c076 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
# commit's log message, unless -C is used, in which case
# keep only this commit's message; -c is same as -C but
# opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# create a merge commit using the original merge commit's
# message (or the oneline, if no original merge commit was
# specified); use -c <commit> to reword the commit message
# u, update-ref <ref> = track a placeholder for the <ref> to be updated
# to this position in the new commits. The <ref> is
# updated at the end of the rebase
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#</pre>
You can see all the commits which will be re-applied after the branch has been rolled back and commits from main been added.

These all have the word <code>pick</code> infront of them, meaning they will be committed as is, with the same message, and no editing happening.

You can look over the other commands to see what you can do, but how it works is by replacing <code>pick</code> with something else, e.g. <code>fixup</code> or <code>reword</code>.

=== The Fixup Commit ===

This is a weird option when committing, that can be useful when you have an MR and you are wanting to keep a neat commit log, but are responding to review feedback.

How this works, is lets say you’ve fixed something with a previous commit in your MR and were wanting to (when merging) squash this fix into that commit. But you want to have it as a separate commit to help the reviewer to see that you have fixed it.

To do this you simply run:

<syntaxhighlight lang="sh">git commit --fixup="amend:<git_commit_id>"</syntaxhighlight>
Which will create a new commit with the message:

<pre>amend! Add information for publicising events from BCSS and where to go

> Add information for publicising events from BCSS and where to go</pre>
Which you can then push as its own commit.

Then when you come to merge you can run:

<syntaxhighlight lang="sh">git rebase -i main --autosquash</syntaxhighlight>
Which will automatically move your amend commit to be <code>fixup</code> above the commit you were amending.

== Merging ==

Normally merging is done via the interface on the remote repository system you are using. However it can be done in the command line as well (it can also be used in place of <code>rebase</code> so you don’t have to use the dangerous <code>git push --force</code>).

It copies all the commits which have been added on a branch to the branch you are currently on (e.g. <code>main</code>).

So the process to merge a branch into main is:

<syntaxhighlight lang="sh">git checkout main
git merge branch-name
git branch -d branch-name # Deletes the branch If you don't need the branch any
# more
git push</syntaxhighlight>
Conflicts can still happen, see [[Git/Advanced#dealing-with-conflicts|above]] for more information about how to manage them.

{{Note|text=
When merging, cherry-picking or other, and you are dealing with conflicts you need to use the subcommand with <code>--continue</code>, for example <code>git merge --continue</code> or <code>git cherry-pick --continue</code>.
}}

== Cherry-picking ==

Cherry picking allows you to bring a single commit (or multiple, see [[#more-on-commit-ids|here]]) from another branch to your current one.

To do this, it is as simple as:

<syntaxhighlight lang="sh">git cherry-pick commit-id
git push</syntaxhighlight>
== Submodules ==

Submodules are normally used in what is called a “monorepo”, a repo which stores multiple difference projects or git repositories.

It is also useful for refactoring some files. E.g. if you need the same files in multiple different projects (e.g. standardised tests or config files), it is common to add a “meta” repo which stores these projects, then add this as a submodule to each project which uses it.

To add a submodule you can run:

<syntaxhighlight lang="sh">git submodule add repo-url folder/to/store</syntaxhighlight>
This will clone the module inside the folder <code>folder/to/store</code> and will add a <code>.gitmodule</code> file in the base of the repo.

When cloning the repo on other devices, you must remember to recursively clone all the submodules as well via:

<syntaxhighlight lang="sh">git clone --recurse-submodules -j8 project-url</syntaxhighlight>

{{Note|text=
<code>-j8</code> is an performance optimisation see [https://stackoverflow.com/questions/3796927/how-do-i-git-clone-a-repo-including-its-submodules#4438292 this Stack Overflow answer]
}}

Or if you have already cloned the repo and were wanting to update all the submodules, you can run:

<syntaxhighlight lang="sh">git submodule update --init --recursive</syntaxhighlight>

=== Managing submodules ===

Once you have cloned a submodule, you will note that any time you pull the latest changes to it, you need to make another commit in the base repo with the update.

This is so that the submodules are locked on specific commits until you specifically say “yes this next commit is fine”.

== Extra Configuration ==

Sometimes there are configuration options that git will recommend when they become a problem, for example:

<syntaxhighlight lang="sh">git config --global pull.rebase true</syntaxhighlight>
This means that when pulling from a branch which has new changes, it will rebase instead of merging the new commits.

=== Commit Signing ===

Commit signing is used to verify if you are who you say you are when committing (e.g. with your email address).

I won’t go into much depth on this, instead just know it exists and is quite good practice to have but not necessary.

You can read more [https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits here].

Be aware that there are some consequences which come along with this:

* If you have setup a strict mode with signing, you cannot commit if you loose access to your signing key
* When rebasing your changes, you have to resign all the commits (meaning you are the only one who can do it)

== See also ==
{{Special:PrefixIndex|prefix=Git/|namespace=0|hideredirects=1}}
[[Category:Tutorial]]

Git

2026-06-05T13:53:55Z

Hw2210: Don't show redirects in see also

[https://git-scm.com/ Git] is a “version control software” which can be [https://git-scm.com/downloads downloaded on their website]. It allows for version management:

* Git keeps track and history of all changes and version of the software you are writing, and lets you:
** Revert and jump back and forth in history
** Create different versions or “branches” of your software
** Manage versions in a better way than “LATEST_COURSEWORK_X.zip”
** Easily see who last edited a line of code
* It makes collaborating on the same codebase easier

A lot of programs have ways to help you interact with git so you don’t have to go through the terminal for most things. Some programs that integrate well:

* VSCode
* All JetBrains products
* Vim with fugitive
* GitHub Desktop

If you want a UI in the terminal on:

* Lazygit (Linux/MacOS)
* GitUI (Linux/MacOS/Windows)
* And many more…

Here we will mostly focus on the parts of git and using the command line tool as it more easily translates to any of the tools above. Plus for the more complicated routines, you can only do it via the terminal.

== Alternative tutorials ==
This page is covers a lot of topics (including some commands which are a bit more advanced but useful for managing projects), and can be read all in one go or can just be used as reference. But there are also other resources which may be a bit more your speed:

* [https://www.w3schools.com/git/default.asp W3Schools git tutorials]
* [https://www.atlassian.com/git/glossary Atlassians indepth git tutorials]
* [https://rogerdudler.github.io/git-guide/ The colourful intro]
* [https://code.visualstudio.com/docs/sourcecontrol/intro-to-git Git in VSCode]
* [https://ohshitgit.com/ Git Troubleshooting]
* [https://www.youtube.com/playlist?list=PLT4sucrjsqF8kvZFcRvxfhWuTvjKrXYzE BCSS Technical Labs]

== Setup ==
When you first download git you need to do a bit of configuring before you start. This is just so when you start committing, it will come up with your name and email address.<syntaxhighlight lang="sh">
git config user.email "your_email@abc.com"
git config user.name "Your Name"
</syntaxhighlight>{{Note|text=Make sure the email is also linked to your account on GitHub or GitLab, see [[Git/GitLab|our GitLab documentation]].}}
You can also configure the name of the default branch to <code>main</code> instead of <code>master</code>. [https://sfconservancy.org/news/2020/jun/23/gitbranchname/ See why].<syntaxhighlight lang="sh">
git config --global init.defaultBranch main
</syntaxhighlight>

== Remote Repos ==
These are the web applications such as GitLab or GitHub. For bathcs.com, we use the [https://gitlab.bath.ac.uk|Bath’s GitLab instance] and have instructions [[Git/GitLab|here]]. But also note Bath has a GitHub enterprise instance running [https://github.bath.ac.uk/ here].

They normally provide many features and tools for version control, collaboration and CI/CD.

* You can <code>push</code> your local repository to them.
* Then on another device you can <code>pull</code> this code from them.
* Collaborators that work on your code can also <code>push</code> and <code>pull</code>
* You can collaborate with other people, on the same codebase
* Provides web UI for <code>git</code>
* Acts as a “cloud backup”

=== HTTP vs SSH ===
In the rest of the page, we will be using URLs to pull from remote repositories.

On both GitHub and GitLab you can do this in two ways:

* <code>http</code>, where urls look like: <code><nowiki>https://gitlab.bath.ac.uk/cs/wiki</nowiki></code>
* <code>ssh</code>, where urls look like: <code>git@gitlab.bath.ac.uk:cs/wiki</code>
{{Note|text=To convert from <code>http</code> to <code>ssh</code>, you replace <code>https://</code> with <code>git@</code> (which is the user we are sshing as) and then replace the first <code>/</code> with <code>:</code>, specifying the folder location of the repo we are copying.

It is exactly the same syntax as <code>scp</code> (or ssh copy).}}
In most cases this does not matter. With public repositories <code>http</code> does not require you to log in, however it cannot handle larger repos at lower internet speeds (as it times out).

However when you have setup 2FA, you can’t just use your credentials to login, which is what <code>https</code> normally requires. Therefore, you need to setup <code>ssh</code> to work, which also requires you to have a [[SSH keys]].

You can generate one by [[SSH keys|following our documentation]], which then you copy and paste the public key into the relavent section on GitHub or GitLab (see our [[Git/GitLab|GitLab documentation]], or the [https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account GitHub documentation]).

Once you have set up this ssh key, you should be able to clone and push to a repo without specifying your login details.

== Best Practices ==

# Follow this workflow:
## Pull main
## Create a branch
## push
## Create a pull/merge request
## Merge into main
## Pull main
# Don’t push to main directly
# Add a <code>README.md</code>
# Add <code>.gitignore</code> file: you can choose what files and folders git should ignore, for example, <code>.env</code> file containing API keys.
# Write descriptive commit and PR/MR messages
# Git is simple in the basics, and very complicated but very powerful once you dive into it. Basics are enough to have version control. Use them!
# It’s hard to permanently lose data in git. Almost always you can restore in some way. Don’t panic. Use your favourite search engine or Dang It Git

== See also ==

{{Special:PrefixIndex|prefix=Git/|namespace=0|hideredirects=1}}

[[Category:Tutorial]]

Main Page

2026-06-05T13:51:46Z

Hw2210: Protected "Main Page": High traffic page ([Edit=Allow only autoconfirmed users] (indefinite) [Move=Allow only autoconfirmed users] (indefinite))

Welcome to the wiki hosted by students of the University of Bath. It is managed by [[Bath Open Source Society]], and any student is free to login and edit pages. This is designed as a space where societies can store knowledge that can easily be passed down, or even students can share fun facts they have learnt while studying at Bath.

There is minimal restriction to what can be uploaded to this wiki, we just ask that you be polite and follow the Dignity and Respect policy. All edits are associated with your University login and we will follow standard University procedures in the event of serious misconduct.

If you would like help with anything, please contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk].

== Contributing ==
To get started with this wiki, you just need to log in at the top right. You will then have permission to edit almost any page.

For ideas of places you can contribute, see [[Special:WantedPages]] and [[:Category:Help needed]].

Git/Basic Components

2026-06-05T11:57:52Z

Hw2210: /* Commands */ Add mention of git checkout

Git/Advanced

2026-06-05T11:55:42Z

Hw2210: /* Stashing */ Clarify uses for switch and checkout within stashing

Kubernetes

2026-06-04T14:20:33Z

Hw2210: Add traefik and gateways section

== General knowledge ==
This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

=== Pod vs Container ===
A common confusion is that [https://kubernetes.io/docs/concepts/workloads/pods/ pod]'s are containers in kubernetes. This is not exactly true, a pod is a general group of linux namespaces which can host multiple containers. This means you can have a container that writes to a directory and another container that reads from that directory in the same pod. This can be very powerful, but in a lot of cases can be ignored.

But it is key to point out that a Pod is a resource that is created by other kubernetes resources. They are a group of processes running, once they die the pod is deleted and forgotten about. Therefore you should not be creating pods directly, instead you should be using deployments, statefulsets, cronjobs or even jobs. All these resources create generate a pod as their lifecycle and will restart/recreate the pod if it fails.

=== Statefulset vs deployment ===
Another key understanding is the difference between statefulsets and deployments, as statefulsets can cause some confusion in how they work. The difference is more applicable to multinode clusters but are still key to the structure of kubernetes.

Effectively, a statefulset is a deployment with writable volumes - known as persistent volumes (PV). Having the ability to write to volumes can cause race conditions when multiple pods across nodes are writing to the same file. This is where statefulsets come in, they lock volumes and so they can only be used by one node and one pod, with scaling creating new persistant volumes which are stored separately. This means that if you scale a statefulset that relies on shared knowledge in the volume, half your requests will have one set of data and the other half will have another.

This obviously is quite a big disadvantage and can lead to confusing behaviour when a node is not configured to shutdown safely and taint itself, moving all the statefulsets off of itself before it shutsdown - if PV is locked by a node and pod, it cannot be deployed to another cluster.

Therefore, this is where deployments come in, they, usually, do not have associated persistent volumes, allowing for easy horizontal scaling. For storing shared data, they should connect to a database on another node which can be more compatible with statefulsets when configured correctly.

Both of these resources will create pods and redeploy them if they crash.

==== Liveness/Startup probes ====
Liveness and startup probes can be defined on pods, and these let kubernetes know if a pod has started correctly and if it still is alive. For example, some deployments might take a while to start up and configure everything before it starts serving content and so when restarting, this can cause some downtime. Downtime is what we are trying to avoid and so by using a startup probe, kubernetes knows that this application is ready, and so it will only terminate the previous node once the new one is started up resulting in zero downtime!

The liveness probe on the other hand periodically checks whether the pod is still alive. This means that if it suddenly stops responding due to a long database query, kubernetes can detect that and replace the pod with another further reducing downtime. However, this usually suggests something else is wrong with the application and so this should be investigated and fixed.

==== Security Context ====
{{Note|text=Within [[BOSS/Hosting/Cluster|BOSS's kubernetes cluster]], we define a security policy which requires all pods to correctly define their security context and make sure that it is not running as root.}}
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ security context] defines what privileges the pod has when running, we effectively want this to be as minimal as possible to reduce attack surface area. E.g.

* Run as user
* Don't allow privilege escalation
* Properly define seccomp policy
* Default SELinux container context
* Drop all capabilities

However this can cause issues with third-party applications which commonly do some questionable things, e.g. require running as root or changing the uid. But for our pods you can mostly just copy and paste:<syntaxhighlight lang="terraform">
resource "kubernetes_deployment_v1" "my_deployment" {
# ...
spec {
# ...
template {
# ...
spec {
container {
# ...

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

# ...
}
}
}
}
}
</syntaxhighlight>See [[Terraform]] for more information.

=== What is a CRD? ===
A [https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ Custom Resource Definition (CRD)], allows you to extend kubernetes capabilities and define custome resources. This is usually paired with an operator which reads the resources and performs some actions.

We should never create our own, but third-party ones make it much easier for doing things such as creating ingress routes with traefik or define database clusters with our postgres operator.

K9s and kubectl support these out of the box (as they are basically just schemas for yaml configuration), and you can see all pods by using the name of the resource.

=== Traefik and gateways ===
[[File:Gateway diagram.svg|thumb|368x368px|A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.]]
Kubernetes works by defining services, which give a common endpoint to call potentially multiple pods. These can then be exposed through HTTPRoutes and the [https://kubernetes.io/docs/concepts/services-networking/gateway/ Gateway API], in which traefik implements.

The Gateway API resources are read by traefik, which acts as the implementation, and acts accordingly to the defined configuration. Therefore, in essence, the gateways act only as a means for configuring traefik. But effectively, traefik has configured open ports it can expose, it then looks for Gateways, in the permitted namespaces, for their configuration. The gateways stores a list of ports that the namespace can expose (though it cannot add one that is not included within traefik configuration itself), as well as a list of certificates. At this point, the domain requested must have a certificate configured within the gateway, and all TLS logic is handled by traefik and so all further traefik is effectively decrypted. Notice here that if a certificate is not configured on the gateway, it cannot be served (one of the downsides of the gateway API).

For us, we have decided to have each namespace have their own gateway, due to the protections traefik offers, this means that we do not have to do any cross namespace references for certificates, and do not have to update the main gateway anytime we need to add a certificate. There is an additional issue with this, is that during the time the certificate doesn't exist but is configured (e.g. when first request it), the gateway is deemed invalid and so doesn't route any traefik (even http). There is a plan to help mitigate this through the use of <code>ListenerSets</code> but this is yet to be supported in traefik and still has this issue. Therefore, we want to make sure that a single gateway hosts services for as few applications as possible (preferably only one).

Anyway, the gateway will have a number of child routes (<code>TLSRoute</code>, <code>HTTPRoute</code>, <code>GRPCRoutes</code> and coming in the future <code>TCPRoute</code> and <code>UDPRoute</code>). These routes act as queries to determine when and what traefik should forward to. So for example they act as:<syntaxhighlight lang="text">
if hostname is example.bathcs.com forward to example-service
if the path starts with /api forward to api-service
</syntaxhighlight>Then traefik can request those services, allowing kubernetes to effectively take over, looking at the pods associated with the service and using the defined algorithm to send the request to those pods and using the defined ports.

This does mean there are a number of different places a port can change:
Exposed port -> Traefik internal port -> Service port -> Pod port -> Application port
In most cases you should have the service port, pod port and application port all matching, this makes debugging a lot easier. Additionally there are few reasons why you will want to change traefik's internal port and the exposed port (but there are some!).

==== Certificates ====
The thing with certificates is that we effectively never want to manually create them, the recommended expiry time for certificates is always dropping, with the most recent update at 45 days. This is way too much work for manual requesting and uploading and adds too many layers for it to go wrong. Therefore we use cert manager, which allows defining certificate objects within the cluster, cert manager will then go and do all the requesting for us and store it in a secret. Then it will also track the expiry and automatically update the certificate a week or so before it expires.

There are multiple different methods it can use to validate that we are in fact in charge of the domain:

* DNS - this is the preferred method, as it allows us generating certificates for protected IPs. But this requires a valid cloudflare API token (which is restricted to a single IP).
* HTTP - this is when the certificate authority will request our server from multiple locations, which means the DNS cannot be set to a protected IP. But it means we can generate certificates for domains that we don't control the DNS of (e.g. bath.ac.uk hostnames) but it is at least configured to point to our server. The integration with traefik means that there is no additional work required for the application to get these working.
* Cloudflare Origin - These are very special certificates and cannot be decrypted by the browser. The idea is that by generating these certificates, only cloudflare themselves will be able to decrypt the contents and so only they can proxy your IP. We use this specifically for [[Kubernetes#Cloudflare proxy|Cloudflare proxy]]-ing thought it doesn't provide us the true benefits (given our IP is still public)

Within cert manager's speak, these are known as issuers, and we have cluster issuers defined for each (meaning any namespace in the cluster can use them).
{{Note|text=When cert manager is first requesting the certificate, the configured gateway will be invalid and so no routes attached will forward traffic.|type=reminder}}

==== Cloudflare proxy ====
Cloudflare proxy offers the benefits of caching our content on "edge" servers, meaning that our websites perform much better on average as well as it can protect the IP of the machine, but as explained later we don't use cloudflare proxy everywhere and so lose this advantage. This caching is amazing when the application is configured for it to work well with it (e.g. correctly labelling requests as cachable). But it does not work with every application, especially third-party services which sometimes just break when using it. But it also adds troubling security questions, for example, a login page will also be proxied, and decrypted by cloudflare, resulting in cloudflare having access to all passwords that go through the site. For this reason we limit where we use cloudflare proxying to services that would benefit heavily from it (e.g. this Wiki as authentication is handled offsite).

To setup cloudflare proxying, it is as simple as generating a certificate with the cloudflare origin issuer and exposing a HTTPRoute with the certificate and then enabling proxy in the dns record. Obviously this does not work with internal DNS records (e.g. <code>k8s.bathcs.com</code>) and so our terraform config automatically detects and does not proxy this stuff.

The cloudflare origin issuer then speaks to the cloudflare origin operator which requests a certificate from cloudflare themselves. The generated certificates can be found in the cloudflare dashboard for the domain under "SSL/TLS > Origin Server".

=== Network Policies ===

=== Helm ===

=== Persistant Volumes ===

=== Cronjobs ===

=== K3S ===

== Deployment with Tofu ==

=== Retaining PVs ===

=== Where is the state stored? ===

== General Terraform management ==

== K9s ==

== Migrating storages ==

== Multicluster setups ==

=== Control plane ===

=== How to manage Storage ===

=== Safely shutting down nodes ===

== Backups ==

== Troubleshooting ==

=== Kubernetes event log ===

=== Traefik dashboard ===

=== Shell-ing into pods ===

=== K3S Log ===

=== Emergency Debug pods ===

=== Cloudflare API tokens ===

=== SELinux ===

=== Scenarios ===

==== Deployment/statefulset created but pod not creating ====

==== Pod in cash loop ====

==== Pod cannot request any website ====

==== Pod can access the internet but everything returns self-signed certificate ====

==== Deployment can't access its database/valkey ====

==== Pod is requesting denied system privileges ====

==== Pod is up, but I get a 404 when requesting visiting the page ====

==== Pod is up, but I get a 403 when requesting visiting the page ====

==== Authelia refuses to start up ====

==== Node is crashing often after running out of RAM ====

==== On IPV6 cluster and requests randomly timeout or return 404 ====

==== Statefulset refusing to start pod (PVC) ====

==== A node just crashed and went offline ====

=== Cleaning up ===

File:Gateway diagram.svg

2026-06-04T13:34:56Z

Hw2210:

A digram depicting the the flow of traffic from the internet, to traefik then to each namespace's gateway. Each gateway then looks at all connected certificates to add TLS authentication and then looks at the HTTPRoute to find which one matches based on the rules which then defines what service to forward the traffic to and subsequently the pods.

Terraform

2026-06-04T11:06:15Z

Hw2210: Fix broken link

[[BOSS]] uses terraform (actually [https://opentofu.org/ <code>opentofu</code>]) to deploy all the resources onto [[BOSS/Hosting/Cluster|their kubernetes cluster]], which tries to be the language in which you can deploy anything and everything and if you need to quickly redeploy a whole machine you can with a simple command. In practise it doesn’t work like that, but its good enough for our needs.

You can find the [https://gitlab.bath.ac.uk/cs/int/terraform terraform repo on gitlab], this page was originally taken from the README as it got too long.

== Other resources ==

* [https://developer.hashicorp.com/terraform/tutorials Hashicorp (the maker of the closed source terraform) has some good tutorials on their site]

Basically that’s it really, the rest of this will be looking at how to deploy an application with our configuration.

== Terraform and its woes ==

Before we get into making an app, I must briefly explain terraform and its benefits/issues/confusing behaviours.

This expects you to have a rough idea around how terraform works. But here is a quick explainer: terraform is build around '''resources''', provided by '''providers'''. These resources have a state stored locally in a state file, whether they are deployed, generated values etc. (note that these can literally be anything e.g. from random passwords to HTTP reequests to kubernetes resources to DNS records). These resources can then be organised into modules, which (can) have outputs from values generated by the resources. There are also “data”, but this is basically a reference to another resource which doesn’t have the controls.

When deploying, terraform will then check the state of all the current deployed modules (even pinging servers if needed) and find anything that has changed (e.g. new resources or updated values) and deploy everything.

=== Module structure ===

Modules are the core of terraform and can be a bit tricky to get your head around, as initially they are quite limited (e.g. there is no such thing as a global variable).

But basically a module has a list of inputs as and a list of outputs (and then providers). So it is expected that your module deploys some resources which are then used to output something. E.g. in [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ldap/user|our ldap user module] the module generates a random password, creates the user and assigns them to a group, then outputs the username, email and password to be used later in another resource.

So e.g.

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
// Note passing providers act somewhat like global constants, passing configuration (e.g. what ldap server we mean)
// to the module
lldap = lldap
}
}

// You can then use the email by: module.user.email or password: module.user.password</syntaxhighlight>
'''NOTE''': All files within a module (the folder) are treated as one global space, similar to how Go works. You can reference variables, locals and resources throughout all files within a module which makes it quite difficult to organise nicely.

The modules are usually structure in the way:

* <code>init.tf</code> – Usually where your <code>providers</code> go and if you are lazy (like me), everything else
* <code>vars.tf</code>/<code>variables.tf</code> – Where you put all your variables. As explained in [[#Variable madness|Variable madness]], I really don’t like this and so commonly ignore
* <code>outputs.tf</code> – All your outputs go here
* <code>*.tf</code> – Anything else, if you want to split it out nicely into other files

=== Variable madness ===

Terraform variables suck.

Anyway, so basically terraform requires you do a full definition for every variable:

<syntaxhighlight lang="terraform">variable "my_var" {
type = string

description = "Something"
nullable = false

sensitive = false # If something is sensitive MAKE THIS TRUE
}
// You can then later reference it with var.my_var</syntaxhighlight>
This defines something that must be inputed by the user, either through the module or your <code>tfvars</code> file (if in the root directory).

Due to this verbosity, and sometimes complex nature of the interfaces I like to create, I have used the <code>object</code> type e.g.:

<syntaxhighlight lang="terraform">variable "my_var" {
type = object({
l = list(string)
m = map(bool) # string -> bool. Same syntax as object, just more flexible
s = set(string) # Yes this is different to list but using the same []
option = optional(string, "my_default")
})
// ...
}</syntaxhighlight>
But even then you can’t define nice defaults for each sub item and the description is for the whole variable, so it has to be done as so. This is just raw pain and not particularly great syntax in my opinion. Also you cannot enable/disable sensitive nature of values for sub items, this means the whole object must be defined as sensitive if you have one password.

And due to the lack of global constants, you must define every variable in every sub project and duplicate the types (yes there is '''no''' way to define a type to use throughout the project).

It is also recommended that you put all variables in a <code>vars.tf</code> file. Which sure does make sense for small modules, but if its that small I find it easier to just chuck at the top of the <code>init.tf</code> file (as the terraform syntax highlighter is soooo broken). Then if its large, I find it more useful to put the variables where they are actually used – but then again this is confusing because the syntax and tooling is so bad.

Oh yeah sorry and then there are '''locals''' which are constants you can define from resources/variables and will be calculated when the information is ready. E.g.

<syntaxhighlight lang="terraform">locals {
temp_val = "hi"
}
// Then you can reference with local.temp_val</syntaxhighlight>
=== Depends on and its pains ===

The one issue with terraform is that its really slow with large projects with this. But the nature of the design encourages large projects (as you want to reference things throughout the smaller apps).

This is due to it having to create a dependency graph where objects wait on their dependencies. These dependencies can be defined by <code>depends_on</code> in any resource or just referencing a value from another resource.

This is really useful so deployments are not actually deployed until all the secrets are deployed. Due to my perferable of not repeating myself, I heavily use the inferred dependency from the referring to resource names e.g. <code>kubernetes_secret_v1.secret.metadata[0].name</code> (yes this is why I don’t just do the simple thing and use the shorter name, its good to know where the value comes from).

'''BUT''' you cannot rely on this, as some things take time to actually deploy even if it says its successful. Therefore you may need to use timers instead. I usually don’t bother due to the numerous other issues with terraform making it so its not actually perfect so there’s minimal point actually making it easy to deploy from scratch.

=== Timeouts ===

If something goes wrong during deployment, e.g. you make a typo, you will have to wait the FULL timeout time. This is really painful when you typo the hostname to the db causing the pod to crashloop in a helm config and you have to wait 10 minutes for terraform to give up. You can Ctrl-C, Ctrl-C, but this causes more issues as you will have to manually intervene and delete the helm chart/deployment before you run the command again.

Instead I recommend shortening the timeouts for the deployment/helm to one more applicable to the application. A lot of our first-party stuff usually deploys in a few seconds and if it doesn’t, something has gone very wrong.

=== Commas or no commas? ===

The terraform syntax is… interesting. Commas are optional in most cases. So I would recommend, not typing commas where they optional.

'''BUT''' within lists/sets (basically between <code>[]</code>) you have to type commas, and if this is across multiple lines '''PLEASE ADD TRAILING COMMAS'''. The reason? Git histories look sooooooo much better.

== How to create a basic project ==

=== Creating a namespace ===

There is a handy util module for this:

<syntaxhighlight lang="terraform">module "example_namespace" {
source = "./utils/namespace"

name = "example"

enable_dns = true
enable_mail = true
enable_lldap = true
bkp = {
// ...
}

providers = {
kubernetes = kubernetes
}
}</syntaxhighlight>
This allows you to enable or disable features for your namespace, e.g. if your pods need to communicate with the outside world, enabling the DNS. All of these features are disabled by default and its heavily encouraged to only enable the features if the namespace needs it.

The next thing is to configure backups through this, to reduce our dependence and costs from our s3 provider, it is recommended that backups are disabled for all namespaces whos data can be regenerated (e.g. froom). If you do enable it, it is then encouraged that you disable backups for any database or pvc that you don’t need backing up with the <code>k8up.io/backup=false</code> annotation (you may notice that all <code>valkey</code> instances set this by default if you are using the <code>app/valkey</code> module).

==== Placement ====

Within this repository, it is tradition to put the namespace creation at the highest level, e.g. <code>20_apps.tf</code>. This means that the apps themselves do not control the namespace they are created in. It is mostly just a personal preference from me after years of configuring k8s on terraform.

=== Using Helm ===

Helm is by far the easiest way to deploy third-party tools, and is used throught this repo despite it’s drawbacks when combined with terraform (it’s just so easy).

You just add helm to the providers list (which defines what terraform modules you are integrating with):

<syntaxhighlight lang="terraform">terraform {
required_providers {
helm = {
source = "hashicorp/helm"
version = "~>3.1.1"
}
}
}</syntaxhighlight>
You can then use the <code>helm_release</code> resource, which takes the form:

<syntaxhighlight lang="terraform">resource "helm_release" "my_app" {
name = "my_app"
namespace = var.namespace

repository = "https://charts.example.com"
chart = "the_app"
version = "version"

values = [yamlencode({
// Values go here written within the terraform config language
})]
}</syntaxhighlight>
This deploys all the resources an application needs and manages and restarting if a secret or config map changes, providing all the configuration at your fingertips.

However this comes at a cost:

* '''SECRETS SHOULD NOT GO IN THE HELM CONFIG'''. This is a big one, all values are easily accessible unencrypted on the cluster, therefore any secrets '''MUST''' go in a <code>kubernetes_secrets_v1</code> object and you should use a <code>secretsRef</code> or similar to link it. If the helm chart does not support this '''DO NOT USE IT'''. Helm also stores the history of all values, therefore if you put it temporarily within helm values for testing, you must to a password rotation.
* You have no power over the types of resources and the structure in which it deploys. This means that if a feature or support for our strict network policies are not implemented, you have to either not use the helm chart completely or fork your own (which we definitely don’t want to do).
* If the helm chart gets deleted, all pvc related ''might'' also get deleted (unless they have the <code>Retain</code> policy, which should be the case for everything).
* Sometimes they don’t have the proper security contexts/network policies by default so you will have to add them youself (see the below section)

Overall, helm is pretty good, just use with caution and understand what templates you are inflicting. Note, you will probably have to get pretty good at reading not only default values, but schemas and the templating language of helm itself, as sometimes the charts are not particularly well documented.

=== Using kubernetes ===

For this you need to understand a bit of structure of how kubernetes works. I will assume that you are deploying a pod. If that pod needs storage attached (and not through SQL or Redis), then you will need to use a [https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ <code>StatefulSet</code>]. If you have no storage or are just communicating with a postgres server or redis, you can instead use a [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ <code>Deployment</code>].

The difference between these two concepts are not particularly seen in the world of a single node cluster, but basically deployments are free to spin up another even if the previous one is still terminating or if the node is non responsive. On the other hand, statefulsets must ensure that no two nodes are trying to access the same data, therefore cannot automatically start up if a node goes down.

This also means that it is much easier to scale a deployment to multiple nodes, vs a statefulsets which must have separate volumes per pod.

Anyway, both statefulsets and deployments have a template configuration for creating the pod associated with itself. This pod has a label which is used to monitor and track the associated pods with its parent. So the structure is as follows:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
metadata {
name = "my_deployment"
namespace = var.namespace
}
spec {
replicas = 1
selector {
match_labels = {
app = "the_deployment"
}
}
// This doesn't really matter in a one node cluster with a replicas = 1
strategy {
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "the_deployment"
}
}
spec {
container {
name = "my_deployment"
image = "bathbcss/my_image:latest"
image_pull_policy = "Always" // Should only be set if the above is "latest"

// This should be the default security context to comply with our pod security policies
security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// This is how the pod is checked its alive, so if something happens,
// e.g. job which causes it to become unresponsive, it will be automatically killed off and replaced
liveness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

// This allows the tracking of when the pod starts, so we wait until the pod is ready to receive requests
startup_probe {
http_get {
path = "/healthz"
port = 8080
}
// 3 * 30 = 90 seconds to start
failure_threshold = 30
// If it takes a while to startup, increase this time
period_seconds = 3
}

// If exposing a port the port to expose
port {
container_port = 8080
// Make sure to give it a name so we can use the name in services
name = "web"
}

// env and env_from definitions
}
}
}
}
}</syntaxhighlight>
''Quick side note: <code>kubernetes_\*_v1</code>is the preferred resouce name, any resource that does not have<code>\_v1</code> on the end is deprecated and should not be used.''

It is recommended that the version of the image is actually set and <code>latest</code> is not used, however to reduce the admin overhead, for internal projects it can be easier to set to <code>latest</code> with an image pull policy of <code>Always</code>. However this means if you want the latest version, you must have access to the cluster to restart a pod.

If you are exposing pods, this should be tied with a <code>Service</code>, as seen below.

==== Security context ====

As you will notice in the example above, we have a security context set. This is '''required''' by the pod security contenxt, otherwise it will not deploy. In most cases you can copy either of the two following policies, depending on whether it is within a kubernetes resource or helm/kubernetes manifest resource:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

securityContext = {
runAsUser = 1000
runAsNonRoot = true
allowPrivilegeEscalation = false
seccompProfile = {
type = "RuntimeDefault"
}
capabilities = {
drop = ["ALL"]
}
}

// ....
}
}</syntaxhighlight>
Note that we are running as a user (not root), setting the default seccompProfile (you ''should'' only need the default unless you are doing weird things with the host machine) as well as dropping all capabilities (you may need to add some back in but I will leave to you as you probably know more than me – NOTE: Some are disabled by our pod security policy but can be override with <code>baseline</code>).

==== Liveness and startup probe ====

The liveness and startup probes are not necessary, but is a nice to have. The liveness probe allows the cluster to detect if a pod becomes unresponsive and is then able to kill it if that is the case. Whereas a startup probe makes it so the cluster knows exactly when the pod is able to receive responses.

Please see the [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ kubernetes docs on probes] for more information on the options. But in most cases the HTTP get option should suffice, which just looks for a status 2xx code.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

liveness_probe {
http_get {
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

startup_probe {
http_get {
port = 8080
}
failure_threshold = 30
period_seconds = 3
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

livenessProbe = {
httpGet = {
port = 8080
}
initialDelaySeconds = 5
periodSeconds = 10
}

startupProbe = {
httpGet = {
port = 8080
}
failureThreshold = 30
periodSeconds = 3
}

// ....
}
}</syntaxhighlight>
==== Environmental variables ====

When configuring deployments, you will want to set environmental variables. There are a few ways to do it, but note '''ANY PASSWORDS/API KEYS GO IN SECRETS''' not the environmental variables. As you will see I will example how to do this.

By default the <code>env</code> list can be used to set a single environmental variable e.g.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "TEST"
value = "my_value"
}

// ....
}</syntaxhighlight>
However, this is quite verbose and takes a lot of space, so if you are configuring a lot of variables or have secrets, you will want to use the <code>env_from</code> list. This allows you to reference a config map or secrets (this is the most basic form).

These look like:

<syntaxhighlight lang="terraform">resource "kubernetes_secret_v1" "module" {
metadata {
name = "deployment-db"
namespace = var.namespace
}
data = {
DATABASE_URL = module.database.url
}
type = "Opaque"
}

resource "kubernetes_config_map_v1" "module" {
metadata {
name = "deployment-config"
namespace = var.namespace
}
data = {
PUBLIC_VALUE = "yoooo"
ROCKET_ADDRESS = "::"
}
}

resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env_from {
secret_ref {
name = kubernetes_secret_v1.module.metadata[0].name
}
}
env_from {
config_map_ref {
name = kubernetes_config_map_v1.module.metadata[0].name
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

envFrom = [
{
secretRef = {
name = kubernetes_secret_v1.module.metadata[0].name
}
},
{
configMapRef = {
name = kubernetes_config_map_v1.module.metadata[0].name
}
},
]

// ....
}
}</syntaxhighlight>
Note that you can set the value of a environmental variable from a secret on an individual basis, which can be useful if you are storing environmental variables as well as files inside your secret. E.g:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "DB_PASSWORD"
value_from {
secret_key_ref {
name = kubernetes_secret_v1.module.metadata[0].name
key = "password"
}
}
}

// ....
}</syntaxhighlight>
==== Third-party CRDs ====

Now this is is where terraform becomes less good. Basically when deploying using the kubernetes API, the checks will make sure the CRDs (so the like api and kind are installed and supported on the kubernetes cluster). This means that you won’t even be able to run.

Basically it means that you need to comment out manifests that reference these resources until the CRDs are deployed (usually though a helm chart or something).

==== PVCs ====

Please remember '''ANY PASSWORDS/API KEYS/CERTIFICATES GO IN SECRETS''' not in the storage (this also means its configurable by us and yes they can be mounted as read only volumes).

So, kubernetes storage works around persistant volumes which are requested by persistant volume claims. On our k3s single node, we are just using the k3s filesystem class. This means it’s a bit basic but does the job.

Things to note:

* You probably should be manually creating persistant volume claims (and definitely not persistant volumes), instead using <code>statefulsets</code>
* It’s really hard to change persistant volumes post fact, so please go through testing phase if you are unsure about anything.
* K3s does not support the storage limit, so please '''DON’T RELY ON IT''' to stop abusive behaviour.
* If you are defining yourself, do not accidentally make your deployment depend on the persistant volume claim, as the pvc will not be created until it is used in something. See [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/services/mail/mailserver.tf|the mailserver module] of how to handle it. '''NOTE''': If you reference the pvc config in your deployment, terraform will add that automatically to the <code>depends_on</code> list.
* If it is critical data you will need to '''manually update the pv to “retain” its data''' if the pvc gets deleted. This just adds a bit of safety if you mess up a deployment. This can be done in <code>k9s</code> by an admin, updating the <code>persistentVolumeReclaimPolicy</code> to <code>Retain</code> instead of <code>Delete</code>.

So how you should be using pvc, in statefulsets:

<syntaxhighlight lang="terraform">resource "kubernetes_stateful_set_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
service_name = var.name
replicas = 1
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
}
}
spec {
container {
name = "my_app"
image = "bathbcss/my_app:1.0.0"

// ...

volume_mount {
name = "data"
mount_path = "/data"
}
}
}
}
volume_claim_template {
metadata {
name = "data"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
}
}
}
}</syntaxhighlight>
==== Services ====

Servies are the first way to adding an ingress, and helps give a common endpoint if you have multiple pods running (though we don’t do this). If using within the cluster you can access a service with the address <code>service_name.namespace.svc.cluster.local</code>. Note that if you are accessing it from within the same namespace you can just use the <code>service_name</code> as the hostname (and it is better on a network policy basis).

For a namespace you just need a label to select on (note we have to define an <code>app</code> label for the statefulset/deployment anyway so you can just use this).

So it should look like this:

<syntaxhighlight lang="terraform">resource "kubernetes_service_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
port {
port = 8080
name = "web"
}
selector = {
app = var.name
}
type = "ClusterIP"
}
}</syntaxhighlight>
And then you can just access the port by <code>service_name:8080</code>. Note that you can also set a <code>target_port</code> if you want to change the port from the deployment and service (but like why?).

Also giving it a <code>name</code> is important, so in the ingress we can just use the name instead of the port number itself, increasing readability.

==== Network policies ====

Now comes the pain. If a helm chart has a network policy, use that (but please actually read what permissions it gives).

Network policies, either grant, or block network “ingress” (stuff going into the pod) and “egress” (stuff going out of the pod). By default, due to our security, ingress to our pods is denied (so anything in the cluster), but egress to the outside world is allowed.

You then use either pod selectors or namespace selectors to allow traffic from a pod (which is how <code>enable_dns,lldap,mail</code> works). In most cases egress can be left alone, unless you want to block a node from doing something in particular.

As talked about in the next section, for postgres and valkey, we create these policies, limited to pods with the <code>${name}-${service}-client=true</code> label allowing only inter-namespace communication.

If you do have to write one, which I really hope you don’t, please read [https://kubernetes.io/docs/concepts/services-networking/network-policies/ kubernetes documentation on Network Policies].

=== Postgres and Redis/Valkey ===

We have utilities for [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/postgres|postgres] and redis (through [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/valkey|valkey] due to redis being really hard to run).

If you require these, we highly recommend the above (though postgres needs moving away from bitnami due to the requirement of money for stability). These setup the network policies you need as well as generating secure passwords and you can look at the <code>outputs.tf</code> file to see the outputs you can use from a module.

'''NOTE:''' If using this, for any pod accessing the data, it must be in the same namespace and have the labels <code>${name}-postgresql-client=true</code> and/or <code>${name}-valkey-client=true</code>, otherwise the traffic will be denied by the network policies.

E.g.

<syntaxhighlight lang="terraform">module "database" {
source = "../postgres"

namespace = var.namespace
prefix = var.name
username = "user"
database = "db_name"
size = "250Mi"

providers = {
helm = helm
}
}

// Create a secret with module.database.url

resource "kubernetes_deployment_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
// This **MUST** be defined otherwise you'll get weird errors
froom-pg-postgresql-client = true
}
}
// ...
}
}
}</syntaxhighlight>
Side note: the namespace will also need to have <code>enable_dns</code> to be able to access it

=== Adding ingress ===

Adding ingress to service basically means that you are making it accessible to the outside world.

If you are doing this, please consider security heavily:

* Can any user alter the DB? Are you doing proper type checking on the inputs?
* Do you make sure to not expose any secrets e.g. db urls
* Do you really need to expose this pod? Or can you leave it to k9s port forwarding?

Then you need to ask:

* Just expose it to people within the bath network e.g. on <code>*.k8s.bathcs.com</code>
* Expose it to the whole world <code>*.bathcs.com</code> (note coordination with backstage to get them to update their traefik will be necessary)

Please expose to the utter minimum people.

If exposing to the full network please note the flow is:

* <code>app.bathcs.com</code> <- You should accept this domain in the ingress
* <code>app.bcss.su.bath.ac.uk</code> <- You should accept this domain in the ingress
* <code>app.k8s.bathcs.com</code> <- This is what the certificate you should be giving (due to this is hostname that backstage is requesting)

Depending on the chosen level, please look at the [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ingress|<code>utils/ingress/*</code> modules] as these handle this most of this flow for you.

E.g. making it internal to bath uni only:

<syntaxhighlight lang="terraform">module "dns" {
source = "../../utils/ingress/dns_flow"

subdomain = var.subdomain
# We don't want public one, so don't specify cloudflare_zone_id
k8s_cloudflare_zone_id = var.cloudflare_zone_id

# Expanded for effect
domains = {
uni = var.domains.uni
k8s = var.domains.k8s
}

providers = {
cloudflare = cloudflare,
}
}

module "ingress" {
source = "../../utils/ingress/tls"

name = var.subdomain
entry_points = ["websecure"]
namespace = var.namespace
host = module.dns.hosts.k8s
additional_ingress_hosts = [module.dns.hosts.uni]
service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "http"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
}
}</syntaxhighlight>
If using a helm repo you may have to manually define the certificate, which is quite easy:

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "cert" {
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "Certificate"
metadata = {
name = "${var.name}-cert"
namespace = var.namespace
}
spec = {
secretName = "${var.name}-cert-secret"
issuerRef = var.cert_issuer
dnsNames = [var.host]
}
}
}</syntaxhighlight>
Which outputs the cert to the secret with name <code>${var.name}-cert-secret</code>. You can also use <code>kubernetes_manifest.cert.manifest.spec.secretName</code> (you can guess what I prefer).

If you are making it publically accessible, you can use the full flow which does both dns and ingress records:

<syntaxhighlight lang="terraform">module "ingress" {
source = "../../utils/ingress/full_flow"

cloudflare_zone_id = var.cloudflare_zone_id
namespace = var.namespace
domains = var.domains
subdomain = var.subdomain

service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "web"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
cloudflare = cloudflare
}
}</syntaxhighlight>
==== Putting it behind authelia ====

If the application doesn’t have oauth integration build in and you are wanting to protect it behind authelia you can add middleware of <code>auth-forwardauth-authelia@kubernetescrd</code> to require them to go through authelia.

You will then have to add an access control rule to authelia e.g.

<pre>rules = [
{
domain = "app.k8s.bathcs.com"
subject = ["group:boss-example-group", "user:hw2210"]
},
]</pre>
'''NOTE''': you can create and manage groups on [https://www.bath.ac.uk/groupmanager/ Bath's group manager], which creates a unix group that can be seen in the LDAP server.

=== Cloudflare dns records ===

Cloudflare provider is an actual pain. You would’ve hoped it would be good, but its not amazing (has previously caused updates on every <code>apply</code>). Basically when creating a dns record you should use the whole address e.g. <code>app.bathcs.com</code> instead of <code>app</code>. This is because it will cause an update on the second application, changing the <code>name</code> value (which we use to get the full domain).

It sucks because theres an additional useless variable ontop of the <code>cloudflare_zone_id</code> (yes you can use the “data” thing to solve this but still).

Additionally for long TXT records, it will add additional quotes, and terraform will continually think you need to change that if you don’t yourself add the quotes into the data.

=== Sending mail ===

Sending mail is somewhat weird.

Basically just because I can, the authentication is managed by <code>lldap</code> (a really fast ldap implementation). This should not be confused with the ldap service that Authelia is hooked up with, as it is not.

Once you’ve enabled mail in the namespace you then have to create an ldap user:

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
lldap = lldap
}
}</syntaxhighlight>
You can then use this to access the mail with <code>module.user.name</code>, <code>module.user.password</code> and <code>module.user.email</code>. '''NOTE''' due to it not being exposed, we must use the kubernetes domain certificate, which means that you normally have to disable tls verification it or set the expected domain to <code>bathcs.com</code> (as seen in authelia).

=== Creating an oauth client ===

I have created a [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/oauth/client|OAuth client module] for generating all the secret data you need and generate the client config (under the <code>config</code> output) you can then pass up and then into the authelia module (see <code>grafana</code> as an example).

But basically please read [https://www.authelia.com/integration/openid-connect/introduction/ the authelia oidc docs] for a full explanation of how it works.

Terraform

2026-06-04T11:05:19Z

Hw2210: /* Putting it behind authelia */ Update information about using bath's group manager for authentication

[[BOSS]] uses terraform (actually [https://opentofu.org/ <code>opentofu</code>]) to deploy all the resources onto [[BOSS/Hosting/Cluster their kubernetes cluster]], which tries to be the language in which you can deploy anything and everything and if you need to quickly redeploy a whole machine you can with a simple command. In practise it doesn’t work like that, but its good enough for our needs.

You can find the [https://gitlab.bath.ac.uk/cs/int/terraform terraform repo on gitlab], this page was originally taken from the README as it got too long.

== Other resources ==

* [https://developer.hashicorp.com/terraform/tutorials Hashicorp (the maker of the closed source terraform) has some good tutorials on their site]

Basically that’s it really, the rest of this will be looking at how to deploy an application with our configuration.

== Terraform and its woes ==

Before we get into making an app, I must briefly explain terraform and its benefits/issues/confusing behaviours.

This expects you to have a rough idea around how terraform works. But here is a quick explainer: terraform is build around '''resources''', provided by '''providers'''. These resources have a state stored locally in a state file, whether they are deployed, generated values etc. (note that these can literally be anything e.g. from random passwords to HTTP reequests to kubernetes resources to DNS records). These resources can then be organised into modules, which (can) have outputs from values generated by the resources. There are also “data”, but this is basically a reference to another resource which doesn’t have the controls.

When deploying, terraform will then check the state of all the current deployed modules (even pinging servers if needed) and find anything that has changed (e.g. new resources or updated values) and deploy everything.

=== Module structure ===

Modules are the core of terraform and can be a bit tricky to get your head around, as initially they are quite limited (e.g. there is no such thing as a global variable).

But basically a module has a list of inputs as and a list of outputs (and then providers). So it is expected that your module deploys some resources which are then used to output something. E.g. in [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ldap/user|our ldap user module] the module generates a random password, creates the user and assigns them to a group, then outputs the username, email and password to be used later in another resource.

So e.g.

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
// Note passing providers act somewhat like global constants, passing configuration (e.g. what ldap server we mean)
// to the module
lldap = lldap
}
}

// You can then use the email by: module.user.email or password: module.user.password</syntaxhighlight>
'''NOTE''': All files within a module (the folder) are treated as one global space, similar to how Go works. You can reference variables, locals and resources throughout all files within a module which makes it quite difficult to organise nicely.

The modules are usually structure in the way:

* <code>init.tf</code> – Usually where your <code>providers</code> go and if you are lazy (like me), everything else
* <code>vars.tf</code>/<code>variables.tf</code> – Where you put all your variables. As explained in [[#Variable madness|Variable madness]], I really don’t like this and so commonly ignore
* <code>outputs.tf</code> – All your outputs go here
* <code>*.tf</code> – Anything else, if you want to split it out nicely into other files

=== Variable madness ===

Terraform variables suck.

Anyway, so basically terraform requires you do a full definition for every variable:

<syntaxhighlight lang="terraform">variable "my_var" {
type = string

description = "Something"
nullable = false

sensitive = false # If something is sensitive MAKE THIS TRUE
}
// You can then later reference it with var.my_var</syntaxhighlight>
This defines something that must be inputed by the user, either through the module or your <code>tfvars</code> file (if in the root directory).

Due to this verbosity, and sometimes complex nature of the interfaces I like to create, I have used the <code>object</code> type e.g.:

<syntaxhighlight lang="terraform">variable "my_var" {
type = object({
l = list(string)
m = map(bool) # string -> bool. Same syntax as object, just more flexible
s = set(string) # Yes this is different to list but using the same []
option = optional(string, "my_default")
})
// ...
}</syntaxhighlight>
But even then you can’t define nice defaults for each sub item and the description is for the whole variable, so it has to be done as so. This is just raw pain and not particularly great syntax in my opinion. Also you cannot enable/disable sensitive nature of values for sub items, this means the whole object must be defined as sensitive if you have one password.

And due to the lack of global constants, you must define every variable in every sub project and duplicate the types (yes there is '''no''' way to define a type to use throughout the project).

It is also recommended that you put all variables in a <code>vars.tf</code> file. Which sure does make sense for small modules, but if its that small I find it easier to just chuck at the top of the <code>init.tf</code> file (as the terraform syntax highlighter is soooo broken). Then if its large, I find it more useful to put the variables where they are actually used – but then again this is confusing because the syntax and tooling is so bad.

Oh yeah sorry and then there are '''locals''' which are constants you can define from resources/variables and will be calculated when the information is ready. E.g.

<syntaxhighlight lang="terraform">locals {
temp_val = "hi"
}
// Then you can reference with local.temp_val</syntaxhighlight>
=== Depends on and its pains ===

The one issue with terraform is that its really slow with large projects with this. But the nature of the design encourages large projects (as you want to reference things throughout the smaller apps).

This is due to it having to create a dependency graph where objects wait on their dependencies. These dependencies can be defined by <code>depends_on</code> in any resource or just referencing a value from another resource.

This is really useful so deployments are not actually deployed until all the secrets are deployed. Due to my perferable of not repeating myself, I heavily use the inferred dependency from the referring to resource names e.g. <code>kubernetes_secret_v1.secret.metadata[0].name</code> (yes this is why I don’t just do the simple thing and use the shorter name, its good to know where the value comes from).

'''BUT''' you cannot rely on this, as some things take time to actually deploy even if it says its successful. Therefore you may need to use timers instead. I usually don’t bother due to the numerous other issues with terraform making it so its not actually perfect so there’s minimal point actually making it easy to deploy from scratch.

=== Timeouts ===

If something goes wrong during deployment, e.g. you make a typo, you will have to wait the FULL timeout time. This is really painful when you typo the hostname to the db causing the pod to crashloop in a helm config and you have to wait 10 minutes for terraform to give up. You can Ctrl-C, Ctrl-C, but this causes more issues as you will have to manually intervene and delete the helm chart/deployment before you run the command again.

Instead I recommend shortening the timeouts for the deployment/helm to one more applicable to the application. A lot of our first-party stuff usually deploys in a few seconds and if it doesn’t, something has gone very wrong.

=== Commas or no commas? ===

The terraform syntax is… interesting. Commas are optional in most cases. So I would recommend, not typing commas where they optional.

'''BUT''' within lists/sets (basically between <code>[]</code>) you have to type commas, and if this is across multiple lines '''PLEASE ADD TRAILING COMMAS'''. The reason? Git histories look sooooooo much better.

== How to create a basic project ==

=== Creating a namespace ===

There is a handy util module for this:

<syntaxhighlight lang="terraform">module "example_namespace" {
source = "./utils/namespace"

name = "example"

enable_dns = true
enable_mail = true
enable_lldap = true
bkp = {
// ...
}

providers = {
kubernetes = kubernetes
}
}</syntaxhighlight>
This allows you to enable or disable features for your namespace, e.g. if your pods need to communicate with the outside world, enabling the DNS. All of these features are disabled by default and its heavily encouraged to only enable the features if the namespace needs it.

The next thing is to configure backups through this, to reduce our dependence and costs from our s3 provider, it is recommended that backups are disabled for all namespaces whos data can be regenerated (e.g. froom). If you do enable it, it is then encouraged that you disable backups for any database or pvc that you don’t need backing up with the <code>k8up.io/backup=false</code> annotation (you may notice that all <code>valkey</code> instances set this by default if you are using the <code>app/valkey</code> module).

==== Placement ====

Within this repository, it is tradition to put the namespace creation at the highest level, e.g. <code>20_apps.tf</code>. This means that the apps themselves do not control the namespace they are created in. It is mostly just a personal preference from me after years of configuring k8s on terraform.

=== Using Helm ===

Helm is by far the easiest way to deploy third-party tools, and is used throught this repo despite it’s drawbacks when combined with terraform (it’s just so easy).

You just add helm to the providers list (which defines what terraform modules you are integrating with):

<syntaxhighlight lang="terraform">terraform {
required_providers {
helm = {
source = "hashicorp/helm"
version = "~>3.1.1"
}
}
}</syntaxhighlight>
You can then use the <code>helm_release</code> resource, which takes the form:

<syntaxhighlight lang="terraform">resource "helm_release" "my_app" {
name = "my_app"
namespace = var.namespace

repository = "https://charts.example.com"
chart = "the_app"
version = "version"

values = [yamlencode({
// Values go here written within the terraform config language
})]
}</syntaxhighlight>
This deploys all the resources an application needs and manages and restarting if a secret or config map changes, providing all the configuration at your fingertips.

However this comes at a cost:

* '''SECRETS SHOULD NOT GO IN THE HELM CONFIG'''. This is a big one, all values are easily accessible unencrypted on the cluster, therefore any secrets '''MUST''' go in a <code>kubernetes_secrets_v1</code> object and you should use a <code>secretsRef</code> or similar to link it. If the helm chart does not support this '''DO NOT USE IT'''. Helm also stores the history of all values, therefore if you put it temporarily within helm values for testing, you must to a password rotation.
* You have no power over the types of resources and the structure in which it deploys. This means that if a feature or support for our strict network policies are not implemented, you have to either not use the helm chart completely or fork your own (which we definitely don’t want to do).
* If the helm chart gets deleted, all pvc related ''might'' also get deleted (unless they have the <code>Retain</code> policy, which should be the case for everything).
* Sometimes they don’t have the proper security contexts/network policies by default so you will have to add them youself (see the below section)

Overall, helm is pretty good, just use with caution and understand what templates you are inflicting. Note, you will probably have to get pretty good at reading not only default values, but schemas and the templating language of helm itself, as sometimes the charts are not particularly well documented.

=== Using kubernetes ===

For this you need to understand a bit of structure of how kubernetes works. I will assume that you are deploying a pod. If that pod needs storage attached (and not through SQL or Redis), then you will need to use a [https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ <code>StatefulSet</code>]. If you have no storage or are just communicating with a postgres server or redis, you can instead use a [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ <code>Deployment</code>].

The difference between these two concepts are not particularly seen in the world of a single node cluster, but basically deployments are free to spin up another even if the previous one is still terminating or if the node is non responsive. On the other hand, statefulsets must ensure that no two nodes are trying to access the same data, therefore cannot automatically start up if a node goes down.

This also means that it is much easier to scale a deployment to multiple nodes, vs a statefulsets which must have separate volumes per pod.

Anyway, both statefulsets and deployments have a template configuration for creating the pod associated with itself. This pod has a label which is used to monitor and track the associated pods with its parent. So the structure is as follows:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
metadata {
name = "my_deployment"
namespace = var.namespace
}
spec {
replicas = 1
selector {
match_labels = {
app = "the_deployment"
}
}
// This doesn't really matter in a one node cluster with a replicas = 1
strategy {
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "the_deployment"
}
}
spec {
container {
name = "my_deployment"
image = "bathbcss/my_image:latest"
image_pull_policy = "Always" // Should only be set if the above is "latest"

// This should be the default security context to comply with our pod security policies
security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// This is how the pod is checked its alive, so if something happens,
// e.g. job which causes it to become unresponsive, it will be automatically killed off and replaced
liveness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

// This allows the tracking of when the pod starts, so we wait until the pod is ready to receive requests
startup_probe {
http_get {
path = "/healthz"
port = 8080
}
// 3 * 30 = 90 seconds to start
failure_threshold = 30
// If it takes a while to startup, increase this time
period_seconds = 3
}

// If exposing a port the port to expose
port {
container_port = 8080
// Make sure to give it a name so we can use the name in services
name = "web"
}

// env and env_from definitions
}
}
}
}
}</syntaxhighlight>
''Quick side note: <code>kubernetes_\*_v1</code>is the preferred resouce name, any resource that does not have<code>\_v1</code> on the end is deprecated and should not be used.''

It is recommended that the version of the image is actually set and <code>latest</code> is not used, however to reduce the admin overhead, for internal projects it can be easier to set to <code>latest</code> with an image pull policy of <code>Always</code>. However this means if you want the latest version, you must have access to the cluster to restart a pod.

If you are exposing pods, this should be tied with a <code>Service</code>, as seen below.

==== Security context ====

As you will notice in the example above, we have a security context set. This is '''required''' by the pod security contenxt, otherwise it will not deploy. In most cases you can copy either of the two following policies, depending on whether it is within a kubernetes resource or helm/kubernetes manifest resource:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

securityContext = {
runAsUser = 1000
runAsNonRoot = true
allowPrivilegeEscalation = false
seccompProfile = {
type = "RuntimeDefault"
}
capabilities = {
drop = ["ALL"]
}
}

// ....
}
}</syntaxhighlight>
Note that we are running as a user (not root), setting the default seccompProfile (you ''should'' only need the default unless you are doing weird things with the host machine) as well as dropping all capabilities (you may need to add some back in but I will leave to you as you probably know more than me – NOTE: Some are disabled by our pod security policy but can be override with <code>baseline</code>).

==== Liveness and startup probe ====

The liveness and startup probes are not necessary, but is a nice to have. The liveness probe allows the cluster to detect if a pod becomes unresponsive and is then able to kill it if that is the case. Whereas a startup probe makes it so the cluster knows exactly when the pod is able to receive responses.

Please see the [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ kubernetes docs on probes] for more information on the options. But in most cases the HTTP get option should suffice, which just looks for a status 2xx code.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

liveness_probe {
http_get {
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

startup_probe {
http_get {
port = 8080
}
failure_threshold = 30
period_seconds = 3
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

livenessProbe = {
httpGet = {
port = 8080
}
initialDelaySeconds = 5
periodSeconds = 10
}

startupProbe = {
httpGet = {
port = 8080
}
failureThreshold = 30
periodSeconds = 3
}

// ....
}
}</syntaxhighlight>
==== Environmental variables ====

When configuring deployments, you will want to set environmental variables. There are a few ways to do it, but note '''ANY PASSWORDS/API KEYS GO IN SECRETS''' not the environmental variables. As you will see I will example how to do this.

By default the <code>env</code> list can be used to set a single environmental variable e.g.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "TEST"
value = "my_value"
}

// ....
}</syntaxhighlight>
However, this is quite verbose and takes a lot of space, so if you are configuring a lot of variables or have secrets, you will want to use the <code>env_from</code> list. This allows you to reference a config map or secrets (this is the most basic form).

These look like:

<syntaxhighlight lang="terraform">resource "kubernetes_secret_v1" "module" {
metadata {
name = "deployment-db"
namespace = var.namespace
}
data = {
DATABASE_URL = module.database.url
}
type = "Opaque"
}

resource "kubernetes_config_map_v1" "module" {
metadata {
name = "deployment-config"
namespace = var.namespace
}
data = {
PUBLIC_VALUE = "yoooo"
ROCKET_ADDRESS = "::"
}
}

resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env_from {
secret_ref {
name = kubernetes_secret_v1.module.metadata[0].name
}
}
env_from {
config_map_ref {
name = kubernetes_config_map_v1.module.metadata[0].name
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

envFrom = [
{
secretRef = {
name = kubernetes_secret_v1.module.metadata[0].name
}
},
{
configMapRef = {
name = kubernetes_config_map_v1.module.metadata[0].name
}
},
]

// ....
}
}</syntaxhighlight>
Note that you can set the value of a environmental variable from a secret on an individual basis, which can be useful if you are storing environmental variables as well as files inside your secret. E.g:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "DB_PASSWORD"
value_from {
secret_key_ref {
name = kubernetes_secret_v1.module.metadata[0].name
key = "password"
}
}
}

// ....
}</syntaxhighlight>
==== Third-party CRDs ====

Now this is is where terraform becomes less good. Basically when deploying using the kubernetes API, the checks will make sure the CRDs (so the like api and kind are installed and supported on the kubernetes cluster). This means that you won’t even be able to run.

Basically it means that you need to comment out manifests that reference these resources until the CRDs are deployed (usually though a helm chart or something).

==== PVCs ====

Please remember '''ANY PASSWORDS/API KEYS/CERTIFICATES GO IN SECRETS''' not in the storage (this also means its configurable by us and yes they can be mounted as read only volumes).

So, kubernetes storage works around persistant volumes which are requested by persistant volume claims. On our k3s single node, we are just using the k3s filesystem class. This means it’s a bit basic but does the job.

Things to note:

* You probably should be manually creating persistant volume claims (and definitely not persistant volumes), instead using <code>statefulsets</code>
* It’s really hard to change persistant volumes post fact, so please go through testing phase if you are unsure about anything.
* K3s does not support the storage limit, so please '''DON’T RELY ON IT''' to stop abusive behaviour.
* If you are defining yourself, do not accidentally make your deployment depend on the persistant volume claim, as the pvc will not be created until it is used in something. See [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/services/mail/mailserver.tf|the mailserver module] of how to handle it. '''NOTE''': If you reference the pvc config in your deployment, terraform will add that automatically to the <code>depends_on</code> list.
* If it is critical data you will need to '''manually update the pv to “retain” its data''' if the pvc gets deleted. This just adds a bit of safety if you mess up a deployment. This can be done in <code>k9s</code> by an admin, updating the <code>persistentVolumeReclaimPolicy</code> to <code>Retain</code> instead of <code>Delete</code>.

So how you should be using pvc, in statefulsets:

<syntaxhighlight lang="terraform">resource "kubernetes_stateful_set_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
service_name = var.name
replicas = 1
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
}
}
spec {
container {
name = "my_app"
image = "bathbcss/my_app:1.0.0"

// ...

volume_mount {
name = "data"
mount_path = "/data"
}
}
}
}
volume_claim_template {
metadata {
name = "data"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
}
}
}
}</syntaxhighlight>
==== Services ====

Servies are the first way to adding an ingress, and helps give a common endpoint if you have multiple pods running (though we don’t do this). If using within the cluster you can access a service with the address <code>service_name.namespace.svc.cluster.local</code>. Note that if you are accessing it from within the same namespace you can just use the <code>service_name</code> as the hostname (and it is better on a network policy basis).

For a namespace you just need a label to select on (note we have to define an <code>app</code> label for the statefulset/deployment anyway so you can just use this).

So it should look like this:

<syntaxhighlight lang="terraform">resource "kubernetes_service_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
port {
port = 8080
name = "web"
}
selector = {
app = var.name
}
type = "ClusterIP"
}
}</syntaxhighlight>
And then you can just access the port by <code>service_name:8080</code>. Note that you can also set a <code>target_port</code> if you want to change the port from the deployment and service (but like why?).

Also giving it a <code>name</code> is important, so in the ingress we can just use the name instead of the port number itself, increasing readability.

==== Network policies ====

Now comes the pain. If a helm chart has a network policy, use that (but please actually read what permissions it gives).

Network policies, either grant, or block network “ingress” (stuff going into the pod) and “egress” (stuff going out of the pod). By default, due to our security, ingress to our pods is denied (so anything in the cluster), but egress to the outside world is allowed.

You then use either pod selectors or namespace selectors to allow traffic from a pod (which is how <code>enable_dns,lldap,mail</code> works). In most cases egress can be left alone, unless you want to block a node from doing something in particular.

As talked about in the next section, for postgres and valkey, we create these policies, limited to pods with the <code>${name}-${service}-client=true</code> label allowing only inter-namespace communication.

If you do have to write one, which I really hope you don’t, please read [https://kubernetes.io/docs/concepts/services-networking/network-policies/ kubernetes documentation on Network Policies].

=== Postgres and Redis/Valkey ===

We have utilities for [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/postgres|postgres] and redis (through [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/valkey|valkey] due to redis being really hard to run).

If you require these, we highly recommend the above (though postgres needs moving away from bitnami due to the requirement of money for stability). These setup the network policies you need as well as generating secure passwords and you can look at the <code>outputs.tf</code> file to see the outputs you can use from a module.

'''NOTE:''' If using this, for any pod accessing the data, it must be in the same namespace and have the labels <code>${name}-postgresql-client=true</code> and/or <code>${name}-valkey-client=true</code>, otherwise the traffic will be denied by the network policies.

E.g.

<syntaxhighlight lang="terraform">module "database" {
source = "../postgres"

namespace = var.namespace
prefix = var.name
username = "user"
database = "db_name"
size = "250Mi"

providers = {
helm = helm
}
}

// Create a secret with module.database.url

resource "kubernetes_deployment_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
// This **MUST** be defined otherwise you'll get weird errors
froom-pg-postgresql-client = true
}
}
// ...
}
}
}</syntaxhighlight>
Side note: the namespace will also need to have <code>enable_dns</code> to be able to access it

=== Adding ingress ===

Adding ingress to service basically means that you are making it accessible to the outside world.

If you are doing this, please consider security heavily:

* Can any user alter the DB? Are you doing proper type checking on the inputs?
* Do you make sure to not expose any secrets e.g. db urls
* Do you really need to expose this pod? Or can you leave it to k9s port forwarding?

Then you need to ask:

* Just expose it to people within the bath network e.g. on <code>*.k8s.bathcs.com</code>
* Expose it to the whole world <code>*.bathcs.com</code> (note coordination with backstage to get them to update their traefik will be necessary)

Please expose to the utter minimum people.

If exposing to the full network please note the flow is:

* <code>app.bathcs.com</code> <- You should accept this domain in the ingress
* <code>app.bcss.su.bath.ac.uk</code> <- You should accept this domain in the ingress
* <code>app.k8s.bathcs.com</code> <- This is what the certificate you should be giving (due to this is hostname that backstage is requesting)

Depending on the chosen level, please look at the [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ingress|<code>utils/ingress/*</code> modules] as these handle this most of this flow for you.

E.g. making it internal to bath uni only:

<syntaxhighlight lang="terraform">module "dns" {
source = "../../utils/ingress/dns_flow"

subdomain = var.subdomain
# We don't want public one, so don't specify cloudflare_zone_id
k8s_cloudflare_zone_id = var.cloudflare_zone_id

# Expanded for effect
domains = {
uni = var.domains.uni
k8s = var.domains.k8s
}

providers = {
cloudflare = cloudflare,
}
}

module "ingress" {
source = "../../utils/ingress/tls"

name = var.subdomain
entry_points = ["websecure"]
namespace = var.namespace
host = module.dns.hosts.k8s
additional_ingress_hosts = [module.dns.hosts.uni]
service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "http"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
}
}</syntaxhighlight>
If using a helm repo you may have to manually define the certificate, which is quite easy:

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "cert" {
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "Certificate"
metadata = {
name = "${var.name}-cert"
namespace = var.namespace
}
spec = {
secretName = "${var.name}-cert-secret"
issuerRef = var.cert_issuer
dnsNames = [var.host]
}
}
}</syntaxhighlight>
Which outputs the cert to the secret with name <code>${var.name}-cert-secret</code>. You can also use <code>kubernetes_manifest.cert.manifest.spec.secretName</code> (you can guess what I prefer).

If you are making it publically accessible, you can use the full flow which does both dns and ingress records:

<syntaxhighlight lang="terraform">module "ingress" {
source = "../../utils/ingress/full_flow"

cloudflare_zone_id = var.cloudflare_zone_id
namespace = var.namespace
domains = var.domains
subdomain = var.subdomain

service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "web"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
cloudflare = cloudflare
}
}</syntaxhighlight>
==== Putting it behind authelia ====

If the application doesn’t have oauth integration build in and you are wanting to protect it behind authelia you can add middleware of <code>auth-forwardauth-authelia@kubernetescrd</code> to require them to go through authelia.

You will then have to add an access control rule to authelia e.g.

<pre>rules = [
{
domain = "app.k8s.bathcs.com"
subject = ["group:boss-example-group", "user:hw2210"]
},
]</pre>
'''NOTE''': you can create and manage groups on [https://www.bath.ac.uk/groupmanager/ Bath's group manager], which creates a unix group that can be seen in the LDAP server.

=== Cloudflare dns records ===

Cloudflare provider is an actual pain. You would’ve hoped it would be good, but its not amazing (has previously caused updates on every <code>apply</code>). Basically when creating a dns record you should use the whole address e.g. <code>app.bathcs.com</code> instead of <code>app</code>. This is because it will cause an update on the second application, changing the <code>name</code> value (which we use to get the full domain).

It sucks because theres an additional useless variable ontop of the <code>cloudflare_zone_id</code> (yes you can use the “data” thing to solve this but still).

Additionally for long TXT records, it will add additional quotes, and terraform will continually think you need to change that if you don’t yourself add the quotes into the data.

=== Sending mail ===

Sending mail is somewhat weird.

Basically just because I can, the authentication is managed by <code>lldap</code> (a really fast ldap implementation). This should not be confused with the ldap service that Authelia is hooked up with, as it is not.

Once you’ve enabled mail in the namespace you then have to create an ldap user:

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
lldap = lldap
}
}</syntaxhighlight>
You can then use this to access the mail with <code>module.user.name</code>, <code>module.user.password</code> and <code>module.user.email</code>. '''NOTE''' due to it not being exposed, we must use the kubernetes domain certificate, which means that you normally have to disable tls verification it or set the expected domain to <code>bathcs.com</code> (as seen in authelia).

=== Creating an oauth client ===

I have created a [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/oauth/client|OAuth client module] for generating all the secret data you need and generate the client config (under the <code>config</code> output) you can then pass up and then into the authelia module (see <code>grafana</code> as an example).

But basically please read [https://www.authelia.com/integration/openid-connect/introduction/ the authelia oidc docs] for a full explanation of how it works.

Terraform

2026-06-04T11:02:30Z

Hw2210: Translate from boss.bathcs.com

[[BOSS]] uses terraform (actually [https://opentofu.org/ <code>opentofu</code>]) to deploy all the resources onto [[BOSS/Hosting/Cluster their kubernetes cluster]], which tries to be the language in which you can deploy anything and everything and if you need to quickly redeploy a whole machine you can with a simple command. In practise it doesn’t work like that, but its good enough for our needs.

You can find the [https://gitlab.bath.ac.uk/cs/int/terraform terraform repo on gitlab], this page was originally taken from the README as it got too long.

== Other resources ==

* [https://developer.hashicorp.com/terraform/tutorials Hashicorp (the maker of the closed source terraform) has some good tutorials on their site]

Basically that’s it really, the rest of this will be looking at how to deploy an application with our configuration.

== Terraform and its woes ==

Before we get into making an app, I must briefly explain terraform and its benefits/issues/confusing behaviours.

This expects you to have a rough idea around how terraform works. But here is a quick explainer: terraform is build around '''resources''', provided by '''providers'''. These resources have a state stored locally in a state file, whether they are deployed, generated values etc. (note that these can literally be anything e.g. from random passwords to HTTP reequests to kubernetes resources to DNS records). These resources can then be organised into modules, which (can) have outputs from values generated by the resources. There are also “data”, but this is basically a reference to another resource which doesn’t have the controls.

When deploying, terraform will then check the state of all the current deployed modules (even pinging servers if needed) and find anything that has changed (e.g. new resources or updated values) and deploy everything.

=== Module structure ===

Modules are the core of terraform and can be a bit tricky to get your head around, as initially they are quite limited (e.g. there is no such thing as a global variable).

But basically a module has a list of inputs as and a list of outputs (and then providers). So it is expected that your module deploys some resources which are then used to output something. E.g. in [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ldap/user|our ldap user module] the module generates a random password, creates the user and assigns them to a group, then outputs the username, email and password to be used later in another resource.

So e.g.

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
// Note passing providers act somewhat like global constants, passing configuration (e.g. what ldap server we mean)
// to the module
lldap = lldap
}
}

// You can then use the email by: module.user.email or password: module.user.password</syntaxhighlight>
'''NOTE''': All files within a module (the folder) are treated as one global space, similar to how Go works. You can reference variables, locals and resources throughout all files within a module which makes it quite difficult to organise nicely.

The modules are usually structure in the way:

* <code>init.tf</code> – Usually where your <code>providers</code> go and if you are lazy (like me), everything else
* <code>vars.tf</code>/<code>variables.tf</code> – Where you put all your variables. As explained in [[#Variable madness|Variable madness]], I really don’t like this and so commonly ignore
* <code>outputs.tf</code> – All your outputs go here
* <code>*.tf</code> – Anything else, if you want to split it out nicely into other files

=== Variable madness ===

Terraform variables suck.

Anyway, so basically terraform requires you do a full definition for every variable:

<syntaxhighlight lang="terraform">variable "my_var" {
type = string

description = "Something"
nullable = false

sensitive = false # If something is sensitive MAKE THIS TRUE
}
// You can then later reference it with var.my_var</syntaxhighlight>
This defines something that must be inputed by the user, either through the module or your <code>tfvars</code> file (if in the root directory).

Due to this verbosity, and sometimes complex nature of the interfaces I like to create, I have used the <code>object</code> type e.g.:

<syntaxhighlight lang="terraform">variable "my_var" {
type = object({
l = list(string)
m = map(bool) # string -> bool. Same syntax as object, just more flexible
s = set(string) # Yes this is different to list but using the same []
option = optional(string, "my_default")
})
// ...
}</syntaxhighlight>
But even then you can’t define nice defaults for each sub item and the description is for the whole variable, so it has to be done as so. This is just raw pain and not particularly great syntax in my opinion. Also you cannot enable/disable sensitive nature of values for sub items, this means the whole object must be defined as sensitive if you have one password.

And due to the lack of global constants, you must define every variable in every sub project and duplicate the types (yes there is '''no''' way to define a type to use throughout the project).

It is also recommended that you put all variables in a <code>vars.tf</code> file. Which sure does make sense for small modules, but if its that small I find it easier to just chuck at the top of the <code>init.tf</code> file (as the terraform syntax highlighter is soooo broken). Then if its large, I find it more useful to put the variables where they are actually used – but then again this is confusing because the syntax and tooling is so bad.

Oh yeah sorry and then there are '''locals''' which are constants you can define from resources/variables and will be calculated when the information is ready. E.g.

<syntaxhighlight lang="terraform">locals {
temp_val = "hi"
}
// Then you can reference with local.temp_val</syntaxhighlight>
=== Depends on and its pains ===

The one issue with terraform is that its really slow with large projects with this. But the nature of the design encourages large projects (as you want to reference things throughout the smaller apps).

This is due to it having to create a dependency graph where objects wait on their dependencies. These dependencies can be defined by <code>depends_on</code> in any resource or just referencing a value from another resource.

This is really useful so deployments are not actually deployed until all the secrets are deployed. Due to my perferable of not repeating myself, I heavily use the inferred dependency from the referring to resource names e.g. <code>kubernetes_secret_v1.secret.metadata[0].name</code> (yes this is why I don’t just do the simple thing and use the shorter name, its good to know where the value comes from).

'''BUT''' you cannot rely on this, as some things take time to actually deploy even if it says its successful. Therefore you may need to use timers instead. I usually don’t bother due to the numerous other issues with terraform making it so its not actually perfect so there’s minimal point actually making it easy to deploy from scratch.

=== Timeouts ===

If something goes wrong during deployment, e.g. you make a typo, you will have to wait the FULL timeout time. This is really painful when you typo the hostname to the db causing the pod to crashloop in a helm config and you have to wait 10 minutes for terraform to give up. You can Ctrl-C, Ctrl-C, but this causes more issues as you will have to manually intervene and delete the helm chart/deployment before you run the command again.

Instead I recommend shortening the timeouts for the deployment/helm to one more applicable to the application. A lot of our first-party stuff usually deploys in a few seconds and if it doesn’t, something has gone very wrong.

=== Commas or no commas? ===

The terraform syntax is… interesting. Commas are optional in most cases. So I would recommend, not typing commas where they optional.

'''BUT''' within lists/sets (basically between <code>[]</code>) you have to type commas, and if this is across multiple lines '''PLEASE ADD TRAILING COMMAS'''. The reason? Git histories look sooooooo much better.

== How to create a basic project ==

=== Creating a namespace ===

There is a handy util module for this:

<syntaxhighlight lang="terraform">module "example_namespace" {
source = "./utils/namespace"

name = "example"

enable_dns = true
enable_mail = true
enable_lldap = true
bkp = {
// ...
}

providers = {
kubernetes = kubernetes
}
}</syntaxhighlight>
This allows you to enable or disable features for your namespace, e.g. if your pods need to communicate with the outside world, enabling the DNS. All of these features are disabled by default and its heavily encouraged to only enable the features if the namespace needs it.

The next thing is to configure backups through this, to reduce our dependence and costs from our s3 provider, it is recommended that backups are disabled for all namespaces whos data can be regenerated (e.g. froom). If you do enable it, it is then encouraged that you disable backups for any database or pvc that you don’t need backing up with the <code>k8up.io/backup=false</code> annotation (you may notice that all <code>valkey</code> instances set this by default if you are using the <code>app/valkey</code> module).

==== Placement ====

Within this repository, it is tradition to put the namespace creation at the highest level, e.g. <code>20_apps.tf</code>. This means that the apps themselves do not control the namespace they are created in. It is mostly just a personal preference from me after years of configuring k8s on terraform.

=== Using Helm ===

Helm is by far the easiest way to deploy third-party tools, and is used throught this repo despite it’s drawbacks when combined with terraform (it’s just so easy).

You just add helm to the providers list (which defines what terraform modules you are integrating with):

<syntaxhighlight lang="terraform">terraform {
required_providers {
helm = {
source = "hashicorp/helm"
version = "~>3.1.1"
}
}
}</syntaxhighlight>
You can then use the <code>helm_release</code> resource, which takes the form:

<syntaxhighlight lang="terraform">resource "helm_release" "my_app" {
name = "my_app"
namespace = var.namespace

repository = "https://charts.example.com"
chart = "the_app"
version = "version"

values = [yamlencode({
// Values go here written within the terraform config language
})]
}</syntaxhighlight>
This deploys all the resources an application needs and manages and restarting if a secret or config map changes, providing all the configuration at your fingertips.

However this comes at a cost:

* '''SECRETS SHOULD NOT GO IN THE HELM CONFIG'''. This is a big one, all values are easily accessible unencrypted on the cluster, therefore any secrets '''MUST''' go in a <code>kubernetes_secrets_v1</code> object and you should use a <code>secretsRef</code> or similar to link it. If the helm chart does not support this '''DO NOT USE IT'''. Helm also stores the history of all values, therefore if you put it temporarily within helm values for testing, you must to a password rotation.
* You have no power over the types of resources and the structure in which it deploys. This means that if a feature or support for our strict network policies are not implemented, you have to either not use the helm chart completely or fork your own (which we definitely don’t want to do).
* If the helm chart gets deleted, all pvc related ''might'' also get deleted (unless they have the <code>Retain</code> policy, which should be the case for everything).
* Sometimes they don’t have the proper security contexts/network policies by default so you will have to add them youself (see the below section)

Overall, helm is pretty good, just use with caution and understand what templates you are inflicting. Note, you will probably have to get pretty good at reading not only default values, but schemas and the templating language of helm itself, as sometimes the charts are not particularly well documented.

=== Using kubernetes ===

For this you need to understand a bit of structure of how kubernetes works. I will assume that you are deploying a pod. If that pod needs storage attached (and not through SQL or Redis), then you will need to use a [https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ <code>StatefulSet</code>]. If you have no storage or are just communicating with a postgres server or redis, you can instead use a [https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ <code>Deployment</code>].

The difference between these two concepts are not particularly seen in the world of a single node cluster, but basically deployments are free to spin up another even if the previous one is still terminating or if the node is non responsive. On the other hand, statefulsets must ensure that no two nodes are trying to access the same data, therefore cannot automatically start up if a node goes down.

This also means that it is much easier to scale a deployment to multiple nodes, vs a statefulsets which must have separate volumes per pod.

Anyway, both statefulsets and deployments have a template configuration for creating the pod associated with itself. This pod has a label which is used to monitor and track the associated pods with its parent. So the structure is as follows:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
metadata {
name = "my_deployment"
namespace = var.namespace
}
spec {
replicas = 1
selector {
match_labels = {
app = "the_deployment"
}
}
// This doesn't really matter in a one node cluster with a replicas = 1
strategy {
type = "RollingUpdate"
}
template {
metadata {
labels = {
app = "the_deployment"
}
}
spec {
container {
name = "my_deployment"
image = "bathbcss/my_image:latest"
image_pull_policy = "Always" // Should only be set if the above is "latest"

// This should be the default security context to comply with our pod security policies
security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// This is how the pod is checked its alive, so if something happens,
// e.g. job which causes it to become unresponsive, it will be automatically killed off and replaced
liveness_probe {
http_get {
path = "/healthz"
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

// This allows the tracking of when the pod starts, so we wait until the pod is ready to receive requests
startup_probe {
http_get {
path = "/healthz"
port = 8080
}
// 3 * 30 = 90 seconds to start
failure_threshold = 30
// If it takes a while to startup, increase this time
period_seconds = 3
}

// If exposing a port the port to expose
port {
container_port = 8080
// Make sure to give it a name so we can use the name in services
name = "web"
}

// env and env_from definitions
}
}
}
}
}</syntaxhighlight>
''Quick side note: <code>kubernetes_\*_v1</code>is the preferred resouce name, any resource that does not have<code>\_v1</code> on the end is deprecated and should not be used.''

It is recommended that the version of the image is actually set and <code>latest</code> is not used, however to reduce the admin overhead, for internal projects it can be easier to set to <code>latest</code> with an image pull policy of <code>Always</code>. However this means if you want the latest version, you must have access to the cluster to restart a pod.

If you are exposing pods, this should be tied with a <code>Service</code>, as seen below.

==== Security context ====

As you will notice in the example above, we have a security context set. This is '''required''' by the pod security contenxt, otherwise it will not deploy. In most cases you can copy either of the two following policies, depending on whether it is within a kubernetes resource or helm/kubernetes manifest resource:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

security_context {
run_as_user = 1000
run_as_non_root = true
allow_privilege_escalation = false
seccomp_profile {
type = "RuntimeDefault"
}
capabilities {
drop = ["ALL"]
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

securityContext = {
runAsUser = 1000
runAsNonRoot = true
allowPrivilegeEscalation = false
seccompProfile = {
type = "RuntimeDefault"
}
capabilities = {
drop = ["ALL"]
}
}

// ....
}
}</syntaxhighlight>
Note that we are running as a user (not root), setting the default seccompProfile (you ''should'' only need the default unless you are doing weird things with the host machine) as well as dropping all capabilities (you may need to add some back in but I will leave to you as you probably know more than me – NOTE: Some are disabled by our pod security policy but can be override with <code>baseline</code>).

==== Liveness and startup probe ====

The liveness and startup probes are not necessary, but is a nice to have. The liveness probe allows the cluster to detect if a pod becomes unresponsive and is then able to kill it if that is the case. Whereas a startup probe makes it so the cluster knows exactly when the pod is able to receive responses.

Please see the [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ kubernetes docs on probes] for more information on the options. But in most cases the HTTP get option should suffice, which just looks for a status 2xx code.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

liveness_probe {
http_get {
port = 8080
}
initial_delay_seconds = 5
period_seconds = 10
}

startup_probe {
http_get {
port = 8080
}
failure_threshold = 30
period_seconds = 3
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

livenessProbe = {
httpGet = {
port = 8080
}
initialDelaySeconds = 5
periodSeconds = 10
}

startupProbe = {
httpGet = {
port = 8080
}
failureThreshold = 30
periodSeconds = 3
}

// ....
}
}</syntaxhighlight>
==== Environmental variables ====

When configuring deployments, you will want to set environmental variables. There are a few ways to do it, but note '''ANY PASSWORDS/API KEYS GO IN SECRETS''' not the environmental variables. As you will see I will example how to do this.

By default the <code>env</code> list can be used to set a single environmental variable e.g.

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "TEST"
value = "my_value"
}

// ....
}</syntaxhighlight>
However, this is quite verbose and takes a lot of space, so if you are configuring a lot of variables or have secrets, you will want to use the <code>env_from</code> list. This allows you to reference a config map or secrets (this is the most basic form).

These look like:

<syntaxhighlight lang="terraform">resource "kubernetes_secret_v1" "module" {
metadata {
name = "deployment-db"
namespace = var.namespace
}
data = {
DATABASE_URL = module.database.url
}
type = "Opaque"
}

resource "kubernetes_config_map_v1" "module" {
metadata {
name = "deployment-config"
namespace = var.namespace
}
data = {
PUBLIC_VALUE = "yoooo"
ROCKET_ADDRESS = "::"
}
}

resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env_from {
secret_ref {
name = kubernetes_secret_v1.module.metadata[0].name
}
}
env_from {
config_map_ref {
name = kubernetes_config_map_v1.module.metadata[0].name
}
}

// ....
}</syntaxhighlight>
Or

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "my_deployment" {
manifest = {
// ....

envFrom = [
{
secretRef = {
name = kubernetes_secret_v1.module.metadata[0].name
}
},
{
configMapRef = {
name = kubernetes_config_map_v1.module.metadata[0].name
}
},
]

// ....
}
}</syntaxhighlight>
Note that you can set the value of a environmental variable from a secret on an individual basis, which can be useful if you are storing environmental variables as well as files inside your secret. E.g:

<syntaxhighlight lang="terraform">resource "kubernetes_deployment_v1" "my_deployment" {
// ....

env {
name = "DB_PASSWORD"
value_from {
secret_key_ref {
name = kubernetes_secret_v1.module.metadata[0].name
key = "password"
}
}
}

// ....
}</syntaxhighlight>
==== Third-party CRDs ====

Now this is is where terraform becomes less good. Basically when deploying using the kubernetes API, the checks will make sure the CRDs (so the like api and kind are installed and supported on the kubernetes cluster). This means that you won’t even be able to run.

Basically it means that you need to comment out manifests that reference these resources until the CRDs are deployed (usually though a helm chart or something).

==== PVCs ====

Please remember '''ANY PASSWORDS/API KEYS/CERTIFICATES GO IN SECRETS''' not in the storage (this also means its configurable by us and yes they can be mounted as read only volumes).

So, kubernetes storage works around persistant volumes which are requested by persistant volume claims. On our k3s single node, we are just using the k3s filesystem class. This means it’s a bit basic but does the job.

Things to note:

* You probably should be manually creating persistant volume claims (and definitely not persistant volumes), instead using <code>statefulsets</code>
* It’s really hard to change persistant volumes post fact, so please go through testing phase if you are unsure about anything.
* K3s does not support the storage limit, so please '''DON’T RELY ON IT''' to stop abusive behaviour.
* If you are defining yourself, do not accidentally make your deployment depend on the persistant volume claim, as the pvc will not be created until it is used in something. See [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/services/mail/mailserver.tf|the mailserver module] of how to handle it. '''NOTE''': If you reference the pvc config in your deployment, terraform will add that automatically to the <code>depends_on</code> list.
* If it is critical data you will need to '''manually update the pv to “retain” its data''' if the pvc gets deleted. This just adds a bit of safety if you mess up a deployment. This can be done in <code>k9s</code> by an admin, updating the <code>persistentVolumeReclaimPolicy</code> to <code>Retain</code> instead of <code>Delete</code>.

So how you should be using pvc, in statefulsets:

<syntaxhighlight lang="terraform">resource "kubernetes_stateful_set_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
service_name = var.name
replicas = 1
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
}
}
spec {
container {
name = "my_app"
image = "bathbcss/my_app:1.0.0"

// ...

volume_mount {
name = "data"
mount_path = "/data"
}
}
}
}
volume_claim_template {
metadata {
name = "data"
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "10Gi"
}
}
}
}
}
}</syntaxhighlight>
==== Services ====

Servies are the first way to adding an ingress, and helps give a common endpoint if you have multiple pods running (though we don’t do this). If using within the cluster you can access a service with the address <code>service_name.namespace.svc.cluster.local</code>. Note that if you are accessing it from within the same namespace you can just use the <code>service_name</code> as the hostname (and it is better on a network policy basis).

For a namespace you just need a label to select on (note we have to define an <code>app</code> label for the statefulset/deployment anyway so you can just use this).

So it should look like this:

<syntaxhighlight lang="terraform">resource "kubernetes_service_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
port {
port = 8080
name = "web"
}
selector = {
app = var.name
}
type = "ClusterIP"
}
}</syntaxhighlight>
And then you can just access the port by <code>service_name:8080</code>. Note that you can also set a <code>target_port</code> if you want to change the port from the deployment and service (but like why?).

Also giving it a <code>name</code> is important, so in the ingress we can just use the name instead of the port number itself, increasing readability.

==== Network policies ====

Now comes the pain. If a helm chart has a network policy, use that (but please actually read what permissions it gives).

Network policies, either grant, or block network “ingress” (stuff going into the pod) and “egress” (stuff going out of the pod). By default, due to our security, ingress to our pods is denied (so anything in the cluster), but egress to the outside world is allowed.

You then use either pod selectors or namespace selectors to allow traffic from a pod (which is how <code>enable_dns,lldap,mail</code> works). In most cases egress can be left alone, unless you want to block a node from doing something in particular.

As talked about in the next section, for postgres and valkey, we create these policies, limited to pods with the <code>${name}-${service}-client=true</code> label allowing only inter-namespace communication.

If you do have to write one, which I really hope you don’t, please read [https://kubernetes.io/docs/concepts/services-networking/network-policies/ kubernetes documentation on Network Policies].

=== Postgres and Redis/Valkey ===

We have utilities for [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/postgres|postgres] and redis (through [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/apps/valkey|valkey] due to redis being really hard to run).

If you require these, we highly recommend the above (though postgres needs moving away from bitnami due to the requirement of money for stability). These setup the network policies you need as well as generating secure passwords and you can look at the <code>outputs.tf</code> file to see the outputs you can use from a module.

'''NOTE:''' If using this, for any pod accessing the data, it must be in the same namespace and have the labels <code>${name}-postgresql-client=true</code> and/or <code>${name}-valkey-client=true</code>, otherwise the traffic will be denied by the network policies.

E.g.

<syntaxhighlight lang="terraform">module "database" {
source = "../postgres"

namespace = var.namespace
prefix = var.name
username = "user"
database = "db_name"
size = "250Mi"

providers = {
helm = helm
}
}

// Create a secret with module.database.url

resource "kubernetes_deployment_v1" "module" {
metadata {
name = var.name
namespace = var.namespace
}
spec {
selector {
match_labels = {
app = var.name
}
}
template {
metadata {
labels = {
app = var.name
// This **MUST** be defined otherwise you'll get weird errors
froom-pg-postgresql-client = true
}
}
// ...
}
}
}</syntaxhighlight>
Side note: the namespace will also need to have <code>enable_dns</code> to be able to access it

=== Adding ingress ===

Adding ingress to service basically means that you are making it accessible to the outside world.

If you are doing this, please consider security heavily:

* Can any user alter the DB? Are you doing proper type checking on the inputs?
* Do you make sure to not expose any secrets e.g. db urls
* Do you really need to expose this pod? Or can you leave it to k9s port forwarding?

Then you need to ask:

* Just expose it to people within the bath network e.g. on <code>*.k8s.bathcs.com</code>
* Expose it to the whole world <code>*.bathcs.com</code> (note coordination with backstage to get them to update their traefik will be necessary)

Please expose to the utter minimum people.

If exposing to the full network please note the flow is:

* <code>app.bathcs.com</code> <- You should accept this domain in the ingress
* <code>app.bcss.su.bath.ac.uk</code> <- You should accept this domain in the ingress
* <code>app.k8s.bathcs.com</code> <- This is what the certificate you should be giving (due to this is hostname that backstage is requesting)

Depending on the chosen level, please look at the [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/ingress|<code>utils/ingress/*</code> modules] as these handle this most of this flow for you.

E.g. making it internal to bath uni only:

<syntaxhighlight lang="terraform">module "dns" {
source = "../../utils/ingress/dns_flow"

subdomain = var.subdomain
# We don't want public one, so don't specify cloudflare_zone_id
k8s_cloudflare_zone_id = var.cloudflare_zone_id

# Expanded for effect
domains = {
uni = var.domains.uni
k8s = var.domains.k8s
}

providers = {
cloudflare = cloudflare,
}
}

module "ingress" {
source = "../../utils/ingress/tls"

name = var.subdomain
entry_points = ["websecure"]
namespace = var.namespace
host = module.dns.hosts.k8s
additional_ingress_hosts = [module.dns.hosts.uni]
service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "http"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
}
}</syntaxhighlight>
If using a helm repo you may have to manually define the certificate, which is quite easy:

<syntaxhighlight lang="terraform">resource "kubernetes_manifest" "cert" {
manifest = {
apiVersion = "cert-manager.io/v1"
kind = "Certificate"
metadata = {
name = "${var.name}-cert"
namespace = var.namespace
}
spec = {
secretName = "${var.name}-cert-secret"
issuerRef = var.cert_issuer
dnsNames = [var.host]
}
}
}</syntaxhighlight>
Which outputs the cert to the secret with name <code>${var.name}-cert-secret</code>. You can also use <code>kubernetes_manifest.cert.manifest.spec.secretName</code> (you can guess what I prefer).

If you are making it publically accessible, you can use the full flow which does both dns and ingress records:

<syntaxhighlight lang="terraform">module "ingress" {
source = "../../utils/ingress/full_flow"

cloudflare_zone_id = var.cloudflare_zone_id
namespace = var.namespace
domains = var.domains
subdomain = var.subdomain

service = {
name = kubernetes_service_v1.module.metadata[0].name
port = "web"
}
cert_issuer = var.cert_issuer

providers = {
kubernetes = kubernetes,
cloudflare = cloudflare
}
}</syntaxhighlight>
==== Putting it behind authelia ====

If the application doesn’t have oauth integration build in and you are wanting to protect it behind authelia you can add middleware of <code>auth-forwardauth-authelia@kubernetescrd</code> to require them to go through authelia.

You will then have to add an access control rule to authelia e.g.

<pre>rules = [
{
domain = "app.k8s.bathcs.com"
subject = ["user:hw2210"]
},
]</pre>
'''NOTE''': Sadly we can’t use groups and so we have to limit by user directly.

=== Cloudflare dns records ===

Cloudflare provider is an actual pain. You would’ve hoped it would be good, but its not amazing (has previously caused updates on every <code>apply</code>). Basically when creating a dns record you should use the whole address e.g. <code>app.bathcs.com</code> instead of <code>app</code>. This is because it will cause an update on the second application, changing the <code>name</code> value (which we use to get the full domain).

It sucks because theres an additional useless variable ontop of the <code>cloudflare_zone_id</code> (yes you can use the “data” thing to solve this but still).

Additionally for long TXT records, it will add additional quotes, and terraform will continually think you need to change that if you don’t yourself add the quotes into the data.

=== Sending mail ===

Sending mail is somewhat weird.

Basically just because I can, the authentication is managed by <code>lldap</code> (a really fast ldap implementation). This should not be confused with the ldap service that Authelia is hooked up with, as it is not.

Once you’ve enabled mail in the namespace you then have to create an ldap user:

<syntaxhighlight lang="terraform">module "user" {
source = "../../utils/ldap/user"

display = "Example"
username = "example"
group_ids = var.ldap_group_ids

providers = {
lldap = lldap
}
}</syntaxhighlight>
You can then use this to access the mail with <code>module.user.name</code>, <code>module.user.password</code> and <code>module.user.email</code>. '''NOTE''' due to it not being exposed, we must use the kubernetes domain certificate, which means that you normally have to disable tls verification it or set the expected domain to <code>bathcs.com</code> (as seen in authelia).

=== Creating an oauth client ===

I have created a [https://gitlab.bath.ac.uk/cs/int/terraform/-/blob/main/utils/oauth/client|OAuth client module] for generating all the secret data you need and generate the client config (under the <code>config</code> output) you can then pass up and then into the authelia module (see <code>grafana</code> as an example).

But basically please read [https://www.authelia.com/integration/openid-connect/introduction/ the authelia oidc docs] for a full explanation of how it works.

Kubernetes

2026-06-04T10:56:45Z

Hw2210: Add text for general knowledge section

BOSS/Hosting/Cluster

2026-06-04T10:20:55Z

Hw2210: Add information about BOSS's kubernetes cluster

To make hosting for free as easy as pie, [[Bath Open Source Society|BOSS]] has its own [[Kubernetes]] cluster. This is hosted on servers donated to us by the Department of Computer Science and is hosted within their server room in 1W. This allows us to host databases and any docker container for free, which all is configured within [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo].

== Deployment ==
If you are wanting to use this cluster for hosting projects, please get in contact with committee. However, this should be part of the process with adding a project.

When deploying, a module and related configuration will be added to [https://gitlab.bath.ac.uk/cs/int/terraform our terraform repo], which then is deployed by a sysadmin to the staging cluster. Once we have validated that it works on staging, it will finally be deployed to production.

== Machine Structure ==
{{Note|text=The more technical detail about the cluster is stored within [https://gitlab.bath.ac.uk/cs/boss/int-wiki BOSS's internal wiki in GitLab] and is only accessible by Committee and sysadmins. This page is here to provide some context and maybe provide some interesting information.}}
Due to me (hw2210) bricking an SSD in 6 months on my own 5 node cluster due to disk intensive operations with [https://longhorn.io/ longhorn] (a storage manager that shares drives across nodes), it was decided that we would only host a 1 node cluster with K3S's basic storage manager to extend the life of our finite disks as much as possible.

From experience, it was also decided that we would host the clusters in VMs with immutable distro's to allow more flexibility with moving VMs between machines and stability (if it continuously crashes, we can get access to the console without plugging into host).

Therefore our final structure was:

* Production host machine running FreeBSD with ZFS (for data protection as ZFS is king)
** Production VM with CoreOS running K3S
* Staging host machine running FreeBSD with ZFS
** Staging VM with CoreOS running K3S

The VM's themselves are then backed up to a backup server allowing for easy whole cluster revertions and recovery without any painful redeployment.

=== CoreOS ===
CoreOS was chosen as it is an immutable distro based off of Fedora with [[SELinux]] support and is extremely lightweight. When first botting up, it uses an ignition file to configure itself (including network access), our ignition files can be found in [https://gitlab.bath.ac.uk/cs/int/ignition our ignition repo]. However, once the VM is running, it will not reapply ignition files, and so all changes must be performed by a wheel user.

By immutable, we mean that the root directory cannot be altered, only <code>/var</code> can be altered during the running of the distro. <code>/etc</code> can also be changed, but it can also be reverted during boot. This has a few quirks, for example, home directories are found in <code>/var/home/<username></code> and installing any package requires a reboot of the VM - but you shouldn't really need to install any packages. Additionally, the package manager is now <code>rpm-ostree install ...</code>.

Each alteration to the installed packages or <code>/etc</code> creates a new version, by default, grub will choose the latest version to run, but during boot, there is the option to boot the previous version by using the down arrow. If you are doing a dangerous update that might require you iterating through these versions, you can pin you current version before any updates, so you can go back to it if everything fails:<syntaxhighlight lang="shell">
sudo ostree admin pin 0
rpm-ostree status # See pinned versions
sudo ostree reset 2 # Reset back to the pinned version
</syntaxhighlight>

=== FreeBSD ===
[[wikipedia:FreeBSD|FreeBSD]] was specifically chosen as it offers two benefits:

* The best support for [[wikipedia:ZFS|ZFS]] on root (Linux doesn't like ZFSes license)
* Additional layer of security as something other than Linux (for context the decision was made around the time when we had many Linux vulnerabilities released).

[[wikipedia:ZFS|ZFS]] was also chosen specifically because it has world-class data retention and backup tooling for RAID based systems, making it perfect for servers running with multiple large drives. We chose RAIDZ-1, which means one drive can fail and we can still recover all our data.

FreeBSD is like Linux in many ways as it is Unix-like and subsequently many linux tools exist on FreeBSD and can be installed. However there are minor cultural differences, such as <code>sudo</code> and <code>systemd</code> are disliked. Instead <code>doas</code> or <code>mdo</code> is recommended. Additionally VMs/containers are called jails and have many more useful features.

== K3S ==
[https://k3s.io K3S] is used for the kubernetes implementation as it provides a really easy stable base. Our configuration follows the [https://docs.k3s.io/security/hardening-guide hardened configuration] on a IPv4 only cluster (Bath sadly does not provide an IPv6 support) as well as running with [[SELinux]] enabled. This means that network policies must be defined to permit communication between pods (e.g. the database and the backend) and allows us blocking all outgoing traffic from a specific pod and requiring all pods do not run as root (with a few permitted exceptions).

== Helper services ==
Kubernetes is hard, if you want more general knowledge about how to deploy and manage kubernetes cluster, please see the [[Kubernetes|Kubernetes page]]. To help with this, we also deploy a range of "helper" applications which help with deployment and add cool features to your own application:

* [https://doc.traefik.io/traefik/ Traefik] - this is our http proxy software, managing routing traffic to the right application. This comes with the ability to [https://doc.traefik.io/traefik/expose/kubernetes/advanced/#create-middlewares create middlewares] to filter traffic based off a range of things, and combined with authelia, allows us to add a login page for applications which don't have one.
* [https://cert-manager.io/ Cert manager] - this managers automatically generating and renewing all our certificates, either DNS based, HTTP based or even Cloudflare Origin certificates, allowing for easy proxying with Cloudflare.
* [https://www.authelia.com/ Authelia] - our authentication manager hooked up to Bath's LDAP. This supports OpenConnect ID, allowing us to add authentication with bath credentials to more complex applications that require different groups. We have a range of unix groups which can be edited through [https://www.bath.ac.uk/services/group-manager/ Bath's group manager].
* [https://docker-mailserver.github.io/docker-mailserver/latest/ Docker mailserver] - This is configured with SPS, DKIM and DMARC to allow us to send emails under <code>bathcs.com</code> for free and makes sure the emails are likely to actually reach the destination without going into spam.
* [https://cloudnative-pg.io/ Cloud native's Postgres Operator] - This is just a tool to easily deploy postgres databases within the cluster and manage them.
* [https://prometheus-operator.dev/ Prometheus operator] - This allows easy monitoring of the whole cluster and emails us if anything is going wrong
* [https://grafana.com/ Grafana] - cool graphs right?
* [https://github.com/lldap/lldap LLDAP] - This is an LDAP server written in Rust and allows us to easily generate application specific credentials for our different services.
* [https://docs.k3s.io/upgrades/automated K3S upgrades] - keeps our cluster up to date
* [https://k8up.io/ k8up] - Automatically backs up our critical PVs to a Scaleway S3 bucket.
* [https://github.com/kubernetes-sigs/security-profiles-operator Security Profile Operator] - Allows for managing SELinux policies within kubernetes resources. In most cases this should not need to be touched, and is quite brittle, but allows us to host applications which need access to the host machine itself (e.g. monitoring the node itself)

BOSS/Hosting/Scaleway

2026-06-04T09:03:06Z

Hw2210: Translate from boss.bathcs.com

{{Note|text=BOSS no longer actively uses Scaleway due to setting up [[BOSS/Hosting/Cluster|our own cluster]], this page is left here if you wish to host applications yourself|type=reminder}}

[https://scaleway.com Scaleway] is similar to Google Cloud however it is a company based in Europe and has all the stuff we need to host our website for free. You can find the free tier listed on [https://www.scaleway.com/en/pricing/?tags=available scaleway’s pricing page], under “Serverless Containers”:

* 400,000 GB-s per account per month
* 200,000 vCPU-s per account per month
* 75GB storage space in container registry (note that this '''MUST''' be public and if an image is bigger than 500MB it ''might'' produce charges).

Furthermore if you do somehow go over this, it should not cost that much due to the low prices (and you can always see the estimated charge you will get at the end of the month).

Scaleway’s documentation is also amazing, so if you need help just go [https://www.scaleway.com/en/docs/ their documentation].

{{Note|text=You can have up to 1 instance running 24/7 with minimum memory and CPU under the free tier. However if you set the instance to 0 and use v2 of the sandboxing, it is as fast as if one instance is always running.}}

== Signing up ==
For some reason, to sign up you only need to provide an email (and then specify a password and 2FA in the “profile”).

You will also need access to a card which can be charged 1€ - we recommend using a virtual card which you can then assign £0 budget or deactivate after signing up so that we can pass to other people later on (and they can’t charge your account). Make sure if they do charge you, you can and will pay for it, asking BCSS for money if you have to.

Once signed up, you have to verify the card (yes you haven’t done that yet). Simply click the button which says “verify card” and find the 4 digit code in your statement and enter that into the website.

== Uploading a container ==
[https://youtu.be/zgxcCcKnXR0?feature=shared Scaleway Tutorial]

As mentioned in the video, to get a container running, you must upload it to the container registry.

To do this, you will first have to create a namespace (if you have not created one already there should be a bit “create a namespace” button when you click on the container registry option in the sidebar).
{{Note|text=The free tier requires the images to be public and so set the namespace to be public as well which will reduce the likelihood of accidental costs. Remember there is also a 75GB limit.}}
You can either tag one of your previously built containers as they mention:<syntaxhighlight lang="shell">
podman tag ubuntu:latest rg.fr-par.scw.cloud/$namespace/ubuntu:latest
</syntaxhighlight>or you can build the dockerfile in the current directory with the tag<syntaxhighlight lang="shell">
podman build -t rg.fr-par.scw.cloud/$namespace/ubuntu:latest .
</syntaxhighlight>And then you can push it with:<syntaxhighlight lang="shell">
podman push rg.fr-par.scw.cloud/$namespace/ubuntu:latest
</syntaxhighlight>Now reload the page and you will be able to see your container there

== Hosting a container ==
You can basically just click the 3 dots for the dropdown and click “deploy”.

* You want to choose the namespace (scroll up)
* Set the port to the one your container is listening to (the one you exposed)
* Set the name + description
* For resources don’t go overboard as we want to basically set it to the lowest it will be able to run at but it preferably it should run fast.
* Scaling should be changed to between 0 and 1 instances.
* You will probably have environment variables and secrets to set, which is under “Advanced Options”.
* Under “Advanced Options” you will want to change the sandboxing to v2 as v1 is extremely slow at starting up instances.
* You can then ignore the estimated cost (if you’ve calculated your total VCPUs and memory is under on your ENTIRE account)
* Hit deploy!

You will then have to add a custom endpoint. Note you will have to wait until the container has been fully deployed (which takes a while).

Basically you want to add a <code>CNAME</code> record to the DNS and set its value to the given domain (make sure Cloudflare proxy-ing is off). Once that is done, just click add endpoint and type in the hostname with the record and it’ll generate a certificate for it.

== Updating a container ==
Okay so I’m not 100% sure this is the official method. But basically you can push your updated image and then go onto the container (in scaleway) and click on the deploy tab then just click on the button at the bottom which says “deploy” and it should redeploy like that.

== Monitoring ==
Scaleway uses [[Grafana]] for monitoring which you can access once you have enabled Cockpit.

We do love Grafana, if you don’t know how to use it, that’s fine, just follow the links scaleway gives you and marvel at Grafana’s beautiful and slightly slow interface (I feel like that’s because this is the free tier).

BOSS/Hosting

2026-06-04T08:55:59Z

Hw2210: Update links and information due to our recent cluster

Our approach to hosting is that you will be able to host any low-intensity applications (this should encompass anything we are doing) for free. However, just so you do not get scammed, if any service shouts it from the roof top that their stuff is free, it’s probably a scam unless it’s a legitimate big business who can afford to do it.

== Application structure ==

The structure of your application is extremely important to be able to host stuff for free.

You will want to split it up into three parts: frontend, backend and some sort of storage (e.g. database), this allows each part to be hosted separately and at lower costs. If you do not need a certain part, for example storage, you can just not host anything.

=== Frontend ===

{{Note|text=
If you can do it as a single static website, do it as a static website.
}}

For the frontend, this should always just be a static page. This can use any of the [[BOSS/Hosting/Pages|“Pages” products]], e.g. GitHub, GitLab, [[BOSS/Hosting/Cloudflare|Cloudflare Pages]] (which are all free). For BathCS, we use Cloudflare pages as we do not store our code on a GitLab instance that supports pages.

By “static”, we mean that requesting a URL, it will always return the same JavaScript and HTML (this is a bit more of a loose definition than what it actually means as we can get dynamic features with an API).

A consideration you may want to take is to integrate the frontend into the same system as the backend, which usually means you can host it with one docker container which means an easier deployment and allows for (easier) dynamic pages.

However if you are using something like Svelte or React, it can be quite hard to get pages well optimised for loading.

As a side note, it is becoming quite common nowadays to have authenticated areas (which require dynamic pages and access control) on a different website entirely (normally under a subdomain) e.g. <code>scaleway.com</code> and <code>console.scaleway.com</code>. However, this is only recommended for big and complicated projects (which we want to stay away from).

Wordpress, Wix or Squarespace are a big no-no.

=== Backend ===

A backend is usually only needed when you have some sort of storage and mostly acts as a validation layer and stops people from just deleting everything. Please avoid Python and TypeScript/JavaScript or other languages which don’t have runtime validation. This is for your own sanity as trying to implement the level of security which is required is a pain beyond compare.

These should usually be using Docker containers or a serverless architecture (e.g. [[BOSS/Hosting/Cloudflare#Workers|Cloudflare Workers]]).

==== Docker containers ====

You should take a few extra things into consideration when using a Docker container:

* If you are building a Docker container, please make sure it uses a small base image e.g. alpine (even if you have to install dependencies manually) as nearly all container registries will charge you for hosting large Docker images. If its a compiled language even better.
* Due to the usual lack of cache-able information, you want to be hosting as close to the UK as possible which also helps with GDPR stuff (so just don’t host anything in America – which invalidates many free hosting options sadly).

You can host Docker containers for free on [[BOSS/Hosting/Scaleway|Scaleway]] however we host all ours on [[BOSS/Hosting/Cluster|our own kubernetes cluster]].

=== Storage ===

This one is the hardest to host for free, but there are free solutions out there if you just stick with the typical database structure (e.g. PostgreSQL).

We do recommend PostgreSQL as it is the standard business practice nowadays, which we can host on [[BOSS/Hosting/Cluster|our own kubernetes cluster]].

An alternative to this is MongoDB, whose Atlas program allows you to easily host this for free. However, this is discouraged (and outright shamed if used with TypeScript/JavaScript and Python) due to the issues with validating data before it goes into the database (as it literally just accepts objects).

==== Required considerations ====

* Hosting '''MUST''' be in the UK or Europe (due to GDPR)
* Please also consider how much data you need and if you are storing personal, identifiable information as that causes headaches with GDPR.

In general just do a refresher course on GDPR.

== How [[Bath Open Source Society|BOSS]] hosts stuff ==
Following try to follow this structure for all applications just to make management a bit easier and to have backup options. But all backends are hosted on our [[BOSS/Hosting/Cluster|own kubernetes cluster]] found in the university. This kubernetes cluster is hosted within an immutable VM on a server that we manage.

This means that we are able to host (basically) anything we like for free as long as it fits within our resources budget.

BOSS/Hosting/Pages

2026-06-04T08:43:35Z

Hw2210: Translate from boss.bathcs.com

GitLab and GitHub both offer their “Pages” which can host static web pages (ones that can be built into raw HTML + JavaScript) for free.

These also integrates with their build options, meaning as soon as you push to your git repo, it will build the project (if you are using a static site generator)

== GitLab ==

You can click on the “Deploy > Pages” and go through the steps to create a <code>.gitlab-ci.yml</code> file to automatically run <code>deno task build</code> when you push to build your project.

Once this is set up, you can follow the instructions on the interface to add a hostname. This will then display the page on that hostname.

{{Note|text=
Do not enable Cloudflare proxy for this (it won’t work).
}}

With this, you can have multiple repos with different hostnames.

For more information see [https://docs.gitlab.com/ee/user/project/pages/ GitLab’s pages documentation].

=== Bath GitLab ===

Currently does not support pages sadly. We recommend to use [[BOSS/Hosting/Cloudflare#Pages|Cloudflare pages]] instead.

== GitHub ==

For more information see [https://pages.github.com/ GitHub’s pages documentation]

We have not done much of this. But in most cases it requires you to add a CNAME record to DNS for <code>yourusername.github.io</code> and then add a <code>CNAME</code> file to the repo with your hostname.

BOSS/Hosting/Pages

2026-06-04T08:42:31Z

Hw2210: Translate from boss.bathcs.com

GitLab and GitHub both offer their “Pages” which can host static web pages (ones that can be built into raw HTML + JavaScript) for free.

These also integrates with their build options, meaning as soon as you push to your git repo, it will build the project (if you are using a static site generator)

== GitLab ==

You can click on the “Deploy > Pages” and go through the steps to create a <code>.gitlab-ci.yml</code> file to automatically run <code>bun run build</code> when you push to build your project.

Once this is set up, you can follow the instructions on the interface to add a hostname. This will then display the page on that hostname.

{{Note|text=
Do not enable Cloudflare proxy for this (it won’t work).
}}

With this, you can have multiple repos with different hostnames.

For more information see [https://docs.gitlab.com/ee/user/project/pages/ GitLab’s pages documentation]

=== Bath GitLab ===

Currently does not support pages sadly. We recommend to use [[BOSS/Hosting/Cloudflare#Pages|Cloudflare pages]] instead.

== GitHub ==

For more information see [https://pages.github.com/ GitHub’s pages documentation]

We have not done much of this. But in most cases it requires you to add a CNAME record to DNS for <code>yourusername.github.io</code> and then add a <code>CNAME</code> file to the repo with your hostname.

BOSS/Hosting/Cloudflare

2026-06-04T08:40:41Z

Hw2210: Translate from boss.bathcs.com

Cloudflare offers free hosting (with unlimited access) using their “Pages” (for static pages) and “Workers” (for non-static pages).

{{Note|text=
If you are wanting to do this for <code>bathcs.com</code> you need super-admin access to the BathCS account.
}}

== Pages ==

We will be using Cloudflare Pages for all static pages (due to the university not allowing static page hosting on their GitLab instance due to security concerns).

We integrate these into the CI (using secrets to make sure people cannot just access our account).

We will explain the setup required for this to work, the Pages docs can be found on [https://developers.cloudflare.com/pages/ Cloudflare documentation on pages].

=== Setup ===

{{Note|text=
A student with super-admin access to the Cloudflare account will have to perform these steps. This means a BathCS admin is needed.
}}

We will be directly uploading the output directory to Cloudflare, explained on [https://developers.cloudflare.com/pages/get-started/direct-upload/ Cloudflare’s direct upload documentation] and [https://developers.cloudflare.com/pages/how-to/use-direct-upload-with-continuous-integration/ continuous deployment documentation]

<ol style="list-style-type: decimal;">
<li>You could add <code>wrangler</code> as a development package, but there is usually no point and just using <code>deno</code>
<syntaxhighlight lang="sh">deno add npm:wrangler --dev --frozen=false</syntaxhighlight></li>
<li>You will then have to add a project to Cloudflare. This can be done via the CLI or the dashboard (which will ask you to login to Cloudflare):
<syntaxhighlight lang="sh">deno run --no-lock npm:wrangler pages project create</syntaxhighlight>
Please make sure you set the production branch to <code>main</code> – it will default to the current branch, which might be an issue later. See [https://www.geeksforgeeks.org/how-to-change-git-default-branch-from-master/ geeksforgeeks] for how to change this.</li>
<li>You will then have to create a new API token for the BCSS account which is explained by [https://developers.cloudflare.com/pages/how-to/use-direct-upload-with-continuous-integration/#generate-an-api-token Cloudflare API token documentation]. Please make sure that “BCSS” is the only account included (and created with the shared Cloudflare account and not simply by a member of BCSS).

{{Note|text=
Each website/project should have its own API key for security reasons
}}

Please save this API token in the Admin organisation on VaultTub with its respective name (it will not be shown again).</li>
<li>Add the API token to the GitLab CI/CD variables (go to Settings > CI/CD > Variables).
There should be two variables added:
<ul>
<li><code>CLOUDFLARE_ACCOUNT_ID</code>: Set to the BCSS account id</li>
<li><code>CLOUDFLARE_API_TOKEN</code>: Set this to the API token generated</li></ul>

These should be marked as “Protected” and “Masked”
This is sadly less secure than a secrets manager, however we have yet to find a free one, so please limit Maintainer access to the project.</li>
<li>Update the pipeline to a build script
If you want some good example structures, we use [https://gitlab.bath.ac.uk/cs/components GitLab CI components] for standardised checks and deployment, please see the pipeline in [https://gitlab.bath.ac.uk/cs/services/froom/-/blob/dev/.gitlab-ci.yml?ref_type=heads froom] or [https://gitlab.bath.ac.uk/cs/services/wiki/-/blob/main/.gitlab-ci.yml?ref_type=heads our policies wiki] for examples of how to use them. Below is a raw example of how it works behind the hood (note building should be separated into its own pipeline)
<syntaxhighlight lang="yaml">deploy:
stage: deploy
tags:
- shared_runner
before_script:
- deno install
script:
- deno run build
- deno run -A npm:wrangler pages deploy <out-dir> --project-name=<project-name>
rules:
- if: '$CI_COMMIT_BRANCH == "$CI_DEFAULT_BRANCH"
when: always</syntaxhighlight>
The production branch name should be the branch name you chose in part <code>2.</code>, which hopefully should be <code>main</code>.</li>
<li>Add custom domain for Cloudflare pages. Go to the Cloudflare dashboard and find the project in the “Workers & Pages” section. And then select “Custom domains” and click “Set up a custom domain”. If you type a hostname on the domain “bathcs.com”, it should do everything for you 🎉.</li></ol>

== Workers ==

We have yet to implement Workers properly and so there is a summary of [https://developers.cloudflare.com/workers/ Cloudflare workers documentation]

This has the downside that projects normally need to be written mostly from the ground up to work with workers (so it’s great if you are just about to start a new project).

If you are going to write a project for Workers, we recommend checking out their [https://developers.cloudflare.com/workers/languages/rust/ Rust support] (and using that if you can) as JavaScript is still really bad for security.

You may also want to design your project in a way that you can easily host it as a docker container (e.g. using <code>axum</code> and refactoring significantly). This mostly just makes it easy for debugging and is a backup if Cloudflare decides to make Workers cost money.

=== Resources ===

* See [https://developers.cloudflare.com/workers/ Cloudflare documentation]

BOSS/Hosting

2026-06-04T08:31:31Z

Hw2210: Translate from boss.bathcs.com

Our approach to hosting is that you will be able to host any low-intensity applications (this should encompass anything we are doing) for free. However, just so you do not get scammed, if any service shouts it from the roof top that their stuff is free, it’s probably a scam unless it’s a legitimate big business who can afford to do it.

== Application structure ==

The structure of your application is extremely important to be able to host stuff for free.

You will want to split it up into three parts: frontend, backend and some sort of storage (e.g. database), this allows each part to be hosted separately and at lower costs. If you do not need a certain part, for example storage, you can just not host anything.

=== Frontend ===

{{Note|text=
If you can do it as a single static website, do it as a static website.
}}

For the frontend, this should always just be a static page. This can use any of the [[BOSS/Hosting/Pages|“Pages” products]], e.g. GitHub, GitLab, [[BOSS/Hosting/Cloudflare|Cloudflare Pages]] (which are all free). For BathCS, we use Cloudflare pages as we do not store our code on a GitLab instance that supports pages.

By “static”, we mean that requesting a URL, it will always return the same JavaScript and HTML (this is a bit more of a loose definition than what it actually means as we can get dynamic features with an API).

A consideration you may want to take is to integrate the frontend into the same system as the backend, which usually means you can host it with one docker container which means an easier deployment and allows for (easier) dynamic pages.

However if you are using something like Svelte or React, it can be quite hard to get pages well optimised for loading.

As a side note, it is becoming quite common nowadays to have authenticated areas (which require dynamic pages and access control) on a different website entirely (normally under a subdomain) e.g. <code>scaleway.com</code> and <code>console.scaleway.com</code>. However, this is only recommended for big and complicated projects (which we want to stay away from).

Wordpress, Wix or Squarespace are a big no-no.

=== Backend ===

A backend is usually only needed when you have some sort of storage and mostly acts as a validation layer and stops people from just deleting everything. Please avoid Python and TypeScript/JavaScript or other languages which don’t have runtime validation. This is for your own sanity as trying to implement the level of security which is required is a pain beyond compare.

These should usually be using Docker containers or a serverless architecture (e.g. [[BOSS/Hosting/Cloudlfare#Workers|Cloudflare Workers]]).

==== Docker containers ====

You should take a few extra things into consideration when using a Docker container:

* If you are building a Docker container, please make sure it uses a small base image e.g. alpine (even if you have to install dependencies manually) as nearly all container registries will charge you for hosting large Docker images. If its a compiled language even better.
* Due to the usual lack of cache-able information, you want to be hosting as close to the UK as possible which also helps with GDPR stuff (so just don’t host anything in America – which invalidates many free hosting sadly).

You can host Docker containers for free on [[BOSS/Hosting/Scaleway|Scaleway]] (recommended) or [[BOSS/Hosting/Google Cloud|Google Cloud]]

=== Storage ===

This one is the hardest to host for free, but there are free solutions out there if you just stick with the typical database structure (e.g. PostgreSQL).

We do recommend PostgreSQL as it is the standard business practice nowadays. However free hosting is much harder to find.

An alternative to this is [[BOSS/Hosting/MongoDB Atlas|MongoDB]], whose Atlas program allows you to easily host this for free. However, this is discouraged (and outright shamed if used with TypeScript/JavaScript and Python) due to the issues with validating data before it goes into the database (as it literally just accepts objects).

==== Required considerations ====

* Hosting '''MUST''' be in the UK or Europe (due to GDPR)
* Please also consider how much data you need and if you are storing personal, identifiable information as that causes headaches with GDPR.

In general just do a refresher course on GDPR.

BOSS/Languages/TypeScript

2026-06-04T08:28:29Z

Hw2210: Translate from boss.bathcs.com

Again, please in most cases use TypeScript over JavaScript.

== Overview ==

This should mostly be done by [https://biomejs.dev/ biome] or [https://prettier.io/ prettier] and [https://eslint.org/ eslint], but so you know

<ul>
<li>Try to prioritise readability of code</li>
<li>2 spaces as tabs</li>
<li>Lines should not be longer than 80 characters (install extension)</li>
<li>Trailing commas when the bracket on a new line</li>
<li>NO trailing whitespaces! (Install an extension to remove it for you)</li>
<li>Space after comments e.g. <code>// something</code></li>
<li>Order imports alphabetically</li>
<li>Prefer documentation over comments (see below on how you do that)</li>
<li>Comments should only be written when necessary, mainly to describe why something is done the way it is or to example a really unreadable bit of code (however this normally means you should change the code).</li>
<li>Methods should be ordered alphabetically (with private methods ordered separately and at the bottom of the file)
<syntaxhighlight lang="js">export function aFunc() {}
export function bFunc() {}
export function zFunc() {}
function aPrivateFunc() {}
function bPrivateFunc() {}</syntaxhighlight></li></ul>

These are really strict, if you take anything away remember this:

* Use a formatter and linter: styling should be consistent throughout the project
* Readability of code takes precedent

=== Use a Linter ===

Linters usually will catch most issues, these can be installed as extensions to your preferred editor. E.g. for TypeScript we will be using [https://biomejs.dev/ biome] or [https://eslint.org/ ESLint].

For Biome, both formatting and linting can be configured with <code>biome.json</code>:<syntaxhighlight lang="json">
{
"$schema": "https://biomejs.dev/schemas/2.4.15/schema.json",
"vcs": {
"enabled": true,
"clientKind": "git",
"useIgnoreFile": true
},
"files": {
"includes": ["**", "!!**/dist"]
},
"formatter": {
"enabled": true,
"indentWidth": 2,
"indentStyle": "space"
},
"linter": {
"enabled": true,
"rules": {
"recommended": true
}
},
"javascript": {
"formatter": {
"quoteStyle": "double"
}
},
"assist": {
"enabled": true,
"actions": {
"source": {
"organizeImports": "on"
}
}
},
"css": {
"parser": {
"tailwindDirectives": true
}
},
"overrides": [
{
"includes": ["**/*.svelte", "**/*.astro", "**/*.vue"],
"linter": {
"rules": {
"style": {
"useConst": "off",
"useImportType": "off"
},
"correctness": {
"noUnusedVariables": "off",
"noUnusedImports": "off"
}
}
}
}
]
}
</syntaxhighlight>Example ESLint config (<code>.estlintrc.cjs</code>) which supports, react and typescript:

{{Note|text=
You will want to customise this to your projects needs. But this is a good base
}}

<syntaxhighlight lang="js">import js from "@eslint/js";
import prettier from "eslint-plugin-prettier";
import tsParser from "@typescript-eslint/parser";
import eslintJs from "@eslint/js";
import eslintReact from "@eslint-react/eslint-plugin";
import globals from "globals";
import ts from "@typescript-eslint/eslint-plugin";

export default [
{
files: ["**/*.{ts,tsx,mjs,js,jsx}", "astro.config.mjs"],
languageOptions: {
globals: { ...globals.browser },
sourceType: "module",
ecmaVersion: 2022,
parser: tsParser,
parserOptions: {
project: ["./tsconfig.json"],
ecmaVersion: "latest",
sourceType: "module",
ecmaFeatures: {
jsx: true, // Enable JSX syntax support
},
},
},
plugins: {
eslintJs: eslintJs.configs.recommended,
eslintReact: eslintReact.configs.recommended,
prettier: prettier,
"@typescript-eslint": ts,
},
settings: {
"mdx/code-blocks": true,
},
rules: {
"@typescript-eslint/triple-slash-reference": "off",
},
},
{
...js.configs.recommended,
...ts.configs.recommendedTypeChecked,
files: ["**/*.ts", "**/*.tsx"],
rules: {
"@typescript-eslint/strict-boolean-expressions": [
2,
{
allowString: false,
allowNumber: false,
},
],
},
},
];</syntaxhighlight>
Node that you must add the following dependencies:

<pre class="txt">@eslint/js
eslint-plugin-prettier
@typescript-eslint/parser
@eslint-react/eslint-plugin
@typescript-eslint/eslint-plugin
</pre>
For formatters, we recommend prettier with the following config:

<syntaxhighlight lang="json">{
"trailingComma": "all",
"tabWidth": 2,
"semi": true,
"singleQuote": false,
"quoteProps": "as-needed",
"jsxSingleQuote": false,
"bracketSpacing": true,
"bracketSameLine": false,
"arrowParens": "always",
"printWidth": 80,
"useTabs": false
}</syntaxhighlight>
(you will need to add <code>prettier</code> as a dependency).

{{Note|text=
It is also worth noting that people have started to mention moving towards <code>biome</code> instead of <code>prettier</code> (even the devs behind prettier), so you may want to use that instead
}}

You can then add the following to your deno.jsonc:

<syntaxhighlight lang="json">{
"tasks": {
// Biome requires running /bin/sh which means it effectively wants all
// permissions
"biome": "deno run -A npm:@biomejs/biome",

// ...
"lint": "deno task biome lint",
"lint:fix": "deno task biome lint --write",
"format": "deno task biome check --write",
"format:check": "deno task biome check",

// With ESLINT + Prettier
"eslint": "deno run -A npm:eslint",
"prettier": "deno run -A npm:prettier",
"prettier:files": "deno task prettier 'src/**/*.{tsx,ts,md,mdx,json}' '*.{json,js,mjs,md}' '.prettierrc'",

"lint": "deno task eslint src --report-unused-disable-directives --max-warnings 0",
"lint:fix": "deno task lint --fix",
"format": "deno task prettier:files --write",
"format:check": "deno task prettier:files --check"
},
}</syntaxhighlight>

== Specifically ==

These are normally the typical styling recommendations for JavaScript:

<ul>
<li><code>camelCase</code> for variables + functions. Preferably, if you had an acronym in the name, all letters should be the same case e.g. <code>myNPC</code> or <code>npcPair</code></li>
<li><code>PascalCase</code> for classes</li>
<li><code>SCREAMING_CASE</code> for global constants</li>
<li>Use <code>const</code> by default. Use <code>let</code> only if you need to reassign to the variable.
Constants in JavaScript + TypeScript are not actually constant, they just can’t be reassigned.</li>
<li>Reduce use of <code>any</code>. Unfortunately TypeScript can be quite dumb at times so this can be used for type conversion.</li>
<li>Prefer <code>"</code> over <code>'</code> for strings, unless you have <code>"</code> inside the string
This one is more personal so not necessary (consistency though!)</li>
<li>Starting a new scope with <code>{</code> should be on the same line, not on a new line, e.g. in javascript:
<syntaxhighlight lang="js">function something(my_arg) {
...
}

// NOT
function something(my_arg)
{
...
}</syntaxhighlight></li>
<li><code>else</code> and <code>else if</code> statements should be on the same line as the scope (ignore the lecturers’ preferences for this).
<syntaxhighlight lang="js">if (myVar === 'Something') {
...
} else if (myOtherVar !== 'THING') {
...
} else {
...
}

// NOT
if (myVar === 'Something') {
...
}
else if (myOtherVar !== 'THING') {
...
}
else {
...
}</syntaxhighlight></li>
<li>Prefer creating your own interface over using the <code>object</code> keyword</li>
<li>ESLint basically covers all my other issues</li></ul>

== Documentation ==

TypeScript and JavaScript have actually good documentation (probably the only good thing about them), so please use it.

Basically please read through [https://gamedevacademy.org/javascript-docstrings-tutorial/ this].

== Package manager ==
As an asside, as already alluded to, [[Bath Open Source Society|BOSS]] has a standard of using [https://deno.com/ deno] for its package manage and runtime due to its significant focus on security. However, this comes with some learning barriers as it unlike your regular npm, yarn or bun and goes further than pnpm.

Our configuration of deno is as follows, which can be found in the <code>deno.jsonc</code> of any project:<syntaxhighlight lang="json">{
"$schema": "https://raw.githubusercontent.com/denoland/deno/refs/tags/v2.8.0/cli/schemas/config-file.v1.json",

"tasks": {
// ... Any "scripts" from package.json
},

"minimumDependencyAge": "P2D",
"vendor": true,
"lock": { "frozen": true }
}</syntaxhighlight>What this configuration does is as follows:

* When updating, the chosen dependency verion must be older than 2 days to help mitigate from supply chain attacks
* By default, the lock file is frozen, and so when adding or updating packages, you must explicity include <code>--frozen=false</code>
* <code>vendor</code> folder is used as a local cache for remote modules, decreasing overall disk usage.

=== Core differences with npm ===

* Added packages must include a namespace, e.g. anything from <code>npm</code> (basically anything) should be prefixed with <code>npm:</code>. You can also install from the [https://jsr.io/ JSR] by prefixing with <code>jsr:</code>
* Scripts are not found in <code>package.json</code> instead they should be in deno's config: <code>deno.jsonc</code> under <code>tasks</code>
* Running "tasks" should be done through deno's task cmd: <code>deno task ...</code>
* Running package scripts, the full package name must be used e.g. <code>deno run -A npm:@biomejs/biome</code>
* You must permit applications to use features such as environmental variables or reading your directories. If deploying yourself, it will ask for each permission. You can also permit anything by using <code>deno run -A</code>

BOSS/Languages/Rust

2026-06-04T08:02:42Z

Hw2210: Translate from boss.bathcs.com

The rust compiler is pretty brutal so not much has to go in here.

* Change the default check command in VSCode (instead of check use <code>clippy</code>)
* 4 spaces for tabs
* Macros are cool but don’t overuse them.

== Use rust analyzer + clippy ==

Rust analyzer is great and has a bunch of recommendations which you should follow, so set this up and use it.

== Use rustfmt ==

This just standardises the code format.

BOSS/Languages/Python

2026-06-04T08:01:46Z

Hw2210: Translate from boss.bathcs.com

Should follow [https://peps.python.org/pep-0008/ PEP-8]

== Quick overview ==

<ul>
<li>4 spaces as tabs</li>
<li><code>snake_case</code> for variables, functions and modules</li>
<li><code>PascalCase</code> for classes</li>
<li>Global variables should not be used.</li>
<li>Global constants should be in <code>SCREAMING_CASE</code> when used.</li>
<li>Lines should not be longer than 80 characters</li>
<li>Use trailing commas when the closing bracket is on a new line</li>
<li>Space after comments e.g. <code># something</code></li>
<li>Protected methods and attributes to start with <code>_</code>, private to start with <code>__</code></li>
<li>When a variable is not used, give it the name <code>_</code> e.g.
<syntaxhighlight lang="python">for _ in range(10):
print("Hi")

(important_var, _) = some_func()</syntaxhighlight></li></ul>

== Non-biased opinions ==

<ul>
<li>Try to prioritise readability of code</li>
<li>Order imports alphabetically but split internal imports</li>
<li>Comments should only be written when necessary, mainly to describe why something is done the way it is or to example a really unreadable bit of code (however this normally means you should change the code).</li>
<li>Methods should be ordered alphabetically (with private methods ordered separately and at the bottom of the file) e.g.
<syntaxhighlight lang="python">def a_func():
...

def b_func():
...

def z_func():
...

def _a_private_func():
...

def _b_private_func():
...</syntaxhighlight></li>
<li>Please use statically-typed code when possible: always for functions and parameters and then only for variables when its hard to tell e.g.
<syntaxhighlight lang="python">MY_CONST: dict[str, set[list[(str|int)]]] = {
"something": set(["hi"], [2], [3]),
}

def my_func(param1: int, param2: list[str]) -> (dict[str, int]|None):
my_variable = "hi"
...

def void_func():
...</syntaxhighlight>
However note that we should try to avoid using the <code>str|int</code> type and keep to one type when possible.</li>
<li>Generators are cool but don’t over use them and format them to be more readable with good naming schemes. e.g.
<syntaxhighlight lang="py">cube = np.from_iter((
(y + x for x in range(10))
for y in range(10)
), ...)</syntaxhighlight></li></ul>

== Using Pylint ==

On most IDEs (including vscode) you should be able to get a pylint extension to point out mistakes and annoy you into writing okay code. Please use this as it should help you with the above styling requirements.

You can also use flake8 (however these are really strict and I don’t like some of them), so here is an example configuration (in the <code>.flake8</code> file):

<pre class="toml">[flake8]
extend-ignore = E272,E221,E227,E201,E202
exclude = .git,__pycache__
max-complexity = 10</pre>
== Documentation ==

There is annoyingly no exact style in python for these… Here is something I kinda like.

Documentation should be for all public modules, methods, classes and functions (however please also try to do them for private versions, but they don’t have to be as indepth).

The style of documentation is kinda whatever you want it to be, so here is a somewhat okay layout. (Any better styles are most welcome as long as we are consistent)

Example:

example_module.py

<syntaxhighlight lang="python">"""
Description
"""

def my_func(param1: int) -> str:
"""Description of return which fits in less than 80 chars"""
...

def my_complex_func(param1: list[str]):
"""
Description

Parameters
----------
param1 (type): Description

Return
------
brief_name (type): Description
"""

class MyClass:
"""
Description

Attributes
----------
...
"""

def my_method():
"""..."""
...
</syntaxhighlight>

BOSS/Languages/Markdown

2026-06-04T08:00:55Z

Hw2210: Translate from boss.bathcs.com

This is not a programming language, but all READMEs should be written in markdown, and this whole wiki is.

<ul>
<li>Tab size to 2 spaces</li>
<li>Use <code>-</code> over <code>*</code></li>
<li>Max line length should be 80</li>
<li>Use blank lines to space things out!
<pre class="md">## A title

Some text

- point1
- point2</pre></li>
<li>Code blocks should always have a filetype
<pre class="md">```py
print("Hi")
```</pre></li>
<li>Anything relating to the point above should be indented and blank space left
<pre class="md">- point1

some other paragraph

- point2</pre></li></ul>

Formatters do exist (e.g. [https://prettier.io/ prettier] or [https://biomejs.dev/ biome]) so use them please!

BOSS/Languages

2026-06-04T07:59:21Z

Hw2210: Minor workind update

As this is for students, we want to allow students to use whatever language they feel most comfortable with, especially due to our security and code quality standards.

However having said that we want a consistent style throughout the projects (which is quite difficult with so many people working on that). So for each language we have used in the past we have written up some code quality guidelines.

If you are going to write in a new language, please follow the general code quality guidelines below.

== Recommendations ==

And now as any pedantic programmer, I will give my recommendations for you to be using.

=== Static Sites ===

For static sites we ask that you don’t use a programming language and instead use a static site generator like [https://gohugo.io/ hugo] and write the pages in markdown.

This is just to help the simplicity of the page and make it much easier to edit for non-technical people, while also meaning we don’t use Wordpress (it’s insecure).

Some static site generators:

* [https://astro.build/ astro]
* [https://gohugo.io/ hugo]
* [https://www.getzola.org/ zola]

=== Other Projects ===

We recommend using a language which suites the purposes of your project. As most projects will be websites, we recommend using a language with a good library for this. These include:

* Rust (using [https://rocket.rs/ rocket])
* Go (using [https://github.com/gorilla/mux gorilla/mux])
* Zig
* Any other sane type safe language

Languages and combos that are not good for this purpose, due to the lack of type safety or insecurity:

* TypeScript (unless its just for the frontend or utilising a package which performs runtime type validation)
* Python
* Perl
* JavaScript (use TypeScript instead for our sanity) - this does not include small scripts for pages.
* Ruby (does not implement some parts of http request spec correctly)
* PHP (due to being run as root in most circumstances, this has significant security vulnerabilities). If you don’t know what this means, look up web shells, almost all of them are PHP so it’s better just not to even have the possibility of being compatible with them.
* Any esoteric languages (I feel like they all probably don’t even support this)
* Java (it sucks, use Kotlin instead at least)
* C/C++ (they are not great for webservers): use Zig or Rust instead please

Please also choose a language which another student may be able to read in 5 years time.

== General Guidelines ==

As we don’t have requirements on what language you should be using, we have some general code quality guidelines.

However for the languages we have projects in we have written more specific code guidelines.

<ul>
<li>All functions should be documented (even if it’s just one sentence explaining what it does) using the official documentation standard for a given language e.g. Docstrings for Python, JSDoc for JavaScript + TypeScript etc.
For more complex functions, more descriptive documentation should be included</li>
<li>Prioritise '''readability''' over thousands of comments</li>
<li>Follow language styles and use a linter. For example, in Python follow [https://peps.python.org/pep-0008/ PEP8 standards] with <code>snake_case</code> variable names, in JavaScript and TypeScript follow [https://eslint.org/ ESLint]/[https://biomejs.dev/ Biome] with <code>camelCase</code>.</li>
<li>Please make use of modules + splitting code up into multiple files (sensibly)</li>
<li>'''NO TRAILING WHITESPACE!''' Install an extension to auto remove it please!</li>
<li>Use spaces over tabs (this ensures alignment is correct between machines)</li>
<li>Trailing commas when array or parameters on new lines for all languages which support it (if the language doesn’t support it please question your choice of language). E.g.
<syntaxhighlight lang="py">def my_func(
param1: int,
param2: list[str],
) -> list[int]:
return [
1,
2,
3,
4,
]</syntaxhighlight></li>
<li>Trailing commas when the bracket on a new line</li></ul>

So basically in summary write documentation and follow the language’s style!

BOSS/Languages

2026-06-04T07:57:50Z

Hw2210: Translate from boss.bathcs.com and update

As this is for students, we want to allow students to use whatever language they feel most comfortable with, especially due to our security and code quality standards.

However having said that we want a consistent style throughout the projects (which is quite difficult with so many people working on that). So for each language we have used in the past we have written up some code quality guidelines.

If you are going to write in a new language, please follow the general code quality guidelines below.

== Recommendations ==

And now as any pedantic programmer, I will give my recommendations for you to be using.

=== Static Sites ===

For static sites we ask that you don’t use a programming language and instead use a static site generator like [https://gohugo.io/ hugo] and write the pages in markdown.

This is just to help the simplicity of the page and make it much easier to edit for non-technical people, while also meaning we don’t use Wordpress (it’s insecure).

Some static site generators:

* [https://astro.build/ astro]
* [https://gohugo.io/ hugo]
* [https://www.getzola.org/ zola]

=== Other Projects ===

We recommend using a language which suites the purposes of your project. As most projects will be websites, we recommend using a language with a good library for this. These include:

* Rust (using [https://rocket.rs/ rocket])
* Go (using [https://github.com/gorilla/mux gorilla/mux])
* Zig

Languages and combos that are not good for this purpose, due to the lack of type safety or insecurity:

* TypeScript (unless its just for the frontend or utilising a package which performs runtime type validation)
* Python
* Perl
* JavaScript (use TypeScript instead for our sanity) - this does not include small scripts for pages.
* Ruby (does not implement some parts of http request spec correctly)
* PHP (due to being run as root in most circumstances, this has significant security vulnerabilities). If you don’t know what this means, look up web shells, almost all of them are PHP so it’s better just not to even have the possibility of being compatible with them.
* Any esoteric languages (I feel like they all probably don’t even support this)
* Java (it sucks, use Kotlin instead at least)
* C/C++ (they are not great for webservers): use Zig or Rust instead please

Please also choose a language which another student may be able to read in 5 years time.

== General Guidelines ==

As we don’t have requirements on what language you should be using, we have some general code quality guidelines.

However for the languages we have projects in we have written more specific code guidelines.

<ul>
<li>All functions should be documented (even if it’s just one sentence explaining what it does) using the official documentation standard for a given language e.g. Docstrings for Python, JSDoc for JavaScript + TypeScript etc.
For more complex functions, more descriptive documentation should be included</li>
<li>Prioritise '''readability''' over thousands of comments</li>
<li>Follow language styles and use a linter. For example, in Python follow [https://peps.python.org/pep-0008/ PEP8 standards] with <code>snake_case</code> variable names, in JavaScript and TypeScript follow [https://eslint.org/ ESLint]/[https://biomejs.dev/ Biome] with <code>camelCase</code>.</li>
<li>Please make use of modules + splitting code up into multiple files (sensibly)</li>
<li>'''NO TRAILING WHITESPACE!''' Install an extension to auto remove it please!</li>
<li>Use spaces over tabs (this ensures alignment is correct between machines)</li>
<li>Trailing commas when array or parameters on new lines for all languages which support it (if the language doesn’t support it please question your choice of language). E.g.
<syntaxhighlight lang="py">def my_func(
param1: int,
param2: list[str],
) -> list[int]:
return [
1,
2,
3,
4,
]</syntaxhighlight></li>
<li>Trailing commas when the bracket on a new line</li></ul>

So basically in summary write documentation and follow the language’s style!

BOSS/Design

2026-06-04T07:49:48Z

Hw2210: Rename hosting page

We don’t want to specify any harsh requirements that would prevent students creating a variety of projects.

However, we do need to give some recommendations for designing projects, so that future students can still understand and maintain the projects.

It is also worth noting that due to the requirement of hosting things for free, you will want to read the [[BOSS/Hosting|hosting page]] as this limits what you can do signficantly.

Please make sure to follow the [[BOSS/Languages|coding guidelines]] as well for any programming.

== KISS ==

Keep it simple stupid!

If it doesn’t require a feature, don’t add it.

If it doesn’t require being written in a fancy programming language, don’t use that.

If it doesn’t require data collection, don’t collect people’s data!

== Existing Systems ==

If a system exists, please try to use it unless its completely impractical.

This is because students don’t have much time and don’t want to be managing a completely new system. However please use open source projects (closed source nearly always comes with a cost, either data collection or money both of which we cannot afford)

Note: I’m mostly referring to libraries. I think its quite fun to write simple projects like a link shortener for yourself.

=== Licensing ===

Please see [https://boss.bathcs.com/policies/licenses/|our license policy] for information on how our projects are licensed.

In general, new projects should be licensed under [https://spdx.org/licenses/LLVM-exception.html <code>Apache-2.0 WITH LLVM-Exception</code>], contributors must agree to the [https://developercertificate.org/ Developers Certificate of Origin (DCO)], and any contributions must not violate the [https://www.bath.ac.uk/publications/university-of-bath-ip-policy/ University of Bath’s Intellectual Property policy].

== Keep it general ==

Always try to keep the structure of the project general, so that it can be repurposed easily. Don’t take it too far though—there’s a balance.

Generalisation normally leads to cleaner code and better maintainability.

== Modules ==

Modules are your friend, different languages implement them differently and have different ideas about how they should be organised (e.g. Go vs Rust).

We don’t really have opinions, as you should use the style which fits the programming language you are using. If you don’t like that style, don’t write a project in that language (for maintainers, I’m sorry but deal with it).

If you try to go against the style of the programming language, you will find more barriers and have to implement hacky workarounds which will lead to unreadable, confusing and buggy code.

== Data storage ==

Due to limitations with the fact we will never get any funding, we must create projects which only relies on database storage (which we can host for free, see the [[BOSS/Projects/Database|database page]]).

== Docker ==

For easy hosting for free, we require projects which need hosting to be dockerised (meaning they have a <code>Dockerfile</code>).

These Docker containers cannot interact with the host system at all.

== Websites ==

If all pages can be written in markdown, then use a static site generator.

If a website is dynamic it should always be designed with a backend and frontend to help with code security, which also helps with a website feeling “modern”.

Any data input should be validated on the backend.

BOSS/Design

2026-06-04T07:48:45Z

Hw2210: Translate from boss.bathcs.com written by hw2210 and pcs47

We don’t want to specify any harsh requirements that would prevent students creating a variety of projects.

However, we do need to give some recommendations for designing projects, so that future students can still understand and maintain the projects.

It is also worth noting that due to the requirement of hosting things for free, you will want to read the [[BOSS/Projects/Hosting|hosting page]] as this limits what you can do signficantly.

Please make sure to follow the [[BOSS/Languages|coding guidelines]] as well for any programming.

== KISS ==

Keep it simple stupid!

If it doesn’t require a feature, don’t add it.

If it doesn’t require being written in a fancy programming language, don’t use that.

If it doesn’t require data collection, don’t collect people’s data!

== Existing Systems ==

If a system exists, please try to use it unless its completely impractical.

This is because students don’t have much time and don’t want to be managing a completely new system. However please use open source projects (closed source nearly always comes with a cost, either data collection or money both of which we cannot afford)

Note: I’m mostly referring to libraries. I think its quite fun to write simple projects like a link shortener for yourself.

=== Licensing ===

Please see [https://boss.bathcs.com/policies/licenses/|our license policy] for information on how our projects are licensed.

In general, new projects should be licensed under [https://spdx.org/licenses/LLVM-exception.html <code>Apache-2.0 WITH LLVM-Exception</code>], contributors must agree to the [https://developercertificate.org/ Developers Certificate of Origin (DCO)], and any contributions must not violate the [https://www.bath.ac.uk/publications/university-of-bath-ip-policy/ University of Bath’s Intellectual Property policy].

== Keep it general ==

Always try to keep the structure of the project general, so that it can be repurposed easily. Don’t take it too far though—there’s a balance.

Generalisation normally leads to cleaner code and better maintainability.

== Modules ==

Modules are your friend, different languages implement them differently and have different ideas about how they should be organised (e.g. Go vs Rust).

We don’t really have opinions, as you should use the style which fits the programming language you are using. If you don’t like that style, don’t write a project in that language (for maintainers, I’m sorry but deal with it).

If you try to go against the style of the programming language, you will find more barriers and have to implement hacky workarounds which will lead to unreadable, confusing and buggy code.

== Data storage ==

Due to limitations with the fact we will never get any funding, we must create projects which only relies on database storage (which we can host for free, see the [[BOSS/Projects/Database|database page]]).

== Docker ==

For easy hosting for free, we require projects which need hosting to be dockerised (meaning they have a <code>Dockerfile</code>).

These Docker containers cannot interact with the host system at all.

== Websites ==

If all pages can be written in markdown, then use a static site generator.

If a website is dynamic it should always be designed with a backend and frontend to help with code security, which also helps with a website feeling “modern”.

Any data input should be validated on the backend.

BOSS/Projects

2026-06-04T07:45:36Z

Hw2210: Translate from boss.bathcs.com

[[BOSS]] want to encourage all students to contribute to our organisation [https://github.bath.ac.uk/cs cs], either by improving our current systems or adding a project to the organisation, which then we can potentially host on the bathcs.com domain.

== Mirroring ==

Due to some projects’ complexity and the fact that original author may want to use their code to gain respect on GitHub or use it for jobs, we allow people to just mirror their project into the organisation.

However we do require the project to be stored in the organisation for maintainability and future reference.

If you are mirroring, please make sure that the current version on the GitHub is the version which is hosted on bathcs.com or is currently used. The project description on Bath GitLab must link to the canonical (original) source code.

=== Licensing ===

Please see [https://boss.bathcs.com/policies/licenses/|our license policy] for information on how our projects are licensed.

In general, new projects should be licensed under [https://spdx.org/licenses/LLVM-exception.html <code>Apache-2.0 WITH LLVM-Exception</code>], contributors must agree to the [https://developercertificate.org/ Developers Certificate of Origin (DCO)], and any contributions must not violate the [https://www.bath.ac.uk/publications/university-of-bath-ip-policy/ University of Bath’s Intellectual Property policy].

== Requirements ==

Due to most projects being tested by chaotic Computer Science students as well as hosted publicly, we have a high set of requirements for projects and things which need to be considered.

More requirements are hidden within the pages in this wiki, so please make sure to read any relevant pages. Hopefully we have made this at least somewhat entertaining to read or at least easy to skim.

=== To consider ===

* Hand over procedure: Is there anything which cannot be handed over by our current procedures, if so how?

=== Functional ===

* Include credits for who wrote it and contact details once you leave the University.
* All systems must comply with our data collection + security policies (to be written)
* If an admin account is required for some parts, please use auth.bathcs.com to validate the user - documentation to be written :P
* GitHub main branch should be protected (except on mirrors?)
* Follow the code guidelines in [[./languages/index.md|our code guidelines page]]
* Follow the design principles in [[./design.md|our design page]]
* Get someone to review your code (this helps with not having any security issues)
* All data input should be validated on the backend (not just the frontend)
* Documentation should be included to setup a development environment as well as production

=== Non-functional ===

* Hosting needs to be free (we will not get any funding)
* Webpages should be light and fast to load (so we support all type of devices)
* A webpage should work on mobile (because students are lazy).

Advertising events

2026-06-04T07:41:18Z

Hw2210: Translate from boss.bathcs.com (written by Penn Mckintosh - pm2022)

{{Note|text=
The information in this page may only be applicable to SU-affiliated departmental societies. Other events can be advertised through subsets of the methods on this page.
}}

== Emails ==

We need to send emails via both the department and the SU, such that everyone gets them. A lot of computer science students show up to our events despite not being members, so it’s important to include everyone. However, we don’t want to send the same email to anyone twice. We send emails to all computer science students via the Directors of Studies/Teaching, and then to all remaining members via the SU website.

== Using the templating system ==

# Download the templating tool
## Install [https://www.php.net/manual/en/install.php PHP 8 (command line setup)] and [https://wiki.python.org/moin/BeginnersGuide/Download Python 3]
## Download the latest .zip file from [https://gitlab.bath.ac.uk/cs/bcss/su-mail/-/releases su-mail releases]
## Extract the downloaded file into a folder
# Create an email
## Copy one of the existing emails from the emails folder and increment the number in the filename
## Edit the content of the email
## Make sure to put <code>rel="noopener noreferer"</code> when using <code><a></code> tags
# Generate the email
## Run <code>python main.py emails/xyz.json</code>
## The email’s HTML should be in your clipboard as well as output in the terminal

=== Via the SU ===

The [https://www.thesubath.com/resources/thesu/SU-Committee-member-web-tools-guide/ committee web tools] document has a detailed explanation of how to send an email via the SU website. It also has some basic debugging steps.

For BCSS, the process is slightly more complicated. We don’t want to send emails to any Computer Science students via the SU, as they will receive the email from the Directors of Studies/Teaching. The SU doesn’t have any way to filter these, so it has to be done manually, following these steps:

<ol style="list-style-type: decimal;">
<li>Download the exclusion tool
<ol style="list-style-type: decimal;">
<li>Download the latest .jar file from [https://gitlab.bath.ac.uk/cs/bcss/su-members/-/releases the su-exclude releases page found here]</li>
<li>Ensure you have Java Runtime 1.8 or higher installed</li></ol></li>
<li>Download the sales report
<ol style="list-style-type: decimal;">
<li>Open [https://www.thesubath.com/organisation/salesreports/6566/ the SU’s sales report page].</li>
<li>Set the '''Start date''' to 01/08/YYYY (where YYYY is set so that 01/08/YYYY is within the last year). Make sure that the '''End date''' is in the future (this should be the default).</li>
<li>Click '''Purchasers report'''.</li>
<li>Scroll down and click the '''Save''' icon:
[[File:events-save.png|Save button to the left of a printer icon]]
</li>
<li>Select '''XML file with report data''':
[[File:events-xml.png|Text in the center showing XML file with report data]]
</li>
<li>Download and '''save''' the XML file in your Downloads folder.</li></ol></li>
<li>Connect to the Intranet
<ol style="list-style-type: decimal;">
<li>Either go to campus and connect to eduroam, or</li>
<li>[https://connect.bath.ac.uk/ Connect to the University VPN]</li></ol></li>
<li>Run the tool
<ol style="list-style-type: decimal;">
<li>Put the downloaded JAR and downloaded XML in the same folder/directory</li>
<li>Open a command prompt/terminal in that folder/directory</li>
<li>Run <code>java -jar suexclude-<VERSION>.jar "Product Purchasers Report (System).xml" --deduplicate --include-product 10000610 --exclude-ou DEP-COMP</code>, adjusting filenames as necessary. This step can take a long time to complete, please be patient! If it hangs, you probably forgot to connect the VPN/connect eduroam on campus. If the VPN is connected or you are on the campus Intranet, then please just wait, it will load eventually!</li>
<li>Copy the output to the clipboard (it should be valid JavaScript, otherwise there was an error).</li></ol></li>
<li>Delete the report file
<ol style="list-style-type: decimal;">
<li>Make sure to delete the Product Purchasers Report from your computer.</li>
<li>Clear the recycle bin!</li></ol></li>
<li>Set recipients
<ol style="list-style-type: decimal;">
<li>Open [https://www.thesubath.com/organisation/messages/6566/ the SU messages page].</li>
<li>Create or open a draft.</li>
<li>Click '''Add recipients'''.</li>
<li>Open DevTools using '''Ctrl+Shift+C'''.</li>
<li>If necessary, press '''Escape''' to show the JavaScript console.</li>
<li>Paste the output of the tool into the JavaScript console prompt.</li>
<li>Close DevTools.</li>
<li>Click '''Add recipients'''.</li></ol></li>
<li>Clear your clipboard
<ol style="list-style-type: decimal;">
<li>Copy some non-confidential text.</li>
<li>If you use a clipboard manager, make sure to clear that too.</li></ol></li>
<li>Write your email
<ol style="list-style-type: decimal;">
<li>Paste the HTML from the [[#using-the-templating-system|templator]] into the Source input.</li>
<li>Click '''Save as draft'''.</li>
<li>Once ready, let another committee member '''double-check''' your email.</li>
<li>Send the same email via the Director of Studies/Teaching and wait for them to forward it.</li>
<li>Once the Director of Studies/Teaching have sent it, disable Google Campaign Tracking then click '''Send''' on the SU page.</li></ol></li></ol>

Debugging:

# If Java throws an error not related to the tool, debug it using your favourite search engine.
# If the only output is <code>var r=null;</code> then you downloaded the wrong report, or set the wrong date range.

=== Via the Director of Studies/Teaching ===

Send an email to Zack Lyons or John Benardis, both are happy to send (relevant) emails to the entire department. Zack is often really busy so John has been more responsive lately. It is worth noting Fabio sends emails targetted at PG-T students.

<ol style="list-style-type: decimal;">
<li>Ensure the email HTML is in your clipboard by running the [[#using-the-templating-system|templator]].</li>
<li>Open [https://outlook.office.com/mail/ Outlook on the Web].</li>
<li>Compose a new email to Zack Lyons. ([mailto:zl221@bath.ac.uk zl221@bath.ac.uk]) or John Benardis ([mailto:ib322@bath.ac.uk ib322@bath.ac.uk]).</li>
<li>Type an introduction just saying hi and asking them to forward the below email to students (specify whether it is for UG, PG-T or PG-R).</li>
<li>Where you want the email to go, type the word “test”.</li>
<li>Hold shift and right-click the word “test”.</li>
<li>Click “inspect”. You should see something like this:
[[File:events-html.png|HTML code, highlighting “div” with class “elementToProof”]]
</li>
<li>Right-click anywhere in the blue highlighted area (text “elementToProof”) and select “Edit as HTML”.</li>
<li>Select the entire textbox and paste (replacing the existing content).</li>
<li>Send the email.</li>
<li>After the email has been forwarded to relevant students, make sure to click send on the SU page!</li></ol>

== Discord ==

== Instagram ==

{{Note|text=This section is a work in progress.}}

* Make sure that any Instagram posts have alt-text and a useful caption. Make sure that all information in the photo is duplicated in the caption.

== WhatsApp ==

File:Events-html.png

2026-06-04T07:38:53Z

Hw2210: HTML code, highlighting “div” with class “elementToProof”

== Summary ==
HTML code, highlighting “div” with class “elementToProof”

File:Events-xml.png

2026-06-04T07:38:37Z

Hw2210: Text in the center showing XML file with report data

== Summary ==
Text in the center showing XML file with report data

File:Events-save.png

2026-06-04T07:38:20Z

Hw2210: Save button to the left of a printer icon

== Summary ==
Save button to the left of a printer icon

StrongSwan VPN on Linux

2026-06-04T07:33:23Z

Hw2210: Translate from boss.bathcs.com (written by Peter Chaplin-Smith - pcs47)

The GlobalProtect VPN has been decommissioned by DDaT. Unfortunately, this was the only easily working VPN on Linux. The new strongSwan VPN ''should'' just work on popular distros like Mint with the instructions given at https://connect.bath.ac.uk/. Unfortunately, for the rest of us, it often does not “just work”.

This page contains troubleshooting steps for if a basic connection doesn’t work.

= Connect instructions =

This section covers the basic connection instructions from https://connect.bath.ac.uk and what to expect for a successful connection. This might help your troubleshooting. If it doesn’t, giving DDaT information about what happens here might be useful.

# Install the correct packages. The name of these packages differs from distro to distro, so you might have to do some searching. These packages (in the Ubuntu <code>apt</code> repositories) are <code>libcharon-extra-plugins</code>, <code>network-manager-strongswan</code>, and possibly <code>network-manager-strongswan-gnome</code> (if you’re using the Gnome desktop environment).
# Using the NetworkManager GUI (might just be called network settings or similar), add a VPN connection with the following:
#* Connection Name: University of Bath VPN
#* Server Name: vpn.bath.ac.uk
#* VPN Type: IKEv2
#* Type of sign-in: Username and Password/EAP-MSCHAPv2 or simply “EAP”
#* Other Options: Request an inner IP address, Enforce UDP encapsulation
# You may need to use the university’s DNS servers (<code>138.38.1.1</code> and <code>138.38.1.2</code>) or a public DNS server like Cloudflare’s <code>1.1.1.1</code>.

The instructions also give the following troubleshooting advice:

<blockquote>You may also need to run the command below if you find DNS lookups fail for some university resources:

<syntaxhighlight lang="sh">sudo ln -sfv /run/systemd/resolve/resolv.conf /etc/resolv.conf</syntaxhighlight>
</blockquote>
Though, we’re not sure this is a great idea, nor are we sure this will work on any distro.

== What to expect ==

Now that you’ve followed the university’s instructions, you should be able to connect using the NetworkManager GUI, it should ask you for a username and password (if you haven’t already supplied the username in the configuration), put your university credentials in (the username should be '''without''' the <code>@bath.ac.uk</code>), and you should get a Microsoft authenticator request via the app.

To confirm you are connected to the university VPN correctly, check you can visit any internal site. For example, go to https://2024-25.moodle-archive.bath.ac.uk/. If the page loads, you’re connected! If not, you’ll need to do some troubleshooting.

= Troubleshooting =

First, wipe the slate clean (delete the connection), update your system, and reboot. Once you’ve done that, try again. This should narrow the chances of your system being in some weird state. If it works, great! If not, it’s time to get troubleshooting.

Some of the following sections will require the use of the command line. If you’re unfamiliar, there are plenty of good guides out there and unfortunately teaching the command line is out of scope for this document. We assume basic command line competence from here on out.

== I can’t add the connection whatsoever ==

First of all, check if <code>NetworkManager</code> has the connection saved, you can list connections with:

<syntaxhighlight lang="sh">nmcli connection</syntaxhighlight>
Or, <code>nmcli</code> has short-hand commands:

<syntaxhighlight lang="sh">nmcli c</syntaxhighlight>
This should list all your connections, look for one with the name you gave it (which would be <code>University of Bath VPN</code> if you followed the instructions in [[#connect-instructions|the first section]]). If it does exist, try connecting with:

<syntaxhighlight lang="sh">nmcli c up "University of Bath VPN" --ask</syntaxhighlight>
You’ll be asked for your password and you should get a Microsoft authenticator request. If you get a successful connection as per [[#what-to-expect|this section]], great! If not and you didn’t get a Microsoft authenticator request, carry on reading this section. If you didn’t get a successful connection but did get a Microsoft authenticator request, check the next section.

First, let’s clean the slate. Check your connections with:

<syntaxhighlight lang="sh">nmcli c</syntaxhighlight>
Then delete the VPN connections by their UUID (UUIDs look something like <code>7abdc95a-4b2a-42bb-9a4a-38104d288737</code>, yours will be different):

<syntaxhighlight lang="sh">nmcli c delete <uuid of the VPN connection></syntaxhighlight>
Now, we’d suggest adding the connection via <code>nmcli</code>, the command line for <code>NetworkManager</code>. To do this, you can use the following command. '''Note that you’ll have replace the placeholder text with your Bath username''':

<syntaxhighlight lang="sh">nmcli connection add \
con-name "University of Bath VPN" \
type vpn \
vpn-type org.freedesktop.NetworkManager.strongswan \
ifname "*" \
ipv4.dns "138.38.1.1,138.38.1.2" \
ipv4.ignore-auto-dns yes \
vpn.data "address=vpn.bath.ac.uk, method=eap, user=<your bath username without the @bath.ac.uk>, encap=yes, virtual=yes"</syntaxhighlight>
This sets up a strongSwan VPN connection using the university’s DNS servers and the <code>vpn.bath.ac.uk</code> VPN server. It should also use your username, set UDP encapsulation enforcement, and should request an inner IP.

You can now attempt to connect again, following the same steps as before. If you get a connection, great! If not, look for the section that best describes your issue.

== I’m not getting a Microsoft authenticator request ==

Before you start this section, make sure to follow the instructions in the previous subsection so you’re in a known state.

When you connect with <code>nmcli c up "University of Bath VPN --ask"</code>, if you get an error such as:

<pre>Error: Connection activation failed: No valid secrets</pre>
You might need a certificate, or you may have entered your password incorrectly. To check the latter, delete the VPN connection and try again.

If you’re not getting a Microsoft authenticator request still, you might need a certificate. The university uses Sectigo certificates for the strongSwan VPN. These ''should'' be bundled in your distro’s trust store, but either <code>charon-nm</code> or <code>NetworkManager</code> might not be checking for them properly and so your connection might be failing. Unfortunately, there is no easy way with a VPN connection to get around this.

The following instructions are a bit messy, but are mostly distro-agnostic and ''should'' resolve the issue ''if it is a certificate problem''.

There are some extra steps you can do first to confirm it is a certificate issue, though different distros package software differently and so this isn’t guaranteed to give results:

<syntaxhighlight lang="sh"># look for anything that suggests a certificate is missing or untrusted
journalctl -u NetworkManager | grep charon

# if the above doesn't show anything, try just looking through NetworkManager's logs
journalctl -u NetworkManager

# if that doesn't immediately show anything, you can try the following
journalctl | grep Sectigo</syntaxhighlight>
Anything that ''does'' come up may be useful to DDaT for troubleshooting if our troubleshooting steps don’t resolve the issue.

Firstly, you’ll have to download the <code>Sectigo Public Server Authentication Root R46</code> certificate. This can be found here: https://crt.sh/?d=4256644734. If you don’t trust us or the certificate has changed, find the correct certificate on this page: https://www.sectigo.com/knowledge-base/detail/Sectigo-Public-Intermediates-and-Roots.

Once you’ve got the certificate, move it somewhere you won’t accidentally delete it. We recommend something like <code>~/.uni-vpn/sectigo-root.crt</code>.

Now, delete your VPN connection (again, just to make sure you’re in a known state) and add the following connection. '''You will need to replace your username AND the path to the Sectigo root certificate''':

<syntaxhighlight lang="sh">nmcli connection add \
con-name "University of Bath VPN" \
type vpn \
vpn-type org.freedesktop.NetworkManager.strongswan \
ifname "*" \
ipv4.dns "138.38.1.1,138.38.1.2" \
ipv4.ignore-auto-dns yes \
vpn.data "address=vpn.bath.ac.uk, method=eap, user=<your bath username without the @bath.ac.uk>, encap=yes, virtual=yes, certificate=/full/path/to/sectigo-root.crt"</syntaxhighlight>
If you used our suggested location, you can replace the certificate path with <code>$HOME/.uni-vpn/sectigo-root.crt</code>.

Attempt to connect as before. If you get a Microsoft authenticator request but don’t get a connection afterwards, go to the next section. If you don’t get a Microsoft authenticator request, [[#file-a-ticket|file a ticket]].

== I get a Microsoft authenticator request, but no connection ==

Firstly, try to connect again, accept the Microsoft authenticator request, and then wait ~40-50 seconds without retrying. If you get a secondary Microsoft authenticator request, this may be due to a routing misconfiguration.

First, check for a rule under <code>210</code> or <code>220</code> with the following command:

<syntaxhighlight lang="sh">ip rule</syntaxhighlight>
If you don’t see a line similar to:

<pre>210: not from all fwmark 0xd2 lookup 210</pre>
You may have a configuration or routing issue. Double check you’ve installed all the correct packages for your distro.

To test if routing is your issue, you can add the following route:

<syntaxhighlight lang="sh">ip route add 138.38.3.176/28 via <default gateway> table 210</syntaxhighlight>
Replacing the default gateway with the IP after “default via” from running <code>ip r</code>. Then, try connecting again. If that doesn’t work, [[#file-a-ticket|file a ticket]]. If that works, '''disconnect and delete the route''' by replacing the <code>add</code> with <code>del</code> in the above command. You can configure <code>NetworkManager</code> to create the route for you when you connect to the VPN, this way it won’t be forever lingering on your system.

To do so, you can run the following:

<syntaxhighlight lang="sh">nmcli c modify "University of Bath VPN" ipv4.route-table 210</syntaxhighlight>
You may need to restart the <code>NetworkManager</code> daemon, '''this will temporarily disconnect you'''. On a <code>systemd</code> based system, this can be done with:

<syntaxhighlight lang="sh">systemctl restart NetworkManager.service</syntaxhighlight>
Now, you should be able to connect to the VPN without issues. If not, [[#file-a-ticket|file a ticket]].

= File a ticket =

If the troubleshooting steps given here don’t work, you should file a ticket with DDaT at: https://topdesk.bath.ac.uk/ (<code>DDaT Support: IT & Audio Visual > Report an Issue > Report a Network Issue</code>).

VaultTub

2026-06-04T07:30:26Z

Hw2210: Update to be more accurate to the current system

VaultTub is our self-hosted vaultwarden/bitwarden instance managed by [[Bath Open Source Society|BOSS]] which can be accessed via [https://vault.bathcs.com vault.bathcs.com]. This allows societies to store and share passwords freely amongst themselves.

During handover, you may be messaged to create an account and join the societies organisation on thep platform.

For docs of how to use this, see [https://bitwarden.com/help/ bitwarden’s documentation].

== Quick setup ==

{{Note|text=
Throughout this page we refer to “Owners of organisations”, from this we mean the delegated people who are the Owners of the Bitwarden organisation. Please see our [[#Quick note on owners|quick note on owners]] for who this should be.
}}

You should be message or told by the previous committee that you need to signup so they can invite you. All bath students can freely make an account (though we ask you not to store your personal passwords on there).

<ol style="list-style-type: decimal;">
<li>Go to [https://vault.bathcs.com vault.bathcs.com]</li>
<li>Enter your uni email and click "Use single sign-on"</li>
<li>Once redirected to our authentication site, enter your university username and password.</li>
<li>You will be redirected back and asked to create a master password{{Note|text=If you lose the password you will not be able to recover it|type=warn}}</li>
<li>You may also receive emails during this process, if the email is in '''junk''', there should be a dropdown at the top of the email with the option to make “vault@bathcs.com” never go to your junk folder. Please select this as you will get a lot more emails after this point.</li>
<li>'''REQUIRED''': [[#Two-step login|Set up 2FA]]</li>
<li>'''OPTIONAL''': [[#Keys|Follow security recommendations]]</li>
<li>'''REQUIRED''': '''for [[#Quick note on owners|Owners of organisations]]''': set up [[#Emergency Access|emergency access]]</li></ol>

Previous committees/initiatives will add you to the relevant organisation as described in the handover procedure.

{{Note|type=warn|text=
Please berate them with emails until they do this as (which has to happen before the summer holidays). This is because if they are uncontactable, all the passwords will be lost forever.
}}

== Recommended Settings ==

Once you have signed up with a password you can remember (but don’t use anywhere else), we recommend updating the following settings:

Settings can be found: “Profile Icon in top right > account settings”.

=== Two-step login ===

{{Note|type=warn|text=
This is required for all committee members, please see our [[2FA|2fa page here]] for a more general overview of what this is.
}}

Assuming you don’t have a security key, we recommend using the “Authenticator App” option (see [[2FA#Individuals|the instructions in 2fa section here]] for a more in-depth explanation of how to setup this).

If you have a security key (e.g. Yubikey), use the FIDO2 WebAuthn option. All other options have not been enabled as that takes time and is not worth it.

=== Keys ===

I recommend upgrading to use Argon2id with settings:

* KDF Iterations: 10
* KDF Memory: 64 (maximum that works with iOS)
* KDF Parrallelism 8

=== Emergency Access ===

{{Note|text=
This is a '''requirement''' for all the [[#Quick note on owners|Owners of organisations]]
|type=warn}}

We recommend setting up emergency access with at least 1 other person, this is for safety if you lose access to your account.

Head to [https://vault.bathcs.com/#/settings/emergency-access the emergency access tab in the settings page] and click “Add emergency contact”, entering the users email (Note they need to have a VaultTub account).

This should be required for users with a significant amount of power due to the risk of losing everything (we cannot recover your passwords).

Note this does require about 3 emails of back and forth accepting with the person (I have been fooled into thinking it was over before when it was not).

=== Using the extension/app ===

Bitwarden has a [https://bitwarden.com/download/#downloads-web-browser browser extension] and an [https://bitwarden.com/download/#downloads-mobile app] which supports self-hosted instances (and multiple accounts).

{{Note|text=
If you wish, you can also set it up so you can approve sign-in requests when signing in somewhere else, which you may prefer over TOTP 2FA.
}}

To install the app or the extension:

* Download the extension, from [https://bitwarden.com/download/#downloads-web-browser Bitwarden’s download page]
* Open it up
** if you already have an account you can click the profile icon and then click “Add account”.
* Under the input for the email address (set to “bitwarden” by default), you can select “self-hosted”.
* Input “https://vault.bathcs.com” for the server field and hit “Save” in the top right.
* Enter your login details for VaultTub.

==== Recommended settings ====

You may wish to change the default lockout period or add a pin, which can be done in the “Settings tab”. These are handled on a '''per account''' basis.

You can click “unlock with pin” and enter a pin (unchecking “unlock with master on browser restart” if you don’t want that).

You can also change the “Vault timeout”, however this is '''not''' recommended.

==== Adding a Login ====
You can easily add a login by clicking the "New" button within the app, and filling out the details, along with the "website". Please make sure the owner is set to the '''society this login should live under''', as well as a collection set.

From this interface you can also generate a password to fill in. The recommneded settings are:

* >= 25 characters

<li>include special characters</li>
<li>min numbers: 2</li>
<li>min special characters: 2</li>
<li>uncheck “avoid ambiguous characters”</li>

== Quick note on owners ==

Make sure you have at least two owners to an organisation. This is so we have backups to make sure we don’t lose access to the data.

For societies we recommend roles equivalent to chair and secretary. For other initiatives, we recommend you choose someone to act as the Owner.

{{Note|text=
For security reasons, please do not grant access to external users unless necessary. Make sure to keep data secure, especially where sensitive or personal data is involved.
}}

Owners also require [[#Emergency Access|emergency access]] to be set up with someone who is not another owner of their organisation.

== Recommended Use ==

This should be used for society passwords.

These society passwords should be stored in relevant organisations. You have the power to create as many organisations you like and share them with other people.

{{Note|text=
When you are no longer a member of a society your account will be disabled (but not deleted unless necessary).
}}

=== Organisations ===

To create an organisation, you can go to your “vaults” and click the “New organization” button on the side panel.

Once created you can go to the “Organizations” tab in the top right and choose the organisation to manage, where you can invite new members (via “Members > Invite Member”) or create a new collection (basically a folder which you can choose who has access to it).

For each member you can choose the role and what collections they have permission to access, the rest is up to you on how you organise everything.

{{Note|text=
When adding members, they will have to accept the invitation and then you will have to confirm them in the organisation interface.
}}

== Handover procedures ==

Please see [https://boss.bathcs.com/handover/permissions/ our handover instructions].

=== Rotating passwords ===

As part of the handover procedures, it is recommended that each initiative rotates all the passwords stored in the organisation.

To do this, please follow this rough instruction list:

<ul>
<li>Open the item on VaultTub (either by the extension or website)</li>
<li>Copy the password and temporarily store it somewhere (e.g. in the notes section)</li>
<li>Visit the website</li>
<li>Navigate to the change password section (it’s different for every website)</li>
<li>On the extension, click “edit” on the item and click the “Generate Password” button and confirm that it will override the current password stored then
Password recommendations:
<ul>
<li>>= 25 characters</li>
<li>include special characters</li>
<li>min numbers: 2</li>
<li>min special characters: 2</li>
<li>uncheck “avoid ambiguous characters”</li></ul></li>
<li>Click “Save” on the item to save the new password</li>
<li>Paste the new password in the change password fields + change the password</li>
<li>Log out and log in again to make sure the new password has saved correctly</li>
<li>Delete the temporary storage of the old password</li>
<li>Add a line to the notes saying it was updated on the current date and include your name</li></ul>

== How to deal with the worst case scenarios ==

If it is relating to the website being down, please contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk]. Note that I do not have any access to any of the data stored as it is all encrypted.

=== I lost my password and don’t have Emergency Access ===

In that case, there is nothing we can do. Your account must be deleted and you will need to be re-invited (this means you WILL lose access to any password you have on the account). Organisations passwords can be recovered by other members of the organisation.

This is why you MUST either know your password by hand or store your password in your own password manager which also has a proper recovery procedure (which normally have to be emergency access contacts).

=== The owner of the organisation is not responding ===

If all [[#Quick note on owners|owners of the organisation]] are not responding or have lost access to their account, this is slightly more of an issue so make sure to have multiple owners.

The organisation will have to be deleted and recreated. To save as many passwords as possible, get all other members to see what Collections they have access to and if they have permission to export the vault (found in the settings for the organisation).

You want to then either export the vault data or copy every single password into a new organisation.

== Create an organisation for your society ==
All societies are welcome to have organisations to share passwords as we believe in good password policies. However, only BOSS committee can create a new organisation for you, and so please submit a request by emailing [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk].

They can do this through the [https://vault.bathcs.com/admin/users/overview admin interface] in the "Organisation" tab (however this interface is normally turned off and so has to be activated through a redeployment).

== Instance Management stuff ==

=== Backups ===

Backups are handled by [[Bath Open Source Society|BOSS]] and is one of the few systems that get completely offsite backup on a third-party service along with the typical whole cluster backups.

But the basic idea:

* VaultTub get's backed up every 4 hours to our backup server.
* The whole cluster get's backed up every day to our backup server
* The specific VaultTub data (including passwords), get's backed up to [https://www.scaleway.com/en/ Scaleway] every day

Contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk] if you have any questions. You can also see [https://gitlab.bath.ac.uk/cs/int/terraform/ our configuration within terraform].

=== Emails ===

Emails are sent through my SMTP server on the same network, which cannot receive emails (because of the firewall) but can send them.

I should mention this has dkim, dmarc and spf setup to help with bypassing spam filters as well as being sent from a university IP (improving deliverability). From testing this works fine however may end up in junk folders from now and again.

VaultTub

2026-06-04T07:02:36Z

Hw2210: Translate from boss.bathcs.com

VaultTub is our self-hosted vaultwarden/bitwarden instance which can be accessed via [https://vault.bathcs.com vault.bathcs.com].

During handover, you should be invited to sign up from vault@bathcs.com (it might be in the junk folder).

For docs of how to use this, see [https://bitwarden.com/help/ bitwarden’s documentation].

== Quick setup ==

{{Note|text=
Throughout this page we refer to “Owners of organisations”, from this we mean the delegated people who are the Owners of the Bitwarden organisation. Please see our [[#Quick note on owners|quick note on owners]] for who this should be.
}}

BathCS admins will invite you to VaultTub once you’ve been elected. Hopefully you will get an email from them directing you here.

<ol style="list-style-type: decimal;">
<li>Once invited, go to the junk folder in Outlook and find the email</li>
<li>If the email is in '''junk''', there should be a dropdown at the top of the email with the option to make “vault@bathcs.com” never go to your junk folder. Please select this as you will get a lot more emails after this point.</li>
<li>Follow the link in the email to sign up</li>
<li>Enter your details and create an account:
<ul>
<li>Use your University of Bath email (as it’s by invite only)</li>
<li>Make sure the password is strong (and '''isn’t''' used anywhere else) and you can remember it. Write it down somewhere safe if you are unsure.</li></ul>

{{Note|type=warn|text=
If you lose this password, you cannot recover it.
}}</li>
<li>'''REQUIRED''': [[#Two-step login|Set up 2FA]]</li>
<li>'''OPTIONAL''': [[#Keys|Follow security recommendations]]</li>
<li>'''REQUIRED''': '''for [[#Quick note on owners|Owners of organisations]]''': set up [[#Emergency Access|emergency access]]</li></ol>

Previous committees/initiatives will add you to the relevant organisation as described in the handover procedure.

{{Note|type=warn|text=
Please berate them with emails until they do this as (which has to happen before the summer holidays). This is because if they are uncontactable, all the passwords will be lost forever.
}}

== Recommended Settings ==

Once you have signed up with a password you can remember (but don’t use anywhere else), we recommend updating the following settings:

Settings can be found: “Profile Icon in top right > account settings”.

=== Two-step login ===

{{Note|type=warn|text=
This is required for all committee members, please see our [[2FA|2fa page here]] for a more general overview of what this is.
}}

Assuming you don’t have a security key, we recommend using the “Authenticator App” option (see [[2FA#Individuals|the instructions in 2fa section here]] for a more in-depth explanation of how to setup this).

If you have a security key (e.g. Yubikey), use the FIDO2 WebAuthn option. All other options have not been enabled as that takes time and is not worth it.

=== Keys ===

I recommend upgrading to use Argon2id with settings:

* KDF Iterations: 10
* KDF Memory: 64 (maximum that works with iOS)
* KDF Parrallelism 8

=== Emergency Access ===

{{Note|type|text=
This is a '''requirement''' for all the [[#Quick note on owners|Owners of organisations]]
}}

We recommend setting up emergency access with at least 1 other person, this is for safety if you lose access to your account.

Head to [https://vault.bathcs.com/#/settings/emergency-access the emergency access tab in the settings page] and click “Add emergency contact”, entering the users email (Note they need to have a VaultTub account).

This should be required for users with a significant amount of power due to the risk of losing everything (we cannot recover your passwords).

Note this does require about 3 emails of back and forth accepting with the person (I have been fooled into thinking it was over before when it was not).

=== Using the extension/app ===

Bitwarden has a [https://bitwarden.com/download/#downloads-web-browser browser extension] and an [https://bitwarden.com/download/#downloads-mobile app] which supports self-hosted instances (and multiple accounts).

{{Note|text=
If you wish, you can also set it up so you can approve sign-in requests when signing in somewhere else, which you may prefer over TOTP 2FA.
}}

To install the app or the extension:

* Download the extension, from [https://bitwarden.com/download/#downloads-web-browser Bitwarden’s download page]
* Open it up
** if you already have an account you can click the profile icon and then click “Add account”.
* Under the input for the email address (set to “bitwarden” by default), you can select “self-hosted”.
* Input “https://vault.bathcs.com” for the server field and hit “Save” in the top right.
* Enter your login details for VaultTub.

==== Recommended settings ====

You may wish to change the default lockout period or add a pin, which can be done in the “Settings tab”. These are handled on a '''per account''' basis.

You can click “unlock with pin” and enter a pin (unchecking “unlock with master on browser restart” if you don’t want that).

You can also change the “Vault timeout”, however this is '''not''' recommended.

== Quick note on owners ==

Make sure you have at least two owners to an organisation. This is so we have backups to make sure we don’t lose access to the data.

For societies we recommend roles equivalent to chair and secretary. For other initiatives, we recommend you choose someone to act as the Owner.

{{Note|text=
For security reasons, please do not grant access to external users unless necessary. Make sure to keep data secure, especially where sensitive or personal data is involved.
}}

Owners also require [[#Emergency Access|emergency access]] to be set up with someone who is not another owner of their organisation.

== Recommended Use ==

This should be used for society passwords.

These society passwords should be stored in relevant organisations. You have the power to create as many organisations you like and share them with other people.

{{Note|text=
When you are no longer a member of a society your account will be disabled (but not deleted unless necessary).
}}

=== Organisations ===

To create an organisation, you can go to your “vaults” and click the “New organization” button on the side panel.

Once created you can go to the “Organizations” tab in the top right and choose the organisation to manage, where you can invite new members (via “Members > Invite Member”) or create a new collection (basically a folder which you can choose who has access to it).

For each member you can choose the role and what collections they have permission to access, the rest is up to you on how you organise everything.

{{Note|text=
When adding members, they will have to accept the invitation and then you will have to confirm them in the organisation interface.
}}

== Handover procedures ==

Please see [../handover/permissions.md]

=== Rotating passwords ===

As part of the handover procedures, it is recommended that each initiative rotates all the passwords stored in the organisation.

To do this, please follow this rough instruction list:

<ul>
<li>Open the item on VaultTub (either by the extension or website)</li>
<li>Copy the password and temporarily store it somewhere (e.g. in the notes section)</li>
<li>Visit the website</li>
<li>Navigate to the change password section (it’s different for every website)</li>
<li>On the extension, click “edit” on the item and click the “Generate Password” button and confirm that it will override the current password stored then
Password recommendations:
<ul>
<li>>= 25 characters</li>
<li>include special characters</li>
<li>min numbers: 2</li>
<li>min special characters: 2</li>
<li>uncheck “avoid ambiguous characters”</li></ul></li>
<li>Click “Save” on the item to save the new password</li>
<li>Paste the new password in the change password fields + change the password</li>
<li>Log out and log in again to make sure the new password has saved correctly</li>
<li>Delete the temporary storage of the old password</li>
<li>Add a line to the notes saying it was updated on the current date and include your name</li></ul>

== How to deal with the worst case scenarios ==

If it is relating to the website being down, please contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk]. Note that I do not have any access to any of the data stored as it is all encrypted.

=== I lost my password and don’t have Emergency Access ===

In that case, there is nothing we can do. Your account must be deleted and you will need to be re-invited (this means you WILL lose access to any password you have on the account). Organisations passwords can be recovered by other members of the organisation.

This is why you MUST either know your password by hand or store your password in your own password manager which also has a proper recovery procedure (which normally have to be emergency access contacts).

=== The owner of the organisation is not responding ===

If all [[#Quick note on owners|owners of the organisation]] are not responding or have lost access to their account, this is slightly more of an issue so make sure to have multiple owners.

The organisation will have to be deleted and recreated. To save as many passwords as possible, get all other members to see what Collections they have access to and if they have permission to export the vault (found in the settings for the organisation).

You want to then either export the vault data or copy every single password into a new organisation.

== Instance Management stuff ==

=== Inviting new users ===

Only administrators (with the interface password) can invite new users.

This can be done through the [https://vault.bathcs.com/admin/users/overview admin interface]. You simply type the users email in “Invite User” section.

=== Backups ===

This is handled by me (hw2210) as I am the maintainer of the NAS it is running on.

But the basic idea:

* Has an on-site iterative backup every day
* Has an off-site iterative backup every day

Contact [mailto:su-boss@bath.ac.uk su-boss@bath.ac.uk] if you have any questions.

=== Emails ===

Emails are sent through my SMTP server on the same network, which cannot receive emails (because of the firewall) but can send them.

I should mention this has dkim, dmarc and spf setup to help with bypassing spam filters. From testing this works fine however may end up in junk folders from now and again.

Passwords

2026-06-04T06:55:38Z

Hw2210: Translate from boss.bathcs.com

Password security is important because, without [[2FA]], they are the only factor stopping an attacker from gaining access to your beloved accounts. This page serves as a basic guide to password security. For setting password policies in applications, follow [https://pages.nist.gov/800-63-4/ NIST SP800-63-4] and/or the [https://www.ncsc.gov.uk/collection/passwords NCSC’s advice].

== TL;DR ==

* Use passkeys instead of passwords where possible.
* If you can’t use a passkey, use a passphrase consisting of '''at least''' three words, separated by some symbol. Add numbers randomly. Capitalise if you want. Optionally, change this every so often so your passwords use different formats.
* Store your passphrase in a password manager. Use a longer passphrase and [[2FA]] to log into your password manager.
* Use [[2FA]] everywhere you possibly can.

'''This can all be done with [[VaultTub]] for societies''':

# Log in to https://vault.bathcs.com/.
# On the sidebar, go to <code>Tools > Generator</code>.
# At the top, press <code>Passphrase</code>.
# Set:
#* The number of words to something greater than or equal to three.
#* The word separator to a random valid symbol of your choice.
#* Optionally, change the above two options every so often so your passwords use different formats.
# Copy the password and use it for whatever account.
# Save the login in VaultTub or your own password manager.

{{Note|text=
VaultTub is ''only'' for society usage. For personal passwords, we recommend [https://bitwarden.com/ Bitwarden] which is very similar (VaultTub uses a Bitwarden-compatible service called VaultWarden).
}}

== Attacks ==

When considering password security, it is important to consider what you are up against. Here is a non-exhaustive list of common attacks:

* '''Phishing''': tricking someone into revealing their password by some means (usually a fake website login from a dodgy email).
* '''Leaked Passwords''': using passwords from data breaches. This is commonly associated with '''credential stuffing''', where an attacker will attempt to reuse leaked credentials from one service on other services.
* '''Password Spraying''': where an attacker tries many common passwords against a large number of accounts on a single service.
* '''Brute Force''': the adversary tries a load of different passwords against a single account on a service. There are a few sub-types:
** '''Simple''': try all possible passwords by incrementally cycling through all combinations.
** '''Dictionary''': use a “wordlist” of common passwords found in breaches. The most well-known is [https://weakpass.com/wordlists/rockyou.txt rockyou], though it is pretty small with only a few million passwords. Some lists are in the billions.
** '''Hybrid''': a ''sort of'' combination between dictionary and simple brute force attacks. Modifies words in a wordlist with common changes people make to passwords, for example if <code>password</code> is in a wordlist, the attack might also check <code>Password123</code>.
** '''Rainbow Table''': an ''offline'' brute force attack, meaning the adversary needs the hash of your password, which they often don’t unless a breach has occurred. Essentially they build a lookup table of common passwords and their hashes for quick cracking of passwords.
* '''Shoulder Surfing''': where an attacker either watches or records you typing in your password.
* '''Insecure Storage''': passwords left in unsecured locations. For example, on a sticky note under a keyboard.
* '''Keyloggers''': if a device has been infected with spyware, a keylogger can see what keys have been pressed and the attacker can see your password.

== Mitigations ==

Considering everything in [[#attacks|the attacks section]], what can we do to protect ourselves?

* '''Phishing''':
** '''Use a password manager'''. A good password manager would not auto-fill on an incorrect domain name.
** ''Ideally'', '''use a passkey''' as they '''will not''' work on an invalid domain.
* '''Leaked Passwords''':
** '''Regularly check for your accounts in breaches'''. Some password managers have functionality built in to check all of your stored accounts against known breaches. Otherwise, there is [https://haveibeenpwned.com/ HIBP]. If an account has been found in a breach, rotate it. Otherwise, it is fine to leave it alone [https://pages.nist.gov/800-63-4/sp800-63b.html#passwordusability (see §8.1.2.1 of SP800-63B)].
** '''Use a password manager'''. With a password manager you don’t need to re-use the same password or variations across many accounts. You use '''one long password''' and [[2FA]] to access your password manager, then you can make all your other passwords completely unique, therefore mitigating credential stuffing.
** '''Use passkeys'''. A passkey will almost certainly be unique<ref>except in the '''extremely unlikely''' event a collision occurs.</ref> to the breached site, therefore mitigating credential stuffing.
* '''Password Spraying''':
** '''Use a password manager'''. With a password manager, you can make your passwords ridiculously long without ever having to memorise them all. Spraying attacks only really target common passwords.
** '''Use passkeys'''. There is no concept of “passkey spraying” as passkeys are all unique<ref>except in the '''extremely unlikely''' event a collision occurs.</ref>.
* '''Brute Force''':
** '''Use a password manager'''. Password managers should have built in password/passphrase generators. The [https://www.ncsc.gov.uk/blog-post/three-random-words-or-thinkrandom-0 NCSC recommends passphrases with 3 or more words]. We would also recommend changing the settings of your passphrase generation every so often. If one of your passwords shows up in a breach and the attacker knows the wordlist you use for your passphrases and the format you use (i.e. word separators, capitalisation, number of words), this might speed up brute force efforts for other accounts.
** '''Use passkeys'''. Brute-forcing a passkey can only really be done with a simple brute force attack which should be impossible with current hardware and cryptography.
* '''Shoulder surfing''':
** '''Use a password manager and 2FA'''. For logging into accounts, your password manager should be able to auto-fill, therefore an adversary can’t see you type your password. The one exception to this is the password to your password manager. If you enable [[2FA]], this is less of an issue because they’d then need to brute force anyways.
** '''Use passkeys'''. Can’t shoulder surf something you don’t type.
** '''Privacy screens'''. While not perfect, these can block shoulder surfers from seeing what you’re doing on your computer or phone, therefore making it a bit harder to grab your credentials if you type them in.
* '''Insecure Storage''':
** '''Use a password manager'''. Password managers are meant to be secure storage for passwords.
** '''Use passkeys'''. Passkeys are pretty much always stored securely somewhere. Whether that is within a secure enclave on your device, on your password manager’s infrastructure, or on a physical security key.
* '''Keyloggers''':
** '''Stay up-to-date'''. Make sure you consistently update your devices and software to help prevent malware.
** '''Anti-malware'''. Use anti-malware software. Yes, [https://www.clamav.net/ Linux has anti-malware software too]. No, [https://madaidans-insecurities.github.io/linux.html using Linux does not mean you are secure].
** '''Sandboxing and mandatory access control'''. You should probably sandbox applications and apply mandatory access control. This will depend on your OS.
** '''Don’t download dodgy stuff'''. Pretty self explanatory really.

Following the common themes: you should [[#use-passkeys|use passkeys]] ''instead of'' passwords where possible. Where it isn’t possible, [[#use-a-password-manager|use a password manager]]. Wherever possible, [[2FA|use 2FA]].

== Use Passkeys ==

Passkeys are a more secure alternative to passwords. You don’t need to remember them as they are created and stored safely by software on your device (or hardware with a security key). Passkeys are also [https://www.microsoft.com/en-us/security/blog/2024/12/12/convincing-a-billion-users-to-love-passkeys-ux-design-insights-from-microsoft-to-boost-adoption-and-security/ a ''lot'' faster to use].

=== How Do They Work? ===

They essentially replace your password with public-key cryptography<ref>this is a very simple explanation that misses out on a lot of detail. For a nicer explanation, see [https://www.youtube.com/watch?v=xYfiOnufBSk this computerphile video]. For a more in-depth explanation, see [https://www.youtube.com/watch?v=V-7zMIgGO1U this conference talk], or [https://www.youtube.com/watch?v=sCybe9FAGOc this goto; conference talk].</ref>. You securely store a private key for each account and give the server a public key. The private key is something only you should have access to, whereas anyone can see your public key. To log in, the server creates a challenge using your public key which can only be solved with the private key. Since you should be the only person with the private key, the server knows it must be you.

In the event of a breach, the public key being leaked doesn’t matter because it is public by design. This is another advantage over passwords<ref>unless a [https://en.wikipedia.org/wiki/Password-authenticated_key_agreement PAKE] is in use.</ref>.

=== Usage ===

To use passkeys, we’d recommend either installing an extension for your password manager (if the password manager supports passkeys), or, if you have money to spend, buy a security key and use that instead (YubiKeys are pretty common).

If your browser is up to date, that’s all you really need to do. Whenever you sign up for a new account, navigate to whatever security settings that service has and add a passkey. Make sure it uses your password manager extension to do so. Unfortunately, adoption has been slow, so many accounts will still require a password. It is therefore important to [[#use-a-password-manager|use a password manager]] effectively.

<div class="tip">
We do not recommend storing passkeys on your device (i.e. in the browser or your phone). If you reset or lose your device, your passkeys will be lost too. Using a service like Bitwarden means your private keys are synced between devices.

With physical security keys, we recommend buying two and making them alike where possible. Keep the backup somewhere safe and your main key attached to something that makes it hard to lose.

</div>

== Use a Password Manager ==

Our general guidance for effective use of a password manager is as follows:

* For the master password, use a long passphrase consisting of five or so words, with special character word separators, and added numbers. This is something you will memorise, so make sure it is memorable, but avoid using personal information (such as birthday, your pets’ names, etc.).
* Enable 2FA. For a 2FA app, use something sensible. We’d recommend you find an open source one such as [https://ente.com/auth/ ente auth]. For more information on 2FA see [[2FA|the 2FA Wiki page]].
* Use the browser extension and app for your password manager. This should give you stuff like auto-fill, password generation, and should make saving new accounts easier.
* If using VaultTub or Bitwarden, follow the recommended settings on [[VaultTub|the VaultTub Wiki page]].
* When generating passphrases:
** Set the number of words to something [https://www.ncsc.gov.uk/blog-post/three-random-words-or-thinkrandom-0 greater than or equal to three]. Optionally, change this every so often.
** Enable adding a number.
** Optionally, cycle through word separators every so often.
** Optionally, cycle between using and not using capitalisation every so often.
** Save them to your password manager immediately after generation. This will save you from making an account, realising you didn’t save the passphrase, and then having to immediately reset your password.
* You don’t need to bother with rotating your master passphrase unless you have suspicion of a breach.

== External Resources ==

* [https://www.ncsc.gov.uk/section/advice-guidance/all-topics/passwords NCSC guidance]
* [https://www.ncsc.gov.uk/collection/passwords NCSC guidance for system owners]
* [https://www.nist.gov/cybersecurity/how-do-i-create-good-password NIST guidance]
* [https://en.wikipedia.org/wiki/Password_strength#Common_recommendations Wikipedia]

-----

<references />

2FA

2026-06-04T06:52:29Z

Hw2210: Translate from boss.bathcs.com

Two-factor authentication is key to security, however if set up incorrectly can lead to worse security or just be unusable. Furthermore, you must know and set up backups for if you lose access to your phone.

For all accounts it is highly recommended to setup 2FA when it is an option, but only in a way where it can be accessed by other people in the society easily.

This document will go over the basics for all of the above.

== TOTP ==

TOTP or One Time Password/Passcode are the most common 2FA you will see as they can be easily setup on any device and are somewhat easy to understand.

The most common form of this are codes which change every 30 seconds.

=== Individuals ===

The issue with TOTP is that normally only 1 device has access to the codes (however you can set multiple devices to this but it is hard to keep up to date).

So for your own accounts you can mostly ignore the worry above and just download an app on your phone which handles this e.g:

Apps which can help you store all your 2FAs:

* Authy
* Google Authenticator
* Microsoft Authenticator
* Or any other that supports TOTP

Note that some companies (cough cough Google and Microsoft) like to push you to download their authenticator app when setting up TOTP, this is not actually required because its an open standard, so please for your own sanity just use one app for this.

=== Student-Led Initiatives ===

For student-led initiatves, we need to make sure that in the future we can have someone else in the initiative to access and use the code.

Therefore we recommend using [[../systems/vaulttub.md|our vaultwarden instance]], and storing it in the same place as the username and password (there is be a field called TOTP).

More information can be found [https://bitwarden.com/help/authenticator-keys/ on Bitwarden Documentation].

=== Adding a code ===

Once you know where you are going to store your TOTP codes, adding one is really quite simple (though it may be hard to find).

Note these instructions are generalised so you may need to use your intuition or guesswork to translate these into practical steps.

* Find the 2FA section on the account settings page
** This may sometimes be under “password” or “security”
* You should then be able to click “Add 2FA” or “Add Authenticator App” or “Add TOTP”.
* If the website says to specifically download an app, ignore and press “continue”.
* Once you see a QR code, you need to open the app which will store the code:
** '''Most apps''': click “Add” (you may need to then click “Other accounts” if they ask you to log in). This should bring up a camera where you can scan the QR code.
** '''Bitwarden''': you will need to open the item with the login information and click edit. Then you can click the camera icon near TOTP (on the mobile version) which will allow you to then scan the QR code.
** '''Input Manually''': If you can’t scan the QR code you, you can go back to the page and copy the code below the QR code (it may be hidden behind a menu which says “can’t scan?” or “input manually”). This code you can copy and paste into the TOTP section of your app.
* Once scanned, the app will show you a code and a count down (make sure you are looking at the correct one if there are multiple listed, it should be under the website name).
* You can then write the code in the input box of the website and hit continue (this checks you have set it up correctly).
* Some website will then show a set of backup codes, see [[#backup-codes|Backup Codes]] for more information, but you want to either download these and store them in a secure location or copy then into a Bitwarden note (separate from the original item).

And that’s it!

=== Using a code ===

Using a code is really simple. When you log in to the website, it will ask for a code after entering the password.

You just need to open up the app and copy and paste the code and hit enter!

=== Backups ===

Backing up 2FA codes is not particularly simple unless you copy and paste the manual code somewhere as it depends on your App.

* Bitwarden: Your codes are not on your device so there’s no need.
* Google Authenticator: You can choose to backup your codes to your Google Drive, or you can export up to 10 accounts as a QR code which you can print and store as a hard copy.
* Authy + Microsoft Authenticator: I have no idea.

== Backup Codes ==

Backup codes are normally given after setting up 2FA as a fallback if you lose your 2FA device (so these should not be stored on the same device).

Normally you should download and store in a folder on your computer (for added security encrypt the folder), or you can print them off and store them in a physical folder somewhere.

For societies it has yet to be decided on the method, but ideally you should print them off and store them in the society locker.

== Passkey ==

Passkeys are relatively new and are basically extensions of the security keys, which allow you to sign in without using a password (and are often more secure than a password).

Bitwarden has [https://bitwarden.com/passwordless-passkeys/ good documentation here about using bitwarden to store passkeys], which societies should be able to use.

Personally, I would recommend people not to use these except through bitwarden unless you have a security key such as a yubikey, as if you lose or reset your phone, you cannot access your account.

But if you have a Yubikey, use it everywhere, they are the most awesome thing ever created :)

== 2FAs not to use ==

The following 2FA methods are insecure and are often worse than a weak password and so '''should not be used''' at all cost (however some websites are still stupid and believe these are okay methods):

* SMS
* Email

SSH keys

2026-06-04T06:50:41Z

Hw2210: Translate from boss.bathcs.com

SSH keys are used for proving that you are who you say you are and are used instead of passwords for SSHing into different machines (and so also used pushing and pulling from repositories).

These work by using a public and private key. You store the private key on your computer and sign messages with this (so basically encrypts it). Then another computer will use the public key to decrypt and so validate that you and only you could have encrypted the message.

== Creating a Key ==

[https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent GitHub also has great documentation on creating a key here]

<ul>
<li>Open a terminal ( [https://apps.microsoft.com/detail/9N0DX20HK701 Windows terminal], Powershell or <code>cmd</code> on Windows)</li>
<li>Run <code>ssh-keygen -t ed25519 -b 4096</code></li>

{{Note|text=
This creates a public and private key pair in your <code>.ssh</code> folder using <code>ed25519</code> encryption and 4096 bit long key. If you wish you can change the encryption type by replacing <code>ed25519</code> with something else e.g. <code>rsa</code> (which is for legacy systems)
}}

<ul>
<li>Accept all the defaults</li>
<li>You may be asked for a password. It is good practice to set one. This encrypts the private key with this password and so you will have to type in the password when trying to SSH into another machine for the first time in your current session. (So remember it or write it in a password manager)</li></ul></li>
<li>This should save a private and public key in <code>.ssh</code> in your home directory. Private key <code>~/.ssh/id_rsa</code> and public key <code>~/.ssh/id_rsa.pub</code>.
On windows we recommend navigating to this directory and check “show hidden files” and “show file extensions” - look it up if its not obvious (which it is not). You should be able to tell the difference between <code>id_rsa</code> and <code>id_rsa.pub</code>. Windows will think the <code>.pub</code> is a publisher file (it is not) and so open it with notepad.</li></ul>

You can then use this public key in locations such as “SSH keys” in GitLab and GitHub for security.