Kubernetes - Revision history

Hw2210: /* Deployment with Tofu */ Add note about configmaps requiring deployments to manually be restarted

2026-06-10T09:09:56Z

Deployment with Tofu: Add note about configmaps requiring deployments to manually be restarted

← Older revision		Revision as of 09:09, 10 June 2026
Line 239:		Line 239:

	You can technically, temporarily edit the resources (with <code>e</code> in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).		You can technically, temporarily edit the resources (with <code>e</code> in k9s) to fix your mistake and to make it successfully deploy, so you can actually fix the mistake and redeploy (updating all the resources again) within a shorter time than it takes for tofu to timeout. You do want to be careful with timeouts however, as if you are on poor wifi (e.g. a train) or if the cluster is a bit pinned atm, deployments will take longer and the worst thing is if tofu times out but the deployment actually succeeded (though just reapply should update the state without an actualy redeploymenht unless its helm).

			{{Note\|text=If updating config maps/secrets, you will have to manually restart affected pods for the changes to apply in some cases!}}

	=== Retaining PVs ===		=== Retaining PVs ===

Hw2210: Add info about migration, backups and alerting

2026-06-07T13:22:10Z

Add info about migration, backups and alerting

← Older revision		Revision as of 13:22, 7 June 2026
Line 292:		Line 292:

	== Migrating storages ==		== Migrating storages ==
	When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so:		When you are migrating clusters to a new machine (just copy the VM though) or just moving statefulsets between namespaces (persistant volumes are namespace agnostic though), you may need to migrate persistant volumes. As I have done this multiple times now, and so here is my steps to do so.

			{{Note\|text=All pods using these volumes should be inaccessible during the process of this. This can be done by temporarily deleting any <code>HTTPRoute</code>s to the service.}}

	=== Is it a database? ===		=== Is it a database? ===
Line 305:		Line 307:
	killall kubectl # Stop port-forward		killall kubectl # Stop port-forward
	</syntaxhighlight>		</syntaxhighlight>

			=== Anything else ===
			You will need to investigate permissions and such, but kubectl has this [https://kubernetes.io/docs/reference/kubectl/generated/kubectl_cp/ <code>cp</code> command] which you can use to download whole directories. But before that you should run <code>ls -al</code> while ssh-ed into the machine using the volume to view the permissions of the files.<syntaxhighlight lang="sh">
			kubectl cp my_namespace/pod:/where/is/pv/mounted bkp_dir
			# ...
			kubectl --context new_cluster cp bkp_dir my_namespace/pod:/where/is/pv/mounted
			</syntaxhighlight>If the pod cannot run without the storage being set. You have two options: set it with basic data to be overridden, or create a dummy pod with the PV mounted and copy it to there.

			Remember to mark any new PVs that should retain data [[Kubernetes#Retaining PVs\|with correct reclaim policy]].

	== Multinode clusters ==		== Multinode clusters ==
Line 349:		Line 360:

	== Backups ==		== Backups ==
			Within kubernetes, I'd say there are 3 types of backups:

			* Whole OS/VM backups (really easy if host is running ZFS, but large)
			* Resource configuration backup (saves time but if you are using terraform, not particularly useful): [https://velero.io/ velero]
			* Persistant volume backup (small and application specific): built into some storage classes (e.g. [https://longhorn.io longhorn]) or you can use [https://k8up.io/ k8up]

			All of which you may want to use, however because I velero isn't amazing (It fails to back up PVs and the helm chart depends on paid pods, I prefer mixing a whole VM backup with ZFS snapshots and replication, and then do per volume backups with k8up for really important data to offsite cloud S3 bucket.

	== Alerting ==		== Alerting ==
			Alerting is really important, for this I would simply just point to the [https://prometheus-operator.dev/ prometheus operator] or the [https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack kube-prometheus-stack], as these provide really easy methods to automatically monitor the clusters and send emails when something might be going wrong (but it can be extremely noisy) and node exporter requires additional configuration for SELinux enabled systems. This provides a nice standardised system that a lot of helm charts implement (and offer grafana boards you can import).

			When you get an email, you should investigate what the error actual means and what are the recommended steps to fix. E.g. if a job fails, you can either delete it or rerun it until it works again. For CPUThrottlingHigh errors, I still have absolutely no clue what you are supposed to do about it other than increasing the permitted resources for a pod (which isn't recommended).

	== Troubleshooting ==		== Troubleshooting ==

Hw2210: Add multinode cluster info and some stuff on migrating storage

2026-06-07T12:11:55Z

Add multinode cluster info and some stuff on migrating storage

Show changes

Hw2210: Work on writing the deployment section.

2026-06-05T17:37:22Z

Work on writing the deployment section.

Show changes

Hw2210: Add traefik and gateways section

2026-06-04T14:20:33Z

Add traefik and gateways section

@@ Line 72: / Line 72: @@
 === Traefik and gateways ===
 ==== Cloudflare proxy ====
 === Network Policies ===

Hw2210: Add text for general knowledge section

2026-06-04T10:56:45Z

Add text for general knowledge section

@@ Line 3: / Line 3: @@
 === Pod vs Container ===
 === Statefulset vs deployment ===
-=== Liveness probes ===
+==== Security Context ====
-=== Security Context ===
+* Run as user
 === What is a CRD? ===
 === Traefik and gateways ===
@@ Line 87: / Line 146: @@
 ==== On IPV6 cluster and requests randomly timeout or return 404 ====

Hw2210: Add basic titles and structure to fill in later

2026-06-03T14:33:07Z

Add basic titles and structure to fill in later

New page

== General knowledge ==
This tries to cover some basic concepts, focusing on common confusion, but it will skip over a lot of the general knowledge information such as secrets and configmaps. The kubernete's documentation is pretty good, though difficult to read at some points, but there are loads of great tutorials explaining how kubernetes works.

=== Pod vs Container ===

=== Statefulset vs deployment ===

=== Liveness probes ===

=== Security Context ===

=== What is a CRD? ===

=== Traefik and gateways ===

==== Cloudflare proxy ====

=== Network Policies ===

=== Helm ===

=== Persistant Volumes ===

=== Cronjobs ===

=== K3S ===

== Deployment with Tofu ==

=== Retaining PVs ===

=== Where is the state stored? ===

== General Terraform management ==

== K9s ==

== Migrating storages ==

== Multicluster setups ==

=== Control plane ===

=== How to manage Storage ===

=== Safely shutting down nodes ===

== Backups ==

== Troubleshooting ==

=== Kubernetes event log ===

=== Traefik dashboard ===

=== Shell-ing into pods ===

=== K3S Log ===

=== Emergency Debug pods ===

=== Cloudflare API tokens ===

=== SELinux ===

=== Scenarios ===

==== Deployment/statefulset created but pod not creating ====

==== Pod in cash loop ====

==== Pod cannot request any website ====

==== Pod can access the internet but everything returns self-signed certificate ====

==== Deployment can't access its database/valkey ====

==== Pod is requesting denied system privileges ====

==== Pod is up, but I get a 404 when requesting visiting the page ====

==== Pod is up, but I get a 403 when requesting visiting the page ====

==== Authelia refuses to start up ====

==== Node is crashing often after running out of RAM ====

==== On IPV6 cluster and requests randomly timeout or return 404 ====