We had to bump up kubeadm configuration version (from v1beta3 to v1beta4), and v1beta4 doesn't work with k8s 1.28, which is used for "oldversion" clusters. So we're bumping these to 1.31.
On some versions of pssh, if we use kubectl apply -f- but
nothing is connected to stdin, we get errors. So execute
cat instead of kubectl apply -f- when needed.
Initially we were using Weave because it was super easy to
install (single kubectl apply -f command), but we now have
multiple problems:
- it is not maintained anymore
- its network policy controller seems to have subtle bugs
(I had multiple classes in 2026 where we ran into policies
that didn't work as intended, but worked fine when trying
on another cluster that used Cilium)
- it doesn't support IPV6
We did a test run with Cilium on a few IPv6-only clusters.
It worked fine. Then a test run on dual stack clusters.
Now we're using Cilium on all kubeadm clusters (removing
support for Weave) and we've updated all materials to
either remove or update mentions to Weave.
For now we set to warn+audit on baseline pods,
but don't enforce any restriction yet. This way,
it shouldn't break anything, but will still issue
visible warnings for problematic pods.
vcluster deployment mode needs 2 volumes per cluster (one for the
control plane, one for shpod), so we're switching to a smaller
machine type since that's quickly becoming the limiting factor
rather than CPU/RAM.
(Probably due to K8S version mismatch; vcluster was on 1.33 and the
host cluster was on 1.35. Symptoms: some pods start, all their
containers are ready, the pod shows up as ready, and yet, it's not
considered ready so the deployment says 0/1 and Helm never completes.)
Note that we install a TON of things from GitHub.
Since GitHub isn't available over IPv6, we are using
a custom solution based on cachttps, a caching
proxy to forward requests to GitHub. Our deployment
scripts try to detect a cachttps instance (assuming
it will be available through DNS over cachttps.internal)
and if they find one, they use it. Otherwise they
access GitHub directly - which won't work on IPv6-only
hosts, but will of course work fine on IPv4 and
dual-stack hosts.
The first iteration on Proxmox support relied on a single
template image hosted on shared storage. This new iteration
relies on template images hosted on local storage. It will
detect the template VM to use on each node thanks to its tags.
Note: later, we'll need to expose an easy way to switch
between shared-store and local-store template images.
Multiple small changes to allow deployment in IPv6-only environments.
What we do:
- detect if we are in an IPv6-only environment
- if yes, specify a service CIDR and listening address
(kubeadm will otherwise pick the IPv4 address for the API server)
- switch to Cilium
Also minor changes to pssh and terraform to handle pinging and
connecting to IPv6 addresses.
- detect which EKS version to use
(instead of hard-coding it in the TF config)
- do not issue a CSR on EKS
(because EKS is broken and doesn't support it)
- automatically install a StorageClass on EKS
(because the EBS CSI addon doesn't install one by default)
- put EKS clusters in the default VPC
(instead of creating one VPC per cluster,
since there is a default limit of 5 VPC per region)
- add support to provision VMs on googlecloud
- refactor the way we define the project used by Terraform
(we'll now use the GOOGLE_PROJECT environment variable,
and if it's not set, we'll set it automatically by getting
the default project from the gcloud CLI)
This adds a codeserver action, which installs code-server
and pre-installs a couple of useful extension. It also
installs a systemd user unit in the user account to run it
automatically.
The 'passwords' action has been tweaked so that it also
creates a code-server configuration file to set the password,
so that the same password can be used for SSH access and
for code-server access.
