github/k3k
1
0
Fork 0
mirror of https://github.com/rancher/k3k.git synced 2026-08-18 03:46:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

update kata docs (#1121)

Browse Source
This commit is contained in:
bcraft
2026-08-05 14:29:22 +03:00
committed by GitHub
parent 037eb72bdc
commit 31a316900f
1 changed files with 18 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/howtos/using-kata-containers.md
+18 -205
View File
@@ -4,7 +4,7 @@
Kata Containers runs each pod inside a lightweight QEMU VM, providing stronger isolation than runc. When used with k3k, this means each virtual cluster's server and agent pods run as hardware-virtualized guests on the host node.
This guide covers the full setup: installing Kata, configuring the QEMU runtime and devmapper snapshotter, preparing a custom K3s image, and deploying a cluster.
This guide covers the full setup: installing Kata, configuring the QEMU runtime, optional erofs snapshotter, and deploying a cluster.
# Prerequisites
@@ -12,109 +12,27 @@ This guide covers the full setup: installing Kata, configuring the QEMU runtime
- KVM available on the host node (`/dev/kvm` accessible)
- `vhost_net` and `vhost_vsock` kernel modules available
- `dmsetup` and `losetup` available (device-mapper utilities)
- K3s or RKE2 installed on the host
Load the required kernel modules:
Ensure the required kernel modules are availible:
```bash
sudo modprobe vhost_net
sudo modprobe vhost_vsock
```
To persist these across reboots, add them to `/etc/modules-load.d/kata.conf`.
The shim should handle the loading of these, but to be explicit, add them to `/etc/modules-load.d/kata.conf`.
## 2. Set Up the devmapper Snapshotter
The default snapshotter will use virtio-fs as the container filesystem, the default overlayfs snapshotter does not work with this type, so devmapper is required in order to provide a block device.
The script below creates a thin-pool backed by loopback devices. This is adequate for development and testing. The [device-mapper thin provisioning documentation](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/thin-provisioning.html) covers the full lifecycle, which is more involved than the example here. `erofs` may be a suitable replacement in future to eliminate the need to manage this pool entirely.
The use of the erofs snapshotter is optional but recommended for performance. To confirm the host meets the requirements, you may need the erofs-utils package.
```bash
#!/bin/bash
DIR=/opt/devmapper
CONTAINERD_SOCK="/run/k3s/containerd/containerd.sock"
set -euxo pipefail
name="k3s"
if [ "$EUID" -ne 0 ]; then
echo "Please run as root."
exit 1
fi
create_loopback_device() {
local path=$1
local size=$2
if [[ ! -f "$path" ]]; then
touch "$path"
truncate -s "$size" "$path"
fi
local dev=$(losetup --output NAME --noheadings --associated "$path")
if [[ -z "$dev" ]]; then
dev=$(losetup --find --show $path)
fi
echo $dev
}
pool_create() {
mkdir -p $DIR
local datadev=$(create_loopback_device "$DIR/data" '10G')
local metadev=$(create_loopback_device "$DIR/metadata" '1G')
if dmsetup info "$name" &>/dev/null; then
echo "Thin pool '$name' already exists, ensuring it is active."
dmsetup resume "$name" 2>/dev/null || true
return
fi
local sectorsize=512
local datasize="$(blockdev --getsize64 -q ${datadev})"
local length_sectors=$(bc <<< "${datasize}/${sectorsize}")
local thinp_table="0 ${length_sectors} thin-pool ${metadev} ${datadev} 128 32768 1 skip_block_zeroing"
dmsetup create "$name" --table "${thinp_table}"
}
pool_create
sudo modprobe erofs
mkfs.erofs --version
```
The pool does not survive a reboot. Consider wrapping `pool_create` in a systemd service that runs before `rke2-server.service`.
## 3. Handling emptyDir mounts
The standard K3s image declares `emptyDir` volumes in its image manifest. Kata detects these as tmpfs mounts and fails to start the container. The k3k controller automatically strips `emptyDir` volumes from the pod spec at runtime when the cluster's `runtimeClassName` starts with `kata-`, but the declarations baked into the image manifest will still be in place by default.
We can avoid emptyDir mounts from image manifests by either of the following options:
### Option A: Ignore the image manifest volume declaration (recommended)
Configuring containerd to ignore the image manifest volumes is a simpler option, but note that it is a global setting and will apply to all runtimes on a particular node. **If you use the recommended Helm installation below, this configuration is handled automatically via the `ignore_image_defined_volumes` setting in the `kata-values.yaml`.**
### Option B: Build a custom K3s image
Build a custom image that omits those declarations:
```dockerfile
FROM rancher/k3s:v1.35.3-k3s1 AS rancher
FROM scratch
COPY --from=rancher / /
ENV PATH=/var/lib/rancher/k3s/data/cni:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
ENV CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
```
Push this image to a registry accessible from the host cluster and configure the controller to use it for Kata-based clusters.
# Helm Installation (Recommended)
Using Helm is the quickest and simplest way to install Kata Containers. It automates the distribution of Kata binaries, configures containerd, and registers the necessary RuntimeClasses across your cluster. It has been tested with RKE2 1.35 and Kata 3.32.
Using Helm is the quickest and simplest way to install Kata Containers. It automates the distribution of Kata binaries, configures containerd, and registers the necessary RuntimeClasses across your cluster. It has been tested with RKE2 1.35 and Kata 4.0.0.
Ensure you are using a v3 containerd config. This will be the default on 1.35 if no template is provided, or you can create a blank v3 template at `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.tmpl`:
@@ -126,7 +44,7 @@ Install the `kata-deploy` Helm chart:
```bash
helm upgrade --install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
--version 3.32.0 \
--version 4.0.0 \
--namespace kata-deploy \
--create-namespace \
--values=kata-values.yaml
@@ -141,139 +59,34 @@ containerd:
userDropIn: |
version = 3
[plugins.'io.containerd.cri.v1.runtime']
ignore_image_defined_volumes = true
[plugins.'io.containerd.snapshotter.v1.erofs']
enable_fsverity = false
[plugins.'io.containerd.snapshotter.v1.devmapper']
pool_name = "k3s"
root_path = "/opt/devmapper"
base_image_size = "4GB"
discard_blocks = true
snapshotter:
setup: []
setup: ["erofs"] # omit of not using erofs
shims:
disableAll: true
qemu:
qemu-runtime-rs:
enabled: true
supportedArches:
- amd64
allowedHypervisorAnnotations: []
containerd:
snapshotter: "devmapper"
snapshotter: "erofs" # omit of not using erofs
dropIn: |
[hypervisor.qemu]
disable_block_device_use = false
[runtime]
emptydir_mode = "block-plain"
debug: true
defaultShim:
amd64: qemu
```
# Manual Installation (Advanced)
> **Warning:** Manual installation is highly dependent on your host OS and specific environment. The steps below should be treated as a general guide. Helm is strongly recommended for a simpler and more reliable setup.
## 1. Install Kata Containers
Download and extract the Kata static binary bundle, then symlink the containerd shim:
```bash
curl -sSL https://github.com/kata-containers/kata-containers/releases/download/3.32.0/kata-static-3.32.0-amd64.tar.zst \
| sudo tar --zstd -xvf - -C /
sudo ln -s /opt/kata/runtime-rs/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
```
## 2. Configure the QEMU Runtime
The default Kata QEMU config requires a few changes. The key ones switch the rootfs and block device transports from `virtio-pmem`/`virtio-scsi` to `virtio-blk-pci`, which is necessary because the devmapper snapshotter provides block devices rather than overlay filesystems. `shared_fs` is also disabled since `virtio-fs` is not needed when using block-backed storage.
```bash
sudo sed -i \
-e 's/vm_rootfs_driver = "virtio-pmem"/vm_rootfs_driver = "virtio-blk-pci"/' \
-e 's/shared_fs = "virtio-fs"/shared_fs = "none"/' \
-e 's/block_device_driver = "virtio-scsi"/block_device_driver = "virtio-blk-pci"/' \
-e 's/disable_image_nvdimm = false/disable_image_nvdimm = true/' \
/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-qemu-runtime-rs.toml
```
## 3. Configure containerd
Place the following template at `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.d/custom.toml`. It registers the `kata-qemu` runtime handler and configures the devmapper snapshotter.
```toml
version = 3
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata-qemu]
runtime_type = "io.containerd.kata.v2"
snapshotter = "devmapper"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata-qemu.options]
ConfigPath = "/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-qemu-runtime-rs.toml"
[plugins.'io.containerd.snapshotter.v1.devmapper']
pool_name = "k3s"
root_path = "/opt/devmapper"
base_image_size = "4GB"
discard_blocks = true
```
If you are using Option A from the "Handling emptyDir mounts" section, you must also add the following to `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.d/volumes.toml`:
```toml
version = 3
[plugins.'io.containerd.cri.v1.runtime']
ignore_image_defined_volumes = true
```
### 5.2 Build a custom K3S image
Build a custom image that omits those declarations:
```dockerfile
FROM rancher/k3s:v1.36.2-k3s1 AS rancher
FROM scratch
COPY --from=rancher / /
ENV PATH=/var/lib/rancher/k3s/data/cni:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
ENV CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
```
Push this image to a registry accessible from the host cluster and configure the controller to use it for Kata-based clusters.
An alternative would be an OCI hook to strip the mounts at runtime, but that falls outside the scope of using the Kata project's supplied build artifacts.
## 6. Load Kernel Modules
`vhost_net` and `vhost_vsock` are required for VM networking and host-to-guest communication over vsock:
```bash
modprobe vhost_net
modprobe vhost_vsock
```
To persist these across reboots, add them to `/etc/modules-load.d/kata.conf`.
## 7. Create the RuntimeClass
The `RuntimeClass` name **must start with `kata-`**. This prefix is how the k3k controller identifies Kata-based clusters and applies the necessary pod spec adjustments: stripping `emptyDir` volumes, injecting a `/dev/kmsg` host path mount, and configuring cgroup handling in the K3s startup script.
```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
handler: kata-qemu
metadata:
name: kata-qemu
amd64: qemu-runtime-rs
```
# Create a Cluster
@@ -310,7 +123,7 @@ metadata:
namespace: test-k3k
spec:
mode: virtual
runtimeClassName: kata-qemu
runtimeClassName: kata-qemu-runtime-rs
persistence:
type: ephemeral
servers: 3