Using Kata Containers

Experimental: Kata Containers support is in early development. Expect rough edges.

Kata Containers runs each pod inside a lightweight QEMU VM, providing stronger isolation than runc. When used with k3k, this means each virtual cluster's server and agent pods run as hardware-virtualized guests on the host node.

This guide covers the full setup: installing Kata, configuring the QEMU runtime, optional erofs snapshotter, and deploying a cluster.

Prerequisites

1. Requirements

KVM available on the host node (/dev/kvm accessible)
vhost_net and vhost_vsock kernel modules available
K3s or RKE2 installed on the host

Ensure the required kernel modules are availible:

sudo modprobe vhost_net
sudo modprobe vhost_vsock

The shim should handle the loading of these, but to be explicit, add them to /etc/modules-load.d/kata.conf.

The use of the erofs snapshotter is optional but recommended for performance. To confirm the host meets the requirements, you may need the erofs-utils package.

sudo modprobe erofs
mkfs.erofs --version

Helm Installation (Recommended)

Using Helm is the quickest and simplest way to install Kata Containers. It automates the distribution of Kata binaries, configures containerd, and registers the necessary RuntimeClasses across your cluster. It has been tested with RKE2 1.35 and Kata 4.0.0.

Ensure you are using a v3 containerd config. This will be the default on 1.35 if no template is provided, or you can create a blank v3 template at /var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.tmpl:

{{ template "base" . }}

Install the kata-deploy Helm chart:

helm upgrade --install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version 4.0.0 \
  --namespace kata-deploy \
  --create-namespace \
  --values=kata-values.yaml

Contents of kata-values.yaml:

k8sDistribution: "rke2"

containerd:
  userDropIn: |
    version = 3

    [plugins.'io.containerd.snapshotter.v1.erofs']
      enable_fsverity = false


snapshotter:
  setup: ["erofs"] # omit of not using erofs

shims:
  disableAll: true

  qemu-runtime-rs:
    enabled: true
    supportedArches:
      - amd64
    allowedHypervisorAnnotations: []
    containerd:
      snapshotter: "erofs"  # omit of not using erofs
    dropIn: |
      [hypervisor.qemu]
      disable_block_device_use = false

      [runtime]
      emptydir_mode = "block-plain"

debug: true

defaultShim:
  amd64: qemu-runtime-rs

Create a Cluster

Kata-based clusters require mode: virtual and persistence.type: ephemeral:

Virtual mode is required because Kata needs a full K3s agent with its own container runtime running on the host. Shared mode uses a virtual kubelet that bypasses node-level runtimes entirely.
Ephemeral persistence is required because dynamic persistence creates a PVC that conflicts with how Kata manages block device storage inside the guest VM.

The example below uses an external PostgreSQL datastore via secretMounts to provide persistence.

apiVersion: v1
kind: Namespace
metadata:
  name: test-k3k
---
apiVersion: v1
kind: Secret
metadata:
  name: datastore-config
  namespace: test-k3k
type: Opaque
stringData:
  config.yaml: |
    datastore-endpoint: postgres://username:password@host:5432/dbname
    cluster-init: false
    server: ""
---
apiVersion: k3k.io/v1beta1
kind: Cluster
metadata:
  name: test
  namespace: test-k3k
spec:
  mode: virtual
  runtimeClassName: kata-qemu-runtime-rs
  persistence:
    type: ephemeral
  servers: 3
  secretMounts:
    - name: externaldb-init-config
      secretName: datastore-config
      mountPath: /opt/rancher/k3s/init/config.yaml.d/
      role: server
    - name: externaldb-server-config
      secretName: datastore-config
      mountPath: /opt/rancher/k3s/server/config.yaml.d/

Notes

Currently runtime-rs is not supported when SELinux is enabled.

3.9 KiB Raw Permalink Blame History