open-cluster-management/test/README-e2e.md

E2E Testing Guide

This guide explains how to run e2e tests locally and in CI. The e2e test system now supports both macOS and Linux, using docker or podman instead of imagebuilder.

Quick Start

# Run e2e tests (builds images, loads to kind, and runs tests)
IMAGE_TAG=e2e make test-e2e

Prerequisites

Container Runtime

Install either Docker Desktop or Podman:

macOS - Docker Desktop:

• Download from: https://www.docker.com/products/docker-desktop

macOS - Podman:

brew install podman
podman machine init --memory 8192  # 8GB recommended
podman machine start

Linux:

• Docker is typically pre-installed on CI runners
• Or install podman: follow your distribution's instructions

Kind

brew install kind  # macOS
# or
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64  # Linux
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

Kubectl

brew install kubectl  # macOS
# or install via your package manager on Linux

Create Kind Cluster

kind create cluster

Usage

Full E2E Test (Default)

Builds images, loads them into kind, deploys hub and spoke, and runs tests:

IMAGE_TAG=e2e make test-e2e

Time: ~30-40 minutes (first run), ~20-30 minutes (with cached layers)

Steps executed:

1. Build all 5 images with docker/podman
2. Load images into kind cluster
3. Deploy hub (helm install cluster-manager)
4. Deploy spoke (helm install klusterlet)
5. Run e2e test suite

Skip Image Building

When you only modify test code and don't need to rebuild images:

IMAGE_TAG=e2e SKIP_IMAGE_BUILD=true make test-e2e

Time: ~20-30 minutes (skips 10+ minutes of image building)

Steps executed:

1. Build images (skipped)
2. Load images (skipped)
3. Deploy hub
4. Deploy spoke
5. Run e2e tests

Use this when:

• Only test code changed (test/e2e/*.go)
• Images are already built and loaded in kind
• Iterating on test fixes

Clean Up Test Environment

Properly clean up all e2e resources (ClusterManager, helm releases, namespaces):

make clean-e2e-env

Important: This waits for the ClusterManager controller to clean up resources before removing helm releases, avoiding stuck finalizers.

Individual Steps

Build images only:

IMAGE_TAG=e2e make images-kind

Load images only:

IMAGE_TAG=e2e make load-images-kind

Run tests only (assumes images already loaded):

IMAGE_TAG=e2e make run-e2e

Environment Variables

Variable	Description	Default
IMAGE_TAG	Tag for built images	latest
SKIP_IMAGE_BUILD	Set to true to skip building and loading images	false
CONTAINER_CLI	Override auto-detection (docker or podman)	Auto-detected
BUILD_PLATFORM	Override platform (e.g., linux/arm64)	Auto-detected from kind
KLUSTERLET_DEPLOY_MODE	Deployment mode	Default
REGISTRATION_DRIVER	Registration driver	csr

Deploy Modes

# Default mode
IMAGE_TAG=e2e make test-e2e

# Singleton mode
IMAGE_TAG=e2e KLUSTERLET_DEPLOY_MODE=Singleton make test-e2e

# Hosted mode
IMAGE_TAG=e2e KLUSTERLET_DEPLOY_MODE=SingletonHosted make test-e2e

Registration Drivers

# CSR-based authentication (default)
IMAGE_TAG=e2e make test-e2e

# GRPC-based authentication
IMAGE_TAG=e2e REGISTRATION_DRIVER=grpc make test-e2e

How It Works

Auto-Detection

The build system automatically detects:

• Container Runtime: docker or podman (whichever is available)
• Kind Cluster Architecture: queries the kind cluster to determine amd64 or arm64
• Build Platform: matches the kind cluster architecture

Architecture Handling

• macOS Apple Silicon: Builds images with --platform=linux/arm64
• macOS Intel: Builds images with --platform=linux/amd64
• Linux: Builds images with --platform=linux/amd64

Image Loading

• Docker: Uses kind load docker-image
• Podman: Uses podman save | kind load image-archive (since podman doesn't integrate with kind directly)

Images Built

1. registration-operator - Operator managing cluster-manager and klusterlet
2. registration - Registration agent (includes server binary)
3. work - Work agent
4. placement - Placement controller
5. addon-manager - Add-on manager

Troubleshooting

Out of Disk Space (Podman)

# Clean up unused images
podman image prune -a -f

# Check disk usage
podman system df

Podman Out of Memory

# Increase podman machine memory to 8GB
podman machine stop
podman machine set --memory 8192
podman machine start

Wrong Architecture Images

# Check kind cluster architecture
kubectl get node -o jsonpath='{.items[0].status.nodeInfo.architecture}'

# Verify image architecture
podman inspect quay.io/open-cluster-management/registration:e2e | grep Architecture

Stuck ClusterManager (Finalizer Issue)

If cleanup is interrupted, ClusterManager might get stuck. Clean it manually:

# Remove finalizer
kubectl patch clustermanager cluster-manager -p '{"metadata":{"finalizers":[]}}' --type=merge

# Force delete
kubectl delete clustermanager --all --force --grace-period=0

# Delete namespaces
kubectl delete ns open-cluster-management open-cluster-management-agent open-cluster-management-hub --force --grace-period=0

Then run make clean-e2e-env next time to avoid this.

Clean Start

# Clean up test environment properly
make clean-e2e-env

# Or recreate kind cluster
kind delete cluster
kind create cluster

# Run tests
IMAGE_TAG=e2e make test-e2e

Check Configuration

make show-kind-config

CI/CD Integration

The GitHub Actions e2e workflow has been simplified. All e2e jobs now just run:

- name: Test E2E
  run: IMAGE_TAG=e2e make test-e2e

The Makefile handles all image building, loading, deployment, and testing automatically.

Development Workflow

First Run

# Create kind cluster
kind create cluster

# Run tests
IMAGE_TAG=e2e make test-e2e

Iterating on Application Code

# Make changes to cmd/ or pkg/
# ...

# Full rebuild and test
IMAGE_TAG=e2e make test-e2e

Iterating on Test Code

# Make changes to test/e2e/*.go
# ...

# Skip image rebuild (saves ~10 minutes)
IMAGE_TAG=e2e SKIP_IMAGE_BUILD=true make test-e2e

Cleanup Between Runs

# Proper cleanup
make clean-e2e-env

# Run next test
IMAGE_TAG=e2e make test-e2e