github/capsule
1
0
Fork 0
mirror of https://github.com/projectcapsule/capsule.git synced 2026-08-25 16:07:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add dev guide

Browse Source
This commit is contained in:
Bright Zheng
2021-09-30 21:26:31 +02:00
committed by Dario Tranchitella
parent 405d3ac52d
commit 7a8148bd58
2 changed files with 292 additions and 251 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dev-guide.md
+292
View File
@@ -0,0 +1,292 @@
# Capsule Development Guide
## Prerequisites
### Tools
Make sure you have these tools installed:
- [Go 1.16+](https://golang.org/dl/)
- [OperatorSDK 1.7.2+](https://github.com/operator-framework/operator-sdk), or [Kubebuilder](https://github.com/kubernetes-sigs/kubebuilder)
- [KinD](https://github.com/kubernetes-sigs/kind), or [k3d](https://k3d.io/), with kubectl
- [ngrok](https://ngrok.com/) (if you want to run locally with remote Kubernetes)
- [golangci-lint](https://github.com/golangci/golangci-lint)
- OpenSSL
### Kubernetes Cluster
A lightweight Kubernetes within your laptop can be very handy for Kubernetes-native development like Capsule.
#### By `k3d`
```sh
# Install K3d cli by brew in Mac, or your preferred way
$ brew install k3d
# Export your laptop's IP, e.g. retrieving it by: ifconfig
# Do change this IP to yours
$ export LAPTOP_HOST_IP=192.168.10.101
# Spin up a bare minimum cluster
# Refer to here for more options: https://k3d.io/v4.4.8/usage/commands/k3d_cluster_create/
$ k3d cluster create k3s-capsule --servers 1 --agents 1 --no-lb --k3s-server-arg --tls-san=${K8S_API_IP}
# This will create a cluster with 1 server and 1 worker node
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3d-k3s-capsule-server-0 Ready control-plane,master 2m13s v1.21.2+k3s1
k3d-k3s-capsule-agent-0 Ready <none> 2m3s v1.21.2+k3s1
# Or 2 Docker containers if you view it from Docker perspective
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5c26ad840c62 rancher/k3s:v1.21.2-k3s1 "/bin/k3s agent" 53 seconds ago Up 45 seconds k3d-k3s-capsule-agent-0
753998879b28 rancher/k3s:v1.21.2-k3s1 "/bin/k3s server --t…" 53 seconds ago Up 51 seconds 0.0.0.0:49708->6443/tcp k3d-k3s-capsule-server-0
```
#### By `kind`
```sh
# Prepare a kind config file with necessary customization
$ cat > kind.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
apiServerAddress: "0.0.0.0"
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
metadata:
name: config
apiServer:
certSANs:
- localhost
- 127.0.0.1
- kubernetes
- kubernetes.default.svc
- kubernetes.default.svc.cluster.local
- kind
- 0.0.0.0
- ${LAPTOP_HOST_IP}
- role: worker
EOF
# Spin up a bare minimum cluster with 1 master 1 worker node
$ kind create cluster --name kind-capsule --config kind.yaml
# This will create a cluster with 1 server and 1 worker node
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-capsule-control-plane Ready control-plane,master 84s v1.21.1
kind-capsule-worker Ready <none> 56s v1.21.1
# Or 2 Docker containers if you view it from Docker perspective
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7b329fd3a838 kindest/node:v1.21.1 "/usr/local/bin/entr…" About a minute ago Up About a minute 0.0.0.0:54894->6443/tcp kind-capsule-control-plane
7d50f1633555 kindest/node:v1.21.1 "/usr/local/bin/entr…" About a minute ago Up About a minute kind-capsule-worker
```
## Folk & clone the repository
The `folk-clone-contribute-pr` flow is common for contributing to OSS projects like Kubernetes, Capsule.
Let's assume you've folked it into your GitHub namespace, say `myuser`, and then you can clone it with Git protocol.
Do remember to change the `myuser` to yours.
```sh
$ git clone git@github.com:myuser/capsule.git && cd capsule
```
It's a good practice to add the upsteam as the remote too so we can easily fetch and merge the upstream to our folk:
```sh
$ git remote add upstream https://github.com/clastix/capsule.git
$ git remote -vv
origin git@github.com:myuser/capsule.git (fetch)
origin git@github.com:myuser/capsule.git (push)
upstream https://github.com/clastix/capsule.git (fetch)
upstream https://github.com/clastix/capsule.git (push)
```
## Build & deploy Capsule
```sh
# Download the project dependencies
$ go mod download
# Build the Capsule image
$ make docker-build
# Retrieve the built image version
$ export CAPSULE_IMAGE_VESION=`docker images --format '{{.Tag}}' quay.io/clastix/capsule`
# If k3s, load the image into cluster by
$ k3d image import --cluster k3s-capsule capsule quay.io/clastix/capsule:${CAPSULE_IMAGE_VESION}
# If Kind, load the image into cluster by
$ kind load docker-image --name kind-capsule quay.io/clastix/capsule:${CAPSULE_IMAGE_VESION}
# deploy all the required manifests
# Note: 1) please retry if you saw errors; 2) if you want to clean it up first, run: make remove
$ make deploy
# Make sure the controller is running
$ kubectl get pod -n capsule-system
NAME READY STATUS RESTARTS AGE
capsule-controller-manager-5c6b8445cf-566dc 1/1 Running 0 23s
# Check the logs if needed
$ kubectl -n capsule-system logs --all-containers -l control-plane=controller-manager
# You may have a try to deploy a Tenant too to make sure it works end to end
$ kubectl apply -f - <<EOF
apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
name: oil
spec:
owners:
- name: alice
kind: User
EOF
# There shouldn't be any errors and you should see the newly created tenant
$ kubectl get tenants
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
oil Active 0 14s
```
## Scaling down the deployed Pod
As of now, a complete env has been set up in `kind` or `k3d` powred cluster, and the `capsule-controller-manager` is running as a deployment serving as:
- The reconcilers for CRDs and;
- A series of webhooks
During development, we prefer the code is running within our IDE locally, so the `capsule-controller-manager` running in the Kubernetes will competing with us.
To avoid that, we need to scale the existing replicas of `capsule-controller-manager` to 0:
```sh
$ kubectl -n capsule-system scale deployment capsule-controller-manager --replicas=0
deployment.apps/capsule-controller-manager scaled
```
## Preparing TLS certificate for webhooks
Running webhooks requires TLS, so let's prepare the TLS key pair in our development env to handle HTTPS requests.
```sh
# Create this dir to mimic the Pod mount point
mkdir -p /tmp/k8s-webhook-server/serving-certs
# Generate cert/key under /tmp/k8s-webhook-server/
openssl req -newkey rsa:4096 -days 3650 -nodes -x509 \
-subj "/C=SG/ST=SG/L=SG/O=CAPSULE/CN=CAPSULE" \
-extensions SAN \
-config <( cat $( [[ "Darwin" -eq "$(uname -s)" ]] && echo /System/Library/OpenSSL/openssl.cnf || echo /etc/ssl/openssl.cnf ) \
<(printf "[ SAN ]\nsubjectAltName = \"IP:${LAPTOP_HOST_IP}\"")) \
-keyout /tmp/k8s-webhook-server/serving-certs/tls.key \
-out /tmp/k8s-webhook-server/serving-certs/tls.crt
```
Do note that there is an [known issue in controller-runtime, #900](https://github.com/kubernetes-sigs/controller-runtime/issues/900) when you now try to run the code locally:
```sh
$ go run .
```
As you will get errors:
```log
{"level":"error","ts":"2021-09-26T20:42:23.165+0800","logger":"setup","msg":"problem running manager","error":"[open /var/folders/d2/wlg1hjdj29z9vxc2lhndp8m80000gn/T/k8s-webhook-server/serving-certs/tls.crt: no such file or directory, failed waiting for all runnables to end within grace period of 30s: context deadline exceeded]","errorCauses":[{"error":"open /var/folders/d2/wlg1hjdj29z9vxc2lhndp8m80000gn/T/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"},
...
exit status 1
```
And the workaround is to explicitly set `TMPDIR=/tmp/`:
```sh
$ TMPDIR=/tmp/ go run .
```
## Patching the Webhooks
And we need to _delegate_ the controllers' and wehbooks' services to the code running in our IDE.
So we need to patch the `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration` objects with the right pointers.
```sh
# Export your laptop's IP with the 9443 port exposed by controllers/webhooks' services
$ export WEBHOOK_URL="https://${LAPTOP_HOST_IP}:9443"
# Trust the cert we generated for webhook TLS
$ export CA_BUNDLE=`openssl base64 -in /tmp/k8s-webhook-server/serving-certs/tls.crt | tr -d '\n'`
# Patch the MutatingWebhookConfiguration
$ kubectl patch MutatingWebhookConfiguration capsule-mutating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/0/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/mutate-v1-namespace-owner-reference\",'caBundle':\"${CA_BUNDLE}\"}}]"
# We can verify it by command
$ kubectl get MutatingWebhookConfiguration capsule-mutating-webhook-configuration -o yaml
# Patch the ValidatingWebhookConfiguration
# Note: there is a list of validating webhook endpoints, not just one
$ kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/0/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/cordoning\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/1/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/ingresses\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/2/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/namespaces\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/3/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/networkpolicies\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/4/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/pods\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/5/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/persistentvolumeclaims\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/6/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/services\",'caBundle':\"${CA_BUNDLE}\"}}]" && \
kubectl patch ValidatingWebhookConfiguration capsule-validating-webhook-configuration \
--type='json' -p="[{'op': 'replace', 'path': '/webhooks/7/clientConfig', 'value':{'url':\"${WEBHOOK_URL}/tenants\",'caBundle':\"${CA_BUNDLE}\"}}]"
# We can verify it by command
$ kubectl get ValidatingWebhookConfiguration capsule-validating-webhook-configuration -o yaml
```
## Run Capsule
```sh
$ export NAMESPACE=capsule-system
$ make run
```
## Work in your prefered IDE
Now it's time to work through our familiar inner development loop for code contribution.
For example, if you're using [Visual Studio Code](https://code.visualstudio.com), this `launch.json` file can be a good start.
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "Launch",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${workspaceFolder}",
"args": [
"--zap-encoder=console",
"--zap-log-level=debug",
"--configuration-name=capsule-default"
],
"env": {
"NAMESPACE": "capsule-system",
"TMPDIR": "/tmp/"
}
}
]
}
```
docs/operator/contributing.md
-251
View File
@@ -1,251 +0,0 @@
# How to contribute to Capsule
First, thanks for your interest in Capsule, any contribution is welcome!
The first step is to set up your local development environment as stated below:
## Setting up the development environment
The following dependencies are mandatory:
- [Go 1.16](https://golang.org/dl/)
- [OperatorSDK 1.7.2](https://github.com/operator-framework/operator-sdk)
- [Kubebuilder](https://github.com/kubernetes-sigs/kubebuilder)
- [KinD](https://github.com/kubernetes-sigs/kind)
- [ngrok](https://ngrok.com/) (if you want to run locally)
- [golangci-lint](https://github.com/golangci/golangci-lint)
### Installing Go dependencies
After cloning Capsule on any folder, access it and issue the following command
to ensure all dependencies are properly downloaded.
```
go mod download
```
### Installing Operator SDK
Some operations, like the Docker image build process or the code-generation of
the CRDs manifests, as well the deep copy functions, require _Operator SDK_:
the binary has to be installed into your `PATH`.
### Installing Kubebuilder
With the latest release of OperatorSDK there's a more tighten integration with
Kubebuilder and its opinionated testing suite: ensure to download the latest
binaries available from the _Releases_ GitHub page and place them into the
`/usr/local/kubebuilder/bin` folder, ensuring this is also in your `PATH`.
### Installing KinD
Capsule can run on any certified Kubernetes installation and locally
the whole development is performed on _KinD_, also knows as
[Kubernetes in Docker](https://github.com/kubernetes-sigs/kind).
> N.B.: Docker is a hard requirement since it's based on it
According to your operative system and architecture, download the right binary
and place it on your `PATH`.
Once done, you're ready to bootstrap in a glance of seconds, a fully functional
Kubernetes cluster.
```
# kind create cluster --name capsule
Creating cluster "capsule" ...
✓ Ensuring node image (kindest/node:v1.18.2) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-capsule"
You can now use your cluster with:
kubectl cluster-info --context kind-capsule
Thanks for using kind! 😊
```
The current `KUBECONFIG` will be populated with the `cluster-admin`
certificates and the context changed to the just born Kubernetes cluster.
### Build the Docker image and push it to KinD
From the root path, issue the _make_ recipe:
```
# make docker-build
```
The image `quay.io/clastix/capsule:<tag>` will be available locally. Built image `<tag>` is resulting last one available [release](https://github.com/clastix/capsule/releases).
Push it to _KinD_ with the following command:
```
# kind load docker-image --nodes capsule-control-plane --name capsule quay.io/clastix/capsule:<tag>
```
### Deploy the Kubernetes manifests
With the current `kind-capsule` context enabled, deploy all the required
manifests issuing the following command:
```
make deploy
```
This will install all the required Kubernetes resources, automatically.
You can check if Capsule is running tailing the logs:
```
# kubectl -n capsule-system logs --all-containers -f -l control-plane=controller-manager
```
Since Capsule is built using _OperatorSDK_, logging is handled by the zap
module: log verbosity of the Capsule controller can be increased passing
the `--zap-log-level` option with a value from `1` to `10` or the
[basic keywords](https://godoc.org/go.uber.org/zap/zapcore#Level) although
it is suggested to use the `--zap-devel` flag to get also stack traces.
> CA generation
>
> You could notice a restart of the Capsule pod upon installation, that's ok:
> Capsule is generating the CA and populating the Secret containing the TLS
> certificate to handle the webhooks and there's the need the reload the whole
> application to serve properly HTTPS requests.
### Run Capsule locally
Debugging remote applications is always struggling but Operators just need
access to the Kubernetes API Server.
#### Scaling down the remote Pod
First, ensure the Capsule pod is not running scaling down the Deployment.
```
# kubectl -n capsule-system scale deployment capsule-controller-manager --replicas=0
deployment.apps/capsule-controller-manager scaled
```
> This is mandatory since Capsule uses Leader Election
#### Providing TLS certificate for webhooks
The next step is to replicate the same environment Capsule is expecting in the Pod,
it means creating a fake certificate to handle HTTP requests.
``` bash
mkdir -p /tmp/k8s-webhook-server/serving-certs
kubectl -n capsule-system get secret capsule-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/k8s-webhook-server/serving-certs/tls.crt
kubectl -n capsule-system get secret capsule-tls -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/k8s-webhook-server/serving-certs/tls.key
```
> We're using the certificates generate upon the first installation of Capsule:
> it means the Secret will be populated at the first start-up.
> If you plan to run it locally since the beginning, it means you will require
> to provide a self-signed certificate in the said directory.
#### Starting NGROK
In another session, we need a `ngrok` session, mandatory to debug also webhooks
(YMMV).
```
# ngrok http https://localhost:9443
ngrok by @inconshreveable
Session Status online
Account Dario Tranchitella (Plan: Free)
Version 2.3.35
Region United States (us)
Web Interface http://127.0.01:4040
Forwarding http://cdb72b99348c.ngrok.io -> https://localhost:9443
Forwarding https://cdb72b99348c.ngrok.io -> https://localhost:9443
Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00
```
What we need is the _ngrok_ URL (in this case, `https://cdb72b99348c.ngrok.io`)
since we're going to use this default URL as the `url` parameter for the
_Dynamic Admissions Control Webhooks_.
#### Patching the MutatingWebhookConfiguration
Now it's time to patch the _MutatingWebhookConfiguration_ and the
_ValidatingWebhookConfiguration_ too, adding the said `ngrok` URL as base for
each defined webhook, as following:
```diff
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: capsule-mutating-webhook-configuration
webhooks:
- name: owner.namespace.capsule.clastix.io
failurePolicy: Fail
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["namespaces"]
clientConfig:
+ url: https://cdb72b99348c.ngrok.io/mutate-v1-namespace-owner-reference
- caBundle:
- service:
- namespace: system
- name: capsule
- path: /mutate-v1-namespace-owner-reference
...
```
#### Run Capsule
Finally, it's time to run locally Capsule using your preferred IDE (or not):
from the project root path, you can issue the following command.
```
make run
```
All the logs will start to flow in your standard output, feel free to attach
your debugger to set breakpoints as well!
## Code convention
The changes must follow the Pull Request method where a _GitHub Action_ will
check the `golangci-lint`, so ensure your changes respect the coding standard.
### golint
You can easily check them issuing the _Make_ recipe `golint`.
```
# make golint
golangci-lint run -c .golangci.yml
```
> Enabled linters and related options are defined in the [.golanci.yml file](../../.golangci.yml)
### goimports
Also, the Go import statements must be sorted following the best practice:
```
<STANDARD LIBRARY>
<EXTERNAL PACKAGES>
<LOCAL PACKAGES>
```
To help you out you can use the _Make_ recipe `goimports`
```
# make goimports
goimports -w -l -local "github.com/clastix/capsule" .
```
### Commits
All the Pull Requests must refer to an already open issue: this is the first phase to contribute also for informing maintainers about the issue.
Commit's first line should not exceed 50 columns.
A commit description is welcomed to explain more the changes: just ensure
to put a blank line and an arbitrary number of maximum 72 characters long
lines, at most one blank line between them.
Please, split changes into several and documented small commits: this will help us to perform a better review. Commits must follow the Conventional Commits Specification, a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with Semantic Versioning, by describing the features, fixes, and breaking changes made in commit messages. See [Conventional Commits Specification](https://www.conventionalcommits.org) to learn about Conventional Commits.
> In case of errors or need of changes to previous commits,
> fix them squashing to make changes atomic.
### Miscellanea
Please, add a new single line at end of any file as the current coding style.