mirror of
https://github.com/projectcapsule/capsule.git
synced 2026-05-01 15:06:43 +00:00
338 lines
12 KiB
Markdown
338 lines
12 KiB
Markdown
# Capsule Proxy
|
||
|
||
Capsule Proxy is an add-on for [Capsule](https://github.com/clastix/capsule), the operator providing multi-tenancy in Kubernetes.
|
||
|
||
## The problem
|
||
|
||
Kubernetes RBAC cannot list only the owned cluster-scoped resources since there are no ACL-filtered APIs. For example:
|
||
|
||
```
|
||
$ kubectl get namespaces
|
||
Error from server (Forbidden): namespaces is forbidden:
|
||
User "alice" cannot list resource "namespaces" in API group "" at the cluster scope
|
||
```
|
||
|
||
However, the user can have permissions on some namespaces
|
||
|
||
```
|
||
$ kubectl auth can-i [get|list|watch|delete] ns oil-production
|
||
yes
|
||
```
|
||
|
||
The reason, as the error message reported, is that the RBAC _list_ action is available only at Cluster-Scope and it is not granted to users without appropriate permissions.
|
||
|
||
To overcome this problem, many Kubernetes distributions introduced mirrored custom resources supported by a custom set of ACL-filtered APIs. However, this leads to radically change the user's experience of Kubernetes by introducing hard customizations that make it painful to move from one distribution to another.
|
||
|
||
With **Capsule**, we took a different approach. As one of the key goals, we want to keep the same user's experience on all the distributions of Kubernetes. We want people to use the standard tools they already know and love and it should just work.
|
||
|
||
## How it works
|
||
|
||
This project is an add-on of the main [Capsule](https://github.com/clastix/capsule) operator, so make sure you have a working instance of Caspule before attempting to install it.
|
||
Use the `capsule-proxy` only if you want Tenant Owners to list their own Cluster-Scope resources.
|
||
|
||
The `capsule-proxy` implements a simple reverse proxy that intercepts only specific requests to the APIs server and Capsule does all the magic behind the scenes.
|
||
|
||
Current implementation filters the following requests:
|
||
|
||
* `api/v1/namespaces`
|
||
* `api/v1/nodes`
|
||
* `apis/storage.k8s.io/v1/storageclasses{/name}`
|
||
* `apis/networking.k8s.io/{v1,v1beta1}/ingressclasses{/name}`
|
||
* `api/scheduling.k8s.io/{v1}/priorityclasses{/name}`
|
||
|
||
All other requestes are proxied transparently to the APIs server, so no side-effects are expected. We're planning to add new APIs in the future, so PRs are welcome!
|
||
|
||
## Installation
|
||
|
||
The `capsule-proxy` can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server.
|
||
Optionally, it can be deployed as a sidecar container in the backend of a dashboard.
|
||
Running outside a Kubernetes cluster is also viable, although a valid `KUBECONFIG` file must be provided, using the environment variable `KUBECONFIG` or the default file in `$HOME/.kube/config`.
|
||
|
||
An Helm Chart is available [here](./charts/capsule-proxy/README.md).
|
||
|
||
## Does it work with kubectl?
|
||
|
||
Yes, it works by intercepting all the requests from the `kubectl` client directed to the APIs server. It works with both users who use the TLS certificate authentication and those who use OIDC.
|
||
|
||
## How RBAC is put in place?
|
||
|
||
Each Tenant owner can have their capabilities managed pretty similar to a standard RBAC.
|
||
|
||
```yaml
|
||
apiVersion: capsule.clastix.io/v1beta1
|
||
kind: Tenant
|
||
metadata:
|
||
name: my-tenant
|
||
spec:
|
||
owners:
|
||
- kind: User
|
||
name: alice
|
||
proxySettings:
|
||
- kind: IngressClasses
|
||
operations:
|
||
- List
|
||
```
|
||
|
||
The proxy setting `kind` is an __enum__ accepting the supported resources:
|
||
|
||
- `Nodes`
|
||
- `StorageClasses`
|
||
- `IngressClasses`
|
||
- `PriorityClasses`
|
||
|
||
Each Resource kind can be granted with several verbs, such as:
|
||
|
||
- `List`
|
||
- `Update`
|
||
- `Delete`
|
||
|
||
### Namespaces
|
||
|
||
As tenant owner `alice`, you can use `kubectl` to create some namespaces:
|
||
```
|
||
$ kubectl --context alice-oidc@mycluster create namespace oil-production
|
||
$ kubectl --context alice-oidc@mycluster create namespace oil-development
|
||
$ kubectl --context alice-oidc@mycluster create namespace gas-marketing
|
||
```
|
||
|
||
and list only those namespaces:
|
||
|
||
```
|
||
$ kubectl --context alice-oidc@mycluster get namespaces
|
||
NAME STATUS AGE
|
||
gas-marketing Active 2m
|
||
oil-development Active 2m
|
||
oil-production Active 2m
|
||
```
|
||
|
||
### Nodes
|
||
|
||
The Capsule Proxy gives the owners the ability to access the nodes matching the `.spec.nodeSelector` in the Tenant manifest:
|
||
|
||
```yaml
|
||
apiVersion: capsule.clastix.io/v1beta1
|
||
kind: Tenant
|
||
metadata:
|
||
name: oil
|
||
spec:
|
||
owners:
|
||
- kind: User
|
||
name: alice
|
||
proxySettings:
|
||
- kind: Nodes
|
||
operations:
|
||
- List
|
||
nodeSelector:
|
||
kubernetes.io/hostname: capsule-gold-qwerty
|
||
```
|
||
|
||
```bash
|
||
$ kubectl --context alice-oidc@mycluster get nodes
|
||
NAME STATUS ROLES AGE VERSION
|
||
capsule-gold-qwerty Ready <none> 43h v1.19.1
|
||
```
|
||
|
||
> Warning: when no `nodeSelector` is specified, the tenant owners has access to all the nodes, according to the permissions listed in the `proxySettings` specs.
|
||
|
||
### Storage Classes
|
||
|
||
A Tenant may be limited to use a set of allowed Storage Class resources, as follows.
|
||
|
||
```yaml
|
||
apiVersion: capsule.clastix.io/v1beta1
|
||
kind: Tenant
|
||
metadata:
|
||
name: oil
|
||
spec:
|
||
owners:
|
||
- kind: User
|
||
name: alice
|
||
proxySettings:
|
||
- kind: StorageClasses
|
||
operations:
|
||
- List
|
||
storageClasses:
|
||
allowed:
|
||
- custom
|
||
allowedRegex: "\\w+fs"
|
||
```
|
||
|
||
In the Kubernetes cluster we could have more Storage Class resources, some of them forbidden and non-usable by the Tenant owner.
|
||
|
||
```bash
|
||
$ kubectl --context admin@mycluster get storageclasses
|
||
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
||
cephfs rook.io/cephfs Delete WaitForFirstConsumer false 21h
|
||
custom custom.tls/provisioner Delete WaitForFirstConsumer false 43h
|
||
default(standard) rancher.io/local-path Delete WaitForFirstConsumer false 43h
|
||
glusterfs rook.io/glusterfs Delete WaitForFirstConsumer false 54m
|
||
zol zfs-on-linux/zfs Delete WaitForFirstConsumer false 54m
|
||
```
|
||
|
||
The expected output using `capsule-proxy` is the retrieval of the `custom` Storage Class as well the other ones matching the regex `\w+fs`.
|
||
|
||
```bash
|
||
$ kubectl --context alice-oidc@mycluster get storageclasses
|
||
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
||
cephfs rook.io/cephfs Delete WaitForFirstConsumer false 21h
|
||
custom custom.tls/provisioner Delete WaitForFirstConsumer false 43h
|
||
glusterfs rook.io/glusterfs Delete WaitForFirstConsumer false 54m
|
||
```
|
||
|
||
### Ingress Classes
|
||
|
||
As for Storage Class, also Ingress Class can be enforced.
|
||
|
||
```yaml
|
||
apiVersion: capsule.clastix.io/v1beta1
|
||
kind: Tenant
|
||
metadata:
|
||
name: oil
|
||
spec:
|
||
owners:
|
||
- kind: User
|
||
name: alice
|
||
proxySettings:
|
||
- kind: IngressClasses
|
||
operations:
|
||
- List
|
||
ingressOptions:
|
||
allowedClasses:
|
||
allowed:
|
||
- custom
|
||
allowedRegex: "\\w+-lb"
|
||
```
|
||
|
||
In the Kubernetes cluster we could have more Ingress Class resources, some of them forbidden and non-usable by the Tenant owner.
|
||
|
||
```bash
|
||
$ kubectl --context admin@mycluster get ingressclasses
|
||
NAME CONTROLLER PARAMETERS AGE
|
||
custom example.com/custom IngressParameters.k8s.example.com/custom 24h
|
||
external-lb example.com/external IngressParameters.k8s.example.com/external-lb 2s
|
||
haproxy-ingress haproxy.tech/ingress 4d
|
||
internal-lb example.com/internal IngressParameters.k8s.example.com/external-lb 15m
|
||
nginx nginx.plus/ingress 5d
|
||
```
|
||
|
||
The expected output using `capsule-proxy` is the retrieval of the `custom` Ingress Class as well the other ones matching the regex `\w+-lb`.
|
||
|
||
```bash
|
||
$ kubectl --context alice-oidc@mycluster get ingressclasses
|
||
NAME CONTROLLER PARAMETERS AGE
|
||
custom example.com/custom IngressParameters.k8s.example.com/custom 24h
|
||
external-lb example.com/external IngressParameters.k8s.example.com/external-lb 2s
|
||
internal-lb example.com/internal IngressParameters.k8s.example.com/internal-lb 15m
|
||
```
|
||
|
||
### Priority Classes
|
||
|
||
Allowed PriorityClasses assigned to a Tenant Owner can be enforced as follows.
|
||
|
||
```yaml
|
||
apiVersion: capsule.clastix.io/v1beta1
|
||
kind: Tenant
|
||
metadata:
|
||
name: oil
|
||
spec:
|
||
owners:
|
||
- kind: User
|
||
name: alice
|
||
proxySettings:
|
||
- kind: IngressClasses
|
||
operations:
|
||
- List
|
||
priorityClasses:
|
||
allowed:
|
||
- best-effort
|
||
allowedRegex: "\\w+priority"
|
||
```
|
||
|
||
In the Kubernetes cluster we could have more PriorityClasses resources, some of them forbidden and non-usable by the Tenant owner.
|
||
|
||
```bash
|
||
$ kubectl --context admin@mycluster get priorityclasses.scheduling.k8s.io
|
||
NAME VALUE GLOBAL-DEFAULT AGE
|
||
custom 1000 false 18s
|
||
maxpriority 1000 false 18s
|
||
minpriority 1000 false 18s
|
||
nonallowed 1000 false 8m54s
|
||
system-cluster-critical 2000000000 false 3h40m
|
||
system-node-critical 2000001000 false 3h40m
|
||
```
|
||
|
||
The expected output using `capsule-proxy` is the retrieval of the `custom` PriorityClass as well the other ones matching the regex `\w+priority`.
|
||
|
||
```bash
|
||
$ kubectl --context alice-oidc@mycluster get ingressclasses
|
||
NAME VALUE GLOBAL-DEFAULT AGE
|
||
custom 1000 false 18s
|
||
maxpriority 1000 false 18s
|
||
minpriority 1000 false 18s
|
||
```
|
||
|
||
### Storage/Ingress class and PriorityClass required label
|
||
|
||
For Storage Class, Ingress Class and Priority Class resources, the `name` label reflecting the resource name is mandatory, otherwise filtering of resources cannot be put in place.
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: storage.k8s.io/v1
|
||
kind: StorageClass
|
||
metadata:
|
||
labels:
|
||
name: my-storage-class
|
||
name: my-storage-class
|
||
provisioner: org.tld/my-storage-class
|
||
---
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: IngressClass
|
||
metadata:
|
||
labels:
|
||
name: external-lb
|
||
name: external-lb
|
||
spec:
|
||
controller: example.com/ingress-controller
|
||
parameters:
|
||
apiGroup: k8s.example.com
|
||
kind: IngressParameters
|
||
name: external-lb
|
||
---
|
||
apiVersion: scheduling.k8s.io/v1
|
||
kind: PriorityClass
|
||
metadata:
|
||
labels:
|
||
name: best-effort
|
||
name: best-effort
|
||
value: 1000
|
||
globalDefault: false
|
||
description: "Priority class for best-effort Tenants"
|
||
```
|
||
|
||
## Does it work with kubectl?
|
||
Yes, it works by intercepting all the requests from the `kubectl` client directed to the APIs server. It works with both users who use the TLS certificate authentication and those who use OIDC.
|
||
|
||
As tenant owner `alice`, you are able to use `kubectl` to create some namespaces:
|
||
```
|
||
$ kubectl --context alice-oidc@mycluster create namespace oil-production
|
||
$ kubectl --context alice-oidc@mycluster create namespace oil-development
|
||
$ kubectl --context alice-oidc@mycluster create namespace gas-marketing
|
||
```
|
||
|
||
and list only those namespaces:
|
||
```
|
||
$ kubectl --context alice-oidc@mycluster get namespaces
|
||
NAME STATUS AGE
|
||
gas-marketing Active 2m
|
||
oil-development Active 2m
|
||
oil-production Active 2m
|
||
```
|
||
|
||
# What’s next
|
||
Have a fun with `capsule-proxy`:
|
||
|
||
* [Standalone Installation](./standalone.md)
|
||
* [Sidecar Installation](./sidecar.md)
|
||
* [OIDC Authentication](./oidc-auth.md)
|
||
* [Contributing](./contributing.md)
|