docs: fix minor issues

This commit is contained in:
bsctl
2021-08-23 16:38:17 +02:00
committed by Dario Tranchitella
parent d0530bbbe3
commit bdafbcf90a
45 changed files with 279 additions and 306 deletions
+2 -2
View File
@@ -1,5 +1,5 @@
# Capsule extension for Lens
With Capsule extension for [Lens](https://github.com/lensapp/lens), a cluster administrator can easly manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.
With Capsule extension for [Lens](https://github.com/lensapp/lens), a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.
## Features
Capsule extension for Lens provides these capabilities:
@@ -8,4 +8,4 @@ Capsule extension for Lens provides these capabilities:
- See tenant details and change through the embedded Lens editor
- Check Resources Quota and Budget at both the tenant and namespace level
Please, see the [README](https://raw.githubusercontent.com/clastix/capsule-lens-extension/master/README.md) for details about installation of the Capsule Lens Extension.
Please, see the [README](https://github.com/clastix/capsule-lens-extension) for details about the installation of the Capsule Lens Extension.
+7 -7
View File
@@ -27,7 +27,7 @@ the CRDs manifests, as well the deep copy functions, require _Operator SDK_:
the binary has to be installed into your `PATH`.
### Installing Kubebuilder
With the latest release of OperatorSDK there's a more tightly integration with
With the latest release of OperatorSDK there's a more tighten integration with
Kubebuilder and its opinionated testing suite: ensure to download the latest
binaries available from the _Releases_ GitHub page and place them into the
`/usr/local/kubebuilder/bin` folder, ensuring this is also in your `PATH`.
@@ -97,7 +97,7 @@ You can check if Capsule is running tailing the logs:
```
Since Capsule is built using _OperatorSDK_, logging is handled by the zap
module: log verbosity of the Capsule controller can be increased by passing
module: log verbosity of the Capsule controller can be increased passing
the `--zap-log-level` option with a value from `1` to `10` or the
[basic keywords](https://godoc.org/go.uber.org/zap/zapcore#Level) although
it is suggested to use the `--zap-devel` flag to get also stack traces.
@@ -124,7 +124,7 @@ deployment.apps/capsule-controller-manager scaled
> This is mandatory since Capsule uses Leader Election
#### Providing TLS certificate for webhooks
Next step is to replicate the same environment Capsule is expecting in the Pod,
The next step is to replicate the same environment Capsule is expecting in the Pod,
it means creating a fake certificate to handle HTTP requests.
``` bash
@@ -133,8 +133,8 @@ kubectl -n capsule-system get secret capsule-tls -o jsonpath='{.data.tls\.crt}'
kubectl -n capsule-system get secret capsule-tls -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/k8s-webhook-server/serving-certs/tls.key
```
> We're using the certificates generate upon first installation of Capsule:
> it means the Secret will be populated at first start-up.
> We're using the certificates generate upon the first installation of Capsule:
> it means the Secret will be populated at the first start-up.
> If you plan to run it locally since the beginning, it means you will require
> to provide a self-signed certificate in the said directory.
@@ -242,10 +242,10 @@ A commit description is welcomed to explain more the changes: just ensure
to put a blank line and an arbitrary number of maximum 72 characters long
lines, at most one blank line between them.
Please, split changes into several and documented small commits: this will help us to perform a better review. Commits must follow the the Conventional Commits Specification, a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with Semantic Versioning, by describing the features, fixes, and breaking changes made in commit messages. See [Conventional Commits Specification](https://www.conventionalcommits.org) to learn about Conventional Commits.
Please, split changes into several and documented small commits: this will help us to perform a better review. Commits must follow the Conventional Commits Specification, a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with Semantic Versioning, by describing the features, fixes, and breaking changes made in commit messages. See [Conventional Commits Specification](https://www.conventionalcommits.org) to learn about Conventional Commits.
> In case of errors or need of changes to previous commits,
> fix them squashing to make changes atomic.
### Miscellanea
Please, add a new single line at end of any file as current coding style.
Please, add a new single line at end of any file as the current coding style.
+3 -3
View File
@@ -19,10 +19,10 @@ $ kubectl apply -f https://raw.githubusercontent.com/clastix/capsule/master/conf
It will install the Capsule controller in a dedicated namespace `capsule-system`.
### Install with Helm Chart
Please, refer to the instructions reported into the Capsule Helm Chart [README](https://github.com/clastix/capsule/blob/master/charts/capsule/README.md).
Please, refer to the instructions reported in the Capsule Helm Chart [README](https://github.com/clastix/capsule/blob/master/charts/capsule/README.md).
# Create your first Tenant
In Capsule, a _Tenant_ is an abstraction to group togheter multiple namespaces in a single entity within a set of bundaries defined by the Cluster Administrator. The tenant is then assigned to a user or group of users who is called _Tenant Owner_.
In Capsule, a _Tenant_ is an abstraction to group multiple namespaces in a single entity within a set of boundaries defined by the Cluster Administrator. The tenant is then assigned to a user or group of users who is called _Tenant Owner_.
Capsule defines a Tenant as Custom Resource with cluster scope.
@@ -99,7 +99,7 @@ $ kubectl -n oil-development run nginx --image=docker.io/nginx
$ kubectl -n oil-development get pods
```
but limited to only your own namespaces:
but limited to only your namespaces:
```
$ kubectl -n kube-system get pods
+1 -1
View File
@@ -1,5 +1,5 @@
# Capsule on AWS EKS
This is an example how to install AWS EKS cluster and one user
This is an example of how to install AWS EKS cluster and one user
manged by Capsule.
It is based on [Using IAM Groups to manage Kubernetes access](https://www.eksworkshop.com/beginner/091_iam-groups/intro/)
+1 -3
View File
@@ -1,5 +1,5 @@
# Capsule over Managed Kubernetes
Capsule Operator can be easly installed on a Managed Kubernetes Service. Since in these services, you do not have access to the Kubernetes APIs Server, you should check with your service provider following pre-requisites:
Capsule Operator can be easily installed on a Managed Kubernetes Service. Since in these services, you do not have access to the Kubernetes APIs Server, you should check with your service provider following pre-requisites:
- the default `cluster-admin` ClusterRole is accessible
- the following Admission Webhooks are enabled on the APIs Server:
@@ -8,8 +8,6 @@ Capsule Operator can be easly installed on a Managed Kubernetes Service. Since i
- ResourceQuota
- MutatingAdmissionWebhook
- ValidatingAdmissionWebhook
- PodSecurityPolicy (optional)
- a CNI plugin with support for Network Policies can be used.
* [AWS EKS](./aws-eks.md)
* CoAKS - Capsule over Azure Kubernetes Service
+5 -5
View File
@@ -1,6 +1,6 @@
# Monitoring Capsule
The Capsule dashboard allows you to track the health and performance of Capsule manager and tenants, with particular attention to resources saturation, server responses and latencies.
The Capsule dashboard allows you to track the health and performance of Capsule manager and tenants, with particular attention to resources saturation, server responses, and latencies.
## Requirements
@@ -76,7 +76,7 @@ making sure to select the correct Prometheus data source:
##### Description
This section provides information about the medium time delay between manager client input, side effects and new state determination (reconciliation).
This section provides information about the medium time delay between manager client input, side effects, and new state determination (reconciliation).
##### Dependant variables and available values
@@ -117,7 +117,7 @@ This section provides information about webhook requests response, mainly focusi
##### Description
This section provides information about the medium time delay between webhook trigger, side effects and data written on etcd.
This section provides information about the medium time delay between webhook trigger, side effects, and data written on etcd.
##### Dependant variables and available values
@@ -139,7 +139,7 @@ This section provides information about the medium time delay between webhook tr
##### Description
This section provides information about the medium time delay between all the calls done by the controller and the API server.
Data display may depends on the REST client verb considered and on available REST client URLs.
Data display may depend on the REST client verb considered and on available REST client URLs.
YMMV
@@ -175,7 +175,7 @@ This section provides information about resources, giving a detailed picture of
##### Description
This section provides information about "actions" in queue, particularly:
This section provides information about "actions" in the queue, particularly:
- Workqueue latency: time to complete a series of actions in the queue ;
- Workqueue rate: number of actions per unit time ;
- Workqueue depth: number of pending actions waiting in the queue.
@@ -6,7 +6,7 @@
**Category:** Self-Service Operations
**Description:** Tenants should be able to perform self-service operations by creating own network policies in their namespaces.
**Description:** Tenants should be able to perform self-service operations by creating their own network policies in their namespaces.
**Rationale:** Enables self-service management of network-policies.
@@ -56,7 +56,7 @@ NAME POD-SELECTOR AGE
capsule-oil-0 <none> 7m5s
```
As tenant owner check for permissions to manage networkpolicy for each verb
As a tenant, checks for permissions to manage networkpolicy for each verb
```bash
kubectl --kubeconfig alice auth can-i get networkpolicies
@@ -6,7 +6,7 @@
**Category:** Self-Service Operations
**Description:** Tenants should be able to perform self-service operations by creating own rolebindings in their namespaces.
**Description:** Tenants should be able to perform self-service operations by creating their rolebindings in their namespaces.
**Rationale:** Enables self-service management of roles.
@@ -6,7 +6,7 @@
**Category:** Self-Service Operations
**Description:** Tenants should be able to perform self-service operations by creating own roles in their namespaces.
**Description:** Tenants should be able to perform self-service operations by creating their own roles in their namespaces.
**Rationale:** Enables self-service management of roles.
@@ -37,7 +37,7 @@ kubectl --kubeconfig alice create ns oil-production
kubectl --kubeconfig alice config set-context --current --namespace oil-production
```
As tenant owner check for permissions to manage roles for each verb
As tenant owner, check for permissions to manage roles for each verb
```bash
kubectl --kubeconfig alice auth can-i get roles
@@ -6,7 +6,7 @@
**Category:** Control Plane Isolation
**Description:** Tenants should not be able to view, edit, create, or delete cluster (non-namespaced) resources such Node, ClusterRole, ClusterRoleBinding, etc.
**Description:** Tenants should not be able to view, edit, create or delete cluster (non-namespaced) resources such Node, ClusterRole, ClusterRoleBinding, etc.
**Rationale:** Access controls should be configured for tenants so that a tenant cannot list, create, modify or delete cluster resources
@@ -53,7 +53,7 @@ kubectl --kubeconfig alice auth can-i create namespaces
yes
```
Any kubernetes user can create `SelfSubjectAccessReview` and `SelfSubjectRulesReviews` to checks whether he/she can perform an action. First two exceptions are not an issue.
Any kubernetes user can create `SelfSubjectAccessReview` and `SelfSubjectRulesReviews` to checks whether he/she can act. First, two exceptions are not an issue.
```bash
kubectl --anyuser auth can-i --list
@@ -78,7 +78,7 @@ selfsubjectrulesreviews.authorization.k8s.io [] []
[/version] [] [get]
```
In order to enable namespace self-service provisioning, Capsule intentionally gives permissions to create namespaces to all users belonging to the Capsule group:
To enable namespace self-service provisioning, Capsule intentionally gives permissions to create namespaces to all users belonging to the Capsule group:
```bash
kubectl describe clusterrolebindings capsule-namespace-provisioner
@@ -6,9 +6,9 @@
**Category:** Tenant Isolation
**Description:** Each tenant namespace may contain resources setup by the cluster administrator for multi-tenancy, such as role bindings, and network policies. Tenants should not be allowed to modify the namespaced resources created by the cluster administrator for multi-tenancy. However, for some resources such as network policies, tenants can configure additional instances of the resource for their workloads.
**Description:** Each tenant namespace may contain resources set up by the cluster administrator for multi-tenancy, such as role bindings, and network policies. Tenants should not be allowed to modify the namespaced resources created by the cluster administrator for multi-tenancy. However, for some resources such as network policies, tenants can configure additional instances of the resource for their workloads.
**Rationale:** Tenants can escalate priviliges and impact other tenants if they are able to delete or modify required multi-tenancy resources such as namespace resource quotas or default network policy.
**Rationale:** Tenants can escalate privileges and impact other tenants if they can delete or modify required multi-tenancy resources such as namespace resource quotas or default network policy.
**Audit:**
@@ -99,7 +99,7 @@ spec:
EOF
```
However, due the additive nature of networkpolicies, the `DENY ALL` policy set by the cluster admin, prevents the hijacking.
However, due to the additive nature of networkpolicies, the `DENY ALL` policy set by the cluster admin, prevents hijacking.
As tenant owner list RBAC permissions set by Capsule
@@ -110,13 +110,13 @@ namespace-deleter ClusterRole/capsule-namespace-deleter 11h
namespace:admin ClusterRole/admin 11h
```
As tenant owner, try to change/delete the rolebindings in order to escalate permissions
As tenant owner, try to change/delete the rolebinding to escalate permissions
```bash
kubectl --kubeconfig alice edit/delete rolebinding namespace:admin
```
The rolebindings is immediately recreated by Capsule:
The rolebinding is immediately recreated by Capsule:
```
kubectl --kubeconfig alice get rolebindings
@@ -125,7 +125,7 @@ namespace-deleter ClusterRole/capsule-namespace-deleter 11h
namespace:admin ClusterRole/admin 2s
```
However, the tenant owner can create and assign permissions inside namespace she owns
However, the tenant owner can create and assign permissions inside the namespace she owns
```yaml
kubectl create -f - << EOF
@@ -6,7 +6,7 @@
**Category:** Tenant Isolation
**Description:** Each tenant has its own set of resources, such as namespaces, service accounts, secrets, pods, services, etc. Tenants should not be allowed to access eachother's resources.
**Description:** Each tenant has its own set of resources, such as namespaces, service accounts, secrets, pods, services, etc. Tenants should not be allowed to access each other's resources.
**Rationale:** Tenant's resources must be not accessible by other tenants.
@@ -69,10 +69,10 @@ As `oil` tenant owner, try to retrieve the resources in the `gas` tenant namespa
kubectl --kubeconfig alice get serviceaccounts --namespace gas-production
```
You must receive an eror message:
You must receive an error message:
```
Error from server (Forbidden): serviceaccounts is forbidden:
Error from server (Forbidden): serviceaccount is forbidden:
User "oil" cannot list resource "serviceaccounts" in API group "" in the namespace "gas-production"
```
@@ -82,10 +82,10 @@ As `gas` tenant owner, try to retrieve the resources in the `oil` tenant namespa
kubectl --kubeconfig joe get serviceaccounts --namespace oil-production
```
You must receive an eror message:
You must receive an error message:
```
Error from server (Forbidden): serviceaccounts is forbidden:
Error from server (Forbidden): serviceaccount is forbidden:
User "joe" cannot list resource "serviceaccounts" in API group "" in the namespace "oil-production"
```
@@ -8,7 +8,7 @@
**Description:** Control container permissions.
**Rationale:** The security `allowPrivilegeEscalation` setting allows a process to gain more privileges from its parent process. Processes in tenant containers should not be allowed to gain additional priviliges.
**Rationale:** The security `allowPrivilegeEscalation` setting allows a process to gain more privileges from its parent process. Processes in tenant containers should not be allowed to gain additional privileges.
**Audit:**
@@ -8,7 +8,7 @@
**Description:** Avoid a tenant to mount existing volumes`.
**Rationale:** Tenants have to be assured that their Persisten Volumes cannot be reclaimed by other tenants.
**Rationale:** Tenants have to be assured that their Persistent Volumes cannot be reclaimed by other tenants.
**Audit:**
@@ -8,7 +8,7 @@
**Description:** Tenants should not be able to mount host volumes and directories.
**Rationale:** The use of host volumes and directories can be used to access shared data or escalate priviliges and also creates a tight coupling between a tenant workload and a host.
**Rationale:** The use of host volumes and directories can be used to access shared data or escalate privileges and also creates a tight coupling between a tenant workload and a host.
**Audit:**
@@ -8,7 +8,7 @@
**Description:** Force a tenant to use a Storage Class with `reclaimPolicy=Delete`.
**Rationale:** Tenants have to be assured that their Persisten Volumes cannot be reclaimed by other tenants.
**Rationale:** Tenants have to be assured that their Persistent Volumes cannot be reclaimed by other tenants.
**Audit:**
@@ -8,7 +8,7 @@
**Description:** Control container permissions.
**Rationale:** Processes in containers run as the root user (uid 0), by default. To prevent potential compromise of container hosts, specify a least privileged user ID when building the container image and require that application containers run as non root users.
**Rationale:** Processes in containers run as the root user (uid 0), by default. To prevent potential compromise of container hosts, specify a least-privileged user ID when building the container image and require that application containers run as non-root users.
**Audit:**
+2 -2
View File
@@ -1,7 +1,7 @@
# Meet the multi-tenancy benchmark MTB
Actually, there's no yet a real standard for the multi-tenancy model in Kubernetes, although the [SIG multi-tenancy group](https://github.com/kubernetes-sigs/multi-tenancy) is working on that. SIG multi-tenancy drafted a generic validation schema appliable to generic multi-tenancy projects. Multi-Tenancy Benchmarks [MTB](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/benchmarks) are guidelines for multi-tenant configuration of Kubernetes clusters. Capsule is an open source multi-tenacy operator and we decided to meet the requirements of MTB.
Actually, there's no yet a real standard for the multi-tenancy model in Kubernetes, although the [SIG multi-tenancy group](https://github.com/kubernetes-sigs/multi-tenancy) is working on that. SIG multi-tenancy drafted a generic validation schema appliable to generic multi-tenancy projects. Multi-Tenancy Benchmarks [MTB](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/benchmarks) are guidelines for multi-tenant configuration of Kubernetes clusters. Capsule is an open source multi-tenancy operator and we decided to meet the requirements of MTB.
> N.B. At time of writing, the MTB are in development and not ready for usage. Strictly speaking, we do not claim an official conformance to MTB, but just to adhere to the multi-tenancy requirements and best practices promoted by MTB.
> N.B. At the time of writing, the MTB is in development and not ready for usage. Strictly speaking, we do not claim official conformance to MTB, but just to adhere to the multi-tenancy requirements and best practices promoted by MTB.
|MTB Benchmark |MTB Profile|Capsule Version|Conformance|Notes |
|--------------|-----------|---------------|-----------|-------|
+1 -1
View File
@@ -3,7 +3,7 @@
* [Getting Started](./getting-started.md)
* [Use Cases](./use-cases/overview.md)
* [SIG Multi-tenancy benchmark](./mtb/sig-multitenancy-bench.md)
* [Run on Managed Kubernetes Services](./managed-kubernetes/getting-started-amazon-eks.md)
* [Run on Managed Kubernetes Services](./managed-kubernetes/overview.md)
* [Monitoring Capsule](./monitoring.md)
* [References](./references.md)
* [Contributing](./contributing.md)
+5 -34
View File
@@ -9,7 +9,7 @@
## Custom Resource Definition
Capsule operator uses a Custom Resources Definition (CRD) for _Tenants_. In Caspule, Tenants are cluster wide resources. You need for cluster level permissions to work with tenants.
Capsule operator uses a Custom Resources Definition (CRD) for _Tenants_. In Capsule, Tenants are cluster wide resources. You need cluster level permissions to work with tenants.
You can learn about tenant CRD by the `kubectl explain` command:
@@ -24,15 +24,11 @@ DESCRIPTION:
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
@@ -205,7 +201,7 @@ capsule-mutating-webhook-configuration 1 2h
## Command Options
The Capsule operator provides following command options:
The Capsule operator provides the following command options:
Option | Description | Default
--- | --- | ---
@@ -213,33 +209,8 @@ Option | Description | Default
`--enable-leader-election` | Start a leader election client and gain leadership before executing the main loop. | `true`
`--zap-log-level` | The log verbosity with a value from 1 to 10 or the basic keywords. | `4`
`--zap-devel` | The flag to get the stack traces for deep debugging. | `null`
`--configuration-name` | The Capsule Configuration CRD name, a default is installed automatically | `capsule-default`
`--configuration-name` | The Capsule Configuration CRD name, default is installed automatically | `capsule-default`
## Capsule Configuration
The Capsule configuration can be piloted by a Custom Resource definition named `CapsuleConfiguration`.
```yaml
apiVersion: capsule.clastix.io/v1alpha1
kind: CapsuleConfiguration
metadata:
name: default
spec:
userGroups: ["capsule.clastix.io"]
forceTenantPrefix: false
protectedNamespaceRegex: ""
```
Option | Description | Default
--- | --- | ---
`.spec.forceTenantPrefix` | Force the tenant name as prefix for namespaces: `<tenant_name>-<namespace>`. | `false`
`.spec.userGroups` | Array of Capsule groups to which all tenant owners must belong. | `[capsule.clastix.io]`
`.spec.protectedNamespaceRegex` | Disallows creation of namespaces matching the passed regexp. | `null`
Upon installation using Kustomize or Helm, a `default` resource will be created.
The reference to this configuration is managed by the CLI flag `--configuration-name`.
>>>>>>> 0f3f6da... docs: updates structure for documentation
## Created Resources
Once installed, the Capsule operator creates the following resources in your cluster:
+5 -5
View File
@@ -3,13 +3,13 @@
Bill needs to cordon a Tenant and its Namespaces for several reasons:
- Avoid accidental resource modification(s) including deletion during a Production Freeze Window
- During Kubernetes upgrade, to prevent any workload updates
- During the Kubernetes upgrade, to prevent any workload updates
- During incidents or outages
- During planned maintenance of a dedicated nodes pool in a BYOD scenario
With this said, the Tenant Owner and the related Service Account living into managed Namespaces, cannot proceed to any update, create or delete action.
This is possible just labelling the Tenant as follows:
This is possible just labeling the Tenant as follows:
```shell
kubectl label tenant oil capsule.clastix.io/cordon=enabled
@@ -20,13 +20,13 @@ Any operation performed by Alice, the Tenant Owner, will be rejected.
```shell
$ kubectl --as alice --as-group capsule.clastix.io -n oil-dev create deployment nginx --image nginx
error: failed to create deployment: admission webhook "cordoning.tenant.capsule.clastix.io" denied the request: tenant oil is freezed: please, reach out to the system administrator
error: failed to create deployment: admission webhook "cordoning.tenant.capsule.clastix.io" denied the request: tenant oil is frozen: please, reach out to the system administrator
$ kubectl --as alice --as-group capsule.clastix.io -n oil-dev delete ingress,deployment,serviceaccount --all
error: failed to create deployment: admission webhook "cordoning.tenant.capsule.clastix.io" denied the request: tenant oil is freezed: please, reach out to the system administrator
error: failed to create deployment: admission webhook "cordoning.tenant.capsule.clastix.io" denied the request: tenant oil is frozen: please, reach out to the system administrator
```
Uncordoning can be done removing the said label:
Uncordoning can be done by removing the said label:
```shell
$ kubectl label tenant oil capsule.clastix.io/cordon-
+2 -2
View File
@@ -1,11 +1,11 @@
# Create namespaces
Alice, once logged with her own credentials, can create a new namespace in her tenant, as simply issuing:
Alice, once logged with her credentials, can create a new namespace in her tenant, as simply issuing:
```
kubectl create ns oil-production
```
Alice started the name of the namespace prepended by the name of the tenant: this is not a strict requirement but it is highly suggested because it is likely that many different tenants would like to call their namespaces as `production`, `test`, or `demo`, etc.
Alice started the name of the namespace prepended by the name of the tenant: this is not a strict requirement but it is highly suggested because it is likely that many different tenants would like to call their namespaces `production`, `test`, or `demo`, etc.
The enforcement of this naming convention is optional and can be controlled by the cluster administrator with the `--force-tenant-prefix` option as an argument of the Capsule controller.
+2 -2
View File
@@ -1,5 +1,5 @@
# Create Custom Resources
Capsule grants admin permissions to the tenant owners but only limited to their namespaces. To achieve that, it assigns the ClusterRole [admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) to the tenant owner. This ClusterRole does not permit the installation of custom resources in the namespaces.
Capsule grants admin permissions to the tenant owners but is only limited to their namespaces. To achieve that, it assigns the ClusterRole [admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) to the tenant owner. This ClusterRole does not permit the installation of custom resources in the namespaces.
In order to leave the tenant owner to create Custom Resources in their namespaces, the cluster admin defines a proper Cluster Role. For example:
@@ -75,4 +75,4 @@ With the above example, Capsule is leaving the tenant owner to create namespaced
> Take Note: a tenant owner having the admin scope on its namespaces only, does not have the permission to create Custom Resources Definitions (CRDs) because this requires a cluster admin permission level. Only Bill, the cluster admin, can create CRDs. This is a known limitation of any multi-tenancy environment based on a single Kubernetes cluster.
# Whats next
See how Bill, the cluster admin, can set taints on the Alice's namespaces. [Taint namespaces](./taint-namespaces.md).
See how Bill, the cluster admin, can set taints on Alice's namespaces. [Taint namespaces](./taint-namespaces.md).
@@ -1,5 +1,5 @@
# Control hostname collision in ingresses
In a multi-tenant environment, as more and more ingresses are defined, there is a chance of collision on the hostname leading to an unpredictable behaviour of the Ingress Controller. Bill, the cluster admin, can enforce hostname collision detection at different scope level:
In a multi-tenant environment, as more and more ingresses are defined, there is a chance of collision on the hostname leading to unpredictable behavior of the Ingress Controller. Bill, the cluster admin, can enforce hostname collision detection at different scope levels:
1. Cluster
2. Tenant
@@ -25,7 +25,7 @@ EOF
When a tenant owner creates an Ingress resource, Capsule will check the collision of hostname in the current ingress with all the hostnames already used, depending on the defined scope.
For example, Alice, one of the tenant owner, creates an Ingress
For example, Alice, one of the tenant owners, creates an Ingress
```yaml
+1 -1
View File
@@ -4,7 +4,7 @@ Bill is a cluster admin providing a Container as a Service platform using shared
Alice, a Tenant Owner, can start container images using private images: according to the Kubernetes architecture, the `kubelet` will download the layers on its cache.
Bob, an attacker, could try to schedule a Pod on the same node where Alice is running her Pods backed by private images: they could start new Pods using `ImagePullPolicy=IfNotPresent` and able to start them, even without required authentication since the image is cached on the node.
Bob, an attacker, could try to schedule a Pod on the same node where Alice is running her Pods backed by private images: they could start new Pods using `ImagePullPolicy=IfNotPresent` and be able to start them, even without required authentication since the image is cached on the node.
To avoid this kind of attack, Bill, the cluster admin, can force Alice, the tenant owner, to start her Pods using only the allowed values for `ImagePullPolicy`, enforcing the `kubelet` to check the authorization first.
+3 -3
View File
@@ -1,7 +1,7 @@
# Assign Trusted Images Registries
Bill, the cluster admin, can set a strict policy on the applications running into Alice's tenant: he'd like to allow running just images hosted on a list of specific container registries.
The spec `containerRegistries` addresses this task and can provide combination with hard enforcement using a list of allowed values.
The spec `containerRegistries` addresses this task and can provide a combination with hard enforcement using a list of allowed values.
```yaml
@@ -21,14 +21,14 @@ spec:
allowedRegex: 'internal.registry.\\w.tld'
```
> In case of `non FQDI` (non fully qualified Docker image) and official images hosted on Docker Hub,
> In case of `non-FQDI` (non fully qualified Docker image) and official images hosted on Docker Hub,
> Capsule is going to retrieve the registry even if it's not explicit: a `busybox:latest` Pod
> running on a Tenant allowing `docker.io` will not be blocked, even if the image
> field is not explicit as `docker.io/busybox:latest`.
A Pod running `internal.registry.foo.tld` as registry will be allowed, as well `internal.registry.bar.tld` since these are matching the regular expression.
> A catch-all regex entry as `.*` allows every kind of registry, that would be the same result of unsetting `containerRegistries` at all.
> A catch-all regex entry as `.*` allows every kind of registry, which would be the same result of unsetting `containerRegistries` at all.
Any attempt of Alice to use a not allowed `containerRegistries` value is denied by the Validation Webhook enforcing it.
+1 -1
View File
@@ -46,7 +46,7 @@ spec:
EOF
```
Any attempt of Alice to use a non valid Ingress Class, or missing it, is denied by the Validation Webhook enforcing it.
Any attempt of Alice to use a non-valid Ingress Class, or missing it, is denied by the Validation Webhook enforcing it.
# Whats next
See how Bill, the cluster admin, can assign a set of dedicated ingress hostnames to Alice's tenant. [Assign Ingress Hostnames](./ingress-hostnames.md).
+2 -2
View File
@@ -1,5 +1,5 @@
# Assign Ingress Hostnames
Bill can control ingress hostnames to the `oil` tenant in order to force the applications to be published only using the given hostname or set of hostnames:
Bill can control ingress hostnames to the `oil` tenant to force the applications to be published only using the given hostname or set of hostnames:
```yaml
kubectl apply -f - << EOF
@@ -47,7 +47,7 @@ spec:
EOF
```
Any attempt of Alice to use a non valid hostname is denied by the Validation Webhook enforcing it.
Any attempt of Alice to use a non-valid hostname is denied by the Validation Webhook enforcing it.
# Whats next
See how Bill, the cluster admin, can control the hostname collision in Ingresses. [Control hostname collision in ingresses](./hostname-collision.md).
+4 -4
View File
@@ -1,7 +1,7 @@
# Assign multiple tenants to an owner
In some scenarios, it's likely that a single team is responsible for multiple lines of business. For example, in our sample organization Acme Corp., Alice is responsible for both the Oil and Gas lines of business. It's more likely that Alice requires two different tenants, for example `oil` and `gas` to keep things isolated.
In some scenarios, a single team is likely responsible for multiple lines of business. For example, in our sample organization Acme Corp., Alice is responsible for both the Oil and Gas lines of business. It's more likely that Alice requires two different tenants, for example, `oil` and `gas` to keep things isolated.
By design, the Capsule operator does not permit hierarchy of tenants, since all tenants are at the same levels. However, we can assign the ownership of multiple tenants to the same user or group of users.
By design, the Capsule operator does not permit a hierarchy of tenants, since all tenants are at the same levels. However, we can assign the ownership of multiple tenants to the same user or group of users.
Bill, the cluster admin, creates multiple tenants having `alice` as owner:
@@ -63,7 +63,7 @@ spec:
EOF
```
The two tenants still remain isolated each other in terms of resources assignments, e.g. _ResourceQuota_, _Nodes Pool_, _Storage Calsses_ and _Ingress Classes_, and in terms of governance, e.g. _NetworkPolicies_, _PodSecurityPolicies_, _Trusted Registries_, etc.
The two tenants remain isolated from each other in terms of resources assignments, e.g. _ResourceQuota_, _Nodes Pool_, _Storage Calsses_ and _Ingress Classes_, and in terms of governance, e.g. _NetworkPolicies_, _PodSecurityPolicies_, _Trusted Registries_, etc.
When Alice logs in, she has access to all namespaces belonging to both the `oil` and `gas` tenants.
@@ -73,7 +73,7 @@ kubectl create ns oil-production
kubectl create ns gas-production
```
When the enforcement of the naming convention with the `--force-tenant-prefix` option, is enabled, the namespaces are automatically assigned to the right tenant by Capsule because the operator does a lookups on the tenant names. If the `--force-tenant-prefix` option, is not set, Alice needs to specify the tenant name as a label `capsule.clastix.io/tenant=<desired_tenant>` in the namespace manifest:
When the enforcement of the naming convention with the `--force-tenant-prefix` option, is enabled, the namespaces are automatically assigned to the right tenant by Capsule because the operator does a lookup on the tenant names. If the `--force-tenant-prefix` option, is not set, Alice needs to specify the tenant name as a label `capsule.clastix.io/tenant=<desired_tenant>` in the namespace manifest:
```yaml
kubectl apply -f - << EOF
+2 -2
View File
@@ -3,7 +3,7 @@ Kubernetes network policies control network traffic between namespaces and betwe
To meet this requirement, Bill needs to define network policies that deny pods belonging to Alice's namespaces to access pods in namespaces belonging to other tenants, e.g. Bob's tenant `water`, or in system namespaces, e.g. `kube-system`.
Also, Bill can make sure pods belonging to a tenant namespace cannot access other network infrastructure like cluster nodes, load balancers, and virtual machines running other services.
Also, Bill can make sure pods belonging to a tenant namespace cannot access other network infrastructures like cluster nodes, load balancers, and virtual machines running other services.
Bill can set network policies in the tenant manifest, according to the requirements:
@@ -96,7 +96,7 @@ And delete the namespace network policies
kubectl -n oil-production delete networkpolicy production-network-policy
```
Any attempt of Alice to delete the tenant network policy defined in the tenant manifest, is denied by the Validation Webhook enforcing it.
Any attempt of Alice to delete the tenant network policy defined in the tenant manifest is denied by the Validation Webhook enforcing it.
# Whats next
See how Bill can enforce the Pod containers image pull policy to `Always` to avoid leaking of private images when running on shared nodes. [Enforcing Pod containers image PullPolicy](./images-pullpolicy.md)
+2 -2
View File
@@ -51,9 +51,9 @@ spec:
hardware: gpu
```
Any attempt of Alice to change the selector on the pods will result an error from the `PodNodeSelector` Admission Controller plugin.
Any attempt of Alice to change the selector on the pods will result in an error from the `PodNodeSelector` Admission Controller plugin.
Also RBAC prevents Alice to change the annotation on the namespace:
Also, RBAC prevents Alice to change the annotation on the namespace:
```
kubectl auth can-i edit ns -n oil-production
+7 -7
View File
@@ -1,9 +1,9 @@
# Onboard a new tenant
Bill, the cluster admin, receives a new request from Acme Corp.'s CTO asking for a new tenant to be onboarded and Alice user will be the tenant owner. Bill then assigns Alice's identity of `alice` in Acme Corp. identity management system. Since Alice is a tenant owner, Bill needs to assign `alice` the Capsule group defined by `--capsule-user-group` option, which defaults to `capsule.clastix.io`.
Bill, the cluster admin, receives a new request from Acme Corp.'s CTO asking for a new tenant to be onboarded and Alice user will be the tenant owner. Bill then assigns Alice's identity of `alice` in the Acme Corp. identity management system. Since Alice is a tenant owner, Bill needs to assign `alice` the Capsule group defined by `--capsule-user-group` option, which defaults to `capsule.clastix.io`.
To keep the things simple, we assume that Bill just creates a client certificate for authentication using X.509 Certificate Signing Request, so Alice's certificate has `"/CN=alice/O=capsule.clastix.io"`.
To keep things simple, we assume that Bill just creates a client certificate for authentication using X.509 Certificate Signing Request, so Alice's certificate has `"/CN=alice/O=capsule.clastix.io"`.
Bill creates a new tenant `oil` in the CaaS manangement portal according to the tenant's profile:
Bill creates a new tenant `oil` in the CaaS management portal according to the tenant's profile:
```yaml
kubectl create -f - << EOF
@@ -32,7 +32,7 @@ oil Active 0 33m
Once the new tenant `oil` is in place, Bill sends the login credentials to Alice.
Alice can log in using her own credentials and check if she can create a namespace
Alice can log in using her credentials and check if she can create a namespace
```
kubectl auth can-i create namespaces
@@ -84,7 +84,7 @@ spec:
EOF
```
However, it's more likely thet Bill assigns the ownership of the `oil` tenant to a group of users instead of a single one. Bill creates a new group account `oil-users` in the Acme Corp. identity management system and then he assigns Alice and Bob identities to the `oil-users` group.
However, it's more likely that Bill assigns the ownership of the `oil` tenant to a group of users instead of a single one. Bill creates a new group account `oil-users` in the Acme Corp. identity management system and then he assigns Alice and Bob identities to the `oil-users` group.
The tenant manifest is modified as in the following:
@@ -101,7 +101,7 @@ spec:
EOF
```
With the configuration above, any user belonging to the `oil-users` group will be owner of the `oil` tenant with the same permissions of Alice. For example, Bob can login with his own credentials and issue
With the configuration above, any user belonging to the `oil-users` group will be the owner of the `oil` tenant with the same permissions of Alice. For example, Bob can log in with his credentials and issue
```
kubectl auth can-i create namespaces
@@ -109,7 +109,7 @@ yes
```
## Assign a robot account as tenant owner
As GitOps metodology is gaining more and more adoption everywhere, it's more likely that an application (Service Account) should act as Tenant Owner. In Capsule, a Tenant can also be owned by a Kubernetes _ServiceAccount_ identity.
As GitOps methodology is gaining more and more adoption everywhere, it's more likely that an application (Service Account) should act as Tenant Owner. In Capsule, a Tenant can also be owned by a Kubernetes _ServiceAccount_ identity.
The tenant manifest is modified as in the following:
+4 -4
View File
@@ -1,8 +1,8 @@
# Use cases for Capsule
Using Capsule, a cluster admin can implement complex multi-tenant scenarios for both public and private deployments. Here a list of common scenarios addressed by Capsule.
Using Capsule, a cluster admin can implement complex multi-tenant scenarios for both public and private deployments. Here is a list of common scenarios addressed by Capsule.
# Container as a Service (CaaS)
***Acme Corp***, our sample organization, built a Container as a Service platform (CaaS), based on Kubernetes to serve multiple lines of business. Each line of business, has its own team of engineers that are responsible for development, deployment, and operating their digital products.
***Acme Corp***, our sample organization, built a Container as a Service platform (CaaS), based on Kubernetes to serve multiple lines of business. Each line of business has its team of engineers that are responsible for the development, deployment, and operating of their digital products.
To simplify the usage of Capsule in this scenario, we'll work with the following actors:
@@ -10,13 +10,13 @@ To simplify the usage of Capsule in this scenario, we'll work with the following
he is the cluster administrator from the operations department of Acme Corp. and he is in charge of administration and maintains the CaaS platform.
* ***Alice***:
she works as the IT Project Leader in the Oil & Gas Business Units. These are two new lines of business at Acme Corp. Alice is responsible for all the strategic IT projects in the two LOB's. She also is responsible for a team made of different job responsibilities (developers, administrators, SRE engineers, etc.) working in separate departments.
she works as the IT Project Leader in the Oil & Gas Business Units. These are two new lines of business at Acme Corp. Alice is responsible for all the strategic IT projects in the two LOBs. She also is responsible for a team made of different job responsibilities (developers, administrators, SRE engineers, etc.) working in separate departments.
* ***Joe***:
he works at Acme Corp, as a lead developer of a distributed team in Alice's organization. Joe is responsible for developing a mission-critical project in the Oil market.
* ***Bob***:
he is the head of Engineering for the Water Business Unit, the main and historical line of business at Acme Corp. He is responsible for development, deployment, and operating multiple digital products in production for a large set of customers.
he is the head of Engineering for the Water Business Unit, the main and historical line of business at Acme Corp. He is responsible for the development, deployment, and operation of multiple digital products in production for a large set of customers.
Use Capsule to address any of the following scenarios:
+3 -3
View File
@@ -1,11 +1,11 @@
# Assign permissions
Alice acts as the tenant admin. Other users can operate inside the tenant with different levels of permissions and authorizations. Alice is responsible for creating additional roles and assigning these roles to other users to work in the same tenant.
One of the key design principles of the Capsule is the self-provisioning management from the tenant owner's perspective. Alice, the tenant owner, does not need to interact with Bill, the cluster admin, to complete her day-by-day duties. On the other side, Bill does not have to deal with multiple requests coming from multiple tenant owners that probably will overwhelm him.
One of the key design principles of the Capsule is self-provisioning management from the tenant owner's perspective. Alice, the tenant owner, does not need to interact with Bill, the cluster admin, to complete her day-by-day duties. On the other side, Bill does not have to deal with multiple requests coming from multiple tenant owners that probably will overwhelm him.
Capsule leaves Alice, and the other tenant owners, the freedom to create RBAC roles at the namespace level, or using the pre-defined cluster roles already available in Kubernetes. Since roles and rolebindings are limited to a namespace scope, Alice can assign the roles to the other users accessing the same tenant only after the namespace is created. This gives Alice the power to administer the tenant without the intervention of the cluster admin.
From the cluster admin perspective, the only required action for Bill is to provision the other identities, eg. `joe` in the Identity Management system. This task can be done once, when onboarding the tenant and the number of users accessing the tenant can be part of the tenant business profile.
From the cluster admin perspective, the only required action for Bill is to provide the other identities, eg. `joe` in the Identity Management system. This task can be done once when onboarding the tenant and the number of users accessing the tenant can be part of the tenant business profile.
Alice can create Roles and RoleBindings only in the namespaces she owns
@@ -51,4 +51,4 @@ no
> Please, note the user `joe`, in the example above, is not acting as tenant owner. He can just operate in `oil-development` namespace as admin.
# Whats next
See how Bill, the cluster admin, set resources quota and limits for Alice's tenant. [Enforce resources quota and limits](./resources-quota-limits.md).
See how Bill, the cluster admin, sets resources quota and limits for Alice's tenant. [Enforce resources quota and limits](./resources-quota-limits.md).
@@ -1,5 +1,5 @@
# Assign Pod Security Policies
Bill, the cluster admin, can assign a dedicated Pod Security Policy (PSP) to the Alice's tenant. This is likely to be a requirement in a multi-tenancy environment.
Bill, the cluster admin, can assign a dedicated Pod Security Policy (PSP) to Alice's tenant. This is likely to be a requirement in a multi-tenancy environment.
The cluster admin creates a PSP:
@@ -80,4 +80,4 @@ roleRef:
With the above example, Capsule is forbidding any authenticated user in `oil-production` namespace to run privileged pods and to perform privilege escalation as declared by the Cluster Role `psp:privileged`.
# Whats next
See how Bill, the cluster admin, can assign to Alice the permissions to create custom resources in her tenant. [Create Custom Resources](./custom-resources.md).
See how Bill, the cluster admin, can assign to Alice the permissions to create custom resources in her tenant. [Create Custom Resources](./custom-resources.md).
@@ -1,5 +1,5 @@
# Enforce resources quota and limits
With help of Capsule, Bill, the cluster admin, can set and enforce resources quota and limits for the Alice's tenant.
With help of Capsule, Bill, the cluster admin, can set and enforce resources quota and limits for Alice's tenant.
## Resources quota
Set resources quota for each namespace in the Alice's tenant by defining them in the tenant spec:
@@ -39,7 +39,7 @@ spec:
EOF
```
The resources quotas above will be inherited by all the namespaces created by Alice. In our case, when Alice creates the namespace `oil-production`, Capsule creates following resource quotas:
The resource quotas above will be inherited by all the namespaces created by Alice. In our case, when Alice creates the namespace `oil-production`, Capsule creates the following resource quotas:
```yaml
kind: ResourceQuota
@@ -93,7 +93,7 @@ At tenant level, the behaviour is controlled by the `spec.resourceQuotas.scope`
* Namespace
### Enforcement at tenant level
By setting enforcement at tenant level, i.e. `spec.resourceQuotas.scope=Tenant`, Capsule aggregates resources usage for all namespaces in the tenant and adjusts all the `ResourceQuota` usage as aggregate. In such case, Alice can check the used resources at tenant level by inspecting the `annotations` in ResourceQuota object of any namespace in the tenant:
By setting enforcement at tenant level, i.e. `spec.resourceQuotas.scope=Tenant`, Capsule aggregates resources usage for all namespaces in the tenant and adjusts all the `ResourceQuota` usage as aggregate. In such case, Alice can check the used resources at the tenant level by inspecting the `annotations` in ResourceQuota object of any namespace in the tenant:
```yaml
kubectl -n oil-production get resourcequotas capsule-oil-1 -o yaml
@@ -119,7 +119,7 @@ metadata:
...
```
When the aggregate usage for all namespaces crosses the hard quota, then the native `ResourceQuota` Admission Controller in Kubernetes denies the Alice's request to create resources exceeding the quota:
When the aggregate usage for all namespaces crosses the hard quota, then the native `ResourceQuota` Admission Controller in Kubernetes denies Alice's request to create resources exceeding the quota:
```
kubectl -n oil-development create deployment nginx --image nginx:latest --replicas 10
@@ -151,11 +151,11 @@ nginx-55649fd747-tkv7m 1/1 Running 0 22m
### Enforcement at namespace level
By setting enforcement at namespace level, i.e. `spec.resourceQuotas.scope=Namespace`, Capsule does not aggregate the resources usage and all enforcement is done at namespace level.
By setting enforcement at the namespace level, i.e. `spec.resourceQuotas.scope=Namespace`, Capsule does not aggregate the resources usage and all enforcement is done at the namespace level.
## Pods and containers limits
Bill, the cluster admin, can also set Limit Ranges for each namespace in the Alice's tenant by defining limits for pods and containers in the tenant spec:
Bill, the cluster admin, can also set Limit Ranges for each namespace in Alice's tenant by defining limits for pods and containers in the tenant spec:
```yaml
apiVersion: capsule.clastix.io/v1alpha1
@@ -234,7 +234,7 @@ spec:
> Note: being the limit range specific of single resources, there is no aggregate to count.
Alice doesn't have permissions to change or delete the resources according to the assigned RBAC profile.
Alice doesn't have permission to change or delete the resources according to the assigned RBAC profile.
```
kubectl -n oil-production auth can-i patch resourcequota
+4 -4
View File
@@ -1,8 +1,8 @@
# Disable Service Types
Bill, the cluster admin, can prevent creation of services with specific service types.
Bill, the cluster admin, can prevent the creation of services with specific service types.
## NodePort
When dealing with a _shared multi-tenant_ scenario, multiple _NodePort_ services can start becoming cumbersome to manage. Reason behind this could be related to the overlapping needs by the Tenant owners, since a _NodePort_ is going to be open on all nodes and, when using `hostNetwork=true`, accessible to any _Pod_ although any specific `NetworkPolicy`.
When dealing with a _shared multi-tenant_ scenario, multiple _NodePort_ services can start becoming cumbersome to manage. The reason behind this could be related to the overlapping needs by the Tenant owners, since a _NodePort_ is going to be open on all nodes and, when using `hostNetwork=true`, accessible to any _Pod_ although any specific `NetworkPolicy`.
Bill, the cluster admin, can block the creation of services with `NodePort` service type for a given tenant
@@ -25,7 +25,7 @@ EOF
With the above configuration, any attempt of Alice to create a Service of type `NodePort` is denied by the Validation Webhook enforcing it. Default value is `true`.
## ExternalName
Service with type of `ExternalName` has been found subject to many security issue. To prevent tenant owners to create services with type of `ExternalName`, the cluster admin can prevent a tenant to create them:
Service with the type of `ExternalName` has been found subject to many security issues. To prevent tenant owners to create services with the type of `ExternalName`, the cluster admin can prevent a tenant to create them:
```yaml
kubectl apply -f - << EOF
@@ -47,7 +47,7 @@ With the above configuration, any attempt of Alice to create a Service of type `
## LoadBalancer
Same as previously, Service of type of `LoadBalancer` could be blocked for various reasons. To prevent tenant owners to create these kinds of services, the cluster admin can prevent a tenant to create them:
Same as previously, the Service of type of `LoadBalancer` could be blocked for various reasons. To prevent tenant owners to create these kinds of services, the cluster admin can prevent a tenant to create them:
```yaml
kubectl apply -f - << EOF
+1 -1
View File
@@ -38,7 +38,7 @@ spec:
EOF
```
Any attempt of Alice to use a non valid Storage Class, or missing it, is denied by the Validation Webhook enforcing it.
Any attempt of Alice to use a non-valid Storage Class, or missing it, is denied by the Validation Webhook enforcing it.
# Whats next
See how Bill, the cluster admin, can assign Network Policies to Alice's tenant. [Assign Network Policies](./network-policies.md).
+1 -1
View File
@@ -1,5 +1,5 @@
# Taint namespaces
With Capsule, Bill can _"taint"_ the namespaces created by Alice with an additional labels and/or annotations. There is no specific semantic assigned to these labels and annotations: they just will be assigned to the namespaces in the tenant as they are created by Alice. This can help the cluster admin to implement specific use cases. As for example, it can be used to implement backup as a service for namespaces in the tenant.
With Capsule, Bill can _"taint"_ the namespaces created by Alice with additional labels and/or annotations. There is no specific semantic assigned to these labels and annotations: they just will be assigned to the namespaces in the tenant as they are created by Alice. This can help the cluster admin to implement specific use cases. As it can be used to implement backup as a service for namespaces in the tenant.
Bill assigns additional labels and annotations to all namespaces created in the `oil` tenant:
+2 -2
View File
@@ -1,5 +1,5 @@
# Taint services
With Capsule, Bill can _"taint"_ the services created by Alice with an additional labels and/or annotations. There is no specific semantic assigned to these labels and annotations: they just will be assigned to the services in the tenant as they are created by Alice. This can help the cluster admin to implement specific use cases.
With Capsule, Bill can _"taint"_ the services created by Alice with additional labels and/or annotations. There is no specific semantic assigned to these labels and annotations: they just will be assigned to the services in the tenant as they are created by Alice. This can help the cluster admin to implement specific use cases.
Bill assigns additional labels and annotations to all services created in the `oil` tenant:
@@ -25,6 +25,6 @@ EOF
When Alice creates a service in a namespace, this will inherit the given label and/or annotation.
# Whats next
This end our tour in Capsule use cases. As we improve Capsule, more use cases about multi-tenancy, policy admission control, and cluster governance will be covered in the future.
This ends our tour in Capsule use cases. As we improve Capsule, more use cases about multi-tenancy, policy admission control, and cluster governance will be covered in the future.
Stay tuned!
@@ -1,8 +1,8 @@
# Velero Backup Restoration
Velero is a backup system that perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.
Velero is a backup system that performs disaster recovery and migrates Kubernetes cluster resources and persistent volumes.
Using this in a Kubernetes cluster where Capsule is installed can lead to an incomplete restore of the cluster's Tenants. This is due to the fact that Velero omits the `ownerReferences` section from the tenant's namespace manifests when backup them.
Using this in a Kubernetes cluster where Capsule is installed can lead to an incomplete restore of the cluster's Tenants. This is because Velero omits the `ownerReferences` section from the tenant's namespace manifests when backup them.
To avoid this problem you can use the script `velero-restore.sh` under the `hack/` folder.
@@ -14,7 +14,7 @@ In case of a data loss, the right thing to do is to restore the cluster with **V
Running this command, we are going to patch the tenant's namespaces manifests that are actually `ownerReferences`-less. Once the command has finished its run, you got the cluster back.
Additionally you can also specify a selected range of tenants to be restored:
Additionally, you can also specify a selected range of tenants to be restored:
```bash
./velero-restore.sh --tenant "gas oil" restore
+8 -4
View File
@@ -8,11 +8,11 @@ You should setup your development environment as following:
> Please, refer to the general coding style rules for Capsule.
### Run locally for test and debug
## Run locally for test and debug
This guide helps new contributors to locally debug in _out or cluster_ mode the project.
1. You need to run a Kind cluster and find the endpoint port of `kind-control-plane` using `docker ps`:
1. You need to run a kind cluster and find the endpoint port of `kind-control-plane` using `docker ps`:
```bash
docker ps
@@ -32,7 +32,11 @@ localhost-key.pem localhost.pem
3. Run the proxy with the following options
```bash
go run main.go --ssl-cert-path=/tmp/localhost.pem --ssl-key-path=/tmp/localhost-key.pem --enable-ssl=true --kubeconfig=<YOUR KUBERNETES CONFIGURATION FILE>
go run main.go \
--ssl-cert-path=/tmp/localhost.pem \
--ssl-key-path=/tmp/localhost-key.pem \
--enable-ssl=true \
--kubeconfig=<YOUR KUBERNETES CONFIGURATION FILE>
```
5. Edit the `KUBECONFIG` file (you should make a copy and work on it) as follows:
@@ -42,7 +46,7 @@ go run main.go --ssl-cert-path=/tmp/localhost.pem --ssl-key-path=/tmp/localhost-
6. Now you should be able to run kubectl using the proxy!
### Debug in a remote Kubernetes cluster
## Debug in a remote Kubernetes cluster
In some cases, you would need to debug the in-cluster mode and [`delve`](https://github.com/go-delve/delve) plays a big role here.
+154
View File
@@ -0,0 +1,154 @@
# OIDC Authentication
The `capsule-proxy` works with `kubectl` users with a token-based authentication, e.g. OIDC or Bearer Token. In the following example, we'll use Keycloak as OIDC server capable to provides JWT tokens.
### Configuring Keycloak
Configure Keycloak as OIDC server:
- Add a realm called `caas`, or use any existing realm instead
- Add a group `capsule.clastix.io`
- Add a user `alice` assigned to group `capsule.clastix.io`
- Add an OIDC client called `kubernetes`
- For the `kubernetes` client, create protocol mappers called `groups` and `audience`
If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as `alice` user:
```
$ KEYCLOAK=sso.clastix.io
$ REALM=caas
$ OIDC_ISSUER=${KEYCLOAK}/auth/realms/${REALM}
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \
-d grant_type=password \
-d response_type=id_token \
-d scope=openid \
-d client_id=${OIDC_CLIENT_ID} \
-d client_secret=${OIDC_CLIENT_SECRET} \
-d username=${USERNAME} \
-d password=${PASSWORD} | jq
```
The result will include an `ACCESS_TOKEN`, a `REFRESH_TOKEN`, and an `ID_TOKEN`. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires.
```json
{
"access_token":"ACCESS_TOKEN",
"refresh_token":"REFRESH_TOKEN",
"id_token": "ID_TOKEN",
"token_type":"bearer",
"scope": "openid groups profile email"
}
```
To introspect the `ID_TOKEN` token run:
```
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \
-d token=${ID_TOKEN} \
--user ${OIDC_CLIENT_ID}:${OIDC_CLIENT_SECRET} | jq
```
The result will be like the following:
```json
{
"exp": 1601323086,
"iat": 1601322186,
"aud": "kubernetes",
"typ": "ID",
"azp": "kubernetes",
"preferred_username": "alice",
"email_verified": false,
"acr": "1",
"groups": [
"capsule.clastix.io"
],
"client_id": "kubernetes",
"username": "alice",
"active": true
}
```
### Configuring Kubernetes API Server
Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) for details and examples. Most likely, your `kube-apiserver.yaml` manifest will looks like the following:
```yaml
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=${OIDC_CLIENT_SECRET}
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
```
### Configuring kubectl
There are two options to use `kubectl` with OIDC:
- OIDC Authenticator
- Use the `--token` option
To use the OIDC Authenticator, add an `oidc` user entry to your `kubeconfig` file:
```
$ kubectl config set-credentials oidc \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \
--auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \
--auth-provider-arg=client-id=${OIDC_CLIENT_ID} \
--auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \
--auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
--auth-provider-arg=id-token=${ID_TOKEN} \
--auth-provider-arg=extra-scopes=groups
```
To use the --token option:
```
$ kubectl config set-credentials oidc --token=${ID_TOKEN}
```
Point the kubectl to the URL where the `capsule-proxy` service is reachable:
```
$ kubectl config set-cluster mycluster \
--server=https://kube.clastix.io \
--certificate-authority=~/.kube/ca.crt
```
Create a new context for the OIDC authenticated users:
```
$ kubectl config set-context alice-oidc@mycluster \
--cluster=mycluster \
--user=oidc
```
As user `alice`, you should be able to use `kubectl` to create some namespaces:
```
$ kubectl --context alice-oidc@mycluster create namespace oil-production
$ kubectl --context alice-oidc@mycluster create namespace oil-development
$ kubectl --context alice-oidc@mycluster create namespace gas-marketing
```
and list only those namespaces:
```
$ kubectl --context alice-oidc@mycluster get namespaces
NAME STATUS AGE
gas-marketing Active 2m
oil-development Active 2m
oil-production Active 2m
```
When logged as cluster-admin power user you should be able to see all namespaces:
```
$ kubectl get namespaces
NAME STATUS AGE
default Active 78d
kube-node-lease Active 78d
kube-public Active 78d
kube-system Active 78d
gas-marketing Active 2m
oil-development Active 2m
oil-production Active 2m
```
_Nota Bene_: once your `ID_TOKEN` expires, the `kubectl` OIDC Authenticator will attempt to refresh automatically your `ID_TOKEN` using the `REFRESH_TOKEN`, the `OIDC_CLIENT_ID` and the `OIDC_CLIENT_SECRET` storing the new values for the `REFRESH_TOKEN` and `ID_TOKEN` in your `kubeconfig` file. In case the OIDC uses a self signed CA certificate, make sure to specify it with the `idp-certificate-authority` option in your `kubeconfig` file, otherwise you'll not able to refresh the tokens. Once the `REFRESH_TOKEN` is expired, you will need to refresh tokens manually.
+6 -4
View File
@@ -107,8 +107,7 @@ oil-production Active 2m
### Nodes
When a Tenant defines a `.spec.nodeSelector`, the nodes matching that labels can be easily retrieved.
The annotation `capsule.clastix.io/enable-node-listing` allows the ability for the owners to retrieve the node list (useful in shared HW scenarios).
The Capsule Proxy gives the owners the ability to access the nodes matching the `.spec.nodeSelector` in the Tenant manifest:
```yaml
apiVersion: capsule.clastix.io/v1beta1
@@ -133,6 +132,8 @@ NAME STATUS ROLES AGE VERSION
capsule-gold-qwerty Ready <none> 43h v1.19.1
```
> Warning: when no `nodeSelector` is specified, the tenant owners has access to all the nodes, according to the permissions listed in the `proxySettings` specs.
### Storage Classes
A Tenant may be limited to use a set of allowed Storage Class resources, as follows.
@@ -327,9 +328,10 @@ oil-development Active 2m
oil-production Active 2m
```
# Whats next
Have a fun with `capsule-proxy`:
* [Standalone Installation](./standalone.md)
* [Sidecar Installation](./sidecar.md)
* [Sidecar Installation](./sidecar.md)
* [OIDC Authentication](./oidc-auth.md)
* [Contributing](./contributing.md)
-156
View File
@@ -55,164 +55,8 @@ $ helm install capsule-proxy clastix/capsule-proxy \
The `capsule-proxy` should be exposed with an Ingress in SSL passthrough mode and reachable at `https://kube.clastix.io`.
## TLS Client Authentication
Users using a TLS client based authentication with certificate and key are able to talks with `capsule-proxy` since the current implementation of the reverse proxy is able to forward client certificates to the Kubernetes APIs server.
## OIDC Authentication
The `capsule-proxy` works with `kubectl` users with a token-based authentication, e.g. OIDC or Bearer Token. In the following example, we'll use Keycloak as OIDC server capable to provides JWT tokens.
### Configuring Keycloak
Configure Keycloak as OIDC server:
- Add a realm called `caas`, or use any existing realm instead
- Add a group `capsule.clastix.io`
- Add a user `alice` assigned to group `capsule.clastix.io`
- Add an OIDC client called `kubernetes`
- For the `kubernetes` client, create protocol mappers called `groups` and `audience`
If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as `alice` user:
```
$ KEYCLOAK=sso.clastix.io
$ REALM=caas
$ OIDC_ISSUER=${KEYCLOAK}/auth/realms/${REALM}
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \
-d grant_type=password \
-d response_type=id_token \
-d scope=openid \
-d client_id=${OIDC_CLIENT_ID} \
-d client_secret=${OIDC_CLIENT_SECRET} \
-d username=${USERNAME} \
-d password=${PASSWORD} | jq
```
The result will include an `ACCESS_TOKEN`, a `REFRESH_TOKEN`, and an `ID_TOKEN`. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires.
```json
{
"access_token":"ACCESS_TOKEN",
"refresh_token":"REFRESH_TOKEN",
"id_token": "ID_TOKEN",
"token_type":"bearer",
"scope": "openid groups profile email"
}
```
To introspect the `ID_TOKEN` token run:
```
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \
-d token=${ID_TOKEN} \
--user ${OIDC_CLIENT_ID}:${OIDC_CLIENT_SECRET} | jq
```
The result will be like the following:
```json
{
"exp": 1601323086,
"iat": 1601322186,
"aud": "kubernetes",
"typ": "ID",
"azp": "kubernetes",
"preferred_username": "alice",
"email_verified": false,
"acr": "1",
"groups": [
"capsule.clastix.io"
],
"client_id": "kubernetes",
"username": "alice",
"active": true
}
```
### Configuring Kubernetes API Server
Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) for details and examples. Most likely, your `kube-apiserver.yaml` manifest will looks like the following:
```yaml
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=${OIDC_CLIENT_SECRET}
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
```
### Configuring kubectl
There are two options to use `kubectl` with OIDC:
- OIDC Authenticator
- Use the `--token` option
To use the OIDC Authenticator, add an `oidc` user entry to your `kubeconfig` file:
```
$ kubectl config set-credentials oidc \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \
--auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \
--auth-provider-arg=client-id=${OIDC_CLIENT_ID} \
--auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \
--auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
--auth-provider-arg=id-token=${ID_TOKEN} \
--auth-provider-arg=extra-scopes=groups
```
To use the --token option:
```
$ kubectl config set-credentials oidc --token=${ID_TOKEN}
```
Point the kubectl to the URL where the `capsule-proxy` service is reachable:
```
$ kubectl config set-cluster mycluster \
--server=https://kube.clastix.io \
--certificate-authority=~/.kube/ca.crt
```
Create a new context for the OIDC authenticated users:
```
$ kubectl config set-context alice-oidc@mycluster \
--cluster=mycluster \
--user=oidc
```
As user `alice`, you should be able to use `kubectl` to create some namespaces:
```
$ kubectl --context alice-oidc@mycluster create namespace oil-production
$ kubectl --context alice-oidc@mycluster create namespace oil-development
$ kubectl --context alice-oidc@mycluster create namespace gas-marketing
```
and list only those namespaces:
```
$ kubectl --context alice-oidc@mycluster get namespaces
NAME STATUS AGE
gas-marketing Active 2m
oil-development Active 2m
oil-production Active 2m
```
When logged as cluster-admin power user you should be able to see all namespaces:
```
$ kubectl get namespaces
NAME STATUS AGE
default Active 78d
kube-node-lease Active 78d
kube-public Active 78d
kube-system Active 78d
gas-marketing Active 2m
oil-development Active 2m
oil-production Active 2m
```
_Nota Bene_: once your `ID_TOKEN` expires, the `kubectl` OIDC Authenticator will attempt to refresh automatically your `ID_TOKEN` using the `REFRESH_TOKEN`, the `OIDC_CLIENT_ID` and the `OIDC_CLIENT_SECRET` storing the new values for the `REFRESH_TOKEN` and `ID_TOKEN` in your `kubeconfig` file. In case the OIDC uses a self signed CA certificate, make sure to specify it with the `idp-certificate-authority` option in your `kubeconfig` file, otherwise you'll not able to refresh the tokens. Once the `REFRESH_TOKEN` is expired, you will need to refresh tokens manually.
## RBAC Considerations
Currently, the service account used for `capsule-proxy` needs to have `cluster-admin` permissions.