diff --git a/slides/k8s/aws-eks.md b/slides/k8s/aws-eks.md index 3cb870cb..2ca6e067 100644 --- a/slides/k8s/aws-eks.md +++ b/slides/k8s/aws-eks.md @@ -1,27 +1,554 @@ -# EKS +# Amazon EKS -- Strategies to provision clusters (eksctl) +- Elastic Kubernetes Service -- (2FA?) +- AWS runs the Kubernetes control plane -- IAM + (all we see is an API server endpoint) -- Direct auth vs. role-bound auth +- Pods can run on any combination of: -- Mapping users/roles to Kubernetes users + - EKS-managed nodes -- LoadBalancer controller, TLS, HTTP, WSS + - self-managed nodes -- ALB + - Fargate -- ExternalIP +- Leverages and integrates with AWS services and APIs -- CNI +--- + +## Some integrations + +- Authenticate with IAM users and roles + +- Associate IAM roles to Kubernetes ServiceAccounts + +- Load balance traffic with ALB/ELB/NLB + +- Persist data with EBS/EFS + +- Label nodes with instance ID, instance type, region, AZ ... + +- Pods can be "first class citizens" of VPC + +--- + +## Pros/cons + +- Fully managed control plane + +- Handles deployment, upgrade, scaling of the control plane + +- Available versions and features tend to lag a bit + +- Doesn't fit the most demanding users + + ("demanding" starts somewhere between 100 and 1000 nodes) + +--- + +## Good to know ... + +- Some integrations are specific to EKS + + (some authentication models) + +- Many integrations are *not* specific to EKS + +- The Cloud Controller Manager can run outside of EKS + + (and provide LoadBalancer services, EBS volumes, and more) + +--- + +# Provisioning clusters + +- AWS console, API, CLI + +- `eksctl` + +- Infrastructure-as-Code + +--- + +## AWS "native" provisioning + +- AWS web console + + - click-click-click! + + - difficulty: low + +- AWS API or CLI + + - must provide subnets, ARNs + + - difficulty: medium + +--- + +## `eksctl` + +- Originally developed by Weave + + (back when AWS "native" provisioning wasn't very good) + +- `eksctl create cluster` just works™ + +- Has been "adopted" by AWS + + (is listed in official documentations) + +--- + +## Infrastructure-as-Code + +- Cloud Formation + +- Terraform + + [terraform-aws-eks](https://github.com/terraform-aws-modules/terraform-aws-eks) + by the community + ([example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/basic)) + + [terraform-provider-aws](https://github.com/hashicorp/terraform-provider-aws) + by Hashicorp + ([example](https://github.com/hashicorp/terraform-provider-aws/tree/main/examples/eks-getting-started)) + + [Kubestack](https://www.kubestack.com/) + +--- + +# IAM → EKS authentication + +- Access EKS clusters using IAM users and roles + +- No special role, permission, or policy is needed in IAM + +- Users and roles need to be explicitly listed in the cluster + +- Configuration is done through a ConfigMap in the cluster + +--- + +## Setting it up + +- Nothing to do when creating the cluster + + (feature is always enabled) + +- Users and roles are *mapped* to Kubernetes users and groups + + (through the `aws-auth` ConfigMap in `kube-system`) + +- That's it! + +--- + +## Mapping + +- The `aws-auth` ConfigMap can contain two entries: + + - `mapRoles` (map IAM roles) + + - `mapUsers` (map IAM users) + +- Each entry is a YAML file + +- Each entry includes: + + - `rolearn` or `userarn` to map + + - `username` (as a string) + + - `groups` (as a list; can be empty) + +--- + +## Example + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + namespace: kube-system + name: aws-auth +data: + mapRoles: `|` + - rolearn: arn:aws:iam::111122223333:role/blah + username: blah + groups: [ devs, ops ] + mapUsers: `|` + - userarn: arn:aws:iam::111122223333:user/alice + username: alice + groups: [ system:masters ] + - userarn: arn:aws:iam::111122223333:user/bob + username: bob + groups: [ system:masters ] +``` + +--- + +## Client setup + +- We need either the `aws` CLI or the `aws-iam-authenticator` + +- We use them as `exec` plugins in `~/.kube/config` + +- Done automatically by `eksctl` + +- Or manually with `aws eks update-kubeconfig` + +--- + +class: extra-details + +## How it works + +- The helper generates a token + + (with `aws eks get-token` or `aws-iam-authenticator token`) + +- Note: these calls will always succeed! + + (even if AWS API keys are invalid) + +- The token is used to authenticate with the Kubernetes API + +- AWS' Kubernetes API server will decode and validate the token + + (and map the underlying user or role accordingly) + +--- + +## Read The Fine Manual + +https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html + +--- + +# EKS → IAM authentication + +- Access AWS services from workloads running on EKS + + (e.g.: access S3 bucket from code running in a Pod) + +- This works by associating an IAM role to a K8S ServiceAccount + +- There are also a few specific roles used internally by EKS + + (e.g. to let the nodes establish network configurations) + +- ... We won't talk about these + +--- + +## The big picture + +- One-time setup task + + ([create and associate an OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html)) + +- Annotate service accounts to map them to a role + + `eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/some-iam-role` + +- Create (or re-create) pods using that ServiceAccount + +- The pods can now use that role! + +--- + +## The little details + +- When pods are created, they are processed by a mutating webhook + + (typically named `pod-identity-webhook`) + +- Pods using a ServiceAccount with the right annotation get: + + - an extra token +
+ (mounted in `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`) + + - a few env vars +
+ (`AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN`) + +- AWS client libraries and tooling will work this that + + (see [this list](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html) for supported versions) + +--- + +# CNI + +- EKS is a compliant Kubernetes implementation + + (which means we can use a wide range of CNI plugins) + +- However, the recommended CNI plugin is the "AWS VPC CNI" + + (https://github.com/aws/amazon-vpc-cni-k8s) + +- Pods are then "first class citizens" of AWS VPC + +--- + +## AWS VPC CNI + +- Each Pod gets an address in a VPC subnet + +- No overlay network, no encapsulation, no overhead + + (other than AWS network fabric, obviously) + +- Probably the fastest network option when running on AWS + +- Allows "direct" load balancing (more on that later) + +- Can use security groups with Pod traffic + +- But: limits the number of Pods per Node + +- But: more complex configuration (more on that later) + +--- + +## Number of Pods per Node + +- Each Pod gets an IP address on an ENI + + (Elastic Network Interface) + +- EC2 instances can only have a limited number of ENIs + + (the exact limit depends on the instance type) + +- ENIs can only have a limited number of IP addresses + + (with variations here as well) + +- This gives limits of e.g. 35 pods on `t3.large`, 29 on `c5.large` ... + + (see + [full list of limits per instance type](https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt +) + and + [ENI/IP details](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/awsutils/vpc_ip_resource_limit.go +)) + +--- + +## Limits? + +- These limits might seem low + +- They're not *that* low if you compute e.g. the RAM/Pod ratio + +- Except if you're running lots if tiny pods + +- Bottom line: do the math! + +--- + +class: extra-details + +## Pre-loading + +- It can take a little while to allocate/attach an ENI + +- The AWS VPC CNI can keep a few extra addresses on each Node + + (by default, one ENI worth of IP addresses) + +- This is tunable if needed + + (see [the docs](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md +) for details) + +--- + +## Better load balancing + +- The default path for inbound traffic is: + + Load balancer → NodePort → Pod + +- With the AWS VPC CNI, it becomes possible to do: + + Load balancer → Pod + +- More on that in the load balancing section! + +--- + +## Configuration complexity + +- The AWS VPC CNI is a very good solution when running EKS + +- It brings optimized solutions to various use-cases: + + - direct load balancing + - user authentication + - interconnection with other infrastructure + - etc. + +- Keep in mind that all these solutions are AWS-specific + +- They can require a non-trivial amount of specific configuration + +- Especially when moving from a simple POC to an IAC deployment! + +--- + +# Load Balancers + +- Here be dragons! + +- Multiple options, each with different pros/cons + +- It's necessary to know both AWS products and K8S concepts + +--- + +## AWS load balancers + +- CLB / Classic Load Balancer (formerly known as ELB) + + - can work in L4 (TCP) or L7 (HTTP) mode + - can do TLS unrolling + - can't do websockets, HTTP/2, content-based routing ... + +- NLB / Network Load Balancer + + - high-performance L4 load balancer with TLS support + +- ALB / Application Load Balancer + + - HTTP load balancer + - can do TLS unrolling + - can do websockets, HTTP/2, content-based routing ... + +--- + +## Load balancing modes + +- "IP targets" + + - send traffic directly from LB to Pods + + - Pods must use the AWS VPC CNI + + - compatible with Fargate Pods + +- "Instance targets" + + - send traffic to a NodePort (generally incurs an extra hop) + + - Pods can use any CNI + + - not compatible with Fargate Pods + +- Each LB (Service) can use a different mode, if necessary + +--- + +## Kubernetes load balancers + +- Service (L4) + + - ClusterIP: internal load balancing + - NodePort: external load balancing on ports >30000 + - LoadBalancer: external load balancing on the port you want + - ExternalIP: external load balancing directly on nodes + +- Ingress (L7 HTTP) + + - partial content-based routing (`Host` header, request path) + - requires an Ingress Controller (in front) + - works with Services (in back) + +--- + +## Two controllers are available + +- Kubernetes "in-tree" load balancer controller + + - always available + - used by default for LoadBalancer Services + - creates CLB by default; can also do NLB + - can only do "instance targets" + - can use extra CLB features (TLS, HTTP) + +- AWS Load Balancer Controller (fka AWS ALB Ingress Controller) + + - optional add-on (requires additional config) + - primarily meant to be an Ingress Controller + - creates NLB and ALB + - can do "instance targets" and "IP targets" + - can also be used for LoadBalancer Services with type `nlb-ip` + +- They can run side by side + +--- + +## Which one should we use? + +- AWS Load Balancer Controller supports "IP targets" + + (which means direct routing of traffic to Pods) + +- It can be used as an Ingress controller + +- It *seems* to be the perfect solution for EKS! + +- However ... + +--- + +## Caveats + +- AWS Load Balancer Controller requires extensive configuration + + - a few hours to a few days to get it to work in a POC ... + + - a few days to a few weeks to industrialize that process? + +- It's AWS-specific + +- It still introduces an extra hop, even if that hop is invisible + +- Other ingress controllers can have interesting features + + (canary deployment, A/B testing ...) + +--- + +## Noteworthy annotations and docs + +- LoadBalancer Service with "IP targets" ([docs](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/nlb_ip_mode/)) + + `service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip` + +- Internal load balancer (for private VPC) + + `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` + +- Opt for NLB instead of CLB with in-tree controller + + `service.beta.kubernetes.io/aws-load-balancer-type: nlb` -- Cluster autoscaler ??? +## Cluster autoscaling + +## Logging + +https://docs.aws.amazon.com/eks/latest/userguide/logging-using-cloudtrail.html + :EN:- Working with EKS :EN:- Cluster and user provisioning :EN:- Networking and load balancing