github/container.training

Fork 0

mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-19 04:49:19 +00:00

Files

Jerome Petazzoni ed5009c769 Tweaks after Caen

2020-01-30 14:44:44 -06:00

16 KiB

Raw Permalink Blame History

Exposing HTTP services with Ingress resources

Services give us a way to access a pod or a set of pods
Services can be exposed to the outside world:
- with type NodePort (on a port >30000)
- with type LoadBalancer (allocating an external load balancer)
What about HTTP services?
- how can we expose webui, rng, hasher?
- the Kubernetes dashboard?
- a new version of webui?

Exposing HTTP services

If we use NodePort services, clients have to specify port numbers

(i.e. http://xxxxx:31234 instead of just http://xxxxx)
LoadBalancer services are nice, but:
- they are not available in all environments
- they often carry an additional cost (e.g. they provision an ELB)
- they require one extra step for DNS integration
  (waiting for the LoadBalancer to be provisioned; then adding it to DNS)
We could build our own reverse proxy

Building a custom reverse proxy

There are many options available:

Apache, HAProxy, Hipache, NGINX, Traefik, ...

(look at jpetazzo/aiguillage for a minimal reverse proxy configuration using NGINX)
Most of these options require us to update/edit configuration files after each change
Some of them can pick up virtual hosts and backends from a configuration store
Wouldn't it be nice if this configuration could be managed with the Kubernetes API?

Enter.red[¹] Ingress resources!

Ingress resources

Kubernetes API resource (kubectl get ingress/ingresses/ing)
Designed to expose HTTP services
Basic features:
- load balancing
- SSL termination
- name-based virtual hosting
Can also route to different services depending on:
- URI path (e.g. /api→api-service, /static→assets-service)
- Client headers, including cookies (for A/B testing, canary deployment...)
- and more!

Principle of operation

Step 1: deploy an ingress controller
- ingress controller = load balancer + control loop
- the control loop watches over ingress resources, and configures the LB accordingly
Step 2: set up DNS
- associate DNS entries with the load balancer address
Step 3: create ingress resources
- the ingress controller picks up these resources and configures the LB
Step 4: profit!

Ingress in action

We will deploy the Traefik ingress controller
- this is an arbitrary choice
- maybe motivated by the fact that Traefik releases are named after cheeses
For DNS, we will use nip.io
- *.1.2.3.4.nip.io resolves to 1.2.3.4
We will create ingress resources for various HTTP services

Deploying pods listening on port 80

We want our ingress load balancer to be available on port 80
The best way to do that would be with a LoadBalancer service

... but it requires support from the underlying infrastructure
Instead, we are going to use the hostNetwork mode on the Traefik pods
Let's see what this hostNetwork mode is about ...

Without `hostNetwork`

Normally, each pod gets its own network namespace

(sometimes called sandbox or network sandbox)
An IP address is assigned to the pod
This IP address is routed/connected to the cluster network
All containers of that pod are sharing that network namespace

(and therefore using the same IP address)

With `hostNetwork: true`

No network namespace gets created
The pod is using the network namespace of the host
It "sees" (and can use) the interfaces (and IP addresses) of the host
The pod can receive outside traffic directly, on any port
Downside: with most network plugins, network policies won't work for that pod
- most network policies work at the IP address level
- filtering that pod = filtering traffic from the node

Other techniques to expose port 80

We could use pods specifying hostPort: 80

... but with most CNI plugins, this doesn't work or requires additional setup
We could use a NodePort service

... but that requires changing the --service-node-port-range flag in the API server
We could create a service with an external IP

... this would work, but would require a few extra steps

(figuring out the IP address and adding it to the service)

Running Traefik

The Traefik documentation tells us to pick between Deployment and Daemon Set
We are going to use a Daemon Set so that each node can accept connections
We will do two minor changes to the YAML provided by Traefik:
- enable hostNetwork
- add a toleration so that Traefik also runs on node1

Taints and tolerations

A taint is an attribute added to a node
It prevents pods from running on the node
... Unless they have a matching toleration
When deploying with kubeadm:
- a taint is placed on the node dedicated to the control plane
- the pods running the control plane have a matching toleration

Checking taints on our nodes

Check our nodes specs:

kubectl get node node1 -o json | jq .spec
kubectl get node node2 -o json | jq .spec

]

We should see a result only for node1 (the one with the control plane):

  "taints": [
    {
      "effect": "NoSchedule",
      "key": "node-role.kubernetes.io/master"
    }
  ]

Understanding a taint

The key can be interpreted as:
- a reservation for a special set of pods
  (here, this means "this node is reserved for the control plane")
- an error condition on the node
  (for instance: "disk full," do not start new pods here!)
The effect can be:
- NoSchedule (don't run new pods here)
- PreferNoSchedule (try not to run new pods here)
- NoExecute (don't run new pods and evict running pods)

Checking tolerations on the control plane

Check tolerations for CoreDNS:

kubectl -n kube-system get deployments coredns -o json |
        jq .spec.template.spec.tolerations

]

The result should include:

  {
    "effect": "NoSchedule",
    "key": "node-role.kubernetes.io/master"
  }

It means: "bypass the exact taint that we saw earlier on node1."

Special tolerations

Check tolerations on kube-proxy:

kubectl -n kube-system get ds kube-proxy -o json | 
        jq .spec.template.spec.tolerations

]

The result should include:

  {
    "operator": "Exists"
  }

This one is a special case that means "ignore all taints and run anyway."

Running Traefik on our cluster

We provide a YAML file (k8s/traefik.yaml) which is essentially the sum of:
- Traefik's Daemon Set resources (patched with hostNetwork and tolerations)
- Traefik's RBAC rules allowing it to watch necessary API objects

Apply the YAML:

kubectl apply -f ~/container.training/k8s/traefik.yaml

]

Checking that Traefik runs correctly

If Traefik started correctly, we now have a web server listening on each node

Check that Traefik is serving 80/tcp:
```
curl localhost
```

]

We should get a 404 page not found error.

This is normal: we haven't provided any ingress rule yet.

Setting up DNS

To make our lives easier, we will use nip.io
Check out http://cheddar.A.B.C.D.nip.io

(replacing A.B.C.D with the IP address of node1)
We should get the same 404 page not found error

(meaning that our DNS is "set up properly", so to speak!)

Traefik web UI

Traefik provides a web dashboard
With the current install method, it's listening on port 8080

Go to http://node1:8080 (replacing node1 with its IP address)

]

Setting up host-based routing ingress rules

We are going to use errm/cheese images

(there are 3 tags available: wensleydale, cheddar, stilton)
These images contain a simple static HTTP server sending a picture of cheese
We will run 3 deployments (one for each cheese)
We will create 3 services (one for each deployment)
Then we will create 3 ingress rules (one for each service)
We will route <name-of-cheese>.A.B.C.D.nip.io to the corresponding deployment

Running cheesy web servers

Run all three deployments:

kubectl create deployment cheddar --image=errm/cheese:cheddar
kubectl create deployment stilton --image=errm/cheese:stilton
kubectl create deployment wensleydale --image=errm/cheese:wensleydale

Create a service for each of them:

kubectl expose deployment cheddar --port=80
kubectl expose deployment stilton --port=80
kubectl expose deployment wensleydale --port=80

]

What does an ingress resource look like?

Here is a minimal host-based ingress resource:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: cheddar
spec:
  rules:
  - host: cheddar.`A.B.C.D`.nip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: cheddar
          servicePort: 80

(It is in k8s/ingress.yaml.)

Creating our first ingress resources

Edit the file ~/container.training/k8s/ingress.yaml
Replace A.B.C.D with the IP address of node1
Apply the file
Open http://cheddar.A.B.C.D.nip.io

]

(An image of a piece of cheese should show up.)

Creating the other ingress resources

Edit the file ~/container.training/k8s/ingress.yaml
Replace cheddar with stilton (in name, host, serviceName)
Apply the file
Check that stilton.A.B.C.D.nip.io works correctly
Repeat for wensleydale

]

Using multiple ingress controllers

You can have multiple ingress controllers active simultaneously

(e.g. Traefik and NGINX)
You can even have multiple instances of the same controller

(e.g. one for internal, another for external traffic)
The kubernetes.io/ingress.class annotation can be used to tell which one to use
It's OK if multiple ingress controllers configure the same resource

(it just means that the service will be accessible through multiple paths)

Ingress: the good

The traffic flows directly from the ingress load balancer to the backends
- it doesn't need to go through the ClusterIP
- in fact, we don't even need a ClusterIP (we can use a headless service)
The load balancer can be outside of Kubernetes

(as long as it has access to the cluster subnet)
This allows the use of external (hardware, physical machines...) load balancers
Annotations can encode special features

(rate-limiting, A/B testing, session stickiness, etc.)

Ingress: the bad

Aforementioned "special features" are not standardized yet
Some controllers will support them; some won't
Even relatively common features (stripping a path prefix) can differ:
- traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
- ingress.kubernetes.io/rewrite-target: /
This should eventually stabilize

(remember that ingresses are currently apiVersion: networking.k8s.io/v1beta1)

A special feature in action

We're going to see how to implement canary releases with Traefik
This feature is available on multiple ingress controllers
... But it is configured very differently on each of them

Canary releases

A canary release (or canary launch or canary deployment) is a release that will process only a small fraction of the workload
After deploying the canary, we compare its metrics to the normal release
If the metrics look good, the canary will progressively receive more traffic

(until it gets 100% and becomes the new normal release)
If the metrics aren't good, the canary is automatically removed
When we deploy a bad release, only a tiny fraction of traffic is affected

Various ways to implement canary

Example 1: canary for a microservice
- 1% of all requests (sampled randomly) are sent to the canary
- the remaining 99% are sent to the normal release
Example 2: canary for a web app
- 1% of users are sent to the canary web site
- the remaining 99% are sent to the normal release
Example 3: canary for shipping physical goods
- 1% of orders are shipped with the canary process
- the reamining 99% are shipped with the normal process
We're going to implement example 1 (per-request routing)

Canary releases with Traefik

We need to deploy the canary and expose it with a separate service
Then, in the Ingress resource, we need:
- multiple paths entries (one for each service, canary and normal)
- an extra annotation indicating the weight of each service
If we want, we can send requests to more than 2 services
Let's send requests to our 3 cheesy services!

Create the resource shown on the next slide

]

The Ingress resource

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: cheeseplate
  annotations:
    traefik.ingress.kubernetes.io/service-weights: |
      cheddar: 50%
      wensleydale: 25%
      stilton: 25%
spec:
  rules:
  - host: cheeseplate.`A.B.C.D`.nip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: cheddar
          servicePort: 80
      - path: /
        backend:
          serviceName: wensledale
          servicePort: 80
      - path: /
        backend:
          serviceName: stilton
          servicePort: 80

]

Testing the canary

Let's check the percentage of requests going to each service

Continuously send HTTP requests to the new ingress:

  while sleep 0.1; do
    curl -s http://cheeseplate.A.B.C.D.nip.io/
  done

]

We should see a 50/25/25 request mix.

Load balancing fairness

Note: if we use odd request ratios, the load balancing algorithm might appear to be broken on a small scale (when sending a small number of requests), but on a large scale (with many requests) it will be fair.

For instance, with a 11%/89% ratio, we can see 79 requests going to the 89%-weighted service, and then requests alternating between the two services; then 79 requests again, etc.

Other ingress controllers

Just to illustrate how different things are ...

With the NGINX ingress controller:
- define two ingress ressources
  (specifying rules with the same host+path)
- add nginx.ingress.kubernetes.io/canary annotations on each
With Linkerd2:
- define two services
- define an extra service for the weighted aggregate of the two
- define a TrafficSplit (this is a CRD introduced by the SMI spec)

We need more than that

What we saw is just one of the multiple building blocks that we need to achieve a canary release.

We also need:

metrics (latency, performance ...) for our releases
automation to alter canary weights

(increase canary weight if metrics look good; decrease otherwise)
a mechanism to manage the lifecycle of the canary releases

(create them, promote them, delete them ...)

For inspiration, check flagger by Weave.

16 KiB Raw Permalink Blame History