From ad835c8deda3c5a51569d72d0d678232bd3e8b16 Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Sun, 18 Mar 2018 14:54:01 +0200 Subject: [PATCH] Canary Deployments and A/B Testing guide --- README.md | 2 + docs/5-canary.md | 203 +++++++++++++++++++++++++++++++++++++++++++++++ docs/index.md | 1 + 3 files changed, 206 insertions(+) create mode 100644 docs/5-canary.md diff --git a/README.md b/README.md index 7131ccb..2e30f65 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,7 @@ Web API: * `GET /readyz` used by Kubernetes readiness probe * `POST /readyz/enable` signals the Kubernetes LB that this instance is ready to receive traffic * `POST /readyz/disable` signals the Kubernetes LB to stop sending requests to this instance +* `GET /error` returns code 500 and logs the error * `GET /panic` crashes the process with exit code 255 * `POST /echo` echos the posted content, logs the SHA1 hash of the content * `GET /echoheaders` prints the request HTTP headers @@ -38,3 +39,4 @@ Web API: * [Horizontal Pod Auto-scaling](docs/2-autoscaling.md) * [Monitoring and alerting with Prometheus](docs/3-monitoring.md) * [StatefulSets with local persistent volumes](docs/4-statefulsets.md) +* [Canary Deployments and A/B Testing](docs/5-canary.md) diff --git a/docs/5-canary.md b/docs/5-canary.md new file mode 100644 index 0000000..774b921 --- /dev/null +++ b/docs/5-canary.md @@ -0,0 +1,203 @@ +# Canary Deployments and A/B Testing + +Canary Deployment and A/B testing with Ambassador's Envoy API Gateway. + +![StatefulSets](diagrams/canary.png) + +### Deploy Ambassador + +Deploy Ambassador and expose it with a LoadBalancer service or NodePort if you are running on-prem: + +```bash +helm repo add sp https://stefanprodan.github.io/k8s-podinfo + +helm upgrade --install --wait envoy \ + --set service.type=LoadBalancer \ + --namespace default \ + sp/ambassador +``` + +Find the the LoadBalancer IP and store it in an environment variable for later use: + +```bash +export ENVOY=$(kubectl get svc --namespace default envoy-ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +``` + +Test that Envoy is listening on port 80: + +```bash +curl -sI http://$ENVOY | grep envoy +server: envoy +``` + +### Expose services via the API gateway + +Deploy podinfo version 0.0.9 as the general available release: + +```yaml +kubectl apply -f ./deploy/canary/ga-dep.yaml +``` + +Expose the GA release via Ambassador on the `podinfo.test` domain: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: ga-podinfo + annotations: + getambassador.io/config: | + --- + apiVersion: ambassador/v0 + kind: Mapping + name: ga-podinfo + prefix: / + host: podinfo.test + service: ga-podinfo.default:9898 +spec: + type: ClusterIP + ports: + - port: 9898 + targetPort: 9898 + protocol: TCP + selector: + app: ga-podinfo +``` + +Apply the GA service: + +```bash +kubectl apply -f ./deploy/canary/ga-svc.yaml +``` + +Test that v0.0.9 is available at `podinfo.test` with curl: + +```bash +curl -H 'Host: podinfo.test' -sS http://$ENVOY/version | grep version +version: 0.0.9 +``` + +# Canary deployment + +Let's assume you have an insiders program for your users and some of them enrolled. +Once enrolled your users requests will have a HTTP header like `X-User: insider` attached to every request. + +Simulate an insider user call with: + +```bash +curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version | grep version +version: 0.0.9 +``` + +Deploy podinfo version 0.1.0 as a release candidate: + +```bash +kubectl apply -f ./deploy/canary/canarya-dep.yaml +``` + +Create a service named `canarya-podinfo` and instruct Ambassador to shift the insiders to the RC deployment: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: canarya-podinfo + annotations: + getambassador.io/config: | + --- + apiVersion: ambassador/v0 + kind: Mapping + name: canarya-podinfo + prefix: / + host: podinfo.test + service: canarya-podinfo.default:9898 + headers: + X-User: insider +spec: + type: ClusterIP + ports: + - port: 9898 + targetPort: 9898 + protocol: TCP + selector: + app: canarya-podinfo +``` + +Apply the service with: + +```bash +kubectl apply -f ./deploy/canary/canarya-svc.yaml +``` + +Now if a normal user calls podinfo nothing changed, but if an insider calls podinfo he/she will be routed to version 0.1.0: + +```bash +curl -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version +version: 0.0.9 + +curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version +version: 0.1.0 +``` + +# A/B testing + +Let's assume you have a new release candidate version that you want to test on a small subset of your +insiders. + +Deploy podinfo version 0.1.1 as a release candidate: + +```bash +kubectl apply -f ./deploy/canary/canaryb-dep.yaml +``` + +Create a service named `canaryb-podinfo` and instruct Ambassador to shift ten percent of +the insiders traffic to v0.1.1: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: canaryb-podinfo + annotations: + getambassador.io/config: | + --- + apiVersion: ambassador/v0 + kind: Mapping + name: canaryb-podinfo + prefix: / + host: podinfo.test + service: canaryb-podinfo.default:9898 + weight: 10 + headers: + X-User: insider +spec: + type: ClusterIP + ports: + - port: 9898 + targetPort: 9898 + protocol: TCP + selector: + app: canaryb-podinfo +``` + +Apply the service with: + +```bash +kubectl apply -f ./deploy/canary/canaryb-svc.yaml +``` + +Now let's call the service in a while loop, one in ten calls will be routed to v0.1.1: + +```bash +while true; do sleep 1; curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version; done +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.0 +version: 0.1.1 +``` diff --git a/docs/index.md b/docs/index.md index b16af97..810cf42 100644 --- a/docs/index.md +++ b/docs/index.md @@ -9,3 +9,4 @@ that showcases best practices of running microservices in Kubernetes. * [Horizontal Pod Auto-scaling](2-autoscaling.md) * [Monitoring and alerting with Prometheus](3-monitoring.md) * [StatefulSets with local persistent volumes](4-statefulsets.md) +* [Canary Deployments and A/B Testing](5-canary.md)