Canary Deployments and A/B Testing guide

README.md

@@ -24,6 +24,7 @@ Web API:
 * `GET /readyz` used by Kubernetes readiness probe
 * `POST /readyz/enable` signals the Kubernetes LB that this instance is ready to receive traffic
 * `POST /readyz/disable` signals the Kubernetes LB to stop sending requests to this instance
+* `GET /error` returns code 500 and logs the error
 * `GET /panic` crashes the process with exit code 255
 * `POST /echo` echos the posted content, logs the SHA1 hash of the content
 * `GET /echoheaders` prints the request HTTP headers
@@ -38,3 +39,4 @@ Web API:
 * [Horizontal Pod Auto-scaling](docs/2-autoscaling.md)
 * [Monitoring and alerting with Prometheus](docs/3-monitoring.md)
 * [StatefulSets with local persistent volumes](docs/4-statefulsets.md)
+* [Canary Deployments and A/B Testing](docs/5-canary.md)

docs/5-canary.md

@@ -0,0 +1,203 @@
+# Canary Deployments and A/B Testing
+
+Canary Deployment and A/B testing with Ambassador's Envoy API Gateway. 
+
+![StatefulSets](diagrams/canary.png)
+
+### Deploy Ambassador
+
+Deploy Ambassador and expose it with a LoadBalancer service or NodePort if you are running on-prem:
+
+```bash
+helm repo add sp https://stefanprodan.github.io/k8s-podinfo
+
+helm upgrade --install --wait envoy \
+    --set service.type=LoadBalancer \
+    --namespace default \
+    sp/ambassador
+```
+
+Find the the LoadBalancer IP and store it in an environment variable for later use:
+
+```bash
+export ENVOY=$(kubectl get svc --namespace default envoy-ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+```
+
+Test that Envoy is listening on port 80:
+
+```bash
+curl -sI http://$ENVOY | grep envoy
+server: envoy
+``` 
+
+### Expose services via the API gateway
+
+Deploy podinfo version 0.0.9 as the general available release:
+
+```yaml
+kubectl apply -f ./deploy/canary/ga-dep.yaml
+```
+
+Expose the GA release via Ambassador on the `podinfo.test` domain:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: ga-podinfo
+  annotations:
+    getambassador.io/config: |
+      ---
+      apiVersion: ambassador/v0
+      kind: Mapping
+      name: ga-podinfo
+      prefix: /
+      host: podinfo.test
+      service: ga-podinfo.default:9898
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9898
+      targetPort: 9898
+      protocol: TCP
+  selector:
+    app: ga-podinfo
+```
+
+Apply the GA service:
+
+```bash
+kubectl apply -f ./deploy/canary/ga-svc.yaml
+```
+
+Test that v0.0.9 is available at `podinfo.test` with curl:
+
+```bash
+curl -H 'Host: podinfo.test' -sS http://$ENVOY/version | grep version
+version: 0.0.9
+```
+
+# Canary deployment
+
+Let's assume you have an insiders program for your users and some of them enrolled.
+Once enrolled your users requests will have a HTTP header like `X-User: insider` attached to every request.
+
+Simulate an insider user call with:
+
+```bash
+curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version | grep version
+version: 0.0.9
+```
+
+Deploy podinfo version 0.1.0 as a release candidate:
+
+```bash
+kubectl apply -f ./deploy/canary/canarya-dep.yaml
+```
+
+Create a service named `canarya-podinfo` and instruct Ambassador to shift the insiders to the RC deployment:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: canarya-podinfo
+  annotations:
+    getambassador.io/config: |
+      ---
+      apiVersion: ambassador/v0
+      kind: Mapping
+      name: canarya-podinfo
+      prefix: /
+      host: podinfo.test
+      service: canarya-podinfo.default:9898
+      headers:
+        X-User: insider
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9898
+      targetPort: 9898
+      protocol: TCP
+  selector:
+    app: canarya-podinfo
+```
+
+Apply the service with:
+
+```bash
+kubectl apply -f ./deploy/canary/canarya-svc.yaml
+```
+
+Now if a normal user calls podinfo nothing changed, but if an insider calls podinfo he/she will be routed to version 0.1.0:
+
+```bash
+curl -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version
+version: 0.0.9
+
+curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version
+version: 0.1.0
+```
+
+# A/B testing
+
+Let's assume you have a new release candidate version that you want to test on a small subset of your 
+insiders. 
+
+Deploy podinfo version 0.1.1 as a release candidate:
+
+```bash
+kubectl apply -f ./deploy/canary/canaryb-dep.yaml
+```
+
+Create a service named `canaryb-podinfo` and instruct Ambassador to shift ten percent of
+the insiders traffic to v0.1.1:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: canaryb-podinfo
+  annotations:
+    getambassador.io/config: |
+      ---
+      apiVersion: ambassador/v0
+      kind: Mapping
+      name: canaryb-podinfo
+      prefix: /
+      host: podinfo.test
+      service: canaryb-podinfo.default:9898
+      weight: 10
+      headers:
+        X-User: insider
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9898
+      targetPort: 9898
+      protocol: TCP
+  selector:
+    app: canaryb-podinfo
+```
+
+Apply the service with:
+
+```bash
+kubectl apply -f ./deploy/canary/canaryb-svc.yaml
+```
+
+Now let's call the service in a while loop, one in ten calls will be routed to v0.1.1:
+
+```bash
+while true; do sleep 1; curl -H 'X-User: insider' -H 'Host: podinfo.test' -sS http://$ENVOY/version|grep version; done
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.0
+version: 0.1.1
+```

docs/index.md

@@ -9,3 +9,4 @@ that showcases best practices of running microservices in Kubernetes.
 * [Horizontal Pod Auto-scaling](2-autoscaling.md)
 * [Monitoring and alerting with Prometheus](3-monitoring.md)
 * [StatefulSets with local persistent volumes](4-statefulsets.md)
+* [Canary Deployments and A/B Testing](5-canary.md)