mirror of
https://github.com/fluxcd/flagger.git
synced 2026-04-15 06:57:34 +00:00
346 lines
44 KiB
HTML
346 lines
44 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en-US">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>Linkerd Canary Deployments | Flagger</title>
|
|
<meta name="description" content="Progressive Delivery operator for Kubernetes (Canary, A/B Testing and Blue/Green deployments)">
|
|
<link rel="icon" href="/favicon.png">
|
|
<link rel="stylesheet" href="/website.css">
|
|
<meta name="keywords" content="gitops kubernetes flagger istio linkerd appmesh contour gloo nginx">
|
|
<meta name="twitter:card" content="summary_large_image">
|
|
<meta name="twitter:title" content="Flagger">
|
|
<meta name="twitter:description" content="Progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments)">
|
|
<meta name="twitter:image:src" content="https://flagger.app/flagger-overview.png">
|
|
|
|
<link rel="preload" href="/assets/css/0.styles.a0ada997.css" as="style"><link rel="preload" href="/assets/js/app.9b46ae46.js" as="script"><link rel="preload" href="/assets/js/2.ba5bf77c.js" as="script"><link rel="preload" href="/assets/js/26.b2d4f402.js" as="script"><link rel="prefetch" href="/assets/js/10.bf6f38d4.js"><link rel="prefetch" href="/assets/js/11.3af38cfe.js"><link rel="prefetch" href="/assets/js/12.eca4b89c.js"><link rel="prefetch" href="/assets/js/13.9104349a.js"><link rel="prefetch" href="/assets/js/14.a4f3d810.js"><link rel="prefetch" href="/assets/js/15.aada2caa.js"><link rel="prefetch" href="/assets/js/16.3012a10b.js"><link rel="prefetch" href="/assets/js/17.a98b6aa9.js"><link rel="prefetch" href="/assets/js/18.e5188926.js"><link rel="prefetch" href="/assets/js/19.bb9498f7.js"><link rel="prefetch" href="/assets/js/20.2b1bebf5.js"><link rel="prefetch" href="/assets/js/21.e1068843.js"><link rel="prefetch" href="/assets/js/22.1c5d14b9.js"><link rel="prefetch" href="/assets/js/23.b3dcc355.js"><link rel="prefetch" href="/assets/js/24.4f54d289.js"><link rel="prefetch" href="/assets/js/25.914acd1d.js"><link rel="prefetch" href="/assets/js/27.ea9f1f9c.js"><link rel="prefetch" href="/assets/js/28.deae273d.js"><link rel="prefetch" href="/assets/js/29.ce4011f4.js"><link rel="prefetch" href="/assets/js/3.8b254d5d.js"><link rel="prefetch" href="/assets/js/30.edda19d9.js"><link rel="prefetch" href="/assets/js/31.fbfb336c.js"><link rel="prefetch" href="/assets/js/32.591c3f2b.js"><link rel="prefetch" href="/assets/js/33.c456709f.js"><link rel="prefetch" href="/assets/js/34.5a071e36.js"><link rel="prefetch" href="/assets/js/35.fc27c09e.js"><link rel="prefetch" href="/assets/js/36.144098aa.js"><link rel="prefetch" href="/assets/js/4.26a3069c.js"><link rel="prefetch" href="/assets/js/5.0c32bdf9.js"><link rel="prefetch" href="/assets/js/6.ee5ffca3.js"><link rel="prefetch" href="/assets/js/7.6bf27a87.js"><link rel="prefetch" href="/assets/js/8.8b72a8d3.js"><link rel="prefetch" href="/assets/js/9.7919fc1d.js">
|
|
<link rel="stylesheet" href="/assets/css/0.styles.a0ada997.css">
|
|
</head>
|
|
<body>
|
|
<div id="app" data-server-rendered="true"><div class="theme-container"><header class="navbar"><div class="sidebar-button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"><path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path></svg></div> <a href="/" class="home-link router-link-active"><!----> <span class="site-name">Flagger</span></a> <div class="links"><div class="search-box"><input aria-label="Search" autocomplete="off" spellcheck="false" value=""> <!----></div> <nav class="nav-links can-hide"><div class="nav-item"><a href="https://docs.flagger.app" target="_blank" rel="noopener noreferrer" class="nav-link external">
|
|
Docs
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div><div class="nav-item"><a href="https://github.com/weaveworks/flagger/blob/master/CHANGELOG.md" target="_blank" rel="noopener noreferrer" class="nav-link external">
|
|
Changelog
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div> <a href="https://github.com/weaveworks/flagger" target="_blank" rel="noopener noreferrer" class="repo-link">
|
|
GitHub
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></nav></div></header> <div class="sidebar-mask"></div> <aside class="sidebar"><nav class="nav-links"><div class="nav-item"><a href="https://docs.flagger.app" target="_blank" rel="noopener noreferrer" class="nav-link external">
|
|
Docs
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div><div class="nav-item"><a href="https://github.com/weaveworks/flagger/blob/master/CHANGELOG.md" target="_blank" rel="noopener noreferrer" class="nav-link external">
|
|
Changelog
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div> <a href="https://github.com/weaveworks/flagger" target="_blank" rel="noopener noreferrer" class="repo-link">
|
|
GitHub
|
|
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></nav> <ul class="sidebar-links"><li><a href="/" class="sidebar-link">Home</a></li><li><section class="sidebar-group depth-0"><a href="/intro/" class="sidebar-heading clickable"><span>Introduction</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/intro/" class="sidebar-link">Get Started</a></li><li><a href="/intro/faq.html" class="sidebar-link">FAQ</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/install/flagger-install-on-kubernetes" class="sidebar-heading clickable"><span>Install</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/install/flagger-install-on-kubernetes.html" class="sidebar-link">On Kubernetes</a></li><li><a href="/install/flagger-install-on-google-cloud.html" class="sidebar-link">On GKE Istio</a></li><li><a href="/install/flagger-install-on-eks-appmesh.html" class="sidebar-link">On EKS App Mesh</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/usage/how-it-works" class="sidebar-heading clickable"><span>Usage</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/usage/how-it-works.html" class="sidebar-link">How it works</a></li><li><a href="/usage/deployment-strategies.html" class="sidebar-link">Deployment Strategies</a></li><li><a href="/usage/metrics.html" class="sidebar-link">Metrics Analysis</a></li><li><a href="/usage/webhooks.html" class="sidebar-link">Webhooks</a></li><li><a href="/usage/alerting.html" class="sidebar-link">Alerting</a></li><li><a href="/usage/monitoring.html" class="sidebar-link">Monitoring</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/tutorials/istio-progressive-delivery" class="sidebar-heading clickable open"><span>Tutorials</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/tutorials/istio-progressive-delivery.html" class="sidebar-link">Istio Canaries</a></li><li><a href="/tutorials/istio-ab-testing.html" class="sidebar-link">Istio A/B Testing</a></li><li><a href="/tutorials/linkerd-progressive-delivery.html" class="active sidebar-link">Linkerd Canaries</a><ul class="sidebar-sub-headers"><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#prerequisites" class="sidebar-link">Prerequisites</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#bootstrap" class="sidebar-link">Bootstrap</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#automated-canary-promotion" class="sidebar-link">Automated canary promotion</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#automated-rollback" class="sidebar-link">Automated rollback</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#custom-metrics" class="sidebar-link">Custom metrics</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#linkerd-ingress" class="sidebar-link">Linkerd Ingress</a></li><li class="sidebar-sub-header"><a href="/tutorials/linkerd-progressive-delivery.html#a-b-testing" class="sidebar-link">A/B Testing</a></li></ul></li><li><a href="/tutorials/appmesh-progressive-delivery.html" class="sidebar-link">App Mesh Canaries</a></li><li><a href="/tutorials/nginx-progressive-delivery.html" class="sidebar-link">NGINX Ingress Canaries</a></li><li><a href="/tutorials/gloo-progressive-delivery.html" class="sidebar-link">Gloo Canaries</a></li><li><a href="/tutorials/contour-progressive-delivery.html" class="sidebar-link">Contour Canaries</a></li><li><a href="/tutorials/kubernetes-blue-green.html" class="sidebar-link">Kubernetes Blue/Green</a></li><li><a href="/tutorials/canary-helm-gitops.html" class="sidebar-link">Canaries with Helm charts and GitOps</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/dev/dev-guide" class="sidebar-heading clickable"><span>Dev</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/dev/dev-guide.html" class="sidebar-link">Development Guide</a></li><li><a href="/dev/release-guide.html" class="sidebar-link">Release Guide</a></li><li><a href="/dev/upgrade-guide.html" class="sidebar-link">Upgrade Guide</a></li></ul></section></li></ul> </aside> <main class="page"> <div class="theme-default-content content__default"><h1 id="linkerd-canary-deployments"><a href="#linkerd-canary-deployments" aria-hidden="true" class="header-anchor">#</a> Linkerd Canary Deployments</h1> <p>This guide shows you how to use Linkerd and Flagger to automate canary deployments.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-linkerd-traffic-split.png" alt="Flagger Linkerd Traffic Split"></p> <h2 id="prerequisites"><a href="#prerequisites" aria-hidden="true" class="header-anchor">#</a> Prerequisites</h2> <p>Flagger requires a Kubernetes cluster <strong>v1.11</strong> or newer and Linkerd <strong>2.4</strong> or newer.</p> <p>Install Flagger in the linkerd namespace:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -k github.com/weaveworks/flagger//kustomize/linkerd
|
|
</code></pre></div><p>Note that you'll need kubectl 1.14 or newer to run the above command.</p> <h2 id="bootstrap"><a href="#bootstrap" aria-hidden="true" class="header-anchor">#</a> Bootstrap</h2> <p>Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
|
|
then creates a series of objects (Kubernetes deployments, ClusterIP services and SMI traffic split).
|
|
These objects expose the application inside the mesh and drive the canary analysis and promotion.</p> <p>Create a test namespace and enable Linkerd proxy injection:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl create ns <span class="token builtin class-name">test</span>
|
|
kubectl annotate namespace <span class="token builtin class-name">test</span> linkerd.io/inject<span class="token operator">=</span>enabled
|
|
</code></pre></div><p>Install the load testing service to generate traffic during the canary analysis:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -k github.com/weaveworks/flagger//kustomize/tester
|
|
</code></pre></div><p>Create a deployment and a horizontal pod autoscaler:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
|
|
</code></pre></div><p>Create a canary custom resource for the podinfo deployment:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> flagger.app/v1beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Canary
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
|
|
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># deployment reference</span>
|
|
<span class="token key atrule">targetRef</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> apps/v1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Deployment
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token comment"># HPA reference (optional)</span>
|
|
<span class="token key atrule">autoscalerRef</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> autoscaling/v2beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> HorizontalPodAutoscaler
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token comment"># the maximum time in seconds for the canary deployment</span>
|
|
<span class="token comment"># to make progress before it is rollback (default 600s)</span>
|
|
<span class="token key atrule">progressDeadlineSeconds</span><span class="token punctuation">:</span> <span class="token number">60</span>
|
|
<span class="token key atrule">service</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># ClusterIP port number</span>
|
|
<span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">9898</span>
|
|
<span class="token comment"># container port number or name (optional)</span>
|
|
<span class="token key atrule">targetPort</span><span class="token punctuation">:</span> <span class="token number">9898</span>
|
|
<span class="token key atrule">analysis</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># schedule interval (default 60s)</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 30s
|
|
<span class="token comment"># max number of failed metric checks before rollback</span>
|
|
<span class="token key atrule">threshold</span><span class="token punctuation">:</span> <span class="token number">5</span>
|
|
<span class="token comment"># max traffic percentage routed to canary</span>
|
|
<span class="token comment"># percentage (0-100)</span>
|
|
<span class="token key atrule">maxWeight</span><span class="token punctuation">:</span> <span class="token number">50</span>
|
|
<span class="token comment"># canary increment step</span>
|
|
<span class="token comment"># percentage (0-100)</span>
|
|
<span class="token key atrule">stepWeight</span><span class="token punctuation">:</span> <span class="token number">5</span>
|
|
<span class="token comment"># Linkerd Prometheus checks</span>
|
|
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>success<span class="token punctuation">-</span>rate
|
|
<span class="token comment"># minimum req success rate (non 5xx responses)</span>
|
|
<span class="token comment"># percentage (0-100)</span>
|
|
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">min</span><span class="token punctuation">:</span> <span class="token number">99</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>duration
|
|
<span class="token comment"># maximum req duration P99</span>
|
|
<span class="token comment"># milliseconds</span>
|
|
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">max</span><span class="token punctuation">:</span> <span class="token number">500</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 30s
|
|
<span class="token comment"># testing (optional)</span>
|
|
<span class="token key atrule">webhooks</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> acceptance<span class="token punctuation">-</span>test
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> pre<span class="token punctuation">-</span>rollout
|
|
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
|
|
<span class="token key atrule">timeout</span><span class="token punctuation">:</span> 30s
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> bash
|
|
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">"curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> load<span class="token punctuation">-</span>test
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> rollout
|
|
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">"hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"</span>
|
|
</code></pre></div><p>Save the above resource as podinfo-canary.yaml and then apply it:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -f ./podinfo-canary.yaml
|
|
</code></pre></div><p>When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
|
|
The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every half a minute.</p> <p>After a couple of seconds Flagger will create the canary objects:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token comment"># applied</span>
|
|
deployment.apps/podinfo
|
|
horizontalpodautoscaler.autoscaling/podinfo
|
|
ingresses.extensions/podinfo
|
|
canary.flagger.app/podinfo
|
|
|
|
<span class="token comment"># generated</span>
|
|
deployment.apps/podinfo-primary
|
|
horizontalpodautoscaler.autoscaling/podinfo-primary
|
|
service/podinfo
|
|
service/podinfo-canary
|
|
service/podinfo-primary
|
|
trafficsplits.split.smi-spec.io/podinfo
|
|
</code></pre></div><p>After the boostrap, the podinfo deployment will be scaled to zero and the traffic to <code>podinfo.test</code>
|
|
will be routed to the primary pods.
|
|
During the canary analysis, the <code>podinfo-canary.test</code> address can be used to target directly the canary pods.</p> <h2 id="automated-canary-promotion"><a href="#automated-canary-promotion" aria-hidden="true" class="header-anchor">#</a> Automated canary promotion</h2> <p>Flagger implements a control loop that gradually shifts traffic to the canary while measuring
|
|
key performance indicators like HTTP requests success rate, requests average duration and pod health.
|
|
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png" alt="Flagger Canary Stages"></p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
|
|
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.1
|
|
</code></pre></div><p>Flagger detects that the deployment revision changed and starts a new rollout:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
|
|
|
|
Status:
|
|
Canary Weight: 0
|
|
Failed Checks: 0
|
|
Phase: Succeeded
|
|
Events:
|
|
New revision detected! Scaling up podinfo.test
|
|
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
|
|
Pre-rollout check acceptance-test passed
|
|
Advance podinfo.test canary weight 5
|
|
Advance podinfo.test canary weight 10
|
|
Advance podinfo.test canary weight 15
|
|
Advance podinfo.test canary weight 20
|
|
Advance podinfo.test canary weight 25
|
|
Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
|
|
Advance podinfo.test canary weight 30
|
|
Advance podinfo.test canary weight 35
|
|
Advance podinfo.test canary weight 40
|
|
Advance podinfo.test canary weight 45
|
|
Advance podinfo.test canary weight 50
|
|
Copying podinfo.test template spec to podinfo-primary.test
|
|
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
|
|
Promotion completed! Scaling down podinfo.test
|
|
</code></pre></div><p><strong>Note</strong> that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.</p> <p>A canary deployment is triggered by changes in any of the following objects:</p> <ul><li>Deployment PodSpec (container image, command, ports, env, resources, etc)</li> <li>ConfigMaps mounted as volumes or mapped to environment variables</li> <li>Secrets mounted as volumes or mapped to environment variables</li></ul> <p>You can monitor all canaries with:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> kubectl get canaries --all-namespaces
|
|
|
|
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
|
|
<span class="token builtin class-name">test</span> podinfo Progressing <span class="token number">15</span> <span class="token number">2019</span>-06-30T14:05:07Z
|
|
prod frontend Succeeded <span class="token number">0</span> <span class="token number">2019</span>-06-30T16:15:07Z
|
|
prod backend Failed <span class="token number">0</span> <span class="token number">2019</span>-06-30T17:05:07Z
|
|
</code></pre></div><h2 id="automated-rollback"><a href="#automated-rollback" aria-hidden="true" class="header-anchor">#</a> Automated rollback</h2> <p>During the canary analysis you can generate HTTP 500 errors and high latency to
|
|
test if Flagger pauses and rolls back the faulted version.</p> <p>Trigger another canary deployment:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
|
|
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.2
|
|
</code></pre></div><p>Exec into the load tester pod with:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">exec</span> -it flagger-loadtester-xx-xx <span class="token function">sh</span>
|
|
</code></pre></div><p>Generate HTTP 500 errors:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> -n <span class="token number">1</span> <span class="token function">curl</span> http://podinfo-canary.test:9898/status/500
|
|
</code></pre></div><p>Generate latency:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> -n <span class="token number">1</span> <span class="token function">curl</span> http://podinfo-canary.test:9898/delay/1
|
|
</code></pre></div><p>When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
|
|
the canary is scaled to zero and the rollout is marked as failed.</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
|
|
|
|
Status:
|
|
Canary Weight: 0
|
|
Failed Checks: 10
|
|
Phase: Failed
|
|
Events:
|
|
Starting canary analysis for podinfo.test
|
|
Pre-rollout check acceptance-test passed
|
|
Advance podinfo.test canary weight 5
|
|
Advance podinfo.test canary weight 10
|
|
Advance podinfo.test canary weight 15
|
|
Halt podinfo.test advancement success rate 69.17% < 99%
|
|
Halt podinfo.test advancement success rate 61.39% < 99%
|
|
Halt podinfo.test advancement success rate 55.06% < 99%
|
|
Halt podinfo.test advancement request duration 1.20s > 0.5s
|
|
Halt podinfo.test advancement request duration 1.45s > 0.5s
|
|
Rolling back podinfo.test failed checks threshold reached 5
|
|
Canary failed! Scaling down podinfo.test
|
|
</code></pre></div><h2 id="custom-metrics"><a href="#custom-metrics" aria-hidden="true" class="header-anchor">#</a> Custom metrics</h2> <p>The canary analysis can be extended with Prometheus queries.</p> <p>Let's a define a check for not found errors. Edit the canary analysis and add the following metric:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code> <span class="token key atrule">analysis</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> <span class="token string">"404s percentage"</span>
|
|
<span class="token key atrule">threshold</span><span class="token punctuation">:</span> <span class="token number">3</span>
|
|
<span class="token key atrule">query</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">
|
|
100 - sum(
|
|
rate(
|
|
response_total{
|
|
namespace="test",
|
|
deployment="podinfo",
|
|
status_code!="404",
|
|
direction="inbound"
|
|
}[1m]
|
|
)
|
|
)
|
|
/
|
|
sum(
|
|
rate(
|
|
response_total{
|
|
namespace="test",
|
|
deployment="podinfo",
|
|
direction="inbound"
|
|
}[1m]
|
|
)
|
|
)
|
|
* 100</span>
|
|
</code></pre></div><p>The above configuration validates the canary version by checking if the HTTP 404 req/sec percentage
|
|
is below three percent of the total traffic.
|
|
If the 404s rate reaches the 3% threshold, then the analysis is aborted and the canary is marked as failed.</p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
|
|
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.3
|
|
</code></pre></div><p>Generate 404s:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> -n <span class="token number">1</span> <span class="token function">curl</span> http://podinfo-canary:9898/status/404
|
|
</code></pre></div><p>Watch Flagger logs:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n linkerd logs deployment/flagger -f | jq .msg
|
|
|
|
Starting canary deployment for podinfo.test
|
|
Pre-rollout check acceptance-test passed
|
|
Advance podinfo.test canary weight 5
|
|
Halt podinfo.test advancement 404s percentage 6.20 > 3
|
|
Halt podinfo.test advancement 404s percentage 6.45 > 3
|
|
Halt podinfo.test advancement 404s percentage 7.22 > 3
|
|
Halt podinfo.test advancement 404s percentage 6.50 > 3
|
|
Halt podinfo.test advancement 404s percentage 6.34 > 3
|
|
Rolling back podinfo.test failed checks threshold reached 5
|
|
Canary failed! Scaling down podinfo.test
|
|
</code></pre></div><p>If you have Slack configured, Flagger will send a notification with the reason why the canary failed.</p> <h2 id="linkerd-ingress"><a href="#linkerd-ingress" aria-hidden="true" class="header-anchor">#</a> Linkerd Ingress</h2> <p>There are two ingress controllers that are compatible with both Flagger and Linkerd: NGINX and Gloo.</p> <p>Install NGINX:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>helm upgrade -i nginx-ingress stable/nginx-ingress <span class="token punctuation">\</span>
|
|
--namespace ingress-nginx
|
|
</code></pre></div><p>Create an ingress definition for podinfo that rewrites the incoming header
|
|
to the internal service name (required by Linkerd):</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> extensions/v1beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Ingress
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
|
|
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">app</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">annotations</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">kubernetes.io/ingress.class</span><span class="token punctuation">:</span> <span class="token string">"nginx"</span>
|
|
<span class="token key atrule">nginx.ingress.kubernetes.io/configuration-snippet</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">
|
|
proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:9898;
|
|
proxy_hide_header l5d-remote-ip;
|
|
proxy_hide_header l5d-server-id;</span>
|
|
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">rules</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">host</span><span class="token punctuation">:</span> app.example.com
|
|
<span class="token key atrule">http</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">paths</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">backend</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">serviceName</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">servicePort</span><span class="token punctuation">:</span> <span class="token number">9898</span>
|
|
</code></pre></div><p>When using an ingress controller, the Linkerd traffic split does not apply to incoming traffic
|
|
since NGINX in running outside of the mesh. In order to run a canary analysis for a frontend app,
|
|
Flagger creates a shadow ingress and sets the NGINX specific annotations.</p> <h2 id="a-b-testing"><a href="#a-b-testing" aria-hidden="true" class="header-anchor">#</a> A/B Testing</h2> <p>Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
|
|
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
|
|
This is particularly useful for frontend applications that require session affinity.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-nginx-linkerd.png" alt="Flagger Linkerd Ingress"></p> <p>Edit podinfo canary analysis, set the provider to <code>nginx</code>, add the ingress reference,
|
|
remove the max/step weight and add the match conditions and iterations:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> flagger.app/v1beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Canary
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
|
|
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># ingress reference</span>
|
|
<span class="token key atrule">provider</span><span class="token punctuation">:</span> nginx
|
|
<span class="token key atrule">ingressRef</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> extensions/v1beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Ingress
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">targetRef</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> apps/v1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Deployment
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">autoscalerRef</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> autoscaling/v2beta1
|
|
<span class="token key atrule">kind</span><span class="token punctuation">:</span> HorizontalPodAutoscaler
|
|
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
|
|
<span class="token key atrule">service</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># container port</span>
|
|
<span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">9898</span>
|
|
<span class="token key atrule">analysis</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
|
|
<span class="token key atrule">threshold</span><span class="token punctuation">:</span> <span class="token number">10</span>
|
|
<span class="token key atrule">iterations</span><span class="token punctuation">:</span> <span class="token number">10</span>
|
|
<span class="token key atrule">match</span><span class="token punctuation">:</span>
|
|
<span class="token comment"># curl -H 'X-Canary: always' http://app.example.com</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">headers</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">x-canary</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">exact</span><span class="token punctuation">:</span> <span class="token string">"always"</span>
|
|
<span class="token comment"># curl -b 'canary=always' http://app.example.com</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">headers</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">cookie</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">exact</span><span class="token punctuation">:</span> <span class="token string">"canary"</span>
|
|
<span class="token comment"># Linkerd Prometheus checks</span>
|
|
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>success<span class="token punctuation">-</span>rate
|
|
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">min</span><span class="token punctuation">:</span> <span class="token number">99</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>duration
|
|
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">max</span><span class="token punctuation">:</span> <span class="token number">500</span>
|
|
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 30s
|
|
<span class="token key atrule">webhooks</span><span class="token punctuation">:</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> acceptance<span class="token punctuation">-</span>test
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> pre<span class="token punctuation">-</span>rollout
|
|
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
|
|
<span class="token key atrule">timeout</span><span class="token punctuation">:</span> 30s
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> bash
|
|
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">"curl -sd 'test' http://podinfo-canary:9898/token | grep token"</span>
|
|
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> load<span class="token punctuation">-</span>test
|
|
<span class="token key atrule">type</span><span class="token punctuation">:</span> rollout
|
|
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
|
|
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
|
|
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">"hey -z 2m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com"</span>
|
|
</code></pre></div><p>The above configuration will run an analysis for ten minutes targeting users that have
|
|
a <code>canary</code> cookie set to <code>always</code> or those that call the service using the <code>X-Canary: always</code> header.</p> <p><strong>Note</strong> that the load test now targets the external address and uses the canary cookie.</p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
|
|
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.4
|
|
</code></pre></div><p>Flagger detects that the deployment revision changed and starts the A/B testing:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
|
|
|
|
Events:
|
|
Starting canary deployment for podinfo.test
|
|
Pre-rollout check acceptance-test passed
|
|
Advance podinfo.test canary iteration 1/10
|
|
Advance podinfo.test canary iteration 2/10
|
|
Advance podinfo.test canary iteration 3/10
|
|
Advance podinfo.test canary iteration 4/10
|
|
Advance podinfo.test canary iteration 5/10
|
|
Advance podinfo.test canary iteration 6/10
|
|
Advance podinfo.test canary iteration 7/10
|
|
Advance podinfo.test canary iteration 8/10
|
|
Advance podinfo.test canary iteration 9/10
|
|
Advance podinfo.test canary iteration 10/10
|
|
Copying podinfo.test template spec to podinfo-primary.test
|
|
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
|
|
Promotion completed! Scaling down podinfo.test
|
|
</code></pre></div><p>The above procedure can be extended with <a href="/usage/metrics.html">custom metrics</a> checks,
|
|
<a href="/usage/webhooks.html">webhooks</a>,
|
|
<a href="/usage/webhooks.html#manual-gating">manual promotion</a> approval and
|
|
<a href="/usage/alerting.html">Slack or MS Teams</a> notifications.</p></div> <footer class="page-edit"><!----> <!----></footer> <div class="page-nav"><p class="inner"><span class="prev">
|
|
←
|
|
<a href="/tutorials/istio-ab-testing.html" class="prev">
|
|
Istio A/B Testing
|
|
</a></span> <span class="next"><a href="/tutorials/appmesh-progressive-delivery.html">
|
|
App Mesh Canaries
|
|
</a>
|
|
→
|
|
</span></p></div> </main></div><div class="global-ui"></div></div>
|
|
<script src="/assets/js/app.9b46ae46.js" defer></script><script src="/assets/js/2.ba5bf77c.js" defer></script><script src="/assets/js/26.b2d4f402.js" defer></script>
|
|
</body>
|
|
</html>
|