github/flagger
1
0
Fork 0
mirror of https://github.com/fluxcd/flagger.git synced 2026-04-15 06:57:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c9c97c379ddcb7be537aec9c8e609c133ab8aec7
flagger/tutorials/nginx-progressive-delivery.html
T
weaveworksbot c9c97c379d Publish website
2020-08-20 06:16:09 +00:00

325 lines
42 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="en-US">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>NGINX Canary Deployments | Flagger</title>
<meta name="description" content="Progressive Delivery operator for Kubernetes (Canary, A/B Testing and Blue/Green deployments)">
<link rel="icon" href="/favicon.png">
<link rel="stylesheet" href="/website.css">
<meta name="keywords" content="gitops kubernetes flagger istio linkerd appmesh contour gloo nginx">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="Flagger">
<meta name="twitter:description" content="Progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments)">
<meta name="twitter:image:src" content="https://flagger.app/flagger-overview.png">
<link rel="preload" href="/assets/css/0.styles.a0ada997.css" as="style"><link rel="preload" href="/assets/js/app.9b46ae46.js" as="script"><link rel="preload" href="/assets/js/2.ba5bf77c.js" as="script"><link rel="preload" href="/assets/js/27.ea9f1f9c.js" as="script"><link rel="prefetch" href="/assets/js/10.bf6f38d4.js"><link rel="prefetch" href="/assets/js/11.3af38cfe.js"><link rel="prefetch" href="/assets/js/12.eca4b89c.js"><link rel="prefetch" href="/assets/js/13.9104349a.js"><link rel="prefetch" href="/assets/js/14.a4f3d810.js"><link rel="prefetch" href="/assets/js/15.aada2caa.js"><link rel="prefetch" href="/assets/js/16.3012a10b.js"><link rel="prefetch" href="/assets/js/17.a98b6aa9.js"><link rel="prefetch" href="/assets/js/18.e5188926.js"><link rel="prefetch" href="/assets/js/19.bb9498f7.js"><link rel="prefetch" href="/assets/js/20.2b1bebf5.js"><link rel="prefetch" href="/assets/js/21.e1068843.js"><link rel="prefetch" href="/assets/js/22.1c5d14b9.js"><link rel="prefetch" href="/assets/js/23.b3dcc355.js"><link rel="prefetch" href="/assets/js/24.4f54d289.js"><link rel="prefetch" href="/assets/js/25.914acd1d.js"><link rel="prefetch" href="/assets/js/26.b2d4f402.js"><link rel="prefetch" href="/assets/js/28.deae273d.js"><link rel="prefetch" href="/assets/js/29.ce4011f4.js"><link rel="prefetch" href="/assets/js/3.8b254d5d.js"><link rel="prefetch" href="/assets/js/30.edda19d9.js"><link rel="prefetch" href="/assets/js/31.fbfb336c.js"><link rel="prefetch" href="/assets/js/32.591c3f2b.js"><link rel="prefetch" href="/assets/js/33.c456709f.js"><link rel="prefetch" href="/assets/js/34.5a071e36.js"><link rel="prefetch" href="/assets/js/35.fc27c09e.js"><link rel="prefetch" href="/assets/js/36.144098aa.js"><link rel="prefetch" href="/assets/js/4.26a3069c.js"><link rel="prefetch" href="/assets/js/5.0c32bdf9.js"><link rel="prefetch" href="/assets/js/6.ee5ffca3.js"><link rel="prefetch" href="/assets/js/7.6bf27a87.js"><link rel="prefetch" href="/assets/js/8.8b72a8d3.js"><link rel="prefetch" href="/assets/js/9.7919fc1d.js">
<link rel="stylesheet" href="/assets/css/0.styles.a0ada997.css">
</head>
<body>
<div id="app" data-server-rendered="true"><div class="theme-container"><header class="navbar"><div class="sidebar-button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"><path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path></svg></div> <a href="/" class="home-link router-link-active"><!----> <span class="site-name">Flagger</span></a> <div class="links"><div class="search-box"><input aria-label="Search" autocomplete="off" spellcheck="false" value=""> <!----></div> <nav class="nav-links can-hide"><div class="nav-item"><a href="https://docs.flagger.app" target="_blank" rel="noopener noreferrer" class="nav-link external">
Docs
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div><div class="nav-item"><a href="https://github.com/weaveworks/flagger/blob/master/CHANGELOG.md" target="_blank" rel="noopener noreferrer" class="nav-link external">
Changelog
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div> <a href="https://github.com/weaveworks/flagger" target="_blank" rel="noopener noreferrer" class="repo-link">
GitHub
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></nav></div></header> <div class="sidebar-mask"></div> <aside class="sidebar"><nav class="nav-links"><div class="nav-item"><a href="https://docs.flagger.app" target="_blank" rel="noopener noreferrer" class="nav-link external">
Docs
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div><div class="nav-item"><a href="https://github.com/weaveworks/flagger/blob/master/CHANGELOG.md" target="_blank" rel="noopener noreferrer" class="nav-link external">
Changelog
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></div> <a href="https://github.com/weaveworks/flagger" target="_blank" rel="noopener noreferrer" class="repo-link">
GitHub
<svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg></a></nav> <ul class="sidebar-links"><li><a href="/" class="sidebar-link">Home</a></li><li><section class="sidebar-group depth-0"><a href="/intro/" class="sidebar-heading clickable"><span>Introduction</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/intro/" class="sidebar-link">Get Started</a></li><li><a href="/intro/faq.html" class="sidebar-link">FAQ</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/install/flagger-install-on-kubernetes" class="sidebar-heading clickable"><span>Install</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/install/flagger-install-on-kubernetes.html" class="sidebar-link">On Kubernetes</a></li><li><a href="/install/flagger-install-on-google-cloud.html" class="sidebar-link">On GKE Istio</a></li><li><a href="/install/flagger-install-on-eks-appmesh.html" class="sidebar-link">On EKS App Mesh</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/usage/how-it-works" class="sidebar-heading clickable"><span>Usage</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/usage/how-it-works.html" class="sidebar-link">How it works</a></li><li><a href="/usage/deployment-strategies.html" class="sidebar-link">Deployment Strategies</a></li><li><a href="/usage/metrics.html" class="sidebar-link">Metrics Analysis</a></li><li><a href="/usage/webhooks.html" class="sidebar-link">Webhooks</a></li><li><a href="/usage/alerting.html" class="sidebar-link">Alerting</a></li><li><a href="/usage/monitoring.html" class="sidebar-link">Monitoring</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/tutorials/istio-progressive-delivery" class="sidebar-heading clickable open"><span>Tutorials</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/tutorials/istio-progressive-delivery.html" class="sidebar-link">Istio Canaries</a></li><li><a href="/tutorials/istio-ab-testing.html" class="sidebar-link">Istio A/B Testing</a></li><li><a href="/tutorials/linkerd-progressive-delivery.html" class="sidebar-link">Linkerd Canaries</a></li><li><a href="/tutorials/appmesh-progressive-delivery.html" class="sidebar-link">App Mesh Canaries</a></li><li><a href="/tutorials/nginx-progressive-delivery.html" class="active sidebar-link">NGINX Ingress Canaries</a><ul class="sidebar-sub-headers"><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#prerequisites" class="sidebar-link">Prerequisites</a></li><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#bootstrap" class="sidebar-link">Bootstrap</a></li><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#automated-canary-promotion" class="sidebar-link">Automated canary promotion</a></li><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#automated-rollback" class="sidebar-link">Automated rollback</a></li><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#custom-metrics" class="sidebar-link">Custom metrics</a></li><li class="sidebar-sub-header"><a href="/tutorials/nginx-progressive-delivery.html#a-b-testing" class="sidebar-link">A/B Testing</a></li></ul></li><li><a href="/tutorials/gloo-progressive-delivery.html" class="sidebar-link">Gloo Canaries</a></li><li><a href="/tutorials/contour-progressive-delivery.html" class="sidebar-link">Contour Canaries</a></li><li><a href="/tutorials/kubernetes-blue-green.html" class="sidebar-link">Kubernetes Blue/Green</a></li><li><a href="/tutorials/canary-helm-gitops.html" class="sidebar-link">Canaries with Helm charts and GitOps</a></li></ul></section></li><li><section class="sidebar-group depth-0"><a href="/dev/dev-guide" class="sidebar-heading clickable"><span>Dev</span> <!----></a> <ul class="sidebar-links sidebar-group-items"><li><a href="/dev/dev-guide.html" class="sidebar-link">Development Guide</a></li><li><a href="/dev/release-guide.html" class="sidebar-link">Release Guide</a></li><li><a href="/dev/upgrade-guide.html" class="sidebar-link">Upgrade Guide</a></li></ul></section></li></ul> </aside> <main class="page"> <div class="theme-default-content content__default"><h1 id="nginx-canary-deployments"><a href="#nginx-canary-deployments" aria-hidden="true" class="header-anchor">#</a> NGINX Canary Deployments</h1> <p>This guide shows you how to use the NGINX ingress controller and Flagger to automate canary deployments and A/B testing.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-nginx-overview.png" alt="Flagger NGINX Ingress Controller"></p> <h2 id="prerequisites"><a href="#prerequisites" aria-hidden="true" class="header-anchor">#</a> Prerequisites</h2> <p>Flagger requires a Kubernetes cluster <strong>v1.11</strong> or newer and NGINX ingress <strong>0.24</strong> or newer.</p> <p>Install NGINX with Helm v3:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl create ns ingress-nginx
helm upgrade -i nginx-ingress stable/nginx-ingress <span class="token punctuation">\</span>
--namespace ingress-nginx <span class="token punctuation">\</span>
--set controller.stats.enabled<span class="token operator">=</span>true <span class="token punctuation">\</span>
--set controller.metrics.enabled<span class="token operator">=</span>true <span class="token punctuation">\</span>
--set controller.podAnnotations.<span class="token string">&quot;prometheus\.io/scrape&quot;</span><span class="token operator">=</span>true <span class="token punctuation">\</span>
--set controller.podAnnotations.<span class="token string">&quot;prometheus\.io/port&quot;</span><span class="token operator">=</span><span class="token number">10254</span>
</code></pre></div><p>Install Flagger and the Prometheus add-on in the same namespace as NGINX:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>helm repo <span class="token function">add</span> flagger https://flagger.app
helm upgrade -i flagger flagger/flagger <span class="token punctuation">\</span>
--namespace ingress-nginx <span class="token punctuation">\</span>
--set prometheus.install<span class="token operator">=</span>true <span class="token punctuation">\</span>
--set <span class="token assign-left variable">meshProvider</span><span class="token operator">=</span>nginx
</code></pre></div><p>Optionally you can enable Slack notifications:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>helm upgrade -i flagger flagger/flagger <span class="token punctuation">\</span>
--reuse-values <span class="token punctuation">\</span>
--namespace ingress-nginx <span class="token punctuation">\</span>
--set slack.url<span class="token operator">=</span>https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK <span class="token punctuation">\</span>
--set slack.channel<span class="token operator">=</span>general <span class="token punctuation">\</span>
--set slack.user<span class="token operator">=</span>flagger
</code></pre></div><h2 id="bootstrap"><a href="#bootstrap" aria-hidden="true" class="header-anchor">#</a> Bootstrap</h2> <p>Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and canary ingress).
These objects expose the application outside the cluster and drive the canary analysis and promotion.</p> <p>Create a test namespace:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl create ns <span class="token builtin class-name">test</span>
</code></pre></div><p>Create a deployment and a horizontal pod autoscaler:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
</code></pre></div><p>Deploy the load testing service to generate traffic during the canary analysis:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>helm upgrade -i flagger-loadtester flagger/loadtester <span class="token punctuation">\</span>
--namespace<span class="token operator">=</span>test
</code></pre></div><p>Create an ingress definition (replace <code>app.example.com</code> with your own domain):</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> extensions/v1beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Ingress
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
<span class="token key atrule">labels</span><span class="token punctuation">:</span>
<span class="token key atrule">app</span><span class="token punctuation">:</span> podinfo
<span class="token key atrule">annotations</span><span class="token punctuation">:</span>
<span class="token key atrule">kubernetes.io/ingress.class</span><span class="token punctuation">:</span> <span class="token string">&quot;nginx&quot;</span>
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">rules</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">host</span><span class="token punctuation">:</span> app.example.com
<span class="token key atrule">http</span><span class="token punctuation">:</span>
<span class="token key atrule">paths</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">backend</span><span class="token punctuation">:</span>
<span class="token key atrule">serviceName</span><span class="token punctuation">:</span> podinfo
<span class="token key atrule">servicePort</span><span class="token punctuation">:</span> <span class="token number">80</span>
</code></pre></div><p>Save the above resource as podinfo-ingress.yaml and then apply it:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -f ./podinfo-ingress.yaml
</code></pre></div><p>Create a canary custom resource (replace <code>app.example.com</code> with your own domain):</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> flagger.app/v1beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Canary
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">provider</span><span class="token punctuation">:</span> nginx
<span class="token comment"># deployment reference</span>
<span class="token key atrule">targetRef</span><span class="token punctuation">:</span>
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> apps/v1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Deployment
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
<span class="token comment"># ingress reference</span>
<span class="token key atrule">ingressRef</span><span class="token punctuation">:</span>
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> extensions/v1beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> Ingress
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
<span class="token comment"># HPA reference (optional)</span>
<span class="token key atrule">autoscalerRef</span><span class="token punctuation">:</span>
<span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> autoscaling/v2beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> HorizontalPodAutoscaler
<span class="token key atrule">name</span><span class="token punctuation">:</span> podinfo
<span class="token comment"># the maximum time in seconds for the canary deployment</span>
<span class="token comment"># to make progress before it is rollback (default 600s)</span>
<span class="token key atrule">progressDeadlineSeconds</span><span class="token punctuation">:</span> <span class="token number">60</span>
<span class="token key atrule">service</span><span class="token punctuation">:</span>
<span class="token comment"># ClusterIP port number</span>
<span class="token key atrule">port</span><span class="token punctuation">:</span> <span class="token number">80</span>
<span class="token comment"># container port number or name</span>
<span class="token key atrule">targetPort</span><span class="token punctuation">:</span> <span class="token number">9898</span>
<span class="token key atrule">analysis</span><span class="token punctuation">:</span>
<span class="token comment"># schedule interval (default 60s)</span>
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 10s
<span class="token comment"># max number of failed metric checks before rollback</span>
<span class="token key atrule">threshold</span><span class="token punctuation">:</span> <span class="token number">10</span>
<span class="token comment"># max traffic percentage routed to canary</span>
<span class="token comment"># percentage (0-100)</span>
<span class="token key atrule">maxWeight</span><span class="token punctuation">:</span> <span class="token number">50</span>
<span class="token comment"># canary increment step</span>
<span class="token comment"># percentage (0-100)</span>
<span class="token key atrule">stepWeight</span><span class="token punctuation">:</span> <span class="token number">5</span>
<span class="token comment"># NGINX Prometheus checks</span>
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>success<span class="token punctuation">-</span>rate
<span class="token comment"># minimum req success rate (non 5xx responses)</span>
<span class="token comment"># percentage (0-100)</span>
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
<span class="token key atrule">min</span><span class="token punctuation">:</span> <span class="token number">99</span>
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
<span class="token comment"># testing (optional)</span>
<span class="token key atrule">webhooks</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> acceptance<span class="token punctuation">-</span>test
<span class="token key atrule">type</span><span class="token punctuation">:</span> pre<span class="token punctuation">-</span>rollout
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
<span class="token key atrule">timeout</span><span class="token punctuation">:</span> 30s
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">type</span><span class="token punctuation">:</span> bash
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">&quot;curl -sd 'test' http://podinfo-canary/token | grep token&quot;</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> load<span class="token punctuation">-</span>test
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
<span class="token key atrule">timeout</span><span class="token punctuation">:</span> 5s
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">&quot;hey -z 1m -q 10 -c 2 http://app.example.com/&quot;</span>
</code></pre></div><p>Save the above resource as podinfo-canary.yaml and then apply it:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl apply -f ./podinfo-canary.yaml
</code></pre></div><p>After a couple of seconds Flagger will create the canary objects:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token comment"># applied </span>
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo
<span class="token comment"># generated </span>
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
ingresses.extensions/podinfo-canary
</code></pre></div><h2 id="automated-canary-promotion"><a href="#automated-canary-promotion" aria-hidden="true" class="header-anchor">#</a> Automated canary promotion</h2> <p>Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png" alt="Flagger Canary Stages"></p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.1
</code></pre></div><p>Flagger detects that the deployment revision changed and starts a new rollout:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 0
Phase: Succeeded
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger New revision detected podinfo.test
Normal Synced 3m flagger Scaling up podinfo.test
Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Normal Synced 3m flagger Advance podinfo.test canary weight 5
Normal Synced 3m flagger Advance podinfo.test canary weight 10
Normal Synced 3m flagger Advance podinfo.test canary weight 15
Normal Synced 2m flagger Advance podinfo.test canary weight 20
Normal Synced 2m flagger Advance podinfo.test canary weight 25
Normal Synced 1m flagger Advance podinfo.test canary weight 30
Normal Synced 1m flagger Advance podinfo.test canary weight 35
Normal Synced 55s flagger Advance podinfo.test canary weight 40
Normal Synced 45s flagger Advance podinfo.test canary weight 45
Normal Synced 35s flagger Advance podinfo.test canary weight 50
Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
</code></pre></div><p><strong>Note</strong> that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.</p> <p>You can monitor all canaries with:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> kubectl get canaries --all-namespaces
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
<span class="token builtin class-name">test</span> podinfo Progressing <span class="token number">15</span> <span class="token number">2019</span>-05-06T14:05:07Z
prod frontend Succeeded <span class="token number">0</span> <span class="token number">2019</span>-05-05T16:15:07Z
prod backend Failed <span class="token number">0</span> <span class="token number">2019</span>-05-04T17:05:07Z
</code></pre></div><h2 id="automated-rollback"><a href="#automated-rollback" aria-hidden="true" class="header-anchor">#</a> Automated rollback</h2> <p>During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses and rolls back the faulted version.</p> <p>Trigger another canary deployment:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.2
</code></pre></div><p>Generate HTTP 500 errors:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> <span class="token function">curl</span> http://app.example.com/status/500
</code></pre></div><p>When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 10
Phase: Failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger Starting canary deployment for podinfo.test
Normal Synced 3m flagger Advance podinfo.test canary weight 5
Normal Synced 3m flagger Advance podinfo.test canary weight 10
Normal Synced 3m flagger Advance podinfo.test canary weight 15
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% &lt; 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% &lt; 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% &lt; 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% &lt; 99%
Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% &lt; 99%
Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
</code></pre></div><h2 id="custom-metrics"><a href="#custom-metrics" aria-hidden="true" class="header-anchor">#</a> Custom metrics</h2> <p>The canary analysis can be extended with Prometheus queries.</p> <p>The demo app is instrumented with Prometheus so you can create a custom check that will use the
HTTP request duration histogram to validate the canary.</p> <p>Create a metric template and apply it on the cluster:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code><span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> flagger.app/v1beta1
<span class="token key atrule">kind</span><span class="token punctuation">:</span> MetricTemplate
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> latency
<span class="token key atrule">namespace</span><span class="token punctuation">:</span> test
<span class="token key atrule">spec</span><span class="token punctuation">:</span>
<span class="token key atrule">provider</span><span class="token punctuation">:</span>
<span class="token key atrule">type</span><span class="token punctuation">:</span> prometheus
<span class="token key atrule">address</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>promethues.ingress<span class="token punctuation">-</span>nginx<span class="token punctuation">:</span><span class="token number">9090</span>
<span class="token key atrule">query</span><span class="token punctuation">:</span> <span class="token punctuation">|</span><span class="token scalar string">
histogram_quantile(0.99,
sum(
rate(
http_request_duration_seconds_bucket{
kubernetes_namespace=&quot;{{ namespace }}&quot;,
kubernetes_pod_name=~&quot;{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)&quot;
}[1m]
)
) by (le)
)</span>
</code></pre></div><p>Edit the canary analysis and add the latency check:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code> <span class="token key atrule">analysis</span><span class="token punctuation">:</span>
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> <span class="token string">&quot;latency&quot;</span>
<span class="token key atrule">templateRef</span><span class="token punctuation">:</span>
<span class="token key atrule">name</span><span class="token punctuation">:</span> latency
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
<span class="token key atrule">max</span><span class="token punctuation">:</span> <span class="token number">0.5</span>
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
</code></pre></div><p>The threshold is set to 500ms so if the average request duration in the last minute goes over half a second
then the analysis will fail and the canary will not be promoted.</p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.3
</code></pre></div><p>Generate high response latency:</p> <div class="language-bash extra-class"><pre class="language-bash"><code><span class="token function">watch</span> <span class="token function">curl</span> http://app.exmaple.com/delay/2
</code></pre></div><p>Watch Flagger logs:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n nginx-ingress logs deployment/flagger -f | jq .msg
Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement latency 1.20 &gt; 0.5
Halt podinfo.test advancement latency 1.45 &gt; 0.5
Halt podinfo.test advancement latency 1.60 &gt; 0.5
Halt podinfo.test advancement latency 1.69 &gt; 0.5
Halt podinfo.test advancement latency 1.70 &gt; 0.5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
</code></pre></div><p>If you have alerting configured, Flagger will send a notification with the reason why the canary failed.</p> <h2 id="a-b-testing"><a href="#a-b-testing" aria-hidden="true" class="header-anchor">#</a> A/B Testing</h2> <p>Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.</p> <p><img src="https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-abtest-steps.png" alt="Flagger A/B Testing Stages"></p> <p>Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:</p> <div class="language-yaml extra-class"><pre class="language-yaml"><code> <span class="token key atrule">analysis</span><span class="token punctuation">:</span>
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
<span class="token key atrule">threshold</span><span class="token punctuation">:</span> <span class="token number">10</span>
<span class="token key atrule">iterations</span><span class="token punctuation">:</span> <span class="token number">10</span>
<span class="token key atrule">match</span><span class="token punctuation">:</span>
<span class="token comment"># curl -H 'X-Canary: insider' http://app.example.com</span>
<span class="token punctuation">-</span> <span class="token key atrule">headers</span><span class="token punctuation">:</span>
<span class="token key atrule">x-canary</span><span class="token punctuation">:</span>
<span class="token key atrule">exact</span><span class="token punctuation">:</span> <span class="token string">&quot;insider&quot;</span>
<span class="token comment"># curl -b 'canary=always' http://app.example.com</span>
<span class="token punctuation">-</span> <span class="token key atrule">headers</span><span class="token punctuation">:</span>
<span class="token key atrule">cookie</span><span class="token punctuation">:</span>
<span class="token key atrule">exact</span><span class="token punctuation">:</span> <span class="token string">&quot;canary&quot;</span>
<span class="token key atrule">metrics</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> request<span class="token punctuation">-</span>success<span class="token punctuation">-</span>rate
<span class="token key atrule">thresholdRange</span><span class="token punctuation">:</span>
<span class="token key atrule">min</span><span class="token punctuation">:</span> <span class="token number">99</span>
<span class="token key atrule">interval</span><span class="token punctuation">:</span> 1m
<span class="token key atrule">webhooks</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> load<span class="token punctuation">-</span>test
<span class="token key atrule">url</span><span class="token punctuation">:</span> http<span class="token punctuation">:</span>//flagger<span class="token punctuation">-</span>loadtester.test/
<span class="token key atrule">timeout</span><span class="token punctuation">:</span> 5s
<span class="token key atrule">metadata</span><span class="token punctuation">:</span>
<span class="token key atrule">cmd</span><span class="token punctuation">:</span> <span class="token string">&quot;hey -z 1m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com/&quot;</span>
</code></pre></div><p>The above configuration will run an analysis for ten minutes targeting users that have a <code>canary</code> cookie
set to <code>always</code> or those that call the service using the <code>X-Canary: insider</code> header.</p> <p>Trigger a canary deployment by updating the container image:</p> <div class="language-bash extra-class"><pre class="language-bash"><code>kubectl -n <span class="token builtin class-name">test</span> <span class="token builtin class-name">set</span> image deployment/podinfo <span class="token punctuation">\</span>
<span class="token assign-left variable">podinfod</span><span class="token operator">=</span>stefanprodan/podinfo:3.1.4
</code></pre></div><p>Flagger detects that the deployment revision changed and starts the A/B testing:</p> <div class="language-text extra-class"><pre class="language-text"><code>kubectl -n test describe canary/podinfo
Status:
Failed Checks: 0
Phase: Succeeded
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger New revision detected podinfo.test
Normal Synced 3m flagger Scaling up podinfo.test
Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
Normal Synced 2m flagger Advance podinfo.test canary iteration 4/10
Normal Synced 2m flagger Advance podinfo.test canary iteration 5/10
Normal Synced 1m flagger Advance podinfo.test canary iteration 6/10
Normal Synced 1m flagger Advance podinfo.test canary iteration 7/10
Normal Synced 55s flagger Advance podinfo.test canary iteration 8/10
Normal Synced 45s flagger Advance podinfo.test canary iteration 9/10
Normal Synced 35s flagger Advance podinfo.test canary iteration 10/10
Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
</code></pre></div><p>The above procedure can be extended with <a href="/usage/metrics.html">custom metrics</a> checks,
<a href="/usage/webhooks.html">webhooks</a>,
<a href="/usage/webhooks.html#manual-gating">manual promotion</a> approval and
<a href="/usage/alerting.html">Slack or MS Teams</a> notifications.</p></div> <footer class="page-edit"><!----> <!----></footer> <div class="page-nav"><p class="inner"><span class="prev">
←
<a href="/tutorials/appmesh-progressive-delivery.html" class="prev">
App Mesh Canaries
</a></span> <span class="next"><a href="/tutorials/gloo-progressive-delivery.html">
Gloo Canaries
</a>
→
</span></p></div> </main></div><div class="global-ui"></div></div>
<script src="/assets/js/app.9b46ae46.js" defer></script><script src="/assets/js/2.ba5bf77c.js" defer></script><script src="/assets/js/27.ea9f1f9c.js" defer></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink