diff --git a/chapters/07-tutorial-deployment.md b/chapters/07-tutorial-deployment.md index b979037..3404167 100644 --- a/chapters/07-tutorial-deployment.md +++ b/chapters/07-tutorial-deployment.md @@ -1,20 +1,14 @@ ## 4.5 Preparing the Cloud Services -Our project supports three clouds: Amazon AWS, Google Cloud Platform -(GCP), and DigitalOcean (DO). AWS is, by far, the most popular, but -likely the most expensive to run Kubernetes. DigitalOcean is the easiest -to use, while Google Cloud sits comfortably in the middle. +Our project supports three clouds: Amazon AWS, Google Cloud Platform (GCP), and DigitalOcean (DO). AWS is, by far, the most popular, but likely the most expensive to run Kubernetes. DigitalOcean is the easiest to use, while Google Cloud sits comfortably in the middle. ### 4.5.1 Provision a Kubernetes Cluster -In this tutorial, we’ll use a three-node Kubernetes cluster; you can -pick a different size, though. You’ll need at least three nodes to run -an effective canary deployment with rolling updates. +In this tutorial, we’ll use a three-node Kubernetes cluster; you can pick a different size, though. You’ll need at least three nodes to run an effective canary deployment with rolling updates. **DigitalOcean Cluster** -DO calls its service *Kubernetes*. Since DigitalOcean doesn’t have a -private registry\[9\], we’ll use Docker Hub. To create a registry: +DO calls its service *Kubernetes*. Since DigitalOcean doesn’t have a private registry\[9\], we’ll use Docker Hub. To create a registry: - Sign up for a free account on `hub.docker.com`. - Create a public repository called “semaphore-demo-cicd-kubernetes” @@ -23,24 +17,19 @@ To create the Kubernetes cluster: - Sign up for an account on `digitalocean.com`. - Create a *New Project*. - - Create a *Kubernetes* cluster: select the latest version and choose - one of the available regions. Name your cluster - “semaphore-demo-cicd-kubernetes”. + - Create a *Kubernetes* cluster: select the latest version and choose one of the available regions. Name your cluster “semaphore-demo-cicd-kubernetes”. - Go to the *API* menu and generate a *Personal Access Token*. We have to store the DigitalOcean Access Token in secret: 1. Login to `semaphoreci.com`. -2. On the main page, under *Configuration* select *Secrets* and click - on the *Create New Secret* button. +2. On the main page, under *Configuration* select *Secrets* and click on the *Create New Secret* button. 3. The name of the secret is “do-key” 4. Add the following variables: - - `DO_ACCESS_TOKEN` set its value to your DigitalOcean access - token. + - `DO_ACCESS_TOKEN` set its value to your DigitalOcean access token. 5. Click on *Save changes*. -Repeat the last steps to add the second secret, call it “dockerhub” and -add the following variables: +Repeat the last steps to add the second secret, call it “dockerhub” and add the following variables: - `DOCKER_USERNAME` for your DockerHub user name. - `DOCKER_PASSWORD` with the corresponding password. @@ -50,10 +39,8 @@ add the following variables: GCP calls the service *Kubernetes Engine*. To create the services: - Sign up for a GCP account on `cloud.google.com`. - - Create a *New Project*. In *Project ID* type - “semaphore-demo-cicd-kubernetes”. - - Go to *Kubernetes Engine* \> *Clusters* and create a cluster. Select - “Zonal” in *Location Type* and select one of the available zones. + - Create a *New Project*. In *Project ID* type “semaphore-demo-cicd-kubernetes”. + - Go to *Kubernetes Engine* \> *Clusters* and create a cluster. Select “Zonal” in *Location Type* and select one of the available zones. - Name your cluster “semaphore-demo-cicd-kubernetes”. - Go to *IAM* \> *Service Accounts*. - Generate an account with “Project Owner” permissions. @@ -62,30 +49,23 @@ GCP calls the service *Kubernetes Engine*. To create the services: Create a secret for your GCP Access Key file: 1. Login to `semaphoreci.com`. -2. On the main page, under *Cconfiguration* select *Secrets* and click - on the *Create New Secret* button. +2. On the main page, under *Cconfiguration* select *Secrets* and click on the *Create New Secret* button. 3. Name the secret “gcp-key” 4. Add the following file: - - `/home/semaphore/gcp-key.json` and upload the GCP Access JSON - from your computer. + - `/home/semaphore/gcp-key.json` and upload the GCP Access JSON from your computer. 5. Click on *Save changes*. **AWS Cluster** -AWS calls its service *Elastic Kubernetes Service* (EKS). The Docker -private registry is called *Elastic Container Registry* (ECR). +AWS calls its service *Elastic Kubernetes Service* (EKS). The Docker private registry is called *Elastic Container Registry* (ECR). -Creating a cluster on AWS is, unequivocally, a complex, multi-step -affair. So complex, that they created a specialized tool for it: +Creating a cluster on AWS is, unequivocally, a complex, multi-step affair. So complex, that they created a specialized tool for it: - Sign up for an AWS account at `aws.amazon.com`. - Select one of the available regions. - - Find and go to the *ECR* service. Create a new repository called - “semaphore-demo-cicd-kubernetes” and copy its address. - - Install *eksctl* from `eksctl.io` and *awscli* from - `aws.amazon.com/cli` in your machine. - - Find the *IAM* console in AWS and create a user with Administrator - permissions. Get its *Access Key Id* and *Secret Access Key* values. + - Find and go to the *ECR* service. Create a new repository called “semaphore-demo-cicd-kubernetes” and copy its address. + - Install *eksctl* from `eksctl.io` and *awscli* from `aws.amazon.com/cli` in your machine. + - Find the *IAM* console in AWS and create a user with Administrator permissions. Get its *Access Key Id* and *Secret Access Key* values. Open a terminal and sign in to AWS: @@ -107,21 +87,18 @@ $ eksctl create cluster \ **Note**: Select the same region for all AWS services. -Once it finishes, eksctl should have created a kubeconfig file at -`$HOME/.kube/config`. Check the output from eksctl for more details. +Once it finishes, eksctl should have created a kubeconfig file at `$HOME/.kube/config`. Check the output from eksctl for more details. Create a secret to store the AWS Secret Access Key and the kubeconfig: 1. Login to `semaphoreci.com`. -2. On the main page, under *Configuration* select *Secrets* and click - on the *Create New Secret* button. +2. On the main page, under *Configuration* select *Secrets* and click on the *Create New Secret* button. 3. Call the secret “aws-key” 4. Add the following variables: - `AWS_ACCESS_KEY_ID` should have your AWS Access Key ID string. - `AWS_SECRET_ACCESS_KEY` has the AWS Access Secret Key string. 5. Add the following file: - - `/home/semaphore/aws-key.yml` and upload the Kubeconfig file - created by eksctl earlier. + - `/home/semaphore/aws-key.yml` and upload the Kubeconfig file created by eksctl earlier. 6. Click on *Save changes*. ### 4.5.2 Provision a Database @@ -132,11 +109,9 @@ PostgreSQL service. **DigitalOcean Database** - Go to *Databases*. - - Create a PostgreSQL database. Select the same region where the - cluster is running. + - Create a PostgreSQL database. Select the same region where the cluster is running. - In the *Connectivity* tab, whitelist the `0.0.0.0/0` network\[10\]. - - Go to the *Users & Databases* tab and create a database called - “demo” and a user named “demouser”. + - Go to the *Users & Databases* tab and create a database called “demo” and a user named “demouser”. - In the *Overview* tab, take note of the PostgreSQL IP address and port. @@ -144,8 +119,7 @@ PostgreSQL service. - Select *SQL* on the console menu. - Create a new PostgreSQL database instance. - - Select the same region and zone where the Kubernetes cluster is - running. + - Select the same region and zone where the Kubernetes cluster is running. - Enable the *Private IP* network. - Go to the *Users* tab and create a new user called “demouser” - Go to the *Databases* tab and create a new DB called “demo”. @@ -155,13 +129,9 @@ PostgreSQL service. **AWS Database** - Find the service called *RDS*. - - Create a PostgreSQL database called “demo” and type in a secure - password. + - Create a PostgreSQL database called “demo” and type in a secure password. - Choose the same region where the cluster is running. - - Select one of the available *templates*. The free tier is perfect - for demoing the application. Under *Connectivity* select all the - VPCs and subnets where the cluster is running (they should have - appeared in eksctl’s output). + - Select one of the available *templates*. The free tier is perfect for demoing the application. Under *Connectivity* select all the VPCs and subnets where the cluster is running (they should have appeared in eksctl’s output). - Under *Connectivity & Security* take note of the endpoint address and port. @@ -171,35 +141,24 @@ The database secret is the same for all clouds. Create a secret to store the database credentials: 1. Login to `semaphoreci.com`. -2. On the main page, under *Configuration* select *Secrets* and click - on the *Create New Secret* button. +2. On the main page, under *Configuration* select *Secrets* and click on the *Create New Secret* button. 3. The secret name is “db-params” 4. Add the following variables: - `DB_HOST` with the database hostname or IP. - `DB_PORT` points to the database port (default is 5432). - - `DB_SCHEMA` for AWS should be called “postgres”, for the other - clouds its value should be “demo”. + - `DB_SCHEMA` for AWS should be called “postgres”, for the other clouds its value should be “demo”. - `DB_USER` for the database user. - `DB_PASSWORD` should have the corresponding password. - - `DB_SSL` should be “true” for DigitalOcean, it can be empty for - the rest. + - `DB_SSL` should be “true” for DigitalOcean, it can be empty for the rest. 5. Click on *Save changes*. -## 4.6 Releasing the Canary +## 4.6 The Canary Pipeline -Now that we have our cloud services, we’re ready to deploy the canary -for the first time. +Now that we have our cloud services, we’re ready to deploy the canary for the first time. -### 4.6.1 Continuous Deployment Pipeline +Our project includes three ready-to-use reference pipelines for deployment. They should work with the secrets as described earlier. For further details, check the `.semaphore` folder in the project. -Our project includes three ready-to-use reference pipelines for deployment. They should work with the secrets as described earlier. - - - AWS: `.semaphore/deploy-canary-aws.yml` and `.semaphore/deploy-stable-aws.yml` - - GCP: `.semaphore/deploy-canary-gcp.yml` and `.semaphore/deploy-stable-gcp.yml` - - DO: `.semaphore/deploy-canary-digitalocean.yml` and `.semaphore/deploy-stable-digitalocean.yml` - -In this section, we’ll focus on the DO deployment but the process is the same for all -clouds. +In this section, we’ll focus on the DO deployment but the process is the same for all clouds. Open the Workflow Builder again to create the new pipeline. @@ -274,8 +233,28 @@ Type the following commands in the job: kubectl exec -it $(kubectl get pod -l deployment=addressbook-canary -o name | head -n 1) -- npm run ping kubectl exec -it $(kubectl get pod -l deployment=addressbook-canary -o name | head -n 1) -- npm run migrate ``` +## 4.7 Your First Release -### 4.6.2 Your First Release +So far, so good. Let's see where we are: we built the Docker image, and, after testing it, we released it as one-pod canary deployment. If the canary worked, we’re ready to complete the deployment. + +### 4.7.1 The Stable Deployment Pipeline + +The stable deployment pipeline is the last one in the workflow. The pipeline does not introduce anything new. Again, we use `apply.sh` script to start a rolling update and `kubectl delete` to clean the canary deployment. + +Open the Workflow Builder once again and open the canary pipeline. Create a new pipeline branching out from the canary and name it “Deploy Stable (DigitalOcean)”. + +Create the “Deploy to Kubernetes” block with the `do-key`, `db-params`, and `dockerhub` secrets. Also, create the `CLUSTER_NAME` variable and repeat the same commands in the prologue as we did in the previous step. + +In the job command box, type the following lines to make the rolling deployment and delete the canary pods: + +```bash +./apply.sh manifests/deployment.yml addressbook-stable 3 $DOCKER_USERNAME/semaphore-demo-cicd-kubernetes:$SEMAPHORE_WORKFLOW_ID +if kubectl get deployment addressbook-canary; then kubectl delete deployment/addressbook-canary; fi +``` + +We’re done with the release pipeline. + +### 4.7.2 Releasing the Canary This is the moment of truth. Will the canary work? Click on **Run the workflow** and then **Start**. @@ -289,51 +268,15 @@ NAME READY UP-TO-DATE AVAILABLE AGE addressbook-canary 1/1 1 1 8m40s ``` -## 4.7 Releasing the Stable +### 4.7.3 Releasing the Stable -So far, so good. Let's see where we are: we built the Docker image, and, after testing it, we released it as one-pod canary deployment. If the canary worked, we’re ready to complete the deployment. +In tandem with the deployment, we should have a dashboard to monitor errors, user incidents, and performance metrics to compare against the baseline. After some pre-determined amount of time, we would reach a go vs. no-go decision. Is the canaried version good enough to be promoted to stable? If so, the deployment continues. If not, after collecting the necessary error reports and stack traces, we rollback and regroup. -### 4.7.1 The Continuous Deployment Pipeline - -The stable deployment pipeline is the last one in the workflow. The pipeline does not introduce anything new. Again, we use `apply.sh` -script to start a rolling update and `kubectl delete` to clean the canary deployment. - -Open the Workflow Builder once again and open the canary pipeline. Create a new pipeline branching from the canary and name it “Deploy Stable (DigitalOcean)”. - -Create the “Deploy Stable” block with the - - -``` yaml -. . . -jobs: - - name: Deploy - commands: - - ./apply.sh manifests/deployment.yml \ - addressbook-stable \ - 3 \ - $DOCKER_USERNAME/addressbook:$SEMAPHORE_WORKFLOW_ID - - - if kubectl get deployment addressbook-canary; then \ - kubectl delete deployment/addressbook-canary; \ - fi -. . . -``` - -### 4.7.2 Making the Release - -In tandem with the deployment, we should have a dashboard to monitor -errors, user incidents, and performance metrics to compare against the -baseline. After some pre-determined amount of time, we would reach a go -vs. no-go decision. Is the canaried version good enough to be promoted -to stable? If so, the deployment continues. If not, after collecting the -necessary error reports and stack traces, we rollback and regroup. - -Let’s say we decide to go ahead. So go on and hit the *Promote* button. +Let’s say we decide to go ahead. So go on and hit the **Promote** button. ![Stable Pipeline](./figures/05-sem-stable-pipeline.png){ width=60% } -While the block runs, you should get the existing canary and a new -“addressbook-stable” deployment: +While the block runs, you should get the existing canary and a new “addressbook-stable” deployment: ``` bash $ kubectl get deployment @@ -342,8 +285,7 @@ addressbook-canary 1/1 1 1 110s addressbook-stable 0/3 3 0 1s ``` -One at a time, the numbers of replicas should increase until reaching -the target of three: +One at a time, the numbers of replicas should increase until reaching the target of three: ``` bash $ kubectl get deployment @@ -369,8 +311,7 @@ addressbook-lb LoadBalancer 10.120.14.50 35.225.210.248 80:30479/TCP 2 kubernetes ClusterIP 10.120.0.1 443/TCP 49m ``` -We can use curl to test the API endpoint directly. For example, to -create a person in the addressbook: +We can use curl to test the API endpoint directly. For example, to create a person in the addressbook: ``` bash $ curl -w "\n" -X PUT -d "firstName=Sammy&lastName=David Jr" 34.68.150.168/person @@ -401,70 +342,44 @@ $ curl -w "\n" 34.68.150.168/all The deployment was a success, that was no small feat. Congratulations\! -### 4.7.3 The Rollback Pipeline +### 4.7.4 The Rollback Pipeline -Fortunately, Kubernetes and CI/CD make an exceptional team when it comes -to recovering from errors. Our project includes a rollback pipeline. +Fortunately, Kubernetes and CI/CD make an exceptional team when it comes to recovering from errors. Our project includes a rollback pipeline. -Let’s say that we don’t like how the canary performs. In that case, we -can click on the *Promote* button on the “Rollback canary” pipeline: +Let’s say that we don’t like how the canary performs or, even worse, the functional tests at the end of the canary deployment pipeline fails. In that case, wouldn’t be great to have the system go back to the previous state automatically? What about being able to undo the change with a click of a button? This is exactly what we are going to create in this step, a rollback pipeline. + +Open the Workflow Builder once more and go to the end of the canary pipeline. Create a new promotion branching out of it, check the **Enable automatic promotion** box, and set this condition: + +```text +"result = 'failed'" +``` + +The rollback pipeline job is to collect information to diagnose the problem. Create a new block called “Rollback Canary”, import the `do-ctl` secret, and create the `CLUSTER_NAME` variable as before. Repeat the prologue commands like we did before and type these lines in the job: + +```bash +kubectl get all -o wide +kubectl get events +kubectl describe deployment addressbook-canary || true +kubectl logs $(kubectl get pod -l deployment=addressbook-canary -o name | head -n 1) || true +if kubectl get deployment addressbook-stable; then kubectl scale --replicas=3 deployment/addressbook-stable; fi +if kubectl get deployment addressbook-canary; then kubectl delete deployment/addressbook-canary; fibash +``` + +The first four lines print out information about the cluster. The last two, undo the changes in the cluster by scaling up the stable deployment and removing the canary: ![Rollback Pipeline](./figures/05-sem-rollback-canary.png){ width=60% } -Check the pipeline at `.semaphore/rollback-canary-digitalocean.yml` +And we’re back to normal, phew\! Now its time to check the job logs to see what went wrong and fix it before merging to master again. -The rollback pipeline job is to collect information to diagnose the -problem: +**But what if the problem is found after the stable release?** Let’s imagine that a defect sneaked its way into the stable deployment. It can happen, maybe there was some subtle bug that no one found out hours or days in. Or perhaps some error not picked up by the functional test. Is it too late? Can we go back to a previous version? -``` yaml -commands: - - kubectl get all -o wide - - kubectl get events - - kubectl describe deployment addressbook-canary || true - - kubectl logs \ - $(kubectl get pod -l deployment=addressbook-canary -o name | head -n 1) || true +The answer is yes, we can go to the previous version, but manual intervention is required. Do you remember that we tagged each Docker image with a unique ID (the `SEMAPHORE_WORKFLOW_ID`)? We can re-promote the stable deployment pipeline for the last good version in Semaphore. When the Docker image is no longer in the registry can just regenerate it using the *Rerun* button in the top right corner. -``` +### 4.7.5 Troubleshooting and Tips -And then undo the changes by scaling up the stable deployment and -removing the canary: +Even the best plans can fail; failure is certainly an option in the software business. Maybe the canary is presented with some unexpected errors, perhaps it has performance problems, or we merged the wrong branch into master. The important thing is (1) learn something from them, and (2) know how to go back to solid ground. -``` yaml -- if kubectl get deployment addressbook-stable; then \ - kubectl scale --replicas=3 deployment/addressbook-stable; \ - fi - -- if kubectl get deployment addressbook-canary; then \ - kubectl delete deployment/addressbook-canary; \ -fi -``` - -And we’re back to normal, phew\! Now its time to check the job logs to -see what went wrong and fix it before merging to master again. - -**But what if the problem is found after the stable release?** Let’s -imagine that a defect sneaked its way into the stable deployment. It can -happen, maybe there was some subtle bug that no one found out hours or -days in. Or perhaps some error not picked up by the functional test. Is -it too late? Can we go back to a previous version? - -The answer is yes, we can go to the previous version. Do you remember -that we tagged each Docker image with a unique ID (the -`SEMAPHORE_WORKFLOW_ID`)? We can re-promote the stable deployment -pipeline for the last good version in Semaphore. When the Docker image -is no longer in the registry can just regenerate it using the *Rerun* -button in the top right corner. - -### 4.7.2 Troubleshooting and Tips - -Even the best plans can fail; failure is certainly an option in the -software business. Maybe the canary is presented with some unexpected -errors, perhaps it has performance problems, or we merged the wrong -branch into master. The important thing is (1) learn something from -them, and (2) know how to go back to solid ground. - -Kubectl can give us a lot of insights into what is happening. First, get -an overall picture of the resources on the cluster. +Kubectl can give us a lot of insights into what is happening. First, get an overall picture of the resources on the cluster. ``` bash $ kubectl get all -o wide @@ -502,15 +417,13 @@ $ kubectl logs $ kubectl logs --previous ``` -If you need to jump in one of the containers, you can start a shell as -long as the pod is running with: +If you need to jump in one of the containers, you can start a shell as long as the pod is running with: ``` bash $ kubectl exec -it -- bash ``` -To access a pod network from your machine, forward a port with -`port-forward`, for instance: +To access a pod network from your machine, forward a port with `port-forward`, for instance: ``` bash $ kubectl port-forward 8080:80 @@ -518,71 +431,42 @@ $ kubectl port-forward 8080:80 These are some common error messages that you might run into: - - Manifest is invalid: it usually means that the manifest YAML syntax - is incorrect. Use `--dry-run` or `--validate` options verify the - manifest. - - `ImagePullBackOff` or `ErrImagePull`: the requested image is invalid - or was not found. Check that the image is in the registry and that - the reference on the manifest file is correct. - - `CrashLoopBackOff`: the application is crashing, and the pod is - shutting down. Check the logs for application errors. - - Pod never leaves `Pending` status: this could mean that one of the - Kubernetes secrets is missing. - - Log message says that “container is unhealthy”: this message may - show that the pod is not passing a probe. Check that the probe - definitions are correct. - - Log message says that there are “insufficient resources”: this may - happen when the cluster is running low on memory or CPU. + - Manifest is invalid: it usually means that the manifest YAML syntax is incorrect. Use `--dry-run` or `--validate` options verify the manifest. + - `ImagePullBackOff` or `ErrImagePull`: the requested image is invalid or was not found. Check that the image is in the registry and that the reference on the manifest file is correct. + - `CrashLoopBackOff`: the application is crashing, and the pod is shutting down. Check the logs for application errors. + - Pod never leaves `Pending` status: this could mean that one of the Kubernetes secrets is missing. + - Log message says that “container is unhealthy”: this message may show that the pod is not passing a probe. Check that the probe definitions are correct. + - Log message says that there are “insufficient resources”: this may happen when the cluster is running low on memory or CPU. ## 4.8 Summary -You have learned how to put together the puzzle of CI/CD, Docker, and -Kubernetes into a practical application. In this chapter, you have put -in practice all that you’ve learned in this book: +You have learned how to put together the puzzle of CI/CD, Docker, and Kubernetes into a practical application. In this chapter, you have put in practice all that you’ve learned in this book: - - How to setup pipelines in Semaphore CI/CD and use them to deploy to - the cloud. - - How to build Docker images and start a dev environment with the help - of Docker Compose. + - How to setup pipelines in Semaphore CI/CD and use them to deploy to the cloud. + - How to build Docker images and start a dev environment with the help of Docker Compose. - How to do canaried deployments and rolling updates in Kubernetes. - - How to scale deployments and how to recover when things don’t go as - planned. + - How to scale deployments and how to recover when things don’t go as planned. -Each of the pieces had its role: Docker brings portability, Kubernetes -adds orchestration, and Semaphore CI/CD drives the test and deployment -process. +Each of the pieces had its role: Docker brings portability, Kubernetes adds orchestration, and Semaphore CI/CD drives the test and deployment process. ## Footnotes -1. The full pipeline reference can be fount at - +1. The full pipeline reference can be fount at -2. To see all the available machines, go to - +2. To see all the available machines, go to -3. For more details on the Ubuntu image see: - +3. For more details on the Ubuntu image see: -4. You can find the full toolbox reference here: - +4. You can find the full toolbox reference here: -5. sem-service can start a lot of popular database engines, for the - full list check: - +5. sem-service can start a lot of popular database engines, for the full list check: -6. The full environment reference can be found at - +6. The full environment reference can be found at -7. For more details on secrets consult: - +7. For more details on secrets consult: -8. For more information on pipelines check - +8. For more information on pipelines check -9. At the time of writing, DigitalOcean announced a beta for a private - registry offering. For more information, consult the available - documentation: - +9. At the time of writing, DigitalOcean announced a beta for a private registry offering. For more information, consult the available documentation: -10. Later, when everything is working, you can restrict access to the - Kubernetes nodes to increase security +10. Later, when everything is working, you can restrict access to the Kubernetes nodes to increase security