# Running our first containers on Kubernetes - First things first: we cannot run a container -- - We are going to run a pod, and in that pod there will be a single container -- - In that container in the pod, we are going to run a simple `ping` command -- - Sounds simple enough, right? -- - Except ... that the `kubectl run` command changed in Kubernetes 1.18! - We'll explain what has changed, and why --- ## Choose your own adventure - First, let's check which version of Kubernetes we're running .exercise[ - Check our API server version: ```bash kubectl version ``` - Look at the **Server Version** in the second part of the output ] - In the following slides, we will talk about 1.17- or 1.18+ (to indicate "up to Kubernetes 1.17" and "from Kubernetes 1.18") --- ## Starting a simple pod with `kubectl run` - `kubectl run` is convenient to start a single pod - We need to specify at least a *name* and the image we want to use - Optionally, we can specify the command to run in the pod .exercise[ - Let's ping the address of `localhost`, the loopback interface: ```bash kubectl run pingpong --image alpine ping 127.0.0.1 ``` ] --- ## What do we see? - In Kubernetes 1.18+, the output tells us that a Pod is created: ``` pod/pingpong created ``` - In Kubernetes 1.17-, the output is much more verbose: ``` kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/pingpong created ``` - There is a deprecation warning ... - ... And a Deployment was created instead of a Pod 🤔 What does that mean? --- ## Show me all you got! - What resources were created by `kubectl run`? .exercise[ - Let's ask Kubernetes to show us *all* the resources: ```bash kubectl get all ``` ] Note: `kubectl get all` is a lie. It doesn't show everything. (But it shows a lot of "usual suspects", i.e. commonly used resources.) --- ## The situation with Kubernetes 1.18+ ``` NAME READY STATUS RESTARTS AGE pod/pingpong 1/1 Running 0 9s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 3h30m ``` We wanted a pod, we got a pod, named `pingpong`. Great! (We can ignore `service/kubernetes`, it was already there before.) --- ## The situation with Kubernetes 1.17- ``` NAME READY STATUS RESTARTS AGE pod/pingpong-6ccbc77f68-kmgfn 1/1 Running 0 11s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 3h45 NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pingpong 1/1 1 1 11s NAME DESIRED CURRENT READY AGE replicaset.apps/pingpong-6ccbc77f68 1 1 1 11s ``` Our pod is not named `pingpong`, but `pingpong-xxxxxxxxxxx-yyyyy`. We have a Deployment named `pingpong`, and an extra Replica Set, too. What's going on? --- ## From Deployment to Pod We have the following resources: - `deployment.apps/pingpong` This is the Deployment that we just created. - `replicaset.apps/pingpong-xxxxxxxxxx` This is a Replica Set created by this Deployment. - `pod/pingpong-xxxxxxxxxx-yyyyy` This is a *pod* created by the Replica Set. Let's explain what these things are. --- ## Pod - Can have one or multiple containers - Runs on a single node (Pod cannot "straddle" multiple nodes) - Pods cannot be moved (e.g. in case of node outage) - Pods cannot be scaled (except by manually creating more Pods) --- class: extra-details ## Pod details - A Pod is not a process; it's an environment for containers - it cannot be "restarted" - it cannot "crash" - The containers in a Pod can crash - They may or may not get restarted (depending on Pod's restart policy) - If all containers exit successfully, the Pod ends in "Succeeded" phase - If some containers fail and don't get restarted, the Pod ends in "Failed" phase --- ## Replica Set - Set of identical (replicated) Pods - Defined by a pod template + number of desired replicas - If there are not enough Pods, the Replica Set creates more (e.g. in case of node outage; or simply when scaling up) - If there are too many Pods, the Replica Set deletes some (e.g. if a node was disconnected and comes back; or when scaling down) - We can scale up/down a Replica Set - we update the manifest of the Replica Set - as a consequence, the Replica Set controller creates/deletes Pods --- ## Deployment - Replica Sets control *identical* Pods - Deployments are used to roll out different Pods (different image, command, environment variables, ...) - When we update a Deployment with a new Pod definition: - a new Replica Set is created with the new Pod definition - that new Replica Set is progressively scaled up - meanwhile, the old Replica Set(s) is(are) scaled down - This is a *rolling update*, minimizing application downtime - When we scale up/down a Deployment, it scales up/down its Replica Set --- ## `kubectl run` through the ages - When we want to run an app on Kubernetes, we *generally* want a Deployment - Up to Kubernetes 1.17, `kubectl run` created a Deployment - it could also create other things, by using special flags - this was powerful, but potentially confusing - creating a single Pod was done with `kubectl run --restart=Never` - other resources could also be created with `kubectl create ...` - From Kubernetes 1.18, `kubectl run` creates a Pod - other kinds of resources can still be created with `kubectl create` --- ## Creating a Deployment the proper way - Let's destroy that `pingpong` app that we created - Then we will use `kubectl create deployment` to re-create it .exercise[ - On Kubernetes 1.18+, delete the Pod named `pingpong`: ```bash kubectl delete pod pingpong ``` - On Kubernetes 1.17-, delete the Deployment named `pingpong`: ```bash kubectl delete deployment pingpong ``` ] --- ## Running `ping` in a Deployment - When using `kubectl create deployment`, we cannot indicate the command to execute (at least, not in Kubernetes 1.18) - We can: - write a custom YAML manifest for our Deployment -- - (yeah right ... too soon!) -- - use an image that has the command to execute baked in - (much easier!) -- - We will use the image `jpetazzo/ping` (it has a default command of `ping 127.0.0.1`) --- ## Creating a Deployment running `ping` - Let's create a Deployment named `pingpong` - It will use the image `jpetazzo/ping` .exercise[ - Create the Deployment: ```bash kubectl create deployment pingpong --image=jpetazzo/ping ``` - Check the resources that were created: ```bash kubectl get all ``` ] --- ## Viewing container output - Let's use the `kubectl logs` command - We will pass either a *pod name*, or a *type/name* (E.g. if we specify a deployment or replica set, it will get the first pod in it) - Unless specified otherwise, it will only show logs of the first container in the pod (Good thing there's only one in ours!) .exercise[ - View the result of our `ping` command: ```bash kubectl logs deploy/pingpong ``` ] --- ## Streaming logs in real time - Just like `docker logs`, `kubectl logs` supports convenient options: - `-f`/`--follow` to stream logs in real time (à la `tail -f`) - `--tail` to indicate how many lines you want to see (from the end) - `--since` to get logs only after a given timestamp .exercise[ - View the latest logs of our `ping` command: ```bash kubectl logs deploy/pingpong --tail 1 --follow ``` - Leave that command running, so that we can keep an eye on these logs ] --- ## Scaling our application - We can create additional copies of our container (I mean, our pod) with `kubectl scale` .exercise[ - Scale our `pingpong` deployment: ```bash kubectl scale deploy/pingpong --replicas 3 ``` - Note that this command does exactly the same thing: ```bash kubectl scale deployment pingpong --replicas 3 ``` ] Note: what if we tried to scale `replicaset.apps/pingpong-xxxxxxxxxx`? We could! But the *deployment* would notice it right away, and scale back to the initial level. --- ## Log streaming - Let's look again at the output of `kubectl logs` (the one we started before scaling up) - `kubectl logs` shows us one line per second - We could expect 3 lines per second (since we should now have 3 pods running `ping`) - Let's try to figure out what's happening! --- ## Streaming logs of multiple pods - What happens if we restart `kubectl logs`? .exercise[ - Interrupt `kubectl logs` (with Ctrl-C) - Restart it: ```bash kubectl logs deploy/pingpong --tail 1 --follow ``` ] `kubectl logs` will warn us that multiple pods were found, and that it's showing us only one of them. Let's leave `kubectl logs` running while we keep exploring. --- ## Resilience - The *deployment* `pingpong` watches its *replica set* - The *replica set* ensures that the right number of *pods* are running - What happens if pods disappear? .exercise[ - In a separate window, watch the list of pods: ```bash watch kubectl get pods ``` - Destroy the pod currently shown by `kubectl logs`: ``` kubectl delete pod pingpong-xxxxxxxxxx-yyyyy ``` ] --- ## What happened? - `kubectl delete pod` terminates the pod gracefully (sending it the TERM signal and waiting for it to shutdown) - As soon as the pod is in "Terminating" state, the Replica Set replaces it - But we can still see the output of the "Terminating" pod in `kubectl logs` - Until 30 seconds later, when the grace period expires - The pod is then killed, and `kubectl logs` exits --- ## Viewing logs of multiple pods - When we specify a deployment name, only one single pod's logs are shown - We can view the logs of multiple pods by specifying a *selector* - A selector is a logic expression using *labels* - If we check the pods created by the deployment, they all have the label `app=pingpong` (this is just a default label that gets added when using `kubectl create deployment`) .exercise[ - View the last line of log from all pods with the `app=pingpong` label: ```bash kubectl logs -l app=pingpong --tail 1 ``` ] --- ### Streaming logs of multiple pods - Can we stream the logs of all our `pingpong` pods? .exercise[ - Combine `-l` and `-f` flags: ```bash kubectl logs -l app=pingpong --tail 1 -f ``` ] *Note: combining `-l` and `-f` is only possible since Kubernetes 1.14!* *Let's try to understand why ...* --- class: extra-details ### Streaming logs of many pods - Let's see what happens if we try to stream the logs for more than 5 pods .exercise[ - Scale up our deployment: ```bash kubectl scale deployment pingpong --replicas=8 ``` - Stream the logs: ```bash kubectl logs -l app=pingpong --tail 1 -f ``` ] We see a message like the following one: ``` error: you are attempting to follow 8 log streams, but maximum allowed concurency is 5, use --max-log-requests to increase the limit ``` --- class: extra-details ## Why can't we stream the logs of many pods? - `kubectl` opens one connection to the API server per pod - For each pod, the API server opens one extra connection to the corresponding kubelet - If there are 1000 pods in our deployment, that's 1000 inbound + 1000 outbound connections on the API server - This could easily put a lot of stress on the API server - Prior Kubernetes 1.14, it was decided to *not* allow multiple connections - From Kubernetes 1.14, it is allowed, but limited to 5 connections (this can be changed with `--max-log-requests`) - For more details about the rationale, see [PR #67573](https://github.com/kubernetes/kubernetes/pull/67573) --- ## Shortcomings of `kubectl logs` - We don't see which pod sent which log line - If pods are restarted / replaced, the log stream stops - If new pods are added, we don't see their logs - To stream the logs of multiple pods, we need to write a selector - There are external tools to address these shortcomings (e.g.: [Stern](https://github.com/wercker/stern)) --- class: extra-details ## `kubectl logs -l ... --tail N` - If we run this with Kubernetes 1.12, the last command shows multiple lines - This is a regression when `--tail` is used together with `-l`/`--selector` - It always shows the last 10 lines of output for each container (instead of the number of lines specified on the command line) - The problem was fixed in Kubernetes 1.13 *See [#70554](https://github.com/kubernetes/kubernetes/issues/70554) for details.* --- class: extra-details ## Party tricks involving IP addresses - It is possible to specify an IP address with less than 4 bytes (example: `127.1`) - Zeroes are then inserted in the middle - As a result, `127.1` expands to `127.0.0.1` - So we can `ping 127.1` to ping `localhost`! (See [this blog post](https://ma.ttias.be/theres-more-than-one-way-to-write-an-ip-address/ ) for more details.) --- class: extra-details ## More party tricks with IP addresses - We can also ping `1.1` - `1.1` will expand to `1.0.0.1` - This is one of the addresses of Cloudflare's [public DNS resolver](https://blog.cloudflare.com/announcing-1111/) - This is a quick way to check connectivity (if we can reach 1.1, we probably have internet access) --- ## Creating other kinds of resources - Deployments are great for stateless web apps (as well as workers that keep running forever) - Jobs are great for "long" background work ("long" being at least minutes our hours) - CronJobs are great to schedule Jobs at regular intervals (just like the classic UNIX `cron` daemon with its `crontab` files) - Pods are great for one-off execution that we don't care about (because they don't get automatically restarted if something goes wrong) --- ## Creating a Job - A Job will create a Pod - If the Pod fails, the Job will create another one - The Job will keep trying until: - either a Pod succeeds, - or we hit the *backoff limit* of the Job (default=6) .exercise[ - Create a Job that has a 50% chance of success: ```bash kubectl create job flipcoin --image=alpine -- sh -c 'exit $(($RANDOM%2))' ``` ] --- ## Our Job in action - Our Job will create a Pod named `flipcoin-xxxxx` - If the Pod succeeds, the Job stops - If the Pod fails, the Job creates another Pod .exercise[ - Check the status of the Pod(s) created by the Job: ```bash kubectl get pods --selector=job-name=flipcoin ``` ] --- class: extra-details ## More advanced jobs - We can specify a number of "completions" (default=1) - This indicates how many times the Job must be executed - We can specify the "parallelism" (default=1) - This indicates how many Pods should be running in parallel - These options cannot be specified with `kubectl create job` (we have to write our own YAML manifest to use them) --- ## Scheduling periodic background work - A Cron Job is a Job that will be executed at specific intervals (the name comes from the traditional cronjobs executed by the UNIX crond) - It requires a *schedule*, represented as five space-separated fields: - minute [0,59] - hour [0,23] - day of the month [1,31] - month of the year [1,12] - day of the week ([0,6] with 0=Sunday) - `*` means "all valid values"; `/N` means "every N" - Example: `*/3 * * * *` means "every three minutes" --- ## Creating a Cron Job - Let's create a simple job to be executed every three minutes - Careful: make sure that the job terminates! (The Cron Job will not hold if a previous job is still running) .exercise[ - Create the Cron Job: ```bash kubectl create cronjob every3mins --schedule="*/3 * * * *" \ --image=alpine -- sleep 10 ``` - Check the resource that was created: ```bash kubectl get cronjobs ``` ] --- ## Cron Jobs in action - At the specified schedule, the Cron Job will create a Job - The Job will create a Pod - The Job will make sure that the Pod completes (re-creating another one if it fails, for instance if its node fails) .exercise[ - Check the Jobs that are created: ```bash kubectl get jobs ``` ] (It will take a few minutes before the first job is scheduled.) --- class: extra-details ## What about `kubectl run` before v1.18? - Creating a Deployment: `kubectl run` - Creating a Pod: `kubectl run --restart=Never` - Creating a Job: `kubectl run --restart=OnFailure` - Creating a Cron Job: `kubectl run --restart=OnFailure --schedule=...` *Avoid using these forms, as they are deprecated since Kubernetes 1.18!* --- ## Beyond `kubectl create` - As hinted earlier, `kubectl create` doesn't always expose all options - can't express parallelism or completions of Jobs - can't express Pods with multiple containers - can't express healthchecks, resource limits - etc. - `kubectl create` and `kubectl run` are *helpers* that generate YAML manifests - If we write these manifests ourselves, we can use all features and options - We'll see later how to do that!