container.training/slides/k8s/kubectlrun.md

Running our first containers on Kubernetes

• First things first: we cannot run a container

--

• We are going to run a pod, and in that pod there will be a single container

--

• In that container in the pod, we are going to run a simple ping command

--

• Sounds simple enough, right?

--

• Except ... that the kubectl run command changed in Kubernetes 1.18!

• We'll explain what has changed, and why

---

Choose your own adventure

• First, let's check which version of Kubernetes we're running

.exercise[

• Check our API server version:

  kubectl version
  

• Look at the Server Version in the second part of the output

]

• In the following slides, we will talk about 1.17- or 1.18+

  (to indicate "up to Kubernetes 1.17" and "from Kubernetes 1.18")

---

Starting a simple pod with kubectl run

• kubectl run is convenient to start a single pod

• We need to specify at least a name and the image we want to use

• Optionally, we can specify the command to run in the pod

.exercise[

• Let's ping the address of localhost, the loopback interface:

  kubectl run pingpong --image alpine ping 127.0.0.1
  

]

---

What do we see?

• In Kubernetes 1.18+, the output tells us that a Pod is created:

  pod/pingpong created
  

• In Kubernetes 1.17-, the output is much more verbose:

  kubectl run --generator=deployment/apps.v1 is DEPRECATED 
  and will be removed in a future version. Use kubectl run 
  --generator=run-pod/v1 or kubectl create instead.
  deployment.apps/pingpong created
  

• There is a deprecation warning ...

• ... And a Deployment was created instead of a Pod

🤔 What does that mean?

---

Show me all you got!

• What resources were created by kubectl run?

.exercise[

• Let's ask Kubernetes to show us all the resources:

  kubectl get all
  

]

Note: kubectl get all is a lie. It doesn't show everything.

(But it shows a lot of "usual suspects", i.e. commonly used resources.)

---

The situation with Kubernetes 1.18+

NAME           READY   STATUS    RESTARTS   AGE
pod/pingpong   1/1     Running   0          9s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h30m

We wanted a pod, we got a pod, named pingpong. Great!

(We can ignore service/kubernetes, it was already there before.)

---

The situation with Kubernetes 1.17-

NAME                            READY   STATUS        RESTARTS   AGE
pod/pingpong-6ccbc77f68-kmgfn   1/1     Running       0          11s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h45

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pingpong   1/1     1            1           11s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/pingpong-6ccbc77f68   1         1         1       11s

Our pod is not named pingpong, but pingpong-xxxxxxxxxxx-yyyyy.

We have a Deployment named pingpong, and an extra Replica Set, too. What's going on?

---

From Deployment to Pod

We have the following resources:

• deployment.apps/pingpong

  This is the Deployment that we just created.

• replicaset.apps/pingpong-xxxxxxxxxx

  This is a Replica Set created by this Deployment.

• pod/pingpong-xxxxxxxxxx-yyyyy

  This is a pod created by the Replica Set.

Let's explain what these things are.

---

Pod

• Can have one or multiple containers

• Runs on a single node

  (Pod cannot "straddle" multiple nodes)

• Pods cannot be moved

  (e.g. in case of node outage)

• Pods cannot be scaled

  (except by manually creating more Pods)

---

class: extra-details

Pod details

• A Pod is not a process; it's an environment for containers

  • it cannot be "restarted"

  • it cannot "crash"

• The containers in a Pod can crash

• They may or may not get restarted

  (depending on Pod's restart policy)

• If all containers exit successfully, the Pod ends in "Succeeded" phase

• If some containers fail and don't get restarted, the Pod ends in "Failed" phase

---

Replica Set

• Set of identical (replicated) Pods

• Defined by a pod template + number of desired replicas

• If there are not enough Pods, the Replica Set creates more

  (e.g. in case of node outage; or simply when scaling up)

• If there are too many Pods, the Replica Set deletes some

  (e.g. if a node was disconnected and comes back; or when scaling down)

• We can scale up/down a Replica Set

  • we update the manifest of the Replica Set

  • as a consequence, the Replica Set controller creates/deletes Pods

---

Deployment

• Replica Sets control identical Pods

• Deployments are used to roll out different Pods

  (different image, command, environment variables, ...)

• When we update a Deployment with a new Pod definition:

  • a new Replica Set is created with the new Pod definition

  • that new Replica Set is progressively scaled up

  • meanwhile, the old Replica Set(s) is(are) scaled down

• This is a rolling update, minimizing application downtime

• When we scale up/down a Deployment, it scales up/down its Replica Set

---

kubectl run through the ages

• When we want to run an app on Kubernetes, we generally want a Deployment

• Up to Kubernetes 1.17, kubectl run created a Deployment

  • it could also create other things, by using special flags

  • this was powerful, but potentially confusing

  • creating a single Pod was done with kubectl run --restart=Never

  • other resources could also be created with kubectl create ...

• From Kubernetes 1.18, kubectl run creates a Pod

  • other kinds of resources can still be created with kubectl create

---

Creating a Deployment the proper way

• Let's destroy that pingpong app that we created

• Then we will use kubectl create deployment to re-create it

.exercise[

• On Kubernetes 1.18+, delete the Pod named pingpong:

  kubectl delete pod pingpong
  

• On Kubernetes 1.17-, delete the Deployment named pingpong:

  kubectl delete deployment pingpong
  

]

---

Running ping in a Deployment

• When using kubectl create deployment, we cannot indicate the command to execute

  (at least, not in Kubernetes 1.18)

• We can:

  • write a custom YAML manifest for our Deployment

--

• (yeah right ... too soon!)

--

• use an image that has the command to execute baked in

• (much easier!)

--

• We will use the image jpetazzo/ping

  (it has a default command of ping 127.0.0.1)

---

Creating a Deployment running ping

• Let's create a Deployment named pingpong

• It will use the image jpetazzo/ping

.exercise[

• Create the Deployment:

  kubectl create deployment pingpong --image=jpetazzo/ping
  

• Check the resources that were created:

  kubectl get all
  

]

---

Viewing container output

• Let's use the kubectl logs command

• We will pass either a pod name, or a type/name

  (E.g. if we specify a deployment or replica set, it will get the first pod in it)

• Unless specified otherwise, it will only show logs of the first container in the pod

  (Good thing there's only one in ours!)

.exercise[

• View the result of our ping command:

  kubectl logs deploy/pingpong
  

]

---

Streaming logs in real time

• Just like docker logs, kubectl logs supports convenient options:

  • -f/--follow to stream logs in real time (à la tail -f)

  • --tail to indicate how many lines you want to see (from the end)

  • --since to get logs only after a given timestamp

.exercise[

• View the latest logs of our ping command:

  kubectl logs deploy/pingpong --tail 1 --follow
  

• Leave that command running, so that we can keep an eye on these logs

]

---

Scaling our application

• We can create additional copies of our container (I mean, our pod) with kubectl scale

.exercise[

• Scale our pingpong deployment:

  kubectl scale deploy/pingpong --replicas 3
  

• Note that this command does exactly the same thing:

  kubectl scale deployment pingpong --replicas 3
  

]

Note: what if we tried to scale replicaset.apps/pingpong-xxxxxxxxxx?

We could! But the deployment would notice it right away, and scale back to the initial level.

---

Log streaming

• Let's look again at the output of kubectl logs

  (the one we started before scaling up)

• kubectl logs shows us one line per second

• We could expect 3 lines per second

  (since we should now have 3 pods running ping)

• Let's try to figure out what's happening!

---

Streaming logs of multiple pods

• What happens if we restart kubectl logs?

.exercise[

• Interrupt kubectl logs (with Ctrl-C)

• Restart it:

  kubectl logs deploy/pingpong --tail 1 --follow
  

]

kubectl logs will warn us that multiple pods were found, and that it's showing us only one of them.

Let's leave kubectl logs running while we keep exploring.

---

Resilience

• The deployment pingpong watches its replica set

• The replica set ensures that the right number of pods are running

• What happens if pods disappear?

.exercise[

• In a separate window, watch the list of pods:

  watch kubectl get pods
  

• Destroy the pod currently shown by kubectl logs:

  kubectl delete pod pingpong-xxxxxxxxxx-yyyyy
  

]

---

What happened?

• kubectl delete pod terminates the pod gracefully

  (sending it the TERM signal and waiting for it to shutdown)

• As soon as the pod is in "Terminating" state, the Replica Set replaces it

• But we can still see the output of the "Terminating" pod in kubectl logs

• Until 30 seconds later, when the grace period expires

• The pod is then killed, and kubectl logs exits

---

Viewing logs of multiple pods

• When we specify a deployment name, only one single pod's logs are shown

• We can view the logs of multiple pods by specifying a selector

• A selector is a logic expression using labels

• If we check the pods created by the deployment, they all have the label app=pingpong

  (this is just a default label that gets added when using kubectl create deployment)

.exercise[

• View the last line of log from all pods with the app=pingpong label:

  kubectl logs -l app=pingpong --tail 1
  

]

---

Streaming logs of multiple pods

• Can we stream the logs of all our pingpong pods?

.exercise[

• Combine -l and -f flags:

  kubectl logs -l app=pingpong --tail 1 -f
  

]

Note: combining -l and -f is only possible since Kubernetes 1.14!

Let's try to understand why ...

---

class: extra-details

Streaming logs of many pods

• Let's see what happens if we try to stream the logs for more than 5 pods

.exercise[

• Scale up our deployment:

  kubectl scale deployment pingpong --replicas=8
  

• Stream the logs:

  kubectl logs -l app=pingpong --tail 1 -f
  

]

We see a message like the following one:

error: you are attempting to follow 8 log streams,
but maximum allowed concurency is 5,
use --max-log-requests to increase the limit

---

class: extra-details

Why can't we stream the logs of many pods?

• kubectl opens one connection to the API server per pod

• For each pod, the API server opens one extra connection to the corresponding kubelet

• If there are 1000 pods in our deployment, that's 1000 inbound + 1000 outbound connections on the API server

• This could easily put a lot of stress on the API server

• Prior Kubernetes 1.14, it was decided to not allow multiple connections

• From Kubernetes 1.14, it is allowed, but limited to 5 connections

  (this can be changed with --max-log-requests)

• For more details about the rationale, see PR #67573

---

Shortcomings of kubectl logs

• We don't see which pod sent which log line

• If pods are restarted / replaced, the log stream stops

• If new pods are added, we don't see their logs

• To stream the logs of multiple pods, we need to write a selector

• There are external tools to address these shortcomings

  (e.g.: Stern)

---

class: extra-details

kubectl logs -l ... --tail N

• If we run this with Kubernetes 1.12, the last command shows multiple lines

• This is a regression when --tail is used together with -l/--selector

• It always shows the last 10 lines of output for each container

  (instead of the number of lines specified on the command line)

• The problem was fixed in Kubernetes 1.13

See #70554 for details.

---

class: extra-details

Party tricks involving IP addresses

• It is possible to specify an IP address with less than 4 bytes

  (example: 127.1)

• Zeroes are then inserted in the middle

• As a result, 127.1 expands to 127.0.0.1

• So we can ping 127.1 to ping localhost!

(See this blog post for more details.)

---

class: extra-details

More party tricks with IP addresses

• We can also ping 1.1

• 1.1 will expand to 1.0.0.1

• This is one of the addresses of Cloudflare's public DNS resolver

• This is a quick way to check connectivity

  (if we can reach 1.1, we probably have internet access)

---

Creating other kinds of resources

• Deployments are great for stateless web apps

  (as well as workers that keep running forever)

• Jobs are great for "long" background work

  ("long" being at least minutes our hours)

• CronJobs are great to schedule Jobs at regular intervals

  (just like the classic UNIX cron daemon with its crontab files)

• Pods are great for one-off execution that we don't care about

  (because they don't get automatically restarted if something goes wrong)

---

Creating a Job

• A Job will create a Pod

• If the Pod fails, the Job will create another one

• The Job will keep trying until:

  • either a Pod succeeds,

  • or we hit the backoff limit of the Job (default=6)

.exercise[

• Create a Job that has a 50% chance of success:

    kubectl create job flipcoin --image=alpine -- sh -c 'exit $(($RANDOM%2))' 
  

]

---

Our Job in action

• Our Job will create a Pod named flipcoin-xxxxx

• If the Pod succeeds, the Job stops

• If the Pod fails, the Job creates another Pod

.exercise[

• Check the status of the Pod(s) created by the Job:

  kubectl get pods --selector=job-name=flipcoin
  

]

---

class: extra-details

More advanced jobs

• We can specify a number of "completions" (default=1)

• This indicates how many times the Job must be executed

• We can specify the "parallelism" (default=1)

• This indicates how many Pods should be running in parallel

• These options cannot be specified with kubectl create job

  (we have to write our own YAML manifest to use them)

---

Scheduling periodic background work

• A Cron Job is a Job that will be executed at specific intervals

  (the name comes from the traditional cronjobs executed by the UNIX crond)

• It requires a schedule, represented as five space-separated fields:

  • minute [0,59]
  • hour [0,23]
  • day of the month [1,31]
  • month of the year [1,12]
  • day of the week ([0,6] with 0=Sunday)

• * means "all valid values"; /N means "every N"

• Example: */3 * * * * means "every three minutes"

---

Creating a Cron Job

• Let's create a simple job to be executed every three minutes

• Careful: make sure that the job terminates!

  (The Cron Job will not hold if a previous job is still running)

.exercise[

• Create the Cron Job:

    kubectl create cronjob every3mins --schedule="*/3 * * * *" \
            --image=alpine -- sleep 10
  

• Check the resource that was created:

  kubectl get cronjobs
  

]

---

Cron Jobs in action

• At the specified schedule, the Cron Job will create a Job

• The Job will create a Pod

• The Job will make sure that the Pod completes

  (re-creating another one if it fails, for instance if its node fails)

.exercise[

• Check the Jobs that are created:

  kubectl get jobs
  

]

(It will take a few minutes before the first job is scheduled.)

---

class: extra-details

What about kubectl run before v1.18?

• Creating a Deployment:

  kubectl run

• Creating a Pod:

  kubectl run --restart=Never

• Creating a Job:

  kubectl run --restart=OnFailure

• Creating a Cron Job:

  kubectl run --restart=OnFailure --schedule=...

Avoid using these forms, as they are deprecated since Kubernetes 1.18!

---

Beyond kubectl create

• As hinted earlier, kubectl create doesn't always expose all options

  • can't express parallelism or completions of Jobs

  • can't express Pods with multiple containers

  • can't express healthchecks, resource limits

  • etc.

• kubectl create and kubectl run are helpers that generate YAML manifests

• If we write these manifests ourselves, we can use all features and options

• We'll see later how to do that!